Method for convolution acceleration calculation and convolution acceleration calculator, board card and electronic device

Abstract

The present disclosure provides a method and a convolution acceleration calculator, a board card and an electronic device for accelerating convolution calculation, which splits an original convolution kernel into a plurality of one-bit sub-convolution kernels according to bits, and aggregates the sub-convolution kernels to generate a base convolution kernel according to the frequency of the occurrence of a target value at different positions; each base convolution kernel of the original convolution kernel is respectively convolved with an input target activation; and the convolution calculation result of each base convolution kernel and the target activation is calculated according to the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel, to obtain the convolution calculation result of the original convolution kernel and the target activation. The present disclosure can greatly reduce the amount of repeated calculation and improve the overall efficiency of convolution calculation. The computing resources can be more efficiently used for effective calculation, and the utilization rate of computing resources is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method for accelerating convolution calculations, as well as a convolution acceleration calculator, board, and electronic device. Background Technology

[0002] Convolutional neural networks are widely used in the field of terminal vision. To meet the requirements of high throughput and low power consumption of terminal inference devices, neural network quantization methods are widely adopted. Among them, adaptive quantization weight bit width can reduce computing power consumption and storage requirements, and bit serial computing method has also attracted much attention because it can support computing with different weight bit widths and reduce hardware overhead.

[0003] However, convolutional neural networks suffer from severe redundant computation problems when performing bit-by-bit sequential computation. After splitting multi-bit convolutional kernels into 1-bit kernels, zero-value weights lead to zero-value computations, and multiple kernels convolving the same zero-value weight at the same location with the same input feature map results in repeated computations. These redundant computations increase the computational load and reduce computational efficiency.

[0004] Currently, most related technologies focus on eliminating zero-value calculations, neglecting the problem of redundant calculations. Therefore, new technical solutions are urgently needed to address these issues, improve computational efficiency, and meet the needs of terminal devices. Summary of the Invention

[0005] This disclosure provides a method for accelerating convolution calculations, as well as a convolution acceleration calculator, board, and electronic device. Its main purpose is to solve the problem of low computational efficiency caused by a large amount of repetitive calculations in convolution calculations.

[0006] According to a first aspect of this disclosure, a convolution acceleration calculator is provided, comprising: a base convolution kernel storage unit, a plurality of convolution computation units, and a convolution reconstruction unit;

[0007] The output of the base convolution kernel storage unit is connected to the input of multiple convolution calculation units, and the output of the multiple convolution calculation units is connected to the input of the convolution reconstruction unit.

[0008] The basal convolution kernel storage unit is used to store the basal convolution kernel, which is generated by splitting the original convolution kernel into multiple one-bit sub-convolution kernels according to bits, and aggregating them according to the frequency of the target value appearing at different positions.

[0009] The convolutional computation unit is used to perform convolutional computation with the configured base convolutional kernel and the input target activation. Each base convolutional kernel corresponding to the original convolutional kernel is configured to a different convolutional computation unit.

[0010] The convolutional reconstruction unit is used to calculate the convolutional calculation results output by each convolutional calculation unit based on the base convolutional kernel information and its expansion coefficients obtained from the decomposition of the original convolutional kernel, so as to obtain the convolutional calculation result between the original convolutional kernel and the target activation.

[0011] Optionally, the basal convolution kernel storage unit includes: basal convolution kernel memory and sparse weight decoder;

[0012] The output of the base convolution kernel memory is connected to the input of the sparse weight decoder, and the output of the sparse weight decoder is connected to the input of multiple convolution computation units.

[0013] The basal convolution kernel memory is used to store the encoded basal convolution kernels obtained after sparse encoding of the basal convolution kernels;

[0014] The sparse weighted decoder is used to decode the encoded radix kernel based on the width and height information of the radix kernel, so as to restore the coordinates of the target value in the radix kernel and generate the restored radix kernel.

[0015] Optionally, the convolution acceleration calculator also includes: activating memory units;

[0016] The activation storage unit includes: activation memory and sparse activation decoder;

[0017] The output of the activation memory is connected to the input of the sparse activation decoder, and the output of the sparse activation decoder is connected to the input of multiple convolutional computation units.

[0018] The activation memory is used to receive and store non-zero activations, which are obtained by sparse encoding the input feature map.

[0019] The sparse activation decoder is used to restore the non-zero activations to recover the non-zero values ​​and their coordinates, thus obtaining the target activation.

[0020] Optionally, the convolution calculation unit includes: a coordinate calculator, an adder, and an output activation memory;

[0021] The input terminals of the coordinate calculator are connected to the output terminals of the base convolution kernel storage unit and the activation storage unit, respectively. The output terminal of the coordinate calculator is connected to the input terminal of the output activation memory, and the output terminal of the adder is connected to the input terminal of the output activation memory.

[0022] The coordinate calculator is used to perform coordinate matching calculations based on the target value coordinates of the base convolution kernel and the non-zero value coordinates of the target activation, and obtain the matching activation value.

[0023] The adder is used to accumulate the non-zero value of the target activation with the matching activation value to obtain the output activation value;

[0024] The output activation memory is used to store intermediate calculation results of the output activation value.

[0025] Optionally, the convolution acceleration calculator may also include: a convolution kernel decomposition information storage;

[0026] The output of the convolution kernel decomposition information memory is connected to the input of the convolution reconstruction unit; the convolution kernel decomposition information memory is used to receive and store the base convolution kernel information and its expansion coefficients obtained from the original convolution kernel decomposition.

[0027] Optionally, the convolutional reconstruction unit includes: a reconstruction scheduler, multiple activation value memories, and multiple shift accumulators;

[0028] The input of the reconstruction scheduler is connected to the output of the convolution calculation unit, and the output of the reconstruction scheduler is connected to the input of multiple shift accumulators. The multiple shift accumulators are connected to their respective multiple activation value memories.

[0029] The reconstruction scheduler is used to obtain the base convolution kernel information and its expansion coefficients obtained from the decomposition of the original convolution kernel; it reads the output activation value stored in the convolution calculation unit according to the base convolution kernel information, and inputs the output activation value and expansion coefficients into the corresponding shift accumulator for calculation;

[0030] The shift accumulator is used to perform shift accumulation calculations based on the output activation value and the expansion coefficient. The shift accumulation calculation result is input into the activation value memory for storage and updating until the convolution calculation result of the base convolution kernel is accumulated, generating the convolution calculation result of the original convolution kernel and the target activation.

[0031] Optionally, the convolution acceleration calculator may also include: a controller;

[0032] The controller is connected to the base convolution kernel storage unit, multiple convolution calculation units, convolution reconstruction unit, and activation storage unit, respectively.

[0033] The controller is used to control the storage access of the base convolution kernel storage unit and the activation storage unit, as well as to control the flow of computational data of multiple convolutional computation units and convolutional reconstruction units.

[0034] According to a second aspect of this disclosure, a method for accelerating convolution computation is provided, the method comprising:

[0035] The original convolutional kernel is split into multiple one-bit sub-convolutional kernels according to bits, and the sub-convolutional kernels are aggregated according to the frequency of the target value appearing at different positions to generate the base convolutional kernel.

[0036] Each base convolution kernel of the original convolution kernel is convolved with the target activation of the input;

[0037] Based on the base convolution kernel information and its expansion coefficients obtained from the decomposition of the original convolution kernel, the convolution calculation results of each base convolution kernel and the target activation are calculated to obtain the convolution calculation results of the original convolution kernel and the target activation.

[0038] Optionally, the sub-convolutional kernels are aggregated to generate the base convolutional kernel according to the frequency of the target value appearing at different positions, including:

[0039] Calculate the frequency of occurrence of target values ​​at different coordinates in multiple one-bit sub-convolution kernels;

[0040] Based on the frequency of occurrence, determine the target coordinates of at least two sub-convolutional kernels that share the same target value;

[0041] The target value at the target coordinate location and the zero-value weights of the non-common coordinates are aggregated to generate the base convolution kernel.

[0042] Optionally, the method also includes:

[0043] The expansion coefficients of the generated base convolution kernel are calculated based on the expansion coefficients of the base convolution kernel in different sub-convolution kernels.

[0044] Optionally, based on the base convolution kernel information and its expansion coefficients obtained from the original convolution kernel decomposition, the convolution calculation result of each base convolution kernel and the target activation is calculated, including:

[0045] Based on the base convolution kernel information, obtain the output activation value calculated by convolution of the base convolution kernel and the target activation.

[0046] The output activation value and the expansion coefficient are shifted and accumulated to reconstruct the convolution calculation result of the original convolution kernel and the target activation.

[0047] According to a third aspect of this disclosure, a board is provided comprising: a storage device, an interface device, and a controller, as well as the convolution accelerator described in the first aspect; wherein the convolution accelerator is connected to the storage device, the controller, and the interface device respectively; the storage device is used to store data; the interface device is used to realize data transmission between the convolution accelerator and an external device; and the controller is used to monitor and manage the status of the chip.

[0048] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: at least one board as described in the third aspect above.

[0049] This disclosure provides a method for accelerating convolution computation, as well as a convolution acceleration calculator, board, and electronic device, relating to the field of artificial intelligence. In this disclosure, the base convolution kernel is generated by splitting the original convolution kernel bit by bit and then aggregating it based on the frequency of the target value appearing at different positions. This generation method can concentrate the repetitive computation parts of multiple 1-bit sub-convolution kernels into the base convolution kernel. During convolution computation, each convolution computation unit only needs to process one base convolution kernel and calculate the repetitive parts only once. Compared with the traditional bit-sequential computation method, this can significantly reduce the amount of repetitive computation and improve the overall efficiency of convolution computation. Since the base convolution kernel is 1 bit, the weights only have two values: 0 and 1. The computation corresponding to zero-value weights is invalid. In actual computation, the convolution computation unit can skip the computation of these zero-value weights, avoiding the waste of computing resources due to invalid computation, allowing computing resources to be used more efficiently for effective computation and improving the utilization rate of computing resources. During the computation process, multiple convolution computation units reuse activation values, reducing memory access. This method reduces the storage and memory access pressure on the hardware, reduces the complexity of hardware implementation, and thus reduces hardware costs. By eliminating redundant calculations and invalid calculations with zero-value weights, computational power consumption is reduced; simultaneously, reducing memory access also lowers memory access power consumption. The convolution reconstruction unit reconstructs the calculation results of the base convolution kernel into the calculation results of the original convolution kernel. This improves the hardware's throughput and significantly enhances its energy efficiency, thereby improving the overall performance of the convolution accelerator in applications such as neural network inference and better meeting the high throughput and low power consumption requirements of terminal devices.

[0050] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0051] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0052] Figure 1 This is a schematic diagram of the structure of a convolution accelerator calculator provided in an embodiment of the present disclosure;

[0053] Figure 2 This is a schematic diagram of another convolution acceleration calculator provided in an embodiment of the present disclosure;

[0054] Figure 3 This is a schematic diagram of a convolution calculation process provided in an embodiment of the present disclosure;

[0055] Figure 4 A flowchart illustrating a method for accelerating convolution computation provided in an embodiment of this disclosure;

[0056] Figure 5A flowchart illustrating a method for accelerating convolution computation provided in an embodiment of this disclosure;

[0057] Figure 6 This is a flowchart illustrating a method for splitting the original convolutional kernel according to an embodiment of the present disclosure. Detailed Implementation

[0058] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0059] The following description, with reference to the accompanying drawings, describes a method for accelerating convolution calculations, as well as a convolution acceleration calculator, board, and electronic device according to embodiments of the present disclosure.

[0060] Figure 1 This is a schematic diagram of the structure of a convolution accelerator calculator provided in an embodiment of this disclosure.

[0061] like Figure 1 As shown, the convolution acceleration calculator includes: a base convolution kernel storage unit 11, multiple convolution calculation units 12, and a convolution reconstruction unit 13. The output of the base convolution kernel storage unit 11 is connected to the input of the multiple convolution calculation units 12, and the output of the multiple convolution calculation units 12 is connected to the input of the convolution reconstruction unit 13.

[0062] The base convolution kernel storage unit 11 is used to store the base convolution kernel, wherein the base convolution kernel is generated by splitting the original convolution kernel into multiple one-bit sub-convolution kernels according to bits, and aggregating them according to the frequency of the target value appearing at different positions.

[0063] In embodiments of this disclosure, the original convolutional kernel of a convolutional neural network typically has multiple bits during computation. When using a bit-sequential computation method, the original convolutional kernel is first split into multiple one-bit sub-convolutional kernels. For example, as... Figure 2 As shown, a 2-bit original convolutional kernel is split into two 1-bit sub-convolutional kernels, the 0-bit and the 1-bit, with each sub-kernel having a weight value of 0 or 1. This 2-bit original convolution kernel ( Figure 2 The original convolution kernel (shown in the leftmost square of the first row) is split into bits. ×(1 1 1 1 0 1 0 1 0)( Figure 2The sub-convolution kernel shown in the second square of the first row is the sub-convolution kernel of the 1st bit. ×(1 0 1 1 1 0 1 0 1)( Figure 2 The sub-convolution kernel shown in the fourth square is the sub-convolution kernel of the 0th bit. When these sub-convolution kernels are convolved with the input feature map, since a 1-bit sub-convolution kernel only has two weight values, 0 and 1, multiple sub-convolution kernels with a 1-value weight at the same position will be repeatedly calculated when convolved with the same feature map.

[0064] To reduce redundant computation, this disclosure aggregates these one-bit sub-convolutional kernels according to the frequency of the target value appearing at different positions, generating a base convolutional kernel. The target value is the 1-value weight in the sub-convolutional kernel. Weight pattern analysis is performed on all the split 1-bit sub-convolutional kernels to extract the distribution positions of the 1-value weights, forming a candidate weight pattern set. Among all 1-bit sub-convolutional kernels, the frequency of patterns with the same 1-value weight position is counted. The weight pattern with the highest repetition rate is selected as the base convolutional kernel. For example, in... Figure 2 The first original convolutional kernel is split into two sub-convolutional kernels. By statistically analyzing the 1-bit weights in these sub-convolutional kernels at the same coordinate positions (e.g., the 1-value weights appearing repeatedly at the first row, first column, first row, third column, and second row, first column in the example above), these 1-values ​​constitute the base convolutional kernel. For multiple base convolutional kernels, they may originate from the same or different original convolutional kernels. Based on their origin, base convolutional kernels can be divided into three categories: Type 1: Repeated weight patterns from different bit positions of the same original convolutional kernel (e.g., the 1-value positions shared by the 0-bit and 1-bit sub-kernels of the original convolutional kernel 0). Type 2: Repeated weight patterns from the same bit position of different original convolutional kernels (e.g., the 1-value positions shared by the 0-bit sub-kernels of multiple original convolutional kernels). Type 3: Repeated weight patterns spanning different original convolutional kernels and different bit positions. Each 1-bit sub-convolutional kernel is decomposed into a combination of several base convolutional kernels, and the 1-value weights of the base convolutional kernels must completely cover the 1-value weights of the sub-kernel.

[0065] The base convolutional kernel storage unit 11 is specifically used to store these generated base convolutional kernels. During subsequent convolution calculations, the base convolutional kernels stored in this unit are sent to the convolution calculation unit, serving as a crucial data foundation for convolution calculations and supporting efficient convolution calculations. This storage method and base convolutional kernel generation strategy can significantly reduce redundant computations in convolution calculations, improve computational efficiency, and optimize the performance of convolutional neural networks. It should be noted that the generation process of the base convolutional kernels can be based on cloud computing power and distributed to a convolution accelerator; alternatively, it can be generated offline in a computing device equipped with a convolution accelerator. This embodiment of the present disclosure does not limit this approach.

[0066] The convolution calculation unit 12 is used to perform convolution calculations between the configured base convolution kernels and the input target activation. Each base convolution kernel corresponding to the original convolution kernel is configured to a different convolution calculation unit 12.

[0067] In the embodiments of this disclosure, the convolution calculation unit 12 is one of the core components for implementing convolution operations, and its design purpose is to efficiently process the convolution calculation process between the base convolution kernel and the target activation.

[0068] The base convolution kernels stored in the base convolution kernel storage unit 11 are sent to the convolution calculation unit 12. Each convolution calculation unit 12 is specifically configured with one base convolution kernel. This one-to-one configuration avoids resource conflicts and data confusion that may occur when multiple base convolution kernels process simultaneously during the calculation process, ensuring the accuracy and efficiency of the calculation. When the target activation is input into the convolution calculation unit 12, the convolution calculation unit 12 will perform convolution calculation with the configured base convolution kernel and the target activation according to the rules of convolution operation. Since the base convolution kernel is generated by splitting and aggregating the original convolution kernel in a specific way, its weight distribution has certain characteristics (such as high repetition of 1-value weights, i.e., high frequency of 1-value weights), which enables the convolution calculation unit 12 to effectively reduce redundant calculations when processing convolution calculations.

[0069] Multiple convolutional computation units 12 simultaneously perform convolutional computations with the target activation, respectively. This parallel computing approach fully utilizes hardware resources, reuses target activation values, and reduces memory access. Each convolutional computation unit 12 independently processes its own convolutional task with the base convolutional kernel and the target activation. The collaborative work of multiple computation units greatly improves the overall computational efficiency. Compared with traditional convolutional computation methods, more convolutional computation operations can be completed in the same amount of time, thereby accelerating the inference speed of the convolutional neural network, improving the overall system performance, and providing multiple convolutional computation results of the base convolutional kernel and the target activation for subsequent convolutional reconstruction units, laying the foundation for finally obtaining the convolutional computation result of the original convolutional kernel and the target activation.

[0070] The convolution reconstruction unit 13 is used to calculate the convolution calculation results output by each convolution calculation unit 12 based on the base convolution kernel information and its expansion coefficients obtained from the decomposition of the original convolution kernel, so as to obtain the convolution calculation result of the original convolution kernel and the target activation.

[0071] In the embodiments of this disclosure, the convolution reconstruction unit 13 integrates and restores the calculation results of the base convolution kernels to the calculation results of the original convolution kernels. When the original convolution kernel is split into multiple 1-bit sub-convolution kernels bit by bit, and further aggregated to generate base convolution kernels, these base convolution kernels and the target activation perform convolution calculations in multiple convolution calculation units 12 respectively. At this time, the convolution calculation results of each base convolution kernel and the target activation are only intermediate data, which need to be processed by the convolution reconstruction unit 13 to obtain the final result equivalent to the direct convolution of the original convolution kernel and the target activation. During the process of decomposing the original convolution kernel into base convolution kernels, corresponding base convolution kernel information is generated. This base convolution kernel information includes key content such as the composition of the base convolution kernels and their positional relationship within the original convolution kernel. Simultaneously, each base convolution kernel has its corresponding expansion coefficient, which is determined by the expansion coefficient of the 1-bit sub-convolution kernels containing that base convolution kernel, which are split from the original convolution kernel. For example, when splitting a k-bit number into k 1-bit sub-convolutional kernels, each 1-bit sub-convolutional kernel has only 0 and 1 as its weight value, and the m-th 1-bit sub-convolutional kernel has an expansion coefficient equal to the power of 2; the expansion coefficient of the base convolutional kernel generated by each sub-convolutional kernel according to the frequency of occurrence of its weight value is consistent with the expansion coefficient of its corresponding sub-convolutional kernel; if the base convolutional kernels come from different original convolutional kernels, then the expansion coefficients from different sources need to be added together.

[0072] The convolutional reconstruction unit 13 performs calculations based on the information of these base convolutional kernels and their expansion coefficients. Specifically, it obtains the convolution calculation results of each base convolutional kernel and the target activation from each convolutional calculation unit 12, and then performs corresponding operations on these results according to the corresponding expansion coefficients. Since the expansion coefficients are all powers of 2, in hardware implementation, the convolutional reconstruction unit 13 mainly completes the calculations through shifting and accumulation.

[0073] The convolutional reconstruction unit 13, based on the information obtained from the decomposition of the original convolutional kernel, sequentially shifts the calculation results of each base convolutional kernel, with the number of bits shifted determined by the corresponding expansion coefficient. The shifted results are then accumulated, and through this shift-and-add method, the calculation results of each base convolutional kernel are gradually integrated. Once the calculation results of all relevant base convolutional kernels have been shifted and added, the convolutional reconstruction unit 13 obtains a calculation result equivalent to the direct convolution of the original convolutional kernel with the target activation. This achieves the reconstruction from the base convolutional kernel calculation result to the original convolutional kernel calculation result, ensuring the accuracy and completeness of the convolutional calculation after optimization and decomposition.

[0074] This disclosure provides a convolution acceleration calculator. The base convolution kernel is generated by splitting the original convolution kernel bit by bit and then aggregating it based on the frequency of the target value appearing at different positions. This generation method concentrates the repetitive computation parts of multiple 1-bit sub-convolution kernels into the base convolution kernel. During convolution computation, each convolution computation unit only needs to process one base convolution kernel and calculate the repetitive parts only once. Compared with the traditional bit-sequential computation method, this significantly reduces the amount of repetitive computation and improves the overall efficiency of convolution computation. Since the base convolution kernel is 1 bit, the weights only have two values: 0 and 1. The computation corresponding to zero-value weights is invalid. In actual computation, the convolution computation unit can skip the computation of these zero-value weights, avoiding the waste of computing resources due to invalid computation, allowing computing resources to be used more efficiently for effective computation and improving the utilization rate of computing resources. During the computation process, multiple convolution computation units reuse activation values, reducing memory access. This method reduces the storage and memory access pressure on the hardware, reduces the complexity of hardware implementation, and thus reduces hardware costs. By eliminating redundant calculations and invalid calculations with zero-value weights, computational power consumption is reduced; simultaneously, reducing memory access also lowers memory access power consumption. The convolution reconstruction unit reconstructs the calculation results of the base convolution kernel into the calculation results of the original convolution kernel. This improves the hardware's throughput and significantly enhances its energy efficiency, thereby improving the overall performance of the convolution accelerator in applications such as neural network inference and better meeting the high throughput and low power consumption requirements of terminal devices.

[0075] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the basal convolution kernel storage unit 11 includes: a basal convolution kernel memory 111 and a sparse weight decoder 112. The output of the basal convolution kernel memory 111 is connected to the input of the sparse weight decoder 112, and the output of the sparse weight decoder 112 is connected to the input of multiple convolution calculation units 12.

[0076] The base convolution kernel memory 111 is used to store the encoded base convolution kernel obtained after sparse encoding processing of the base convolution kernel; the sparse weight decoder 112 is used to decode the encoded base convolution kernel based on the width and height information of the base convolution kernel to restore the coordinates of the target value in the base convolution kernel and generate the restored base convolution kernel.

[0077] Specifically, in this embodiment, the base convolution kernel storage unit 11 is further subdivided into a base convolution kernel memory 111 and a sparse weight decoder 112, which work together to achieve efficient storage and processing of the base convolution kernel.

[0078] The base convolution kernel memory 111 primarily stores the base convolution kernels. Due to the special structure of the base convolution kernel, its weights only have two values: 0 and 1. Weights with 0 values ​​have no practical effect in the calculation and are considered invalid. To reduce storage resource consumption and improve storage efficiency, sparse encoding is applied to the base convolution kernel. After this processing, the base convolution kernel memory 111 does not store complete base convolution kernel information, but rather the sparsely encoded base convolution kernel, i.e., it only stores information related to weights with 1 values. Furthermore, it uses a relative positional storage method, recording only the number of 0-value weights between two 1-value weights, thus significantly reducing storage requirements. Assuming a 3×3 base convolution kernel, its weight distribution is as follows (1 represents an effective weight, 0 represents an invalid weight):

[0079] [[1, 0, 0],

[0080] [0, 1, 0],

[0081] [0, 0, 1]]

[0082] By expanding in one dimension and marking non-zero positions, the sequence is expanded row by row into a one-dimensional sequence: 1, 0, 0, 0, 1, 0, 0, 0, 1. The absolute position indices of the non-zero (1 value) are: 0 (row 1, column 1), 4 (row 2, column 2), 8 (row 3, column 3).

[0083] The number of zeros between adjacent 1 values ​​(interval step size) is recorded. The first 1 starts at position 0, with an interval step size of 0 (no leading zeros). The second 1 is separated from the previous 1 by 3 zeros (index 0→4, with 3 zeros in between), with an interval step size of 3. The third 1 is also separated from the previous 1 by 3 zeros (index 4→8), with an interval step size of 3. Sparse coding result: stored sequence [0,3, 3], the original 9 weights are compressed into 3 values. The sparse weight decoder 112 is responsible for decoding the encoded base convolution kernel stored in the base convolution kernel memory 111. During the decoding process, it needs to use the width and height information of the base convolution kernel. This is because in sparse coding, the absolute position information of the 1-value weights is not directly stored, only the relative position information is retained. Therefore, when reconstructing the actual coordinates of the 1-value weights in the base convolution kernel, the size information of the base convolution kernel is needed as a reference. The sparse weight decoder 112, based on the acquired width and height information and the relative position data within the encoded base convolutional kernel, uses a specific decoding algorithm to reconstruct the encoded base convolutional kernel into a form usable for convolution calculation, i.e., generating the reconstructed base convolutional kernel. This reconstructed base convolutional kernel contains the accurate coordinate information of the 1-value weights within the base convolutional kernel and can be used by multiple subsequent convolution calculation units 12. For example... Figure 3As shown, when base convolution kernel 1 and base convolution kernel 0 contain four 1-value weights, sparse encoding is used to reduce storage requirements, recording only the number of 0-value weights between two 1-value weights, for example, stored as 1, 5, 3, 5. During the actual computation phase, the sparsely encoded weight data is transmitted to the sparse weight decoder 112. This decoder decodes the sparsely encoded weights in real time based on the width, height, and other information of the base convolution kernels, thereby obtaining the coordinate information of each 1-value weight within the base convolution kernel. After the sparse weight decoding is completed, the 1-value weight information of each base convolution kernel is transmitted to the convolution computation unit 12, providing data support for subsequent convolution computations.

[0084] Through the cooperation of the base convolution kernel memory 111 and the sparse weight decoder 112, from storage optimization to information restoration, the efficiency and accuracy of the base convolution kernel in the storage and processing process are achieved, providing the convolution calculation unit 12 with base convolution kernel data that meets the calculation requirements, effectively supporting the efficient operation of the entire convolution acceleration calculator.

[0085] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the convolution acceleration calculator also includes an activation storage unit 14. The output of the activation storage unit 14 is connected to the input of multiple convolution calculation units 12; the activation storage unit 14 is used to receive and store non-zero activations, and to perform sparse decoding processing on the non-zero activations to obtain the target activation.

[0086] Specifically, in this embodiment of the disclosure, the activation storage unit 14 is mainly responsible for processing and providing the activation data required for convolution calculation, ensuring that the convolution calculation unit 12 can perform convolution operations efficiently.

[0087] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the activation storage unit 14 includes an activation memory 141 and a sparse activation decoder 142. The output of the activation memory 141 is connected to the input of the sparse activation decoder 142, and the output of the sparse activation decoder 142 is connected to the input of multiple convolution calculation units 12.

[0088] The activation memory 141 is used to receive and store non-zero activations, which are obtained by sparse encoding of the input feature map; the sparse activation decoder 142 is used to restore the non-zero activations to restore the non-zero values ​​and their coordinates to obtain the target activation.

[0089] Specifically, in this embodiment, the activation storage unit 14 is composed of an activation memory 141 and a sparse activation decoder 142, which work together to manage and process activation data. In the computational flow of the convolutional neural network, the input feature map undergoes sparse encoding before entering the activation storage unit 14. This is because the input feature map contains a large number of zero-value activations, which do not produce valid results in the convolution calculation and are considered invalid computational parts. Sparse encoding retains only non-zero activation information, greatly reducing data storage and improving storage efficiency. The non-zero activation data obtained after sparse encoding is stored in the activation memory 141. The activation memory 141 is responsible for receiving and saving this non-zero activation data, providing a data foundation for subsequent processing.

[0090] The sparse activation decoder 142 undertakes the crucial decoding task. It retrieves the stored non-zero activation data from the activation memory 141 and then uses a specific decoding algorithm to restore this data. Since non-zero activation values ​​and their coordinates are compressed and stored during sparse encoding, the sparse activation decoder 142 needs to restore the non-zero values ​​and their coordinates from the non-zero activations according to the encoding rules, ultimately obtaining the target activation. This target activation contains complete and accurate non-zero activation information, which can be directly used by multiple convolutional computation units 12. To eliminate invalid computations caused by zero-value activations, a sparse encoding and storage strategy is adopted for the activation data. This encoding method is consistent with the encoding method of the base convolution kernel, both using relative position encoding to store the number of zero values ​​among non-zero values. The difference is that the activation value is usually 8 bits or 16 bits, not just 1; therefore, in addition to recording the relative position, the activation value itself also needs to be stored. Figure 4 As shown, the 3*5 input feature map has 3 non-zero activation values. It is necessary to store 3 relative positions and 3 activation values. The non-zero activations obtained by sparse coding are 1, 2, 5, 15, 6, 8.

[0091] The output of the activation storage unit 14 is connected to the input of multiple convolutional computation units 12. The target activation, restored by the sparse activation decoder 142, is transmitted to each convolutional computation unit 12. In the convolutional computation unit 12, the target activation is convolved with the base convolutional kernel obtained from the base convolutional kernel storage unit 11. In this way, the activation storage unit 14 provides effective activation data for convolutional computation, working closely with other components to optimize and accelerate convolutional computation, thereby improving the overall operating efficiency of the convolutional neural network.

[0092] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the convolution calculation unit 12 includes: a coordinate calculator 121, an adder 122, and an output activation memory 123.

[0093] The input terminal of the coordinate calculator 121 is connected to the output terminal of the base convolution kernel storage unit 11 and the output terminal of the activation storage unit 14, respectively. The output terminal of the coordinate calculator 121 is connected to the input terminal of the output activation memory 123, and the output terminal of the adder 122 is connected to the input terminal of the output activation memory 123.

[0094] The coordinate calculator 121 is used to perform coordinate matching calculations based on the target value coordinates of the base convolution kernel and the non-zero value coordinates of the target activation to obtain the matching activation value; the adder 122 is used to accumulate the non-zero value of the target activation and the matching activation value to obtain the output activation value; the output activation memory 123 is used to store the intermediate calculation results of the output activation value.

[0095] Specifically, in this embodiment of the disclosure, the convolution calculation unit 12, as a key module for implementing convolution operations, is composed of a coordinate calculator 121, an adder 122, and an output activation memory 123 working together. Each of them undertakes a unique function and together completes the convolution calculation task.

[0096] The input terminals of the coordinate calculator 121 are connected to the base convolution kernel storage unit 11 (which stores the 1-bit base convolution kernel of sparse coding) and the activation storage unit 14 (which stores the activation value of sparse coding), respectively, and its output terminal is connected to the address input terminal of the output activation memory 123; the two input terminals of the adder 122 are connected to the value output terminal of the activation storage unit 14 and the value output terminal of the output activation memory 123, respectively, and its output terminal is connected to the value update input terminal of the output activation memory 123.

[0097] The base convolution kernel storage unit 11 outputs the set of coordinates of the base convolution kernel's 1-value weights (coordinates obtained through sparse decoding), and the activation storage unit 14 outputs the set of non-zero coordinates of the activation values ​​and their corresponding values. The coordinate calculator 121, based on the Cartesian coordinate system, performs pairwise matching between the base convolution kernel weight coordinates and the non-zero activation coordinates. Through the spatial correspondence of convolution operations, it calculates the mapping position of each weight coordinate in the input feature map, determining the activation value coordinates participating in the current convolution calculation.

[0098] Accumulation calculation of adder 122: When the weight of the base convolution kernel is 1, the convolution calculation is equivalent to the accumulation of the activation values ​​at the corresponding positions. Adder 122 obtains the activation values ​​(non-zero values) of the matching coordinates from the activation storage unit 14, reads the intermediate accumulation results of the corresponding output coordinates from the output activation memory 123, performs the addition operation, and updates the output activation memory 123.

[0099] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the convolution acceleration calculator also includes: a convolution kernel decomposition information storage 15.

[0100] The output of the convolution kernel decomposition information memory 15 is connected to the input of the convolution reconstruction unit 13; the convolution kernel decomposition information memory 15 is used to receive and store the base convolution kernel information and its expansion coefficients obtained from the original convolution kernel decomposition.

[0101] Specifically, in the embodiments of this disclosure, during the optimization calculation of the convolutional neural network, the original convolutional kernel is decomposed into multiple base convolutional kernels. This decomposition operation is accompanied by the generation of corresponding base convolutional kernel information and the expansion coefficients corresponding to each base convolutional kernel. This information is crucial for accurately restoring the calculation results of the base convolutional kernels to the calculation results of the original convolutional kernels.

[0102] The convolution kernel decomposition information storage 15 is responsible for receiving and storing this key information. It obtains the base convolution kernel information and its expansion coefficients from the module responsible for decomposing the original convolution kernel, and stores them in an orderly manner. This information includes the specific structure of the base convolution kernels, their positional distribution within the original convolution kernels, and the expansion coefficients that each base convolution kernel needs to be multiplied by when reconstructing the original convolution kernel result. The module responsible for decomposing the original convolution kernels can either generate the base convolution kernels and their information, along with the expansion coefficients, based on cloud computing power and distribute them to the convolution accelerator calculator; or it can generate them offline in a computing device equipped with a convolution accelerator calculator; this embodiment does not limit this approach.

[0103] The output of the kernel decomposition information memory 15 is connected to the input of the convolution reconstruction unit 13. When the convolution reconstruction unit 13 needs to integrate the convolution calculation results of each base convolution kernel and the target activation into the convolution calculation result of the original convolution kernel and the target activation, it reads the corresponding base convolution kernel information and its expansion coefficients from the kernel decomposition information memory 15. Based on this information, the convolution reconstruction unit 13 performs corresponding shifting and accumulation operations on the convolution calculation results of each base convolution kernel according to specific calculation rules, such as through components such as shifters and accumulators, thereby accurately reconstructing the convolution result of the original convolution kernel.

[0104] The presence of the convolution kernel decomposition information storage 15 ensures that the convolution reconstruction process can obtain accurate and complete base convolution kernel information and expansion coefficients, providing the necessary data foundation for the convolution reconstruction unit 13. It is an important guarantee for the convolution accelerator to complete the convolution calculation task efficiently and accurately. It enables the entire convolution calculation process to maintain the consistency and accuracy of the calculation results on the basis of optimized decomposition.

[0105] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the convolutional reconstruction unit 13 includes: a reconstruction scheduler 131, multiple activation value memories 132, and multiple shift accumulators 133.

[0106] The input of the reconstruction scheduler 131 is connected to the output of the convolution calculation unit 12, and the output of the reconstruction scheduler 131 is connected to the input of multiple shift accumulators 133. The multiple shift accumulators 133 are interconnected with their corresponding multiple activation value memories 132.

[0107] The reconstruction scheduler 131 is used to obtain the base convolution kernel information and its expansion coefficients obtained by decomposing the original convolution kernel; it reads the output activation value stored in the convolution calculation unit 12 according to the base convolution kernel information, and inputs the output activation value and expansion coefficients into the corresponding shift accumulator 133 for calculation; the shift accumulator 133 is used to perform shift accumulation calculation according to the output activation value and expansion coefficients, and inputs the shift accumulation calculation result into the activation value memory 132 for storage update, until the convolution calculation result of the base convolution kernel is accumulated, and the convolution calculation result of the original convolution kernel and the target activation is generated.

[0108] Specifically, in this embodiment of the disclosure, the convolution reconstruction unit 13, as a key component for restoring the calculation result of the base convolution kernel to the calculation result of the original convolution kernel, is implemented by the reconstruction scheduler 131, multiple activation value memories 132 and multiple shift accumulators 133 working together to achieve its function.

[0109] The reconstruction scheduler 131 plays a coordinating role in the convolution reconstruction process. Its input is connected to the output of the convolution calculation unit 12, through which it obtains the output activation values ​​generated by the convolution calculation unit 12. Simultaneously, the reconstruction scheduler 131 also retrieves the base convolution kernel information and its expansion coefficients obtained from the original convolution kernel decomposition information memory 15. This information is crucial for subsequent convolution reconstruction calculations. Based on the obtained base convolution kernel information, the reconstruction scheduler 131 can determine the storage location of the output activation values ​​corresponding to each base convolution kernel in the convolution calculation unit 12, thereby accurately reading these output activation values. Then, the reconstruction scheduler 131 inputs the read output activation values ​​and corresponding expansion coefficients into the corresponding shift accumulators 133 for calculation. This process strictly follows the convolution kernel decomposition information, ensuring the accuracy and orderliness of the calculations.

[0110] The shift accumulator 133 is the core component for implementing convolutional reconstruction calculations. Since all expansion coefficients are powers of 2, the shift accumulator 133 utilizes this characteristic to shift the input and output activation values, with the number of bits shifted determined by the corresponding expansion coefficients. After the shift is complete, the shift accumulator 133 accumulates the shifted results. Each time a shift accumulation calculation is completed, the result is input into the corresponding activation value memory 132 for storage update. This process continues until the convolution calculation results of all base convolution kernels have been accumulated.

[0111] Multiple activation value memories 132 are interconnected with corresponding shift accumulators 133 to store intermediate results and the final convolution reconstruction result generated by the shift accumulators 133. Each time a calculation result is received from the shift accumulator 133, the activation value memory 132 updates its stored content to ensure data real-time performance and accuracy. Once the convolution calculation results of all base convolution kernels have been accumulated in the shift accumulators 133 and the final result is stored in the activation value memory 132, the convolution calculation result of the original convolution kernel and the target activation is generated. Through the close cooperation of the reconstruction scheduler 131, multiple activation value memories 132, and multiple shift accumulators 133, the convolution reconstruction unit 13 efficiently and accurately completes the reconstruction task from the base convolution kernel calculation result to the original convolution kernel calculation result.

[0112] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the convolution acceleration calculator also includes: controller 16.

[0113] The controller 16 is connected to the base convolution kernel storage unit 11, multiple convolution calculation units 12, convolution reconstruction unit 13, and activation storage unit 14 respectively. The controller 16 is used to control the storage access of the base convolution kernel storage unit 11 and the activation storage unit 14, and to control the flow of calculation data of the multiple convolution calculation units 12 and the convolution reconstruction unit 13.

[0114] Specifically, in this embodiment, the controller 16 is a key control hub responsible for coordinating the work of the main components and ensuring that the entire convolution acceleration calculation process is carried out in an orderly and efficient manner. The controller 16 is connected to the base convolution kernel storage unit 11, multiple convolution calculation units 12, the convolution reconstruction unit 13, and the activation storage unit 14, enabling precise control of each component through these connections. For the base convolution kernel storage unit 11 and the activation storage unit 14, the controller 16 is primarily responsible for controlling their storage access operations. During the convolution calculation process, the base convolution kernel storage unit 11 needs to store sparsely encoded base convolution kernels, and the activation storage unit 14 needs to store sparsely encoded non-zero activation data. The controller 16 sends storage instructions to these two storage units at appropriate times according to the needs of the calculation process, determining the data write or read operation. For example, before the convolution calculation begins, the controller 16 controls the base convolution kernel storage unit 11 to read the base convolution kernel data and send it to the convolution calculation unit 12; simultaneously, it controls the activation storage unit 14 to read the non-zero activation data, providing data support for the convolution calculation.

[0115] When controlling multiple convolutional computation units 12 and convolutional reconstruction units 13, the controller 16 primarily manages the flow of their computational data. The convolutional computation unit 12 is responsible for performing convolutional computation between the base convolutional kernel and the target activation. This process involves retrieving base convolutional kernel data from the base convolutional kernel storage unit 11, retrieving target activation data from the activation storage unit 14, and outputting the computational results to the convolutional reconstruction unit 13. The controller 16 precisely controls the transmission path and timing of this data to ensure accurate and timely data flow between the various components. For example, it sends start signals to each convolutional computation unit 12 sequentially according to the computation order, causing it to begin convolutional computation after receiving the base convolutional kernel and target activation data. The convolutional reconstruction unit 13 is responsible for restoring the convolutional computation results of the base convolutional kernel to the original convolutional kernel's computational results. This process requires retrieving the computational results from the convolutional computation unit 12 and combining them with the base convolutional kernel information and its expansion coefficients in the convolutional kernel decomposition information storage unit 15 for calculation. The controller 16 controls the convolution calculation unit 12 to accurately transmit the calculation results to the convolution reconstruction unit 13, while ensuring that the convolution reconstruction unit 13 obtains the base convolution kernel information and its expansion coefficients in the correct order and manner to achieve accurate convolution reconstruction calculation.

[0116] By controlling the storage access of the base convolution kernel storage unit 11 and the activation storage unit 14 through the controller 16, and managing the data flow of multiple convolution calculation units 12 and convolution reconstruction units 13, all components of the entire convolution accelerator can work together to effectively improve the efficiency and accuracy of convolution calculation and complete complex convolutional neural network calculation tasks.

[0117] Figure 5 This is a flowchart illustrating a method for accelerating convolution computation provided in an embodiment of this disclosure.

[0118] like Figure 5 As shown, the method includes the following steps:

[0119] Step 201: The original convolution kernel is split into multiple one-bit sub-convolution kernels according to the bits, and the sub-convolution kernels are aggregated according to the frequency of the target value appearing at different positions to generate the base convolution kernel.

[0120] In the embodiments of this disclosure, during the computational optimization process of the convolutional neural network, the original convolutional kernel typically has multiple bits. To achieve more efficient computation, it needs to be processed. First, step 201 is executed to split the original convolutional kernel by bit. For example, for a k-bit original convolutional kernel, k one-bit sub-convolutional kernels are obtained, each sub-convolutional kernel consisting of a one-bit value of each weight in the original convolutional kernel at a specific bit. After splitting, the frequency of the target value (usually 1) appearing at different positions in the sub-convolutional kernels is counted. Taking a two-dimensional convolutional kernel as an example, for each coordinate position of each sub-convolutional kernel, the number of times the target value appears at that position in all sub-convolutional kernels is counted. Based on the statistical results, the target values ​​at positions with high frequency are extracted. For positions that are not shared by different sub-convolutional kernels, their weights are set to zero, and then aggregated to generate the base convolutional kernel. This generation method can concentrate the repetitive computation parts in the sub-convolutional kernels into the base convolutional kernel, effectively reducing the amount of subsequent computation.

[0121] Step 202: Perform convolution calculations between each base convolution kernel of the original convolution kernel and the input target activation.

[0122] In the embodiments of this disclosure, after generating the base convolutional kernels, the process proceeds to step 202. In convolution computation, the target activation is crucial data participating in the calculation. Each base convolutional kernel obtained by decomposing the original convolutional kernel is then convolved with the input target activation. In actual hardware implementations, this operation is typically performed by a convolution computation unit. Each convolution computation unit is configured with one base convolutional kernel, and according to the convolution operation rules, the weights of the base convolutional kernels and the target activation value are calculated accordingly to determine the output activation value. This step provides intermediate data for the subsequent calculation results of reconstructing the original convolutional kernels.

[0123] Step 203: Based on the base convolution kernel information and its expansion coefficients obtained from the decomposition of the original convolution kernel, calculate the convolution calculation result between each base convolution kernel and the target activation to obtain the convolution calculation result between the original convolution kernel and the target activation.

[0124] In the embodiments of this disclosure, when the original convolutional kernel is decomposed, corresponding base convolutional kernel information and its expansion coefficients are generated. The base convolutional kernel information includes the composition of the base convolutional kernel, its positional relationship in the original convolutional kernel, etc., while the expansion coefficients are used to adjust the weight of the base convolutional kernel calculation results in the reconstruction process. Since the expansion coefficients are all powers of 2, this characteristic is utilized to perform calculations using a shift-accumulation method. Specifically, based on the base convolutional kernel information, data is extracted from the convolution calculation results of each base convolutional kernel and the target activation. These results are then shifted according to the corresponding expansion coefficients; that is, when the expansion coefficient is 2^n, the result is shifted left by n positions. After shifting, all shifted results are accumulated. When the calculation results of all relevant base convolutional kernels have been shifted and accumulated, the final result obtained is the convolution calculation result of the original convolutional kernel and the target activation. This step, by integrating the base convolutional kernel calculation results, accurately reconstructs the calculation process of the original convolutional kernel, while improving computational efficiency by utilizing the characteristics of the expansion coefficients.

[0125] This disclosure provides a method for accelerating convolution computation. In this method, the base convolution kernel is generated by splitting the original convolution kernel bit by bit and then aggregating it based on the frequency of the target value appearing at different positions. This generation method concentrates the repetitive computation parts of multiple 1-bit sub-convolution kernels into the base convolution kernel. During convolution computation, each convolution computation unit only needs to process one base convolution kernel and calculate the repetitive parts only once. Compared with the traditional bit-sequential computation method, this significantly reduces the amount of repetitive computation and improves the overall efficiency of convolution computation. Since the base convolution kernel is 1 bit, the weights only have two values: 0 and 1. The computation corresponding to zero-value weights is invalid. In actual computation, the convolution computation unit can skip the computation of these zero-value weights, avoiding the waste of computing resources due to invalid computation, allowing computing resources to be used more efficiently for effective computation and improving the utilization rate of computing resources. During the computation process, multiple convolution computation units reuse activation values, reducing memory access. This method reduces the storage and memory access pressure on the hardware, reduces the complexity of hardware implementation, and thus reduces hardware costs. By eliminating redundant calculations and invalid calculations with zero-value weights, computational power consumption is reduced; simultaneously, reducing memory access also lowers memory access power consumption. The convolution reconstruction unit reconstructs the calculation results of the base convolution kernel into the calculation results of the original convolution kernel. This improves the hardware's throughput and significantly enhances its energy efficiency, thereby improving the overall performance of the convolution accelerator in applications such as neural network inference and better meeting the high throughput and low power consumption requirements of terminal devices.

[0126] To clearly illustrate the embodiments of this disclosure, this embodiment provides a flowchart of another method for accelerating convolution computation.

[0127] like Figure 6 As shown, the method includes the following steps:

[0128] Step 301: Split the original convolution kernel into multiple one-bit sub-convolution kernels according to bits.

[0129] Specifically, in step 301, the original convolutional kernel is split into multiple one-bit sub-convolutional kernels. In the computation of convolutional neural networks, the original convolutional kernel typically has multiple bits. When using a bit-sequential computation method, the original convolutional kernel needs to be split bit by bit. For example, a k-bit original convolutional kernel will be split into k one-bit sub-convolutional kernels. Each sub-convolutional kernel consists of a one-bit value of each weight in the original convolutional kernel at a specific bit position. For example, the value of the m-th bit of a certain weight in the original convolutional kernel constitutes the weight value of the corresponding sub-convolutional kernel. This splitting is the basis for subsequent optimization calculations, allowing convolutional computation to be processed based on one-bit weights. For example, Figure 2 The paper demonstrates the process of splitting a 2-bit original convolutional kernel into two 1-bit sub-convolutional kernels and generating a base convolutional kernel based on these two 1-bit sub-convolutional kernels.

[0130] Step 302: Calculate the expansion coefficient of the generated base convolution kernel based on the expansion coefficients of the base convolution kernel in different sub-convolution kernels.

[0131] Specifically, in step 302, the expansion coefficients of the generated base convolution kernel are calculated based on the expansion coefficients of the base convolution kernel in different sub-convolution kernels. After splitting into multiple one-bit sub-convolution kernels in step 301, each sub-convolution kernel has its corresponding expansion coefficient. Typically, the expansion coefficient of the k-th bit sub-convolution kernel is 2^k, such as... Figure 2 In this model, the expansion coefficient of the first bit of the sub-convolutional kernel is 2. The base convolutional kernel is generated by aggregating the partial weights of multiple sub-convolutional kernels. When generating the base convolutional kernel, its expansion coefficient is calculated based on the expansion coefficients of each sub-convolutional kernel that contains the weight pattern of the base convolutional kernel. Specifically, the expansion coefficients of these sub-convolutional kernels are summed to obtain the expansion coefficient of the base convolutional kernel. This expansion coefficient is used in the subsequent convolution reconstruction process to accurately restore the convolution result of the base convolutional kernel to the convolution result of the original convolutional kernel.

[0132] Step 303: Calculate the frequency of occurrence of the target value at different coordinates in multiple one-bit sub-convolution kernels.

[0133] Specifically, in step 303, the frequency of the target value at different coordinates in multiple one-bit sub-convolutional kernels is calculated. After splitting into multiple one-bit sub-convolutional kernels, the number of times the target value (usually 1) appears at each coordinate position in these sub-convolutional kernels is counted. Figure 2Taking the convolution kernel in the example, for each coordinate point of each sub-convolution kernel, the frequency of the target value appearing at that coordinate position is counted across all sub-convolution kernels. This frequency calculation reveals the concentration of target values ​​at different coordinate positions, providing data for subsequently determining the weight distribution of the base convolution kernel.

[0134] Step 304: Based on the frequency of occurrence, determine the target coordinate positions where at least two sub-convolutional kernels share the same target value.

[0135] Specifically, in step 304, based on the frequency of occurrence, the target coordinate positions where at least two sub-convolutional kernels share the same target value are determined. After obtaining the frequency of occurrence of the target value at different coordinates in step 303, those coordinate positions with higher frequencies are selected. These coordinate positions mean that the target value frequently appears in multiple sub-convolutional kernels. That is, finding those coordinate points where at least two sub-convolutional kernels have the target value at the same coordinate position, these coordinate points are the key positions for subsequently constructing the base convolutional kernel. These coordinate positions with the same target value represent the redundant calculation parts between sub-convolutional kernels; concentrating these parts in the base convolutional kernel can effectively reduce the amount of computation. Please refer to [reference needed]. Figure 2 The process of generating the mid-base convolution kernel.

[0136] Step 305: Aggregate the target value at the target coordinate position and the zero-value weights of the non-common coordinates to generate a base convolution kernel.

[0137] Specifically, in step 305, the target values ​​at the target coordinate positions and the zero-value weights of non-common coordinates are aggregated to generate the base convolution kernel. After determining that at least two sub-convolution kernels share the same target value at their target coordinate positions, the target values ​​at these coordinate positions are retained. For coordinate positions that are not shared across different sub-convolution kernels, their weights are set to zero. In this way, the repetitive parts of multiple sub-convolution kernels are extracted and aggregated to form the base convolution kernel. The base convolution kernel contains the weight pattern with the highest repetition among multiple sub-convolution kernels. In subsequent convolution calculations, only the base convolution kernel needs to be calculated, and combined with its expansion coefficients, the convolution result of the original convolution kernel can be reconstructed, thereby reducing a large amount of redundant calculations and improving the efficiency of convolution calculations.

[0138] Step 306: Based on the base convolution kernel information, obtain the output activation value calculated by convolving the base convolution kernel with the target activation kernel.

[0139] Specifically, in step 306, based on the base convolution kernel information, the output activation value obtained from the convolution calculation of the base convolution kernel and the target activation is acquired. After generating the base convolution kernel and performing convolution calculations with the base convolution kernel and the target activation input convolution calculation unit, the results of these calculations need to be obtained. The base convolution kernel information includes the structure of the base convolution kernel, its positional relationship within the original convolution kernel, etc., which are important indexes for obtaining the corresponding calculation results. After completing the convolution calculation of the base convolution kernel and the target activation, the convolution calculation unit stores the calculated output activation value in a specific storage area. Based on the base convolution kernel information, the system can accurately locate and extract the output activation value corresponding to each base convolution kernel from this storage area, providing a data foundation for subsequent calculations. This step ensures the accuracy and completeness of the calculation results used in the convolution reconstruction process and is a necessary prerequisite for further processing the base convolution kernel calculation results to restore the original convolution kernel calculation results.

[0140] Step 307: The output activation value and the expansion coefficient are shifted and accumulated to reconstruct the convolution calculation result of the original convolution kernel and the target activation.

[0141] Specifically, in step 307, the output activation value and the expansion coefficients are shifted and accumulated to reconstruct the convolution calculation result of the original convolution kernel and the target activation. After obtaining the output activation value obtained from the convolution calculation of the base convolution kernel and the target activation, since the base convolution kernel is obtained by decomposing the original convolution kernel, each base convolution kernel has its corresponding expansion coefficient. These expansion coefficients are determined when generating the base convolution kernel based on the expansion coefficient of the sub-convolution kernel containing that base convolution kernel. In this step, taking advantage of the characteristic that the expansion coefficients are all powers of 2, the output activation value is shifted, and the number of bits shifted is determined by the corresponding expansion coefficient. For example, if the expansion coefficient is 2^k, the output activation value is shifted k bits to the left. After the shift is completed, the shifted output activation value is accumulated. By sequentially shifting and accumulating the output activation value of each base convolution kernel, the calculation results of each base convolution kernel are gradually integrated. When the output activation values ​​of all relevant base convolution kernels have been shifted and accumulated, the final result is the convolution calculation result of the original convolution kernel and the target activation. This process, through the clever use of the expansion coefficient, efficiently reconstructs the calculation result from the base convolution kernel to the original convolution kernel, restoring the complete convolution calculation process. At the same time, it reduces redundant operations in the calculation process, improving computational efficiency and system performance.

[0142] It should be noted that the embodiments of this disclosure may include multiple steps. For ease of description, these steps are numbered, but these numbers are not a limitation on the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of this disclosure do not limit this.

[0143] This disclosure provides a board comprising: a storage device, an interface device, and a controller, as well as a convolution accelerator calculator as described in the foregoing embodiments; wherein the convolution accelerator calculator is connected to the storage device, the controller, and the interface device respectively; the storage device is used to store data; the interface device is used to realize data transmission between the convolution accelerator calculator and external devices; and the controller is used to monitor and manage the status of the chip. The foregoing explanations of the embodiments and method embodiments of the convolution accelerator calculator also apply to the board of this embodiment, as the principles are the same, and this embodiment is not further limited thereto. Similarly, the descriptions of the foregoing embodiments also apply to the electronic device provided in this disclosure.

[0144] The various numerical designations such as "first," "second," etc., used in this disclosure are merely for ease of description and are not intended to limit the scope of the embodiments of this disclosure, nor do they indicate a sequential order.

[0145] At least one of the features described in this disclosure can also be described as one or more, and multiple features can be two, three, four or more, and this disclosure does not impose any limitations. In the embodiments of this disclosure, for a technical feature, the technical features in that technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", etc., and there is no sequential order or size order among the technical features described by "first", "second", "third", "A", "B", "C" and "D".

[0146] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A convolution acceleration calculator, characterized in that, include: The system includes a basic convolution kernel storage unit, multiple convolution computation units, a convolution reconstruction unit, and a controller. The controller is connected to the base convolution kernel storage unit, the plurality of convolution calculation units, and the convolution reconstruction unit, respectively. The output of the base convolution kernel storage unit is connected to the input of the plurality of convolution calculation units, and the output of the plurality of convolution calculation units is connected to the input of the convolution reconstruction unit. The base convolution kernel storage unit is used to store base convolution kernels, wherein the base convolution kernel is generated by splitting the original convolution kernel into multiple one-bit sub-convolution kernels according to bits, and aggregating them according to the frequency of the target value appearing at different positions; The convolution calculation unit is used to perform convolution calculation between the configured base convolution kernel and the input target activation, and each of the base convolution kernels corresponding to the original convolution kernel is configured to a different convolution calculation unit; The convolution reconstruction unit is used to calculate the convolution calculation results output by each of the convolution calculation units based on the base convolution kernel information and its expansion coefficients obtained from the decomposition of the original convolution kernel, so as to obtain the convolution calculation result between the original convolution kernel and the target activation.

2. The convolution acceleration calculator according to claim 1, characterized in that, The base convolution kernel storage unit includes: a base convolution kernel memory and a sparse weight decoder; The output of the base convolution kernel memory is connected to the input of the sparse weight decoder, and the output of the sparse weight decoder is connected to the input of the plurality of convolution calculation units. The base convolution kernel memory is used to store the encoded base convolution kernel obtained after sparse encoding processing of the base convolution kernel; The sparse weighted decoder is used to decode the encoded base convolutional kernel based on the width and height information of the base convolutional kernel, so as to restore the coordinates of the target value in the base convolutional kernel and generate the restored base convolutional kernel.

3. The convolution acceleration calculator according to claim 1, characterized in that, The convolution acceleration calculator also includes: an activation storage unit; The activation storage unit includes: an activation memory and a sparse activation decoder; The output of the activation memory is connected to the input of the sparse activation decoder, and the output of the sparse activation decoder is connected to the input of the plurality of convolution calculation units. The activation memory is used to receive and store non-zero activations, which are obtained by sparse encoding the input feature map. The sparse activation decoder is used to restore the non-zero activations to recover the non-zero values ​​and their coordinates, thereby obtaining the target activation.

4. The convolution acceleration calculator according to claim 1, characterized in that, The convolution calculation unit includes: a coordinate calculator, an adder, and an output activation memory; The input terminal of the coordinate calculator is connected to the output terminal of the base convolution kernel storage unit and the output terminal of the activation storage unit, respectively. The output terminal of the coordinate calculator is connected to the input terminal of the output activation memory, and the output terminal of the adder is connected to the input terminal of the output activation memory. The coordinate calculator is used to perform coordinate matching calculations based on the target value coordinates of the base convolution kernel and the non-zero value coordinates of the target activation, to obtain the matching activation value; The adder is used to accumulate the non-zero value of the target activation with the matching activation value to obtain the output activation value; The output activation memory is used to store intermediate calculation results of the output activation value.

5. The convolution acceleration calculator according to claim 1, characterized in that, The convolution acceleration calculator also includes: a convolution kernel decomposition information storage; The output of the convolution kernel decomposition information memory is connected to the input of the convolution reconstruction unit; the convolution kernel decomposition information memory is used to receive and store the base convolution kernel information and its expansion coefficients obtained from the decomposition of the original convolution kernel.

6. The convolution acceleration calculator according to claim 1, characterized in that, The convolutional reconstruction unit includes: a reconstruction scheduler, multiple activation value memories, and multiple shift accumulators; The input of the reconstruction scheduler is connected to the output of the convolution calculation unit, and the output of the reconstruction scheduler is connected to the input of the plurality of shift accumulators. The plurality of shift accumulators are respectively connected to their respective plurality of activation value memories. The reconstruction scheduler is used to obtain the base convolution kernel information and its expansion coefficients obtained by decomposing the original convolution kernel; read the output activation value stored in the convolution calculation unit according to the base convolution kernel information, and input the output activation value and the expansion coefficient into the corresponding shift accumulator for calculation; The shift accumulator is used to perform shift accumulation calculation based on the output activation value and the expansion coefficient, and input the shift accumulation calculation result into the activation value memory for storage and update until the convolution calculation result of the base convolution kernel is accumulated, thereby generating the convolution calculation result of the original convolution kernel and the target activation.

7. The convolution acceleration calculator according to claim 3, characterized in that, The controller is connected to the activation storage unit; The controller is used to control the storage access of the base convolution kernel storage unit and the activation storage unit, and to control the flow of computational data of the plurality of convolution calculation units and the convolution reconstruction unit.

8. A method for accelerating convolution computation, characterized in that, The method is applied to a convolution acceleration calculator as described in any one of claims 1-7, comprising: The original convolution kernel is split into multiple one-bit sub-convolution kernels according to bits, and the sub-convolution kernels are aggregated according to the frequency of the target value appearing at different positions to generate a base convolution kernel, which is stored in the base convolution kernel storage unit. The convolution calculation unit performs convolution calculations between each base convolution kernel of the original convolution kernel and the input target activation. The convolutional reconstruction unit calculates the convolutional result between each base convolutional kernel and the target activation based on the base convolutional kernel information and its expansion coefficients obtained from the decomposition of the original convolutional kernel, thereby obtaining the convolutional result between the original convolutional kernel and the target activation.

9. The method for accelerating convolution computation according to claim 8, characterized in that, The step of aggregating the sub-convolutional kernels to generate the base convolutional kernel according to the frequency of the target value appearing at different positions includes: Calculate the frequency of occurrence of the target value at different coordinates in the plurality of one-bit sub-convolution kernels; Based on the occurrence frequency, at least two of the sub-convolutional kernels are determined to have the same target value. The target value at the target coordinate position and the zero-value weights of the non-common coordinates are aggregated to generate the base convolution kernel.

10. The method for accelerating convolution computation according to claim 8, characterized in that, The method further includes: The expansion coefficients of the generated base convolution kernel are calculated based on the expansion coefficients of the base convolution kernel in different sub-convolution kernels.

11. The method for accelerating convolution computation according to claim 8, characterized in that, The convolutional reconstruction unit calculates the convolutional results of each base convolutional kernel and the target activation based on the base convolutional kernel information obtained from the decomposition of the original convolutional kernel and its expansion coefficients, including: The convolutional reconstruction unit obtains the output activation value calculated by convolving the base convolutional kernel with the target activation kernel based on the base convolutional kernel information; The convolution reconstruction unit performs shift and accumulation processing on the output activation value and the expansion coefficient to reconstruct the convolution calculation result of the original convolution kernel and the target activation.

12. A circuit board, characterized in that, The board includes: a storage device, an interface device, and a controller, as well as a convolution acceleration calculator as described in any one of claims 1-7; wherein the convolution acceleration calculator is connected to the storage device, the controller, and the interface device respectively; the storage device is used to store data; the interface device is used to realize data transmission between the convolution acceleration calculator and external devices; and the controller is used to monitor and manage the status of the convolution acceleration calculator.

13. An electronic device, characterized in that, include: At least one board as described in claim 12.

Images