Convolution acceleration calculation method, convolution acceleration calculator, board card and electronic equipment

Abstract

The invention provides a convolution acceleration calculation method, a convolution acceleration calculator, a board card and electronic equipment, and the method comprises the steps: splitting an original convolution kernel into a plurality of one-bit sub-convolution kernels according to bits, and carrying out the aggregation of the sub-convolution kernels according to the occurrence frequency of a target value at different positions, so as to generate a base convolution kernel; performing convolution calculation on each base convolution kernel of the original convolution kernel and input target activation; and according to base convolution kernel information obtained by decomposing the original convolution kernel and an expansion coefficient of the base convolution kernel information, calculating each base convolution kernel and a convolution calculation result of the target activation to obtain a convolution calculation result of the original convolution kernel and the target activation. According to the method, the repeated calculation amount can be greatly reduced, and the overall efficiency of convolution calculation is improved. The computing resources can be used for effective computing more efficiently, and the utilization rate of the computing resources is increased.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a convolution acceleration calculation method, a convolution acceleration calculator, a board, and an electronic device. Background Art

[0002] Convolutional neural networks are widely used in the field of terminal vision. To meet the high throughput and low power consumption requirements of terminal inference devices, neural network quantization methods are widely adopted. Among them, adaptive quantization weight bit width can reduce computing power consumption and storage requirements. The bit-serial computing method has also attracted much attention because it can support calculations with different weight bit widths and reduce hardware overhead.

[0003] However, convolutional neural networks suffer from a serious problem of redundant computation when performing bit-serial computations. When a multi-bit convolution kernel is split into 1-bit kernels, zero-valued weights result in zero-valued computations. Furthermore, when 1-valued weights at the same position in multiple kernels are convolved with the same input feature map, repeated computations occur. These redundant computations increase computational complexity and reduce efficiency.

[0004] Currently, related technologies focus on eliminating zero-value calculations, ignoring the problem of repeated calculations. Therefore, new technical solutions are urgently needed to solve these problems in order to improve computing efficiency and meet the needs of terminal devices. Summary of the Invention

[0005] The present disclosure provides a method for convolution acceleration calculation, a convolution acceleration calculator, a board, and an electronic device. The main purpose of the method is to solve the problem of low computation efficiency caused by a large amount of repeated calculations in convolution calculations.

[0006] According to a first aspect of the present disclosure, a convolution acceleration calculator is provided, comprising: a base convolution kernel storage unit, a plurality of convolution calculation units, and a convolution reconstruction unit; The output end of the base convolution kernel storage unit is connected to the input end of the multiple convolution calculation units, and the output end of the multiple convolution calculation units is connected to the input end of the convolution reconstruction unit; The base convolution kernel storage unit is used to store the base convolution kernel, where the base convolution kernel is split into multiple one-bit sub-convolution kernels according to the bit position of the original convolution kernel, and is aggregated and generated according to the frequency of the target value appearing at different positions; The convolution calculation unit is used to perform convolution calculation on the configured base convolution kernel and the input target activation. The base convolution kernels corresponding to the original convolution kernel are respectively configured to different convolution calculation units; The convolution reconstruction unit is used to calculate the convolution calculation results output by each convolution calculation unit according to the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel, and obtain the convolution calculation result of the original convolution kernel and the target activation.

[0007] Optionally, the base convolution kernel storage unit includes: a base convolution kernel memory and a sparse weight decoder; The output end of the base convolution kernel memory is connected to the input end of the sparse weight decoder, and the output end of the sparse weight decoder is connected to the input end of the plurality of convolution calculation units; The base convolution kernel memory is used to store the coded base convolution kernel obtained after sparse coding processing of the base convolution kernel; The sparse weight decoder is used to decode the encoded base convolution kernel based on the width information and height information of the base convolution kernel to restore the coordinates of the target value in the base convolution kernel and generate a restored base convolution kernel.

[0008] Optionally, the convolution acceleration calculator further includes: activating a storage unit; The activation storage unit includes: an activation memory and a sparse activation decoder; The output end of the activation memory is connected to the input end of the sparse activation decoder, and the output end of the sparse activation decoder is connected to the input end of the plurality of convolutional computing units; The activation memory is used to receive and store non-zero activations, which are obtained by sparsely encoding the input feature map; The sparse activation decoder is used to restore the non-zero activations to restore the non-zero values and their coordinates in the non-zero activations to obtain the target activations.

[0009] Optionally, the convolution calculation unit includes: a coordinate calculator, an adder, and an output activation memory; The input end of the coordinate calculator is connected to the output end of the base convolution kernel storage unit and the output end of the activation storage unit respectively, the output end of the coordinate calculator is connected to the input end of the output activation memory, and the output end of the adder is connected to the input end of the output activation memory; The coordinate calculator is used to perform coordinate matching calculation based on the target value coordinates of the base convolution kernel and the non-zero value coordinates of the target activation to obtain the matching activation value; The adder is used to accumulate the non-zero value of the target activation and the matching activation value to obtain the output activation value; The output activation memory is used to store the intermediate calculation results of the output activation value.

[0010] Optionally, the convolution acceleration calculator further includes: a convolution kernel decomposition information memory; The output end of the convolution kernel decomposition information memory is connected to the input end of the convolution reconstruction unit; the convolution kernel decomposition information memory is used to receive and store the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel.

[0011] Optionally, the convolution reconstruction unit includes: a reconstruction scheduler, a plurality of activation value memories, and a plurality of shift accumulators; An input end of the reconstruction scheduler is connected to an output end of the convolution calculation unit, an output end of the reconstruction scheduler is connected to input ends of a plurality of shift accumulators, and the plurality of shift accumulators are respectively connected to a plurality of activation value memories corresponding thereto; The reconstruction scheduler is used to obtain the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel; read the output activation value stored in the convolution calculation unit according to the base convolution kernel information, and calculate the output activation value and the expansion coefficient input corresponding shift accumulator; The shift accumulator is used to perform shift accumulation calculation according to the output activation value and the expansion coefficient, and input the shift accumulation calculation result into the activation value memory for storage and update until the convolution calculation result of the base convolution kernel is accumulated, thereby generating the convolution calculation result of the original convolution kernel and the target activation.

[0012] Optionally, the convolution acceleration calculator further includes: a controller; The controller is respectively connected to the base convolution kernel storage unit, the multiple convolution calculation units, the convolution reconstruction unit, and the activation storage unit; The controller is used to control the base convolution kernel storage unit, activate the storage unit for storage access, and control the flow of calculation data of multiple convolution calculation units and convolution reconstruction units.

[0013] According to a second aspect of the present disclosure, a method for convolution acceleration calculation is provided, the method comprising: The original convolution kernel is split into multiple one-bit sub-convolution kernels according to the bit position, and the sub-convolution kernels are aggregated to generate the base convolution kernel according to the frequency of the target value appearing at different positions; Convolve each base convolution kernel of the original convolution kernel with the input target activation respectively; According to the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel, the convolution calculation result of each base convolution kernel and the target activation is calculated to obtain the convolution calculation result of the original convolution kernel and the target activation.

[0014] Optionally, the sub-convolution kernels are aggregated to generate a base convolution kernel according to the frequency of the target value at different positions, including: Calculate the frequency of occurrence of target values at different coordinates in multiple one-bit sub-convolution kernels; Based on the frequency of occurrence, determining target coordinate positions where at least two sub-convolution kernels share the same target value; The target value of the target coordinate position and the zero-value weights of the non-public coordinates are aggregated to generate the base convolution kernel.

[0015] Optionally, the method further includes: According to the expansion coefficients of the base convolution kernel in different sub-convolution kernels, the expansion coefficients of the base convolution kernel are calculated.

[0016] Optionally, based on the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel, the convolution calculation result of each base convolution kernel and the target activation is calculated, including: Based on the base convolution kernel information, obtain the output activation value calculated by convolution of the base convolution kernel and the target activation; The output activation value and the expansion coefficient are shifted and accumulated to reconstruct the convolution calculation result of the original convolution kernel and the target activation.

[0017] According to a third aspect of the present disclosure, a board is provided, comprising: a memory device, an interface device, a control device, and the convolution acceleration calculator described in the first aspect; wherein the convolution acceleration calculator is connected to the memory device, the control device, and the interface device, respectively; the memory device is used to store data; the interface device is used to realize data transmission between the convolution acceleration calculator and an external device; and the control device is used to monitor and manage the status of the chip.

[0018] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: at least one board card as described in the third aspect.

[0019] The present disclosure provides a method for convolution acceleration calculation, as well as a convolution acceleration calculator, board, and electronic device, relating to the field of artificial intelligence. In this disclosure, a base convolution kernel is generated by bit-wise splitting the original convolution kernel and aggregating it based on the frequency of target values appearing at different locations. This generation method can concentrate the repeated computations of multiple 1-bit sub-convolution kernels into the base convolution kernel. During convolution calculation, each convolution calculation unit only needs to process one base convolution kernel and calculate the repeated portion once. Compared with traditional bit-serial calculation methods, this method can significantly reduce repeated computations and improve the overall efficiency of convolution calculations. Because the base convolution kernel is 1 bit, the weights can only have two values, 0 and 1. The calculations corresponding to zero-valued weights are invalid. In actual calculations, the convolution calculation unit can skip the calculation of these zero-valued weights, avoiding the waste of invalid computations on computing resources, allowing computing resources to be more efficiently used for effective calculations, thereby improving computing resource utilization. During the calculation process, multiple convolution calculation units reuse activation values, reducing memory access. This method reduces hardware storage and memory access pressure, reduces hardware implementation complexity, and thus reduces hardware cost. By eliminating duplicate calculations and ineffective computations involving zero-valued weights, computing power consumption is reduced. This also reduces memory accesses, which in turn reduces memory power consumption. The convolution reconstruction unit reconstructs the results of the base convolution kernel into the original convolution kernel. This improves hardware throughput and significantly enhances energy efficiency, thereby enhancing the overall performance of the convolution accelerator in applications such as neural network inference, better meeting the high-throughput, low-power requirements of terminal devices.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure. Figure 1 A schematic diagram of the structure of a convolution acceleration calculator provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of the structure of another convolution acceleration calculator provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of a convolution calculation process provided in an embodiment of the present disclosure; Figure 4 A flowchart of a method for accelerating convolution calculation provided by an embodiment of the present disclosure; Figure 5 A flowchart of a method for accelerating convolution calculation provided by an embodiment of the present disclosure; Figure 6 A flowchart of a method for splitting an original convolution kernel provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] The following describes the convolution acceleration calculation method, convolution acceleration calculator, board, and electronic device according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0024] Figure 1 A schematic diagram of the structure of a convolution acceleration calculator provided in an embodiment of the present disclosure.

[0025] like Figure 1 As shown, the convolution acceleration calculator includes: a base convolution kernel storage unit 11, multiple convolution calculation units 12, and a convolution reconstruction unit 13. The output end of the base convolution kernel storage unit 11 is connected to the input end of the multiple convolution calculation units 12, and the output end of the multiple convolution calculation units 12 is connected to the input end of the convolution reconstruction unit 13.

[0026] The base convolution kernel storage unit 11 is used to store the base convolution kernel, wherein the base convolution kernel is split into multiple one-bit sub-convolution kernels according to the bit position by the original convolution kernel, and is aggregated and generated according to the frequency of the target value appearing at different positions.

[0027] In the embodiments of the present disclosure, when the convolutional neural network is performing calculations, the original convolution kernel usually has multiple bits. When using the bit serial calculation method, the original convolution kernel is first split according to the bit position to obtain multiple one-bit sub-convolution kernels. The multi-bit original convolution kernel is split into multiple 1-bit sub-convolution kernels according to the bit position. For example, Figure 2 As shown in the figure, a 2-bit original convolution kernel is split into two 1-bit sub-convolution kernels at the 0th bit and the 1st bit, and the weight value of each sub-kernel is 0 or 1. This 2-bit original convolution kernel ( Figure 2 The original convolution kernel shown in the leftmost square of the first row in the figure is split into ×(1 1 1 1 0 1 0 1 0) ( Figure 2 The subconvolution kernel shown in the second square of the first row is the 1-bit subconvolution kernel) and ×(1 0 1 1 1 0 1 0 1) ( Figure 2 The subconvolution kernel shown in the fourth square in the figure is the 0-bit subconvolution kernel. When these subconvolution kernels are convolved with the input feature map, since the 1-bit subconvolution kernel has only two weight values 0 and 1, there will be repeated calculations when multiple subconvolution kernels with a weight of 1 at the same position are convolved with the same feature map.

[0028] In order to reduce this repeated calculation, the present invention aggregates these one-bit sub-convolution kernels according to the frequency of the target value appearing in different positions to generate a base convolution kernel. The target value is the 1-value weight in the sub-convolution kernel. The weight pattern analysis is performed on all the 1-bit sub-convolution kernels after the split, and the distribution position of the 1-value weight is extracted to form a set of candidate weight patterns. Among all the 1-bit sub-convolution kernels, the frequency of occurrence of patterns with the same 1-value weight position is counted. The weight pattern with the highest reproducibility is selected as the base convolution kernel. For example, in Figure 2The first original convolution kernel is split into two sub-convolution kernels. By counting the 1-bit sub-convolution kernels that contain 1-valued weights at the same coordinate positions (for example, the 1-valued weights at the first row, first column, first row, third column, and second row, first column positions in the two sub-convolution kernels obtained in the above example), the 1-valued weights at these positions constitute the base convolution kernel. For multiple base convolution kernels, the base convolution kernels may come from the same or different original convolution kernels. Based on the source of the base convolution kernel, the base convolution kernels can be divided into three categories: Type 1: Repeating weight patterns from sub-kernels at different bit positions of the same original convolution kernel (for example, the 1-valued position shared by the 0-bit and 1-bit sub-kernels of the original convolution kernel 0). Type 2: Repeating weight patterns from sub-kernels at the same bit position of different original convolution kernels (for example, the 1-valued position shared by the 0-bit sub-kernels of multiple original convolution kernels). Type 3: Repeating weight patterns across different original convolution kernels and sub-kernels at different bit positions. Each 1-bit sub-convolution kernel is decomposed into a combination of several base convolution kernels, and the 1-valued weight of the base convolution kernel must completely cover the 1-valued weight of the sub-kernel. The base convolution kernel storage unit 11 is specifically used to store these generated base convolution kernels. In the subsequent convolution calculation process, the base convolution kernel stored in the unit will be transmitted to the convolution calculation unit as an important data basis for the convolution calculation, providing support for efficient convolution calculation. This storage method and base convolution kernel generation strategy can significantly reduce the amount of repeated calculations in the convolution calculation, improve computing efficiency, and optimize the performance of the convolution neural network. It should be noted that the generation process of the base convolution kernel can be based on cloud computing power and sent to the convolution acceleration calculator; it can also be generated offline in a computing device equipped with a convolution acceleration calculator; this is not limited in the embodiments of the present disclosure.

[0029] The convolution calculation unit 12 is used to perform convolution calculation on the configured base convolution kernel and the input target activation, and each base convolution kernel corresponding to the original convolution kernel is respectively configured to a different convolution calculation unit 12.

[0030] In the embodiment of the present disclosure, the convolution calculation unit 12 is one of the core components for implementing the convolution operation, and its design purpose is to efficiently process the convolution calculation process between the base convolution kernel and the target activation.

[0031] The base convolution kernel stored in the base convolution kernel storage unit 11 will be transmitted to the convolution calculation unit 12. Each convolution calculation unit 12 is specially configured with a base convolution kernel. This one-to-one configuration avoids resource conflicts and data confusion that may arise from the simultaneous processing of multiple base convolution kernels during the calculation process, thereby ensuring the accuracy and efficiency of the calculation. When the target activation is input into the convolution calculation unit 12, the convolution calculation unit 12 will perform convolution calculation on the configured base convolution kernel and the target activation according to the rules of the convolution operation. Since the base convolution kernel is generated by splitting and aggregating the original convolution kernel in a specific way, its weight distribution has certain characteristics (such as high repetitiveness of 1-value weights, that is, high frequency of 1-value weights), which enables the convolution calculation unit 12 to effectively reduce repeated calculations when processing convolution calculations.

[0032] Multiple convolution calculation units 12 perform convolution calculations with the target activation at the same time. This parallel calculation method makes full use of hardware resources, can reuse target activation values, and reduce memory access. Each convolution calculation unit 12 independently processes the convolution task of its own base convolution kernel and target activation, and multiple calculation units work together to greatly improve the overall computing efficiency. Compared with the traditional convolution calculation method, more convolution calculation operations can be completed in the same time, thereby speeding up the reasoning speed of the convolutional neural network and improving the performance of the entire system. It provides the subsequent convolution reconstruction unit with the convolution calculation results of multiple base convolution kernels and target activation, laying the foundation for finally obtaining the convolution calculation results of the original convolution kernel and target activation.

[0033] The convolution reconstruction unit 13 is used to calculate the convolution calculation results output by each convolution calculation unit 12 according to the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel, and obtain the convolution calculation result of the original convolution kernel and the target activation.

[0034] In an embodiment of the present disclosure, the convolution reconstruction unit 13 integrates and restores the calculation results of the base convolution kernel to the calculation results of the original convolution kernel. When the original convolution kernel is split into multiple 1-bit sub-convolution kernels by bit, and further aggregated to generate a base convolution kernel, these base convolution kernels and the target activation complete the convolution calculation in multiple convolution calculation units 12 respectively. At this time, the convolution calculation results of each base convolution kernel and the target activation are only intermediate data, which need to be processed by the convolution reconstruction unit 13 to obtain the final result equivalent to the direct convolution of the original convolution kernel and the target activation. In the process of decomposing the original convolution kernel into base convolution kernels, corresponding base convolution kernel information is generated. These base convolution kernel information contains key content such as the composition of the base convolution kernel and the positional relationship in the original convolution kernel. At the same time, each base convolution kernel has its corresponding expansion coefficient, which is determined by the expansion coefficient of the 1-bit sub-convolution kernel containing the base convolution kernel split from the original convolution kernel. For example, when a k-bit number is split into k 1-bit sub-convolution kernels by bit, the weight values of each 1-bit sub-convolution kernel are only 0 and 1, and the m-th 1-bit sub-convolution kernel has an expansion coefficient equal to the mth power of 2; the expansion coefficient of the base convolution kernel generated by each sub-convolution kernel according to the frequency of occurrence of the weight value is consistent with the expansion coefficient of the corresponding sub-convolution kernel; if the base convolution kernel comes from different original convolution kernels, then the expansion coefficients from different sources need to be added together.

[0035] The convolution reconstruction unit 13 performs calculations based on these base convolution kernel information and their expansion coefficients. Specifically, it obtains the convolution calculation results of each base convolution kernel and the target activation from each convolution calculation unit 12, and then performs corresponding operations on these results based on the corresponding expansion coefficients. Since the expansion coefficients are all powers of 2, in hardware implementation, the convolution reconstruction unit 13 mainly completes the calculations through shifting and accumulation.

[0036] The convolution reconstruction unit 13 will perform a shift operation on the calculation results of each base convolution kernel in turn according to the information obtained by decomposing the original convolution kernel. The number of bits of the shift is determined by the corresponding expansion coefficient. The shifted results are accumulated, and the calculation results of each base convolution kernel are gradually integrated through this shift addition method. When the calculation results of all relevant base convolution kernels have completed the shift addition operation, the convolution reconstruction unit 13 can obtain a calculation result equivalent to the direct convolution of the original convolution kernel and the target activation, thereby realizing the reconstruction from the calculation result of the base convolution kernel to the calculation result of the original convolution kernel, ensuring the accuracy and integrity of the convolution calculation after the optimized decomposition.

[0037] The present disclosure provides a convolution acceleration calculator. In the present disclosure, the base convolution kernel is generated by splitting the original convolution kernel by bit and aggregating the results based on the frequency of target values appearing at different positions. This generation method can concentrate the repeated calculation parts of multiple 1-bit sub-convolution kernels into the base convolution kernel. During convolution calculation, each convolution calculation unit only needs to process one base convolution kernel and calculate the repeated parts only once. Compared with the traditional bit-serial calculation method, it can significantly reduce the amount of repeated calculations and improve the overall efficiency of convolution calculation. Because the base convolution kernel is 1 bit, the weight has only two values 0 and 1. The calculation corresponding to the zero-value weight is invalid. In actual calculation, the convolution calculation unit can skip the calculation of these zero-value weights, avoiding the waste of computing resources caused by invalid calculations, allowing computing resources to be used more efficiently for effective calculations, thereby improving computing resource utilization. During the calculation process, multiple convolution calculation units reuse activation values to reduce memory access. This method reduces the storage and memory access pressure of the hardware, reduces the complexity of hardware implementation, and thus reduces hardware cost. By eliminating duplicate calculations and ineffective computations involving zero-valued weights, computing power consumption is reduced. This also reduces memory accesses, which in turn reduces memory power consumption. The convolution reconstruction unit reconstructs the results of the base convolution kernel into the original convolution kernel. This improves hardware throughput and significantly enhances energy efficiency, thereby enhancing the overall performance of the convolution accelerator in applications such as neural network inference, better meeting the high-throughput, low-power requirements of terminal devices.

[0038] Furthermore, in a possible implementation of this embodiment, as Figure 3 As shown, the base convolution kernel storage unit 11 includes: a base convolution kernel memory 111 and a sparse weight decoder 112. The output end of the base convolution kernel memory 111 is connected to the input end of the sparse weight decoder 112, and the output end of the sparse weight decoder 112 is connected to the input end of the multiple convolution calculation units 12.

[0039] The base convolution kernel memory 111 is used to store the encoded base convolution kernel obtained after sparse coding of the base convolution kernel; the sparse weight decoder 112 is used to decode the encoded base convolution kernel based on the width information and height information of the base convolution kernel to restore the coordinates of the target value in the base convolution kernel and generate a restored base convolution kernel.

[0040] Specifically, in the embodiment of the present disclosure, the base convolution kernel storage unit 11 is further subdivided into a base convolution kernel memory 111 and a sparse weight decoder 112, which work together to achieve efficient storage and processing of the base convolution kernel.

[0041] The base convolution kernel memory 111 is mainly responsible for the storage function of the base convolution kernel. Due to the special structure of the base convolution kernel, its weight has only two values, 0 and 1. The 0-value weight does not play a substantial role in the calculation and belongs to the invalid calculation part. In order to reduce the occupation of storage resources and improve storage efficiency, sparse coding is used for the base convolution kernel. After this processing, the base convolution kernel memory 111 does not store the complete base convolution kernel information, but the base convolution kernel after sparse coding, that is, it only stores the relevant information of the 1-value weight, and adopts a relative position storage method to only record the number of 0-value weights between two 1-value weights, thereby greatly reducing the storage capacity. Assume a 3×3 base convolution kernel, its weight distribution is as follows (1 represents a valid weight and 0 represents an invalid weight): [[1, 0, 0], [0, 1, 0], [0, 0, 1]] By one-dimensional expansion and marking non-zero positions, it is expanded into a one-dimensional sequence by row: 1, 0, 0, 0, 1, 0, 0, 0, 1. The absolute position index of the non-zero (1 value) mark is: 0 (1st row, 1st column), 4 (2nd row, 2nd column), 8 (3rd row, 3rd column).

[0042] Record the number of zero values between adjacent 1 values (interval step). The first 1 starts at position 0, with an interval step of 0 (no leading zeros). The second 1 is separated from the previous 1 by 3 zeros (index 0→4, with 3 zeros in between), with an interval step of 3. The third 1 is also separated from the previous 1 by 3 zeros (index 4→8), with an interval step of 3. Sparse coding result: The sequence [0, 3, 3] is stored, with the original 9 weights compressed into 3 values. The sparse weight decoder 112 is responsible for decoding the encoded base convolution kernel stored in the base convolution kernel memory 111. During the decoding process, it requires the width and height information of the base convolution kernel. This is because during sparse coding, the absolute position information of the 1-valued weights is not directly stored; only the relative position information is retained. Therefore, when restoring the actual coordinates of the 1-valued weights in the base convolution kernel, the base convolution kernel size information is required as a reference. The sparse weight decoder 112 uses a specific decoding algorithm to restore the encoded base convolution kernel to a form that can be used for convolution calculation based on the acquired width and height information and the relative position data in the encoded base convolution kernel, that is, to generate a restored base convolution kernel. The restored base convolution kernel contains the accurate coordinate information of the 1-valued weight in the base convolution kernel and can be used by subsequent multiple convolution calculation units 12. Figure 3As shown, when base convolution kernel 1 and base convolution kernel 0 contain 4 1-value weights, in order to reduce the amount of storage, sparse coding is used for storage, and only the number of 0-value weights between the 2 1-value weights is recorded, for example, stored as 1, 5, 3, 5. In the actual calculation stage, the sparsely coded weight data is transmitted to the sparse weight decoder 112. The decoder decodes the sparsely coded weights in real time based on information such as the width and height of the base convolution kernel, thereby obtaining the coordinate information of each 1-value weight in the base convolution kernel. After completing the sparse weight decoding, the 1-value weight information of each base convolution kernel will be transmitted to the convolution calculation unit 12 to provide data support for subsequent convolution calculations.

[0043] Through the cooperation of the base convolution kernel memory 111 and the sparse weight decoder 112, from storage optimization to information restoration, the efficiency and accuracy of the base convolution kernel in the storage and processing process are achieved, providing the convolution calculation unit 12 with base convolution kernel data that meets the calculation requirements, effectively supporting the efficient operation of the entire convolution accelerator.

[0044] Furthermore, in a possible implementation of this embodiment, as Figure 3 As shown, the convolution acceleration calculator further includes an activation storage unit 14. The output end of the activation storage unit 14 is connected to the input ends of the multiple convolution calculation units 12; the activation storage unit 14 is used to receive and store non-zero activations, and perform sparse decoding processing on the non-zero activations to obtain target activations.

[0045] Specifically, in the embodiment of the present disclosure, the activation storage unit 14 is mainly responsible for processing and providing activation data required for convolution calculation, ensuring that the convolution calculation unit 12 can perform convolution operation efficiently.

[0046] Furthermore, in a possible implementation of this embodiment, as Figure 3 As shown, the activation storage unit 14 includes: an activation memory 141 and a sparse activation decoder 142. The output of the activation memory 141 is connected to the input of the sparse activation decoder 142, and the output of the sparse activation decoder 142 is connected to the input of the plurality of convolution calculation units 12.

[0047] The activation memory 141 is used to receive and store non-zero activations, which are obtained by sparsely encoding the input feature map; the sparse activation decoder 142 is used to restore the non-zero activations to restore the non-zero values and their coordinates in the non-zero activations to obtain the target activations.

[0048] Specifically, in the embodiment of the present disclosure, the activation storage unit 14 is composed of an activation memory 141 and a sparse activation decoder 142, and the two work together to realize the management and processing of activation data. In the calculation process of the convolutional neural network, the input feature map will be sparsely coded before entering the activation storage unit 14. This is because there are a large number of zero-value activations in the input feature map. These zero-value activations do not produce valid results in the convolution calculation and belong to the invalid calculation part. Through sparse coding, only non-zero activation information is retained, which can greatly reduce the amount of data storage and improve storage efficiency. The non-zero activation data obtained after sparse coding is stored in the activation memory 141. The activation memory 141 is responsible for receiving and saving these non-zero activation data, providing a data basis for subsequent processing.

[0049] The sparse activation decoder 142 is responsible for the key decoding task. It obtains the stored non-zero activation data from the activation memory 141, and then restores these data through a specific decoding algorithm. Since the non-zero activation values and their coordinate information are compressed and stored during sparse coding, the sparse activation decoder 142 needs to restore the non-zero values and their coordinates in the non-zero activation according to the coding rules, and finally obtain the target activation. This target activation contains complete and accurate non-zero activation information and can be directly used by multiple convolution calculation units 12. In order to eliminate invalid calculations caused by zero-value activation, a sparse coding and storage strategy is adopted for the activation data. This encoding method is consistent with the encoding method of the base convolution kernel, and both use relative position encoding to store the number of zero values between non-zero values. The difference is that the activation value is usually 8 bits or 16 bits, not just 1, so in addition to recording the relative position, the activation value itself needs to be stored. As Figure 4 As shown in the figure, the 3*5 input feature map has 3 non-zero activation values, and 3 relative positions and 3 activation values need to be stored. The non-zero activations obtained by sparse coding are 1, 2, 5, 15, 6, and 8.

[0050] The output of the activation storage unit 14 is connected to the input of multiple convolution calculation units 12. The target activation restored by the sparse activation decoder 142 will be transmitted to each convolution calculation unit 12. In the convolution calculation unit 12, the target activation is convolved with the base convolution kernel obtained from the base convolution kernel storage unit 11. In this way, the activation storage unit 14 provides effective activation data for the convolution calculation, and works closely with other components to realize the optimization and acceleration function of the convolution accelerator for the convolution calculation, thereby improving the overall operating efficiency of the convolutional neural network.

[0051] Furthermore, in a possible implementation of this embodiment, as Figure 3 As shown, the convolution calculation unit 12 includes: a coordinate calculator 121, an adder 122, and an output activation memory 123.

[0052] The input end of the coordinate calculator 121 is respectively connected to the output end of the base convolution kernel storage unit 11 and the output end of the activation storage unit 14, the output end of the coordinate calculator 121 is connected to the input end of the output activation memory 123, and the output end of the adder 122 is connected to the input end of the output activation memory 123.

[0053] The coordinate calculator 121 is used to perform coordinate matching calculation based on the target value coordinates of the base convolution kernel and the non-zero value coordinates of the target activation to obtain a matching activation value; the adder 122 is used to accumulate the non-zero value of the target activation and the matching activation value to obtain an output activation value; the output activation memory 123 is used to store the intermediate calculation results of the output activation value.

[0054] Specifically, in the embodiment of the present disclosure, the convolution calculation unit 12 is a key module for implementing the convolution operation, and is collaboratively composed of a coordinate calculator 121, an adder 122 and an output activation memory 123. They each have unique functions and jointly complete the convolution calculation task.

[0055] The input end of the coordinate calculator 121 is respectively connected to the base convolution kernel storage unit 11 (storing the sparsely coded 1-bit base convolution kernel) and the activation storage unit 14 (storing the sparsely coded activation value), and its output end is connected to the address input end of the output activation memory 123; the two input ends of the adder 122 are respectively connected to the numerical output end of the activation storage unit 14 and the numerical output end of the output activation memory 123, and the output end is connected to the numerical update input end of the output activation memory 123.

[0056] The base convolution kernel storage unit 11 outputs the base convolution kernel's 1-value weight coordinate set (the coordinates obtained through sparse decoding), and the activation storage unit 14 outputs the non-zero coordinate set of the activation value and the corresponding value. The coordinate calculator 121 performs a pairwise match between the base convolution kernel weight coordinates and the activation non-zero coordinates based on the Cartesian coordinate system. Based on the spatial correspondence of the convolution operation, it calculates the mapping position of each weight coordinate in the input feature map and determines the activation value coordinate participating in the current convolution calculation.

[0057] Accumulation calculation by adder 122: When the base convolution kernel weight is 1, the convolution calculation is equivalent to the accumulation of activation values at the corresponding position. Adder 122 obtains the activation value (non-zero value) of the matching coordinate from activation storage unit 14, reads the intermediate accumulation result corresponding to the output coordinate from output activation memory 123, performs the addition operation, and updates the output activation memory 123.

[0058] Furthermore, in a possible implementation of this embodiment, as Figure 3 As shown, the convolution acceleration calculator also includes: a convolution kernel decomposition information memory 15.

[0059] The output end of the convolution kernel decomposition information memory 15 is connected to the input end of the convolution reconstruction unit 13; the convolution kernel decomposition information memory 15 is used to receive and store the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel.

[0060] Specifically, in the disclosed embodiments, during the optimization calculation process of a convolutional neural network, the original convolution kernel is decomposed into multiple base convolution kernels. This decomposition operation is accompanied by the generation of corresponding base convolution kernel information and the expansion coefficients corresponding to each base convolution kernel. This information is crucial for accurately restoring the calculation results of the base convolution kernels to the calculation results of the original convolution kernels.

[0061] The convolution kernel decomposition information storage 15 is responsible for receiving and storing these key information. It obtains the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel from the module responsible for decomposing the original convolution kernel, and stores them in order. This information includes the specific composition of the base convolution kernel, the position distribution in the original convolution kernel, and the expansion coefficient that each base convolution kernel needs to be multiplied by when reconstructing the original convolution kernel result. The module responsible for decomposing the original convolution kernel can be based on cloud computing power to perform base convolution kernel, base convolution kernel information, and expansion coefficient, and send them to the convolution acceleration calculator; it can also be generated offline in a computing device equipped with a convolution acceleration calculator; this is not limited in the embodiment of the present disclosure.

[0062] The output of the convolution kernel decomposition information memory 15 is connected to the input of the convolution reconstruction unit 13. When the convolution reconstruction unit 13 needs to integrate the convolution calculation results of each base convolution kernel with the target activation into the convolution calculation result of the original convolution kernel with the target activation, it reads the corresponding base convolution kernel information and its expansion coefficient from the convolution kernel decomposition information memory 15. Based on this information and in accordance with specific calculation rules, the convolution reconstruction unit 13 performs corresponding shift and accumulation operations on the convolution calculation results of each base convolution kernel through components such as shifters and accumulators, thereby accurately reconstructing the convolution result of the original convolution kernel.

[0063] The existence of the convolution kernel decomposition information storage 15 ensures that the convolution reconstruction process can obtain accurate and complete base convolution kernel information and expansion coefficients, provides the necessary data basis for the convolution reconstruction unit 13, and is an important guarantee for the convolution acceleration calculator to complete the convolution calculation task efficiently and accurately. It enables the entire convolution calculation process to maintain the consistency and accuracy of the calculation results on the basis of optimized decomposition.

[0064] Furthermore, in a possible implementation of this embodiment, as Figure 3 As shown, the convolution reconstruction unit 13 includes: a reconstruction scheduler 131, multiple activation value memories 132, and multiple shift accumulators 133.

[0065] The input end of the reconstruction scheduler 131 is connected to the output end of the convolution calculation unit 12, and the output end of the reconstruction scheduler 131 is connected to the input end of multiple shift accumulators 133. The multiple shift accumulators 133 are respectively connected to their corresponding multiple activation value memories 132.

[0066] The reconstruction scheduler 131 is used to obtain the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel; read the output activation value stored in the convolution calculation unit 12 according to the base convolution kernel information, and input the output activation value and the expansion coefficient into the corresponding shift accumulator 133 for calculation; the shift accumulator 133 is used to perform shift accumulation calculation according to the output activation value and the expansion coefficient, and input the shift accumulation calculation result into the activation value memory 132 for storage and update, until the convolution calculation result of the base convolution kernel is accumulated, thereby generating the convolution calculation result of the original convolution kernel and the target activation.

[0067] Specifically in the embodiment of the present disclosure, the convolution reconstruction unit 13 is a key component that restores the calculation result of the base convolution kernel to the calculation result of the original convolution kernel. Its function is realized by the collaborative work of the reconstruction scheduler 131, multiple activation value memories 132 and multiple shift accumulators 133.

[0068] The reconstruction scheduler 131 plays a coordinating role in the convolution reconstruction process. Its input end is connected to the output end of the convolution calculation unit 12, and the output activation value generated by the convolution calculation unit 12 is obtained through this connection. At the same time, the reconstruction scheduler 131 also obtains the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel from the convolution kernel decomposition information memory 15. This information is an important basis for subsequent convolution reconstruction calculations. Based on the obtained base convolution kernel information, the reconstruction scheduler 131 can determine the storage location of the output activation value corresponding to each base convolution kernel in the convolution calculation unit 12, and then accurately read these output activation values. Then, the reconstruction scheduler 131 inputs the read output activation value and the corresponding expansion coefficient into the corresponding shift accumulator 133 for calculation. This process is carried out strictly in accordance with the convolution kernel decomposition information, ensuring the accuracy and orderliness of the calculation.

[0069] The shift accumulator 133 is the core component for implementing convolution reconstruction calculations. Since the expansion coefficients are all powers of 2, the shift accumulator 133 utilizes this property to shift the input output activation values. The number of bits shifted is determined by the corresponding expansion coefficient. After the shift is completed, the shift accumulator 133 accumulates the shifted results. Each time a shift-accumulation calculation is completed, the result is input into the corresponding activation value memory 132 for storage and update. This process continues until the convolution calculation results of all base convolution kernels are accumulated.

[0070] Multiple activation value memories 132 are interconnected with corresponding shift accumulators 133, respectively, and are used to store the intermediate results generated by the shift accumulator 133 and the final convolution reconstruction results. After receiving the calculation results input by the shift accumulator 133 each time, the activation value memory 132 will update the storage content to ensure the real-time and accuracy of the data. When the convolution calculation results of all base convolution kernels are accumulated in the shift accumulator 133 and the final results are stored in the activation value memory 132, the convolution calculation results of the original convolution kernel and the target activation are generated. Through the close cooperation of the reconstruction scheduler 131, multiple activation value memories 132 and multiple shift accumulators 133, the convolution reconstruction unit 13 efficiently and accurately completes the reconstruction task from the base convolution kernel calculation results to the original convolution kernel calculation results.

[0071] Furthermore, in a possible implementation of this embodiment, as Figure 3 As shown, the convolution acceleration calculator also includes: a controller 16.

[0072] The controller 16 is respectively connected to the base convolution kernel storage unit 11, multiple convolution calculation units 12, the convolution reconstruction unit 13, and the activation storage unit 14; the controller 16 is used to control the base convolution kernel storage unit 11 and the activation storage unit 14 for storage access, and to control the flow of calculation data of the multiple convolution calculation units 12 and the convolution reconstruction unit 13.

[0073] Specifically, in the disclosed embodiment, controller 16 is a key control hub responsible for coordinating the operations of various key components to ensure the orderly and efficient execution of the entire convolution acceleration calculation process. Controller 16 establishes connections with the base convolution kernel storage unit 11, multiple convolution calculation units 12, convolution reconstruction unit 13, and activation storage unit 14, enabling precise control of each component. For the base convolution kernel storage unit 11 and activation storage unit 14, controller 16 is primarily responsible for controlling their storage access operations. During the convolution calculation process, base convolution kernel storage unit 11 needs to store sparsely coded base convolution kernels, and activation storage unit 14 needs to store sparsely coded non-zero activation data. Based on the needs of the calculation process, controller 16 sends storage instructions to these two storage units at appropriate times to determine data write or read operations. For example, before the convolution calculation begins, controller 16 controls base convolution kernel storage unit 11 to read the base convolution kernel data and transmit it to convolution calculation unit 12; simultaneously, controller 16 controls activation storage unit 14 to read non-zero activation data to provide data support for the convolution calculation.

[0074] When controlling multiple convolution calculation units 12 and convolution reconstruction units 13, the controller 16 primarily manages the flow of their computational data. The convolution calculation unit 12 is responsible for performing convolution calculations on the base convolution kernel and the target activation. This process involves obtaining base convolution kernel data from the base convolution kernel storage unit 11, obtaining target activation data from the activation storage unit 14, and outputting the calculation results to the convolution reconstruction unit 13. The controller 16 precisely controls the transmission path and timing of this data to ensure accurate and timely data flow between various components. For example, it sends a start signal to each convolution calculation unit 12 in sequence according to the calculation order, causing it to begin convolution calculations after receiving the base convolution kernel and target activation data. The convolution reconstruction unit 13 is responsible for restoring the convolution calculation results of the base convolution kernel to the calculation results of the original convolution kernel. This process requires obtaining the calculation results from the convolution calculation unit 12 and combining them with the base convolution kernel information and its expansion coefficient in the convolution kernel decomposition information storage 15 for calculation. The controller 16 controls the convolution calculation unit 12 to accurately transmit the calculation results to the convolution reconstruction unit 13, and ensures that the convolution reconstruction unit 13 obtains the base convolution kernel information and its expansion coefficient in the correct order and manner to achieve accurate convolution reconstruction calculation.

[0075] Through the controller 16's control of the storage access to the base convolution kernel storage unit 11 and the activation storage unit 14, as well as the management of the calculation data flow of multiple convolution calculation units 12 and the convolution reconstruction unit 13, the various components of the entire convolution accelerator calculator can work together to effectively improve the efficiency and accuracy of convolution calculations and complete complex convolutional neural network calculation tasks.

[0076] Figure 5 A flowchart of a method for convolution acceleration calculation provided by an embodiment of the present disclosure.

[0077] like Figure 5 As shown, the method comprises the following steps: In step 201, the original convolution kernel is split into multiple one-bit sub-convolution kernels according to the bit position, and the sub-convolution kernels are aggregated to generate a base convolution kernel according to the frequency of the target value appearing at different positions.

[0078] In the embodiments of the present disclosure, during the computational optimization of a convolutional neural network, the original convolution kernel typically has multiple bits. To achieve more efficient computation, this kernel must be processed. First, step 201 is performed to split the original convolution kernel by bit. For example, for a k-bit original convolution kernel, k one-bit sub-convolution kernels are generated, each consisting of a one-bit value for each weight in the original convolution kernel at a specific bit position. After the splitting is completed, the frequency of occurrence of the target value (usually 1) at different positions in the sub-convolution kernel is counted. Taking a two-dimensional convolution kernel as an example, for each coordinate position of each sub-convolution kernel, the number of occurrences of the target value at that position across all sub-convolution kernels is counted. Based on the statistical results, the target values at positions with higher frequency are extracted. For positions that are not shared by different sub-convolution kernels, their weights are set to zero, and then aggregated to generate a base convolution kernel. This generation method can concentrate the repetitive calculations in the sub-convolution kernels into the base convolution kernel, effectively reducing the amount of subsequent computation.

[0079] In step 202, each base convolution kernel of the original convolution kernel is convolved with the input target activation.

[0080] In the embodiment of the present disclosure, after the generation of the base convolution kernel is completed, step 202 is entered. In the convolution calculation, the target activation is an important data involved in the operation. Each base convolution kernel obtained by decomposing the original convolution kernel is convolved with the input target activation. In actual hardware implementation, this operation is usually performed by a convolution calculation unit. Each convolution calculation unit is configured with a base convolution kernel, and according to the convolution operation rules, the weight of the base convolution kernel is calculated accordingly with the target activation value to determine the output activation value. This step provides intermediate data for the subsequent reconstruction of the calculation result of the original convolution kernel.

[0081] In step 203, based on the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel, the convolution calculation result of each base convolution kernel and the target activation is calculated to obtain the convolution calculation result of the original convolution kernel and the target activation.

[0082] In the embodiment of the present disclosure, when the original convolution kernel is decomposed, the corresponding base convolution kernel information and its expansion coefficient are generated. The base convolution kernel information includes the composition of the base convolution kernel, the positional relationship in the original convolution kernel, etc., and the expansion coefficient is used to adjust the weight of the base convolution kernel calculation result in the reconstruction process. Since the expansion coefficients are all powers of 2, this characteristic is utilized to use the shift accumulation method in the calculation process. The specific operation is to extract data from the convolution calculation results of each base convolution kernel and the target activation according to the base convolution kernel information, and shift these results according to the corresponding expansion coefficient, that is, when the expansion coefficient is 2^n, the result is shifted left by n bits. After the shift, all shifted results are accumulated. When the calculation results of all relevant base convolution kernels are shifted and accumulated, the final result is the convolution calculation result of the original convolution kernel and the target activation. This step accurately restores the calculation process of the original convolution kernel by integrating the calculation results of the base convolution kernel, and at the same time improves the calculation efficiency by utilizing the characteristics of the expansion coefficient.

[0083] The present disclosure provides a method for accelerating convolution calculations. In the present disclosure, a base convolution kernel is generated by splitting the original convolution kernel by bit and aggregating the results based on the frequency of target values appearing at different locations. This generation method can concentrate the repeated calculations in multiple 1-bit sub-convolution kernels into the base convolution kernel. During convolution calculation, each convolution calculation unit only needs to process one base convolution kernel and calculate the repeated parts only once. Compared with the traditional bit-serial calculation method, this method can significantly reduce the amount of repeated calculations and improve the overall efficiency of convolution calculations. Because the base convolution kernel is 1 bit, the weights have only two values, 0 and 1. The calculations corresponding to zero-valued weights are invalid. In actual calculations, the convolution calculation unit can skip the calculation of these zero-valued weights, avoiding the waste of computing resources caused by invalid calculations, allowing computing resources to be used more efficiently for effective calculations, thereby improving computing resource utilization. During the calculation process, multiple convolution calculation units reuse activation values to reduce memory access. This method reduces hardware storage and memory access pressure, reduces the complexity of hardware implementation, and thus reduces hardware costs. By eliminating duplicate calculations and ineffective computations involving zero-valued weights, computing power consumption is reduced. This also reduces memory accesses, which in turn reduces memory power consumption. The convolution reconstruction unit reconstructs the results of the base convolution kernel into the original convolution kernel. This improves hardware throughput and significantly enhances energy efficiency, thereby enhancing the overall performance of the convolution accelerator in applications such as neural network inference, better meeting the high-throughput, low-power requirements of terminal devices.

[0084] In order to clearly illustrate the embodiments of the present disclosure, this embodiment provides a flowchart of another method for convolution acceleration calculation.

[0085] like Figure 6 As shown, the method comprises the following steps: Step 301: Split the original convolution kernel into multiple one-bit sub-convolution kernels according to the bit position.

[0086] Specifically, in step 301, the original convolution kernel is split into multiple one-bit sub-convolution kernels according to the bit position. In the calculation of convolutional neural networks, the original convolution kernel usually has multiple bits. When using the bit serial calculation method, the original convolution kernel needs to be disassembled according to the bit position. For example, for a k-bit original convolution kernel, it will be split into k one-bit sub-convolution kernels. Each sub-convolution kernel is composed of a one-bit value of each weight in the original convolution kernel at a specific bit position. For example, the value of the m-th bit of a certain weight of the original convolution kernel constitutes the weight value of the corresponding sub-convolution kernel. Such a split is the basis for subsequent optimization calculations, so that convolution calculations can be processed based on one-bit weights. For example, Figure 2 The figure shows how to split the original 2-bit convolution kernel into two 1-bit sub-convolution kernels and generate a base convolution kernel based on these two 1-bit sub-convolution kernels.

[0087] Step 302: Calculate and generate the expansion coefficient of the base convolution kernel according to the expansion coefficients of the base convolution kernel in different sub-convolution kernels.

[0088] Specifically, in step 302, the expansion coefficient of the base convolution kernel is calculated based on the expansion coefficients of the base convolution kernel in different sub-convolution kernels. After the multiple one-bit sub-convolution kernels are split in step 301, each sub-convolution kernel has its corresponding expansion coefficient. Usually, the expansion coefficient of the k-th bit sub-convolution kernel is 2^k, as shown in Figure 2 In the example, the expansion factor of the first-bit subconvolution kernel is 2. The base convolution kernel is generated by aggregating the partial weights of multiple subconvolution kernels. When generating a base convolution kernel, its expansion factor is calculated based on the expansion factors of each subconvolution kernel that contains the weight pattern of the base convolution kernel. The expansion factor of these subconvolution kernels is accumulated to obtain the expansion factor of the base convolution kernel. This expansion factor is used in the subsequent convolution reconstruction process to accurately restore the convolution result of the base convolution kernel to the convolution result of the original convolution kernel.

[0089] Step 303: Calculate the occurrence frequencies of target values at different coordinates in multiple one-bit sub-convolution kernels.

[0090] Specifically, in step 303, the frequency of occurrence of target values at different coordinates in multiple one-bit sub-convolution kernels is calculated. After splitting to obtain multiple one-bit sub-convolution kernels, the number of occurrences of the target value (here the target value is usually 1) at each coordinate position in these sub-convolution kernels is counted. Figure 2For example, for each coordinate point of each sub-convolution kernel, the frequency of the target value at that coordinate position in all sub-convolution kernels is counted. This frequency calculation can understand the concentration of target values at different coordinate positions, providing a data basis for subsequently determining the weight distribution of the base convolution kernel.

[0091] Step 304: Determine target coordinate positions where at least two sub-convolution kernels share the same target value based on the frequency of occurrence.

[0092] Specifically, in step 304, based on the frequency of occurrence, the target coordinate positions where at least two sub-convolution kernels share the same target value are determined. After obtaining the frequency of occurrence of the target values at different coordinates through step 303, the coordinate positions with higher frequency of occurrence are screened out. These coordinate positions mean that the target values frequently appear in multiple sub-convolution kernels. That is, find those coordinate points where at least two sub-convolution kernels have target values at the same coordinate position. These coordinate points are the key positions for the subsequent construction of the base convolution kernel. These coordinate positions that share the same target value represent the repeated calculation parts between the sub-convolution kernels. Concentrating these parts into the base convolution kernel can effectively reduce the amount of calculation. Please refer to Figure 2 The process of generating the medium-based convolution kernel.

[0093] Step 305: Aggregate the target value of the target coordinate position and the zero-value weights of the non-public coordinates to generate a base convolution kernel.

[0094] Specifically, in step 305, the target value of the target coordinate position and the zero-value weight of the non-public coordinate are aggregated to generate a base convolution kernel. After determining the target coordinate position where at least two sub-convolution kernels share the same target value, the target values at these coordinate positions are retained, and for those coordinate positions that are not shared in different sub-convolution kernels, their weights are set to zero. In this way, the repeated parts in multiple sub-convolution kernels are extracted and aggregated to form a base convolution kernel. The base convolution kernel contains the most repetitive weight pattern among multiple sub-convolution kernels. In subsequent convolution calculations, only the base convolution kernel needs to be calculated, and then combined with its expansion coefficient to reconstruct the convolution result of the original convolution kernel, thereby reducing a large amount of repeated calculations and improving the efficiency of convolution calculations.

[0095] Step 306: Based on the base convolution kernel information, obtain the output activation value obtained by convolving the base convolution kernel with the target activation.

[0096] Specifically, in step 306, based on the base convolution kernel information, the output activation value obtained by the convolution calculation of the base convolution kernel and the target activation is obtained. After completing the generation of the base convolution kernel and inputting the base convolution kernel and the target activation into the convolution calculation unit for convolution calculation, it is necessary to obtain the results generated by these calculations. The base convolution kernel information contains the structure of the base convolution kernel, the positional relationship in the original convolution kernel, and other content. This information is an important index for obtaining the corresponding calculation results. After completing the convolution calculation of the base convolution kernel and the target activation, the convolution calculation unit will store the calculated output activation value in a specific storage area. Based on the base convolution kernel information, the system can accurately locate and extract the output activation value corresponding to each base convolution kernel from the storage area, providing a data basis for subsequent calculations. This step ensures the accuracy and completeness of the calculation results used in the convolution reconstruction process, and is a necessary prerequisite for further processing the base convolution kernel calculation results to restore the original convolution kernel calculation results.

[0097] Step 307: Perform shift accumulation processing on the output activation value and the expansion coefficient to reconstruct the convolution calculation result of the original convolution kernel and the target activation.

[0098] Specifically, in step 307, the output activation value and the expansion coefficient are shifted and accumulated to reconstruct the convolution calculation result of the original convolution kernel and the target activation. After obtaining the output activation value obtained by the convolution calculation of the base convolution kernel and the target activation, since the base convolution kernel is decomposed from the original convolution kernel, each base convolution kernel has its own corresponding expansion coefficient. These expansion coefficients are determined when generating the base convolution kernel based on the expansion coefficient of the one-bit sub-convolution kernel containing the base convolution kernel. In this step, the output activation value is shifted by the corresponding expansion coefficient, taking advantage of the fact that the expansion coefficient is a power of 2. The number of bits shifted is determined by the corresponding expansion coefficient. For example, if the expansion coefficient is 2^k, the output activation value is shifted to the left by k bits. After the shift is completed, the shifted output activation value is accumulated. By performing the shift and accumulation process on the output activation value of each base convolution kernel in turn, the calculation results of each base convolution kernel are gradually integrated. When the output activation values of all relevant base convolution kernels have completed the shift and accumulation, the final result is the convolution calculation result of the original convolution kernel and the target activation. This process efficiently realizes the reconstruction from the calculation results of the base convolution kernel to the original convolution kernel through the clever use of the expansion coefficient, restores the complete convolution calculation process, and at the same time reduces redundant operations in the calculation process, thereby improving the calculation efficiency and system performance.

[0099] It should be noted that the embodiments of the present disclosure may include multiple steps. For the convenience of description, these steps are numbered, but these numbers do not limit the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.

[0100] The embodiment of the present disclosure provides a board including: a memory device, an interface device and a control device and a convolution acceleration calculator as in the aforementioned embodiment; wherein the convolution acceleration calculator is connected to the memory device, the control device and the interface device respectively; the memory device is used to store data; the interface device is used to realize data transmission between the convolution acceleration calculator and an external device; the control device is used to monitor and manage the status of the chip. The aforementioned explanations of the embodiment and method embodiment of the convolution acceleration calculator are also applicable to the board of this embodiment, and the principles are the same, and are no longer limited in this embodiment. The same descriptions of the aforementioned embodiments are also applicable to the electronic device provided by the embodiment of the present disclosure.

[0101] The various numerical numbers such as first and second involved in the present disclosure are only for the convenience of description and are not used to limit the scope of the embodiments of the present disclosure, and also indicate the order of precedence.

[0102] The at least one in the present disclosure can also be described as one or more, and the multiple can be two, three, four or more, which is not limited in the present disclosure. In the embodiments of the present disclosure, for a technical feature, the technical features in the technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", and there is no order of precedence or size between the technical features described by "first", "second", "third", "A", "B", "C" and "D".

[0103] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0104] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for accelerating convolution calculation, characterized in that: The method comprises: The original convolution kernel is split into multiple one-bit sub-convolution kernels according to the bit position, and the sub-convolution kernels are aggregated to generate a base convolution kernel according to the frequency of the target value appearing at different positions; Convolution calculation is performed on each base convolution kernel of the original convolution kernel and the input target activation respectively; According to the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel, the convolution calculation result of each base convolution kernel and the target activation is calculated to obtain the convolution calculation result of the original convolution kernel and the target activation.

2. The method for accelerating convolution calculation according to claim 1, wherein: The step of aggregating the sub-convolution kernels to generate a base convolution kernel according to the frequencies at which target values appear at different positions includes: Calculating the occurrence frequencies of target values at different coordinates in the multiple one-bit sub-convolution kernels; Based on the occurrence frequency, determining a target coordinate position where at least two of the sub-convolution kernels share the same target value; The target value of the target coordinate position and the zero-value weights of the non-public coordinates are aggregated to generate the base convolution kernel.

3. The method for convolution acceleration calculation according to claim 1, characterized in that: The method further comprises: The expansion coefficient of the base convolution kernel is calculated and generated according to the expansion coefficient of the base convolution kernel in different sub-convolution kernels.

4. The method for accelerating convolution calculation according to claim 1, wherein: The step of calculating the convolution calculation result of each base convolution kernel with the target activation based on the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel comprises: Based on the base convolution kernel information, obtaining an output activation value obtained by convolving the base convolution kernel with the target activation; The output activation value and the expansion coefficient are shifted and accumulated to reconstruct the convolution calculation result of the original convolution kernel and the target activation.

5. A convolution acceleration calculator, characterized in that: include: Base convolution kernel storage unit, multiple convolution calculation units, and convolution reconstruction unit; The output end of the base convolution kernel storage unit is connected to the input end of the multiple convolution calculation units, and the output end of the multiple convolution calculation units is connected to the input end of the convolution reconstruction unit; The base convolution kernel storage unit is used to store the base convolution kernel, wherein the base convolution kernel is split into multiple one-bit sub-convolution kernels according to the bit position of the original convolution kernel, and is aggregated and generated according to the frequency of the target value appearing at different positions; The convolution calculation unit is used to perform convolution calculation on the configured base convolution kernel and the input target activation, and each base convolution kernel corresponding to the original convolution kernel is respectively configured to a different convolution calculation unit; The convolution reconstruction unit is used to calculate the convolution calculation results output by each of the convolution calculation units according to the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel, so as to obtain the convolution calculation result of the original convolution kernel and the target activation.

6. The convolution acceleration calculator according to claim 5, wherein: The base convolution kernel storage unit includes: a base convolution kernel memory and a sparse weight decoder; The output end of the base convolution kernel memory is connected to the input end of the sparse weight decoder, and the output end of the sparse weight decoder is connected to the input ends of the multiple convolution calculation units; The base convolution kernel memory is used to store the coded base convolution kernel obtained by performing sparse coding processing on the base convolution kernel; The sparse weight decoder is used to decode the encoded base convolution kernel based on the width information and height information of the base convolution kernel to restore the coordinates of the target value in the base convolution kernel and generate the restored base convolution kernel.

7. The convolution acceleration calculator according to claim 5, wherein: The convolution acceleration calculator further includes: activating a storage unit; The activation storage unit includes: an activation memory and a sparse activation decoder; An output of the activation memory is connected to an input of the sparse activation decoder, and an output of the sparse activation decoder is connected to inputs of the plurality of convolutional computing units; The activation memory is used to receive and store non-zero activations, where the non-zero activations are obtained by sparsely encoding the input feature map; The sparse activation decoder is used to restore the non-zero activation to restore the non-zero value and the coordinates thereof in the non-zero activation to obtain the target activation.

8. The convolution acceleration calculator according to claim 5, wherein: The convolution calculation unit includes: a coordinate calculator, an adder, and an output activation memory; The input end of the coordinate calculator is connected to the output end of the base convolution kernel storage unit and the output end of the activation storage unit respectively, the output end of the coordinate calculator is connected to the input end of the output activation memory, and the output end of the adder is connected to the input end of the output activation memory; The coordinate calculator is used to perform coordinate matching calculation according to the target value coordinates of the base convolution kernel and the non-zero value coordinates of the target activation to obtain a matching activation value; The adder is configured to accumulate the non-zero value of the target activation and the matching activation value to obtain an output activation value; The output activation memory is used to store intermediate calculation results of the output activation value.

9. The convolution acceleration calculator according to claim 5, wherein: The convolution acceleration calculator further includes: a convolution kernel decomposition information memory; The output end of the convolution kernel decomposition information memory is connected to the input end of the convolution reconstruction unit; the convolution kernel decomposition information memory is used to receive and store the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel.

10. The convolution acceleration calculator according to claim 5, wherein: The convolution reconstruction unit includes: a reconstruction scheduler, a plurality of activation value memories, and a plurality of shift accumulators; An input end of the reconstruction scheduler is connected to an output end of the convolution calculation unit, an output end of the reconstruction scheduler is connected to input ends of the plurality of shift accumulators, and the plurality of shift accumulators are respectively connected to the plurality of activation value memories corresponding thereto; The reconstruction scheduler is used to obtain the base convolution kernel information and its expansion coefficient obtained by decomposing the original convolution kernel; read the output activation value stored in the convolution calculation unit according to the base convolution kernel information, and input the output activation value and the expansion coefficient into the corresponding shift accumulator for calculation; The shift accumulator is used to perform shift accumulation calculation according to the output activation value and the expansion coefficient, and input the shift accumulation calculation result into the activation value memory for storage and update until the convolution calculation result of the base convolution kernel is accumulated, thereby generating the convolution calculation result of the original convolution kernel and the target activation.

11. The convolution acceleration calculator according to any one of claims 5 to 10, characterized in that: The convolution acceleration calculator further includes: a controller; The controller is respectively connected to the base convolution kernel storage unit, the multiple convolution calculation units, the convolution reconstruction unit, and the activation storage unit; The controller is used to control the base convolution kernel storage unit and the activation storage unit for storage access, and to control the flow of calculation data of the multiple convolution calculation units and the convolution reconstruction unit.

12. A board, characterized in that: The board includes: a storage device, an interface device and a control device, and a convolution acceleration calculator as described in any one of claims 5 to 11; wherein the convolution acceleration calculator is connected to the storage device, the control device and the interface device respectively; the storage device is used to store data; the interface device is used to realize data transmission between the convolution acceleration calculator and an external device; the control device is used to monitor and manage the status of the convolution acceleration calculator.

13. An electronic device, characterized in that: include: At least one board as claimed in claim 12.

Images