Intelligent processing unit and convolution operation method
By using quantization and order-accumulation operations in the intelligent processing unit, the problem of balancing accuracy and cost in convolution operations is solved, achieving efficient convolution operation results that are suitable for processing image and speech data.
Patent Information
- Application Number
- CN202511429511.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-30
- Publication Date
- 2026-02-03
AI Technical Summary
Existing technologies suffer from severe accuracy loss or excessive cost in convolution operations, making it difficult to strike a balance between accuracy and cost.
The system employs an intelligent processing unit, including a memory, a quantization circuit, a multiply-accumulate circuit, a displacement calculation circuit, a parallel accumulation circuit, and a data type conversion circuit. It quantizes floating-point data into fixed-point data, performs multiply-accumulate, displacement calculation, and parallel accumulation operations, and finally converts the data into floating-point output data.
It achieves a balance between accuracy and cost, improves computational efficiency, reduces the reading operation burden of electronic devices, and is applicable to weighted data of various shapes, thereby enhancing the application flexibility and effectiveness of electronic devices.
Smart Images
Figure CN121455445A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of convolution operation technology, specifically to intelligent processing units and convolution operation methods. Background Technology
[0002] Figure 1A and Figure 1B This is a flowchart of existing convolution operations. Figure 1A Corresponding to electronic devices that perform fixed-point arithmetic, Figure 1B This corresponds to electronic devices that perform floating-point operations. Figure 1A and Figure 1B In convolution operations, the data (e.g., input data (or input feature map) and weights (or convolution kernel)) are floating-point data.
[0003] right Figure 1A In the preparation stage, floating-point data is first converted into fixed-point data through statistical quantization (step S110), then configuration parameters are generated based on the fixed-point data and the hardware specifications of the electronic device (step S120), and finally, convolution operation based on fixed-point numbers is performed on the electronic device (step S130).
[0004] right Figure 1B In the preparation phase, the floating-point data is first converted to a new format (step S115, for example, from floating-point FP32 format to floating-point FP16 format), then configuration parameters are generated based on the floating-point data and the hardware specifications of the electronic device (step S125), and finally, a convolution operation based on floating-point numbers is performed on the electronic device (step S135).
[0005] Figure 1A The advantage of this approach is its relatively low cost, but the disadvantage is that it suffers from a relatively significant loss of accuracy. Figure 1B The advantage of this approach is its relatively high accuracy, but its disadvantage is its relatively high cost. Therefore, a solution with relatively high accuracy and relatively low cost is needed. Summary of the Invention
[0006] In view of the shortcomings of the prior art, one objective of this application is to provide an intelligent processing unit and a convolution operation method to improve the shortcomings of the prior art.
[0007] One embodiment of this application provides an intelligent processing unit, including: a memory, a quantization circuit, a multiply-accumulate-add circuit, a displacement calculation circuit, a pair of alignment and addition circuits, and a data type conversion circuit. The memory stores a first input data and a first weight data. The quantization circuit, coupled to the memory, quantizes the first input data to generate multiple second input data and an input data displacement, and quantizes the first weight data to generate multiple second weight data and a weight data displacement. The multiply-accumulate-add circuit, coupled to the quantization circuit, performs a multiply-accumulate-add operation on the second input data and the second weight data to generate an intermediate result. The displacement calculation circuit, coupled to the quantization circuit, generates an intermediate result displacement based on the input data displacement and the weight data displacement. The alignment and addition circuit, coupled to the multiply-accumulate-add circuit and the displacement calculation circuit, performs a pair of alignment and addition operations on the intermediate result and an intermediate addition result based on the intermediate result displacement to generate a final addition result. A data type conversion circuit, coupled to the parallel accumulator circuit and the memory, is used to convert the final accumulation result to produce an output data. The first input data and the first weight data are floating-point data, while the second input data and the second weight data are fixed-point data.
[0008] Another embodiment of this application provides a convolution operation method, including the following steps: (A) quantizing a first weight data to generate a plurality of second weight data and a weight data displacement; (B) quantizing a first input data to generate a plurality of second input data and an input data displacement; (C) performing a multiplication-accumulation operation on the second input data and the second weight data to generate an intermediate result; (D) generating an intermediate result displacement based on the input data displacement and the weight data displacement; (E) performing a one-order accumulation operation on the intermediate result and the intermediate accumulation result based on the intermediate result displacement to generate a final accumulation result; and (F) converting the final accumulation result to generate output data. The first input data and the first weight data are floating-point data, while the second input data and the second weight data are fixed-point data.
[0009] The technical means embodied in the embodiments of this application can improve at least one of the shortcomings of the prior art. Therefore, this application can achieve a better balance between accuracy and cost compared with the prior art.
[0010] The features, implementation, and effects of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description
[0011] Figure 1A and Figure 1B This is a flowchart of the existing convolution operation;
[0012] Figure 2 This is a functional block diagram of an embodiment of the electronic device of this application;
[0013] Figure 3 This is a schematic diagram of the input data, weight data, and final summation result of the convolution operation in this application;
[0014] Figure 4 This is a flowchart of one embodiment of the quantization operation in this application;
[0015] Figure 5 It corresponds to Figure 4 Data flow diagram;
[0016] Figure 6A and Figure 6B This is a flowchart of an embodiment of the order-accumulation operation of this application;
[0017] Figure 7 It corresponds to Figure 6A and Figure 6B Data flow diagram; and
[0018] Figure 8A and Figure 8B This is a flowchart of an embodiment of the convolution operation in this application.
[0019] Symbol explanation:
[0020] S110, S115, S120, S125, S130, S135, S410, S420, S430, S440, S450, S460, S605, S610, S620, S630, S640, S650, S660, S670, S680, S690, S810, S820, S830, S840, S850, S860, S870, S880, S890, S895: Steps;
[0021] 200: Electronic devices;
[0022] 210: External memory;
[0023] 220: Intelligent processing unit;
[0024] 221: Memory;
[0025] 222: Quantization circuit;
[0026] 223: Multiply-accumulate circuit;
[0027] 224: Displacement calculation circuit;
[0028] 225: Alignment accumulator circuit;
[0029] 226: Data type conversion circuit;
[0030] ABf[k]: The accumulated result in the intermediate steps;
[0031] ABf[k-1]: Temporary intermediate accumulation result;
[0032] ABf_F: The final accumulated result;
[0033] ABi: Intermediate result;
[0034] D, H, W: Dimensions;
[0035] IBf, IBi: Input data;
[0036] KBf, KBi: Weight data;
[0037] OBf: Output data;
[0038] ST_AB: Intermediate result displacement;
[0039] ST_IB: Input data displacement;
[0040] ST_KB: Weight data displacement;
[0041] ABf1_A, ABf2_A, ABfM_A: Data blocks;
[0042] IBf_A: Input data block;
[0043] KBf1, KBf2, KBfM: Sub-weight data;
[0044] KBf1_A, KBf2_A, KBfM_A: Weighted data blocks;
[0045] Exp_1, Exp_2, Exp_p, Exp_X: Exponential parts;
[0046] Exp_Dif: Exponential difference;
[0047] Exp_Max: The largest exponential portion;
[0048] Fd_1, Fd_2, Fd_p, Fd_X: fixed-point part;
[0049] ST_x, SH_i, SH_V: Displacement amounts;
[0050] Tmp: Intermediate data;
[0051] xBf_1, xBf_2, xBf_p, xBf_X: tensor;
[0052] xBi: Fixed-point data;
[0053] ABis: Intermediate result after shifting;
[0054] Exp_i: The intermediate exponent portion;
[0055] Fxd_i: The intermediate fixed-point part;
[0056] Fxd_is: The intermediate fixed-point part after shifting;
[0057] Fxd_tar: Number of target fixed points;
[0058] Fxd_tmp: Temporary fixed-point number;
[0059] P2S_Max: A larger exponent;
[0060] P2S_tar: Target index. Detailed Implementation
[0061] The technical terms used in the following description are based on the customary terms in this technical field. If this specification provides explanations or definitions for certain terms, the explanations or definitions in this specification shall prevail.
[0062] The disclosure of this application includes an intelligent processing unit and a convolution operation method. Since some components of the intelligent processing unit may be known individually, details of known components will be omitted in the following description without affecting the full disclosure and implementability of the device invention. Furthermore, the convolution operation method of this application can be executed by the intelligent processing unit of this application or its equivalent device. Without affecting the full disclosure and implementability of the method invention, the following description of the method invention will focus on the steps rather than the hardware.
[0063] Figure 2 This is a functional block diagram of an embodiment of the electronic device of this application. The electronic device 200 includes an external memory 210 (e.g., Dynamic Random Access Memory (DRAM)) and an intelligence processing unit (IPU) 220, which are coupled to each other. The intelligence processing unit 220 includes a memory 221 (e.g., Static Random Access Memory (SRAM)), a quantization circuit 222, a multiply-accumulate circuit 223, a displacement calculation circuit 224, an order-aligned accumulation circuit 225, and a data type conversion circuit 226, which are coupled to each other.
[0064] The intelligent processing unit 220 retrieves data (including input data IBf and weight data KBf) for convolution operations from the external memory 210. Both the input data IBf and the weight data KBf are floating-point data.
[0065] Quantization circuit 222 is coupled to or electrically connected to memory 221, and is used to perform quantization operations on input data IBf and weight data KBf to generate input data IBi, weight data KBi, input data displacement ST_IB, and weight data displacement ST_KB. In one embodiment, input data IBf and weight data KBf are both floating-point data. Input data IBi and weight data KBi are both fixed-point data. Input data displacement ST_IB and weight data displacement ST_KB correspond to input data IBi and weight data KBi, respectively.
[0066] The multiply-accumulate circuit 223 is coupled to or electrically connected to the quantization circuit 222, and is used to multiply and accumulate the input data IBi and the weight data KBi to produce the intermediate result ABi. The multiply-accumulate operation performed by the multiply-accumulate circuit 223 is a fixed-point multiply-accumulate operation, that is, the intermediate result ABi is fixed-point data.
[0067] The displacement calculation circuit 224 is coupled to or electrically connected to the quantization circuit 222, and is used to generate an intermediate result displacement ST_AB based on the input data displacement ST_IB and the weight data displacement ST_KB. For example, the intermediate result displacement ST_AB is equal to the sum of the input data displacement ST_IB and the weight data displacement ST_KB. The intermediate result displacement ST_AB corresponds to the intermediate result ABi.
[0068] The order-accumulating circuit 225 is coupled or electrically connected to the multiplication-accumulating circuit 223 and the displacement calculation circuit 224, and is used to perform order-accumulating operations based on the intermediate result ABi and the intermediate result displacement ST_AB. The intermediate accumulation result ABf[k] is floating-point data. The parameter k represents the k-th intermediate accumulation result (1≦k≦Y, where Y is an integer greater than or equal to 1; Y will be used in conjunction with the following...). Figure 3 (Details to follow). ABf[0] is the initial value, while the final accumulated result ABf_F (i.e. ABf[Y]) is the result of one convolution operation. The intermediate accumulated result ABf[k] and the final accumulated result ABf_F are both floating-point data.
[0069] Data type conversion circuit 226 is coupled or electrically connected to parallel accumulator circuit 225 to perform data conversion on the final accumulation result ABf_F (e.g., from floating-point FP32 format to floating-point FP16 format) to generate output data OBf (or output feature map). Output data OBf is floating-point data. Details of the data type conversion operation are well known to those skilled in the art and will not be described further. Output data OBf is first stored in memory 221 and then moved to external memory 210.
[0070] Figure 3 This diagram illustrates the input data IBf, weight data KBf, and the final accumulated result ABf_F for the convolution operation in this application. It is assumed that the dimension ([W, H, D]) of the input data IBf is [6, 6, X] (where X is an integer greater than or equal to 1); the weight data KBf includes M sub-weight data (KBf1, KBf2, ..., KBfM), and the dimension of each sub-weight data is [3, 3, X]; the dimension of the final accumulated result ABf_F is [4, 4, M]. The Y value mentioned above represents the number of weight data blocks included in a sub-weight data. Figure 3 In the example, since each sub-weight data (KBf1, KBf2, ..., KBfM) includes 9 weight data blocks (each weight data block has a dimension of [1, 1, X], i.e., a size of 1*1*X), Y equals 9 (i.e., the product of the sub-weight data dimension W and the dimension H). The final accumulated result ABf_F corresponds to the sub-weight data KBf1, sub-weight data KBf2, ..., sub-weight data KBfM in dimension D. More specifically, the data in the first (D=1) WH plane is the result of the convolution operation between the input data IBf and the sub-weight data KBf1; similarly, the data in the second (D=2) and the Mth (D=M) WH planes are the results of the convolution operations between the input data IBf and the sub-weight data KBf2 and sub-weight data KBfM, respectively. Those skilled in the art are familiar with the details of convolution operations, so they will not be elaborated further.
[0071] Figure 4 This is a flowchart of one embodiment of the quantization operation in this application. Figure 5 It corresponds to Figure 4 Data flow diagram. Figure 4 The process can be executed by the quantization circuit 222, and Figure 5 The steps are marked in the middle. Figure 5The top layer displays X tensors: xBf_1, xBf_2, ..., xBf_p, ..., and xBf_X (1 ≦ p ≦ X), and each tensor is floating-point data. Each tensor is a data block of input data IBf (e.g., the top-left input data block IBf_A, dimension: [1, 1, X]), or a data block of weight data KBf (e.g., weight data blocks KBf1_A, KBf2_A, or KBfM_A, dimension: [1, 1, X]) with a dimension of [1, 1, 1]. Please refer to the following explanation. Figure 4 and Figure 5 .
[0072] Step S410: Decompose a floating-point tensor into a fixed-point part and an exponent part. For example... Figure 5 As shown, tensor xBf_1 is decomposed into a fixed-point part Fd_1 (i.e., the mantissa, which is a fixed-point number) and an exponent part Exp_1; tensor xBf_2 is decomposed into a fixed-point part Fd_2 and an exponent part Exp_2; tensor xBf_p is decomposed into a fixed-point part Fd_p and an exponent part Exp_p; and tensor xBf_X is decomposed into a fixed-point part Fd_X and an exponent part Exp_X.
[0073] Step S420: Find the largest among all the exponential parts Exp_1 to Exp_X to obtain the largest exponential part Exp_Max.
[0074] Step S430: Shift the fixed-point portion Fd_p to the left by a preset number of bits to obtain the intermediate data Tmp. Specifically, when the input data IBf is FP16, this preset number of bits is 4 bits. When the input data IBf is BP16, this preset number of bits is 7 bits.
[0075] Step S440: Subtract the exponential part Exp_p from the largest exponential part Exp_Max to obtain the exponential difference Exp_Dif.
[0076] Step S450: Shift the intermediate data Tmp to the right based on the exponent difference Exp_Dif to obtain the fixed-point data xBi. Specifically, for converting from floating-point FP16 or BF16 format to fixed-point INT16 format, the number of bits shifted right by the intermediate data Tmp is equal to the exponent difference Exp_Dif. For converting from floating-point FP16 or BF16 format to fixed-point INT8 format, the number of bits shifted right by the intermediate data Tmp is equal to the sum of the exponent difference Exp_Dif and 8.
[0077] Step S460: Subtract a preset value from the largest exponent portion Exp_Max to obtain the displacement ST_x. The purpose of subtracting the preset value is to correct the decimal point position. For converting from floating-point FP16 format to fixed-point INT16 format, the preset value is 29. For converting from floating-point FP16 format to fixed-point INT8 format, the preset value is 21. For converting from floating-point BF16 format to fixed-point INT16 format, the preset value is 141. For converting from floating-point FP16 format to fixed-point INT8 format, the preset value is 133. Those skilled in the art are familiar with how to determine the preset value based on the format before and after conversion, so it will not be described further.
[0078] For the input data, the fixed-point data xBi and the displacement ST_x correspond to respectively... Figure 2 The input data IBi and the input data displacement ST_IB. For the weighted data, the fixed-point data xBi and the displacement ST_x correspond to... Figure 2 The weight data KBi and the weight data displacement ST_KB.
[0079] Please note that, for the sake of simplicity in the accompanying diagrams, Figure 5 The code only displays the intermediate data Tmp and exponential difference Exp_Dif corresponding to the tensor xBf_p. However, there are actually X intermediate data, X exponential differences, and X fixed-point data. That is, after the data block (input data block IBf_A, weight data block KBf1_A, KBf2_A, or KBfM_A) is quantized by the quantization circuit 222, it will yield X fixed-point data xBi and a displacement ST_x. In other words, each input data block (e.g., IBf_A) includes X input data IBi (size 1*1*1), and each weight data block (e.g., KBf1_A, KBf2_A, KBfM_A) includes X weight data KBi (size 1*1*1).
[0080] Please see Figure 3 Because during the convolution operation, the sub-weight data KBf1 (KBf2 or KBfM) slides across the input data IBf, each weight data block of the sub-weight data KBf1 (KBf2 or KBfM) corresponds to L input data blocks (L is an integer greater than or equal to 1). Figure 3In the example, L is 16, which is the product of dimension W and dimension H of the final accumulated result ABf_F. The multiply-accumulate circuit 223 performs a multiply-accumulate operation on X input data IBi and X weight data KBi to obtain an intermediate result ABi. That is, the multiply-accumulate circuit 223 will obtain an intermediate result ABi after performing a multiply-accumulate operation on the input data block IBi and the weight data block KBf1_A. For example, each of the 9 data blocks of sub-weight data KBf1 corresponds to 16 intermediate results ABi (because of window sliding). Therefore, for data block ABf1_A, the order-accumulate circuit 225 performs an order-accumulate operation on the 9 intermediate results ABi corresponding to the 9 data blocks of sub-weight data KBf1 based on the intermediate result displacement ST_AB to obtain data block ABf1_A. Similarly, data blocks ABf2_A and ABfM_A correspond to sub-weight data KBf2 and sub-weight data KBfM, respectively.
[0081] In some embodiments, the quantization circuit 222 simultaneously generates the input data IBi and input data displacement ST_IB of the input data block IBf_A, and the weight data KBi and weight data displacement ST_KB of the weight data blocks KBf1_A, KBf2_A, and KBfM_A in parallel processing. The displacement calculation circuit 224 adds the weight data displacement ST_KB of the weight data blocks KBf1_A, KBf2_A, and KBfM_A to the input data displacement ST_IB of the input data block IBf_A, respectively, to obtain three intermediate result displacements ST_AB corresponding to the weight data blocks KBf1_A, KBf2_A, and KBfM_A. The displacement calculation circuit 224 can be implemented by a fixed-point adder.
[0082] Following the previous section, the order accumulation circuit 225 generates a portion of the data block ABf1_A (ABf2_A or ABfM_A) (i.e., the intermediate accumulation result ABf[k]) based on the intermediate result displacement ST_AB corresponding to the sub-weight data KBf1 (KBf2 or KBfM) and the intermediate result ABi, and temporarily stores this portion in the memory 221.
[0083] Figure 6A and Figure 6B This is a flowchart of an embodiment of the order accumulation operation of this application. Figure 7 It corresponds to Figure 6A and Figure 6B Data flow diagram. Figure 6A and Figure 6B The process can be executed by the alignment accumulator circuit 225, and Figure 7 The steps are marked in the middle.
[0084] Step S605: Read the previous intermediate accumulation result ABf[k] from memory 221 as a temporary intermediate accumulation result ABf[k-1].
[0085] Step S610: Obtain the intermediate fixed-point part Fxd_i and the intermediate exponent part Exp_i based on the temporary intermediate accumulation result ABf[k-1]. For example, when the temporary intermediate accumulation result ABf[k-1] is a number in FP32 format, the intermediate fixed-point part Fxd_i and the intermediate exponent part Exp_i are 24 bits and 10 bits of fixed-point data, respectively.
[0086] Please note that when steps S605 and S610 are executed for the first time, the memory 221 does not yet have the previous intermediate accumulation result ABf[k]. At this time, the alignment accumulation circuit 225 reads the initial value ABf[0] from the memory 221. The format of the initial value is the same as that of the intermediate accumulation result ABf[k] (it can be a custom format or a standard format (e.g., FP32 format)), and it can be generated based on the bias of the convolution operation. Generating the initial value based on the bias is well known to those skilled in the art, and will not be described further.
[0087] Step S620: Compare the intermediate exponent portion Exp_i with the intermediate result displacement ST_AB to generate the larger exponent P2S_Max. That is, the larger exponent P2S_Max is the larger of the intermediate exponent portion Exp_i and the intermediate result displacement ST_AB.
[0088] Step S630: Determine the displacement SH_i based on the larger exponent P2S_Max. The displacement SH_i is the difference between the larger exponent P2S_Max and the other. For example, if the intermediate exponent Exp_i and the intermediate result displacement ST_AB are -7 and -8 respectively, then the larger exponent P2S_Max is -7, and the displacement SH_i = -7 - (-8) = 1.
[0089] Step S640: Shift one of the intermediate fixed-point number part Fxd_i and the intermediate result ABi to the right by a displacement SH_i, and shift the other to the right by 0 bits. This shift corresponds to the intermediate result ABi or the intermediate fixed-point number part Fxd_i with the smaller exponent. Continuing with the previous example (where the intermediate exponent part Exp_i and the intermediate result displacement ST_AB are -7 and -8 respectively), because the intermediate result displacement ST_AB is smaller, this step shifts the intermediate result ABi to the right by a displacement SH_i, and shifts the intermediate fixed-point number part Fxd_i to the right by 0 bits.
[0090] Step S650: Add the shifted intermediate fixed-point number part Fxd_is to the shifted intermediate result ABis to obtain the temporary fixed-point number Fxd_tmp.
[0091] Step S660: Find the leading one (i.e., the highest bit 1) of the temporary fixed-point number Fxd_tmp, and generate a displacement SH_V based on the position of the leading one. For example, if the temporary fixed-point number Fxd_tmp is 0001...0011 (48 bits in total), then the displacement SH_V is equal to 4.
[0092] Step S670: Shift the temporary fixed point number Fxd_tmp to the left based on the displacement SH_V to generate the target fixed point number Fxd_tar, which is to shift the target fixed point number Fxd_tar to the left by the displacement SH_V.
[0093] Step S680: Generate the target exponent P2S_tar based on the displacement SH_V and the larger exponent P2S_Max. More specifically, the target exponent P2S_tar is equal to the larger exponent P2S_Max minus the displacement SH_V.
[0094] Step S690: Generate the intermediate accumulation result ABf[k] based on the target fixed-point number Fxd_tar and the target exponent P2S_tar, where the target fixed-point number Fxd_tar is the fixed-point part of the intermediate accumulation result ABf[k], and the target exponent P2S_tar is the exponent part of the intermediate accumulation result ABf[k].
[0095] Figure 8A and Figure 8B This is a flowchart of an embodiment of the convolution operation in this application. Figure 8A and Figure 8B It can be by Figure 2 The intelligent processing unit 220 executes the following steps.
[0096] Step S810: The quantization circuit 222 reads one weight data block from each of the M sub-weight data from the memory 221 as the target weight data block. That is to say (see [link to relevant documentation]). Figure 3 In this step, the quantization circuit 222 reads one weight data block (e.g., KBf1_A, KBf2_A, ..., KBfM_A) from each of the M sub-weight data (KBf1, KBf2, ..., KBfM) to obtain M target weight data blocks.
[0097] Step S820: The quantization circuit 222 quantizes the M target weight data blocks read, and generates X weight data KBi and a weight data displacement ST_KB for each target weight data block. In some embodiments, the quantization circuit 222 processes the M target weight data blocks in parallel to substantially generate M*X weight data KBi and M weight data displacement ST_KB simultaneously.
[0098] Step S830: Quantization circuit 222 reads an input data block (e.g., IBf_A) from memory 221.
[0099] Step S840: Quantization circuit 222 quantizes the read input data block and generates X input data IBi and one input data displacement ST_IB for the input data block.
[0100] Step S850: For each of the M target weight data blocks, the multiply-accumulate circuit 223 performs a multiply-accumulate operation on the X weight data KBi and the X input data IBi to obtain an intermediate result ABi (a total of M intermediate results ABi are obtained). Figure 3 For example, when this step is executed, M intermediate results ABi are generated, which correspond to data blocks ABf1_A, ABf2_A, ..., ABfM_A respectively.
[0101] Step S860: For each of the M target weight data blocks, the displacement calculation circuit 224 adds the weight data displacement ST_KB to the input data displacement ST_IB to obtain an intermediate result displacement ST_AB. In other words, this step generates a total of M intermediate result displacements ST_AB, corresponding to the M intermediate results ABi respectively.
[0102] Step S870: Based on the displacement amount ST_AB of the M intermediate results, the alignment accumulation circuit 225 performs alignment accumulation operation on the M intermediate results ABi and their corresponding temporary intermediate accumulation results ABf[k-1] to obtain M intermediate accumulation results ABf[k]. The temporary intermediate accumulation result ABf[k-1] is the intermediate accumulation result ABf[k] of the previous alignment accumulation operation and is temporarily stored in memory 221.
[0103] Step S880: The intelligent processing unit 220 determines whether all input data blocks corresponding to the target weight data block (e.g., KBf1_A, KBf2_A, ..., KBfM_A) have been processed. More specifically (in... Figure 3For example, during the convolution operation, the sub-weight data KBf1 (KBf2 or KBfM) slides on the input data IBf, so each weight data block of the sub-weight data KBf1 (KBf2 or KBfM) corresponds to L input data blocks (in...). Figure 3 In the example, L is 16). If the L input data blocks have not been processed, return to step S830 to read the next input data block corresponding to the target weight data block. If the L input data blocks have been processed (meaning that the M target weight data blocks for this round have been processed), proceed to step S890.
[0104] Step S890: The intelligent processing unit 220 determines whether all weighted data blocks have been processed. If not, it returns to step S810 to determine the M target weighted data blocks for the next round; otherwise, it proceeds to step S895. Figure 3 For example, since a sub-weight data includes 9 weight data blocks (i.e., the aforementioned Y values), steps S810 to S890 need to be executed 9 times (i.e., all weight data blocks will be selected as target weight data blocks in turn).
[0105] Step S895: Obtain the final accumulated result ABf_F, and then convert the data type of the final accumulated result ABf_F to generate the output data OBf.
[0106] like Figures 8A-8B As shown, the convolution operation process of this application includes a small loop (steps S830-S880) and a large loop (steps S810-S890). For one convolution operation, the small loop is executed L times, and the large loop is executed Y times. The advantage of this design is that it avoids repeatedly reading weight data, reducing the read operation burden on the intelligent processing unit 220. Figure 3 For example, each sub-weight data (KBf1, KBf2, ..., KBfM) will only be read once (that is, the weight data KBf will only be read once), while the input data IBf will be read at most 9 times (i.e., the aforementioned Y value, which is the number of weight data blocks included in each sub-weight data). For comparison (again using...) Figure 3 For example, in existing convolution operations, the input data IBf is read 9 times, and each sub-weight data is also read 9 times. Therefore, the intelligent processing unit 220 of this application significantly reduces the number of read operations, thereby improving the performance of the electronic device 200.
[0107] Furthermore, because of the large loop of the convolution operation in this application ( Figures 8A-8BSteps S810 to S890 are performed on a weight data block basis (instead of the convolution operation in the prior art which is performed on a sub-weight data block basis), so this application is applicable to weight data KBf of various shapes (e.g., 3*3*X, i.e. Figure 3 (as shown in the examples), 2*4*X, 6*5*X, etc., to enhance the application flexibility of electronic devices 200.
[0108] Furthermore, because the multiply-accumulate circuit 223 processes fixed-point numbers (which are lower in cost and easier to implement), this application significantly reduces the number of floating-point accumulations, thus improving both performance and cost. This advantage is more pronounced as the value of X increases, because a larger X value results in more multiply-accumulate operations performed by the multiply-accumulate circuit 223. Since this application includes both fixed-point and floating-point arithmetic, its precision is higher than that of purely fixed-point arithmetic (…). Figure 1A It is more expensive than pure floating-point arithmetic, but the cost is higher. Figure 1B )Low.
[0109] The intelligent processing unit 220 of this application can perform convolution operations on various types of data (including but not limited to image data or voice data). In other words, the input data IBf and the weight data KBf can be image data or voice data.
[0110] Although the embodiments of this application are described above, these embodiments are not intended to limit this application. Those skilled in the art can make changes to the technical features of this application based on the express or implied content of this application. All such changes may fall within the scope of patent protection sought by this application. In other words, the scope of patent protection of this application shall be determined by the scope of the patent application as defined in this specification.
Claims
1. An intelligent processing unit, characterized by, The application comprises: a memory for storing a first input data and a first weight data; a quantization circuit coupled to the memory for quantizing the first input data to generate a plurality of second input data and an input data bit shift amount, and for quantizing the first weight data to generate a plurality of second weight data and a weight data bit shift amount; a multiply-accumulate circuit coupled to the quantization circuit for performing a multiply-accumulate operation on the plurality of second input data and the plurality of second weight data to generate an intermediate result; a bit shift amount calculation circuit coupled to the quantization circuit for generating an intermediate result bit shift amount based on the input data bit shift amount and the weight data bit shift amount; a de-normalization accumulate circuit coupled to the multiply-accumulate circuit and the bit shift amount calculation circuit for performing a de-normalization accumulate operation on the intermediate result and an intermediate accumulate result based on the intermediate result bit shift amount to generate a final accumulate result; and a data type conversion circuit coupled to the de-normalization accumulate circuit and the memory for converting the final accumulate result to generate an output data; wherein the first input data and the first weight data are floating point data, and the plurality of second input data and the plurality of second weight data are fixed point data. The multiply-accumulate operation is a fixed point multiply-accumulate operation, and the intermediate result is a fixed point number.
2. The intelligent processing unit of claim 1, wherein, The intermediate accumulate result is stored in the memory, and the intermediate accumulate result is a floating point data, and the de-normalization accumulate operation comprises the following steps:
3. The intelligent processing unit of claim 2, wherein, reading a previous intermediate accumulate result from the memory as a temporary intermediate accumulate result; splitting the temporary intermediate accumulate result into an intermediate fixed point number part and an intermediate exponent part; comparing the intermediate exponent part and the intermediate result bit shift amount to generate a larger exponent; determining a first bit shift amount according to the larger exponent; shifting one of the intermediate fixed point number part and the intermediate result according to the first bit shift amount, and shifting the other one by 0 bit to generate a shifted intermediate fixed point number part and a shifted intermediate result; adding the shifted intermediate fixed point number part and the shifted intermediate result to obtain a temporary fixed point number; finding a leading 1 of the temporary fixed point number, and generating a second bit shift amount according to a position of the leading 1; shifting the temporary fixed point number based on the second bit shift amount to generate a target fixed point number; generating a target exponent according to the second bit shift amount and the larger exponent; and generating the intermediate accumulate result according to the target fixed point number and the target exponent. 4. The intelligent processing unit of claim 1, wherein, The intelligent processing unit performs a convolution operation; the first input data includes multiple input data blocks; each input data block includes X second input data; the first weight data includes a sub-weight data; the sub-weight data includes Y weight data blocks, each weight data block including X second weight data; in the convolution operation, each weight data block corresponds to L of the multiple input data blocks; in one multiply-accumulate operation, the multiply-accumulate circuit performs the multiply-accumulate operation on the X second input data and X second weight data of a target weight data block to produce the intermediate result; the target weight data block is one of the Y weight data blocks; after the multiply-accumulate circuit performs the multiply-accumulate operation L times on the target weight data block, the intelligent processing unit changes the target weight data block; X, Y, and L are all integers greater than or equal to 1.
5. The intelligent processing unit of claim 4, wherein, The intelligent processing unit selects one of the Y weight data blocks in turn as the target weight data block; the convolution operation reads the first input data Y times and the first weight data once.
6. The intelligent processing unit of claim 4, wherein, The dimension of each weighted data block is 1*1*X, the dimension of each second input data is 1*1*1, and the dimension of each second weighted data is 1*1*1.
7. The intelligent processing unit of claim 1, wherein, The first input data and the first weight data are either image data or voice data.
8. A method of convolution operation, characterized by, include: (A) Quantize a first weight data to generate multiple second weight data and a weight data displacement; (B) Quantize a first input data to generate multiple second input data and an input data displacement; (C) Perform a multiplication-accumulation-addition operation on the plurality of second input data and the plurality of second weight data to produce an intermediate result; (D) Generate an intermediate result displacement based on the input data displacement and the weight data displacement; (E) Perform a one-order accumulation operation on the intermediate result and an intermediate accumulation result based on the displacement of the intermediate result to produce a final accumulation result; as well as (F) The final accumulated result is transformed to produce an output data; Wherein, the first input data and the first weight data are floating-point data, while the plurality of second input data and the plurality of second weight data are fixed-point data.
9. The method of claim 8, wherein, The multiplication-accumulation operation is a multiplication-accumulation operation with a certain number of points, and the intermediate result is a certain number of points.
10. The method of claim 9, wherein, The intermediate accumulation result is stored in a memory, and the intermediate accumulation result is floating-point data. The order-wise accumulation operation includes the following steps: (G) Read the previous intermediate accumulation result from the memory as a temporary intermediate accumulation result; (H) The temporary intermediate accumulation result is decomposed into an intermediate fixed-point part and an intermediate exponent part; (I) Compare the intermediate index portion with the intermediate result displacement to generate a larger index. (J) A first displacement is determined based on the larger exponent; (K) Shift one of the intermediate fixed-point number portion and the intermediate result according to the first displacement amount, and shift the other to 0 bits to produce a shifted intermediate fixed-point number portion and a shifted intermediate result. (L) Add the shifted intermediate fixed-point number to the shifted intermediate result to obtain a temporary fixed-point number; (M) Find a leading 1 of the temporary fixed point number, and generate a second displacement based on the position of the leading 1; (N) Shift the temporary fixed point number based on the second displacement amount to generate a target fixed point number; (O) Generate a target index based on the second displacement and the larger index; and (P) The intermediate accumulation result is generated based on the target fixed point number and the target index.
11. The method of claim 8, wherein, The first input data includes multiple input data blocks; each input data block includes X second input data; the first weight data includes a sub-weight data; the sub-weight data includes Y weight data blocks, each weight data block including X second weight data; each weight data block corresponds to L of the multiple input data blocks; in one multiplication-accumulation operation, step (C) performs the multiplication-accumulation operation on the X second input data and the X second weight data of a target weight data block to generate the intermediate result; the target weight data block is one of the Y weight data blocks; X, Y, and L are all integers greater than or equal to 1; the method further includes: After performing the multiplication and accumulation operation L times on the target weight data block in step (C), the target weight data block is changed.
12. The method of claim 11, wherein, The Y weighted data blocks are used in turn as the target weighted data block; the convolution operation method reads the first input data Y times and the first weighted data once.