Intelligence processing unit and convolution operation method

Abstract

The intelligent processing unit includes: a memory for storing a first input data (floating-point data) and a first weight data (floating-point data); a quantization circuit for quantizing the first input data to generate a plurality of second input data (fixed-point data) and an input data displacement, and for quantizing the first weight data to generate a plurality of second weight data (fixed-point data) and a weight data displacement; a multiply-accumulate circuit for performing a multiply-accumulate operation on the second input data and the second weight data to generate an intermediate result; a displacement calculation circuit for generating an intermediate result displacement based on the input data displacement and the weight data displacement; a pairwise accumulation circuit for performing a pairwise accumulation operation on the intermediate result and an intermediate accumulation result based on the intermediate result displacement to generate a final accumulation result; and a data type conversion circuit for converting the final accumulation result to generate output data.

Description

Technical Field

[0001] This invention relates to convolution operations, and more particularly to convolution operations on floating-point data. Prior Technology

[0002] Figures 1A and 1B are flowcharts of a conventional convolution operation. Figure 1A corresponds to an electronic device performing fixed-point operations, while Figure 1B corresponds to an electronic device performing floating-point operations. In Figures 1A and 1B, the data for the convolution operation (e.g., input data (or input feature map) and weights (or convolution kernel)) are floating-point data.

[0003] For Figure 1A, in the preparation stage, the floating-point data is first converted into fixed-point data through statistical quantization (step S110), then configuration parameters are generated based on the fixed-point data and the hardware specifications of the electronic device (step S120), and finally, a convolution operation based on fixed-point numbers is performed on the electronic device (S130).

[0004] For Figure 1B, in the preparation stage, the floating-point data is first converted to a new format (step S115, for example, from floating-point FP32 format to floating-point FP16 format), then configuration parameters are generated based on the floating-point data and the hardware specifications of the electronic device (step S125), and finally, a floating-point-based convolution operation is performed on the electronic device (step S135).

[0005] The advantage of the scheme in Figure 1A is its relatively low cost, but its disadvantage is a relatively significant loss of accuracy. The advantage of the scheme in Figure 1B is its relatively high accuracy, but its disadvantage is its relatively high cost. Therefore, a scheme with relatively high accuracy and relatively low cost is needed. Summary of the Invention

[0006] In view of the shortcomings of prior art, one object of the present invention is to provide an intelligent processing unit and a convolution operation method to improve the shortcomings of prior art.

[0007] One embodiment of the present invention provides an intelligent processing unit, comprising: a memory, a quantization circuit, a multiply-accumulate-add circuit, a displacement calculation circuit, a pair of alignment and addition circuits, and a data type conversion circuit. The memory stores a first input data and a first weight data. The quantization circuit is coupled to the memory and is used to quantize the first input data to generate a plurality of second input data and an input data displacement, and to quantize the first weight data to generate a plurality of second weight data and a weight data displacement. The multiply-accumulate-add circuit is coupled to the quantization circuit and is used to perform a multiply-accumulate-add operation on the second input data and the second weight data to generate an intermediate result. The displacement calculation circuit is coupled to the quantization circuit and is used to generate an intermediate result displacement based on the input data displacement and the weight data displacement. The alignment and addition circuit is coupled to the multiply-accumulate-add circuit and the displacement calculation circuit and is used to perform a pair of alignment and addition operations on the intermediate result and an intermediate addition result based on the intermediate result displacement to generate a final addition result. A data type conversion circuit is coupled to the parallel accumulation circuit and the memory to convert the final accumulation result to produce an output data. The first input data and the first weight data are floating-point data, while the second input data and the second weight data are fixed-point data.

[0008] Another embodiment of the present invention provides a convolution operation method, comprising the following steps: (A) quantizing a first weight data to generate a plurality of second weight data and a weight data displacement; (B) quantizing a first input data to generate a plurality of second input data and an input data displacement; (C) performing a multiplication-accumulation operation on the second input data and the second weight data to generate an intermediate result; (D) generating an intermediate result displacement based on the input data displacement and the weight data displacement; (E) performing a one-order accumulation operation on the intermediate result and the intermediate accumulation result based on the intermediate result displacement to generate a final accumulation result; and (F) converting the final accumulation result to generate an output data. The first input data and the first weight data are floating-point data, while the second input data and the second weight data are fixed-point data.

[0009] The technical means embodied in the embodiments of the present invention can improve at least one of the disadvantages of the prior art. Therefore, the present invention can achieve a better balance between accuracy and cost compared with the prior art.

[0010] The features, implementation, and effects of this invention are described in detail below with reference to the accompanying drawings. Simple Explanation of the Diagram

[0011] Figures 1A and 1B are flowcharts of the conventional convolution operation; Figure 2 is a functional block diagram of one embodiment of the electronic device of the present invention; Figure 3 is a schematic diagram of the input data, weight data, and final accumulation result of the convolution operation of this invention; Figure 4 is a flowchart of one embodiment of the quantization operation of the present invention; Figure 5 is a data flow diagram corresponding to Figure 4; Figures 6A and 6B are flowcharts of one embodiment of the order-accumulation operation of the present invention; Figure 7 is a data flow diagram corresponding to Figures 6A and 6B; and Figures 8A and 8B are flowcharts of one embodiment of the convolution operation of the present invention. Implementation

[0012] The technical terms used in the following description are based on the common terminology of this technical field. If this specification provides explanations or definitions for certain terms, the explanations or definitions in this specification shall prevail.

[0013] The disclosure of this invention includes an intelligent processing unit and a convolution operation method. Since some components of the intelligent processing unit of this invention may be known individually, details of known components will be omitted in the following description without affecting the full disclosure and implementability of the device invention. Furthermore, the convolution operation method of this invention can be executed by the intelligent processing unit of this invention or its equivalent device. Without affecting the full disclosure and implementability of the method invention, the following description of the method invention will focus on the steps rather than the hardware.

[0014] Figure 2 is a functional block diagram of one embodiment of the electronic device of the present invention. The electronic device 200 includes external memory 210 (e.g., Dynamic Random Access Memory (DRAM)) and an intelligence processing unit (IPU) 220 coupled to each other. The intelligence processing unit 220 includes memory 221 (e.g., Static Random Access Memory (SRAM)), quantization circuit 222, multiply-accumulate circuit 223, displacement calculation circuit 224, order-aligned accumulation circuit 225, and data type conversion circuit 226 coupled to each other.

[0015] The intelligent processing unit 220 retrieves data (including input data IBf and weight data KBf) for convolution operations from the external memory 210. Both the input data IBf and the weight data KBf are floating-point data.

[0016] Quantization circuit 222 is coupled to or electrically connected to memory 221 to perform quantization operations on input data IBf and weight data KBf, generating input data IBi, weight data KBi, input data displacement ST_IB, and weight data displacement ST_KB. In one embodiment, both input data IBf and weight data KBf are floating-point data. Both input data IBi and weight data KBi are fixed-point data. Input data displacement ST_IB and weight data displacement ST_KB correspond to input data IBi and weight data KBi, respectively.

[0017] The multiply-accumulate circuit 223 is coupled to or electrically connected to the quantization circuit 222, and is used to multiply and accumulate the input data IBi and the weight data KBi to produce the intermediate result ABi. The multiply-accumulate operation performed by the multiply-accumulate circuit 223 is a fixed-point multiply-accumulate operation, that is, the intermediate result ABi is fixed-point data.

[0018] The displacement calculation circuit 224 is coupled to or electrically connected to the quantization circuit 222, and is used to generate an intermediate result displacement ST_AB based on the input data displacement ST_IB and the weight data displacement ST_KB. For example, the intermediate result displacement ST_AB is equal to the sum of the input data displacement ST_IB and the weight data displacement ST_KB. The intermediate result displacement ST_AB corresponds to the intermediate result ABi.

[0019] The order-accumulating circuit 225 is coupled or electrically connected to the multiplication-accumulating circuit 223 and the displacement calculation circuit 224, and is used to perform order-accumulating operations based on the intermediate result ABi and the intermediate result displacement ST_AB. The intermediate accumulation result ABf[k] is floating-point data. The argument k represents the k-th intermediate accumulation result (1≦k≦Y, where Y is an integer greater than or equal to 1; Y will be described in detail below with reference to Figure 3). ABf[0] is the initial value, and the final accumulation result ABf_F (i.e., ABf[Y]) is the result of a convolution operation. Both the intermediate accumulation result ABf[k] and the final accumulation result ABf_F are floating-point data.

[0020] Data type conversion circuit 226 is coupled or electrically connected to parallel accumulation circuit 225 to perform data conversion on the final accumulation result ABf_F (e.g., from floating-point FP32 format to floating-point FP16 format) to generate output data OBf (or output feature map). Output data OBf is floating-point data. Details of the data type conversion operation are well known to those skilled in the art and will not be described further. Output data OBf is first stored in memory 221 and then moved to external memory 210.

[0021] Figure 3 is a schematic diagram of the input data IBf, weight data KBf, and the final accumulated result ABf_F of the convolution operation of this invention. It is assumed that the dimension ([W,H,D]) of the input data IBf is [6,6,X] (X is an integer greater than or equal to 1); the weight data KBf contains M sub-weight data (KBf1, KBf2, ..., KBfM), and the dimension of each sub-weight data is [3,3,X]; the dimension of the final accumulated result ABf_F is [4,4,M]. The Y value mentioned above is the number of weight data blocks contained in a sub-weight data. In the example of Figure 3, since each sub-weight data (KBf1, KBf2, ..., KBfM) contains 9 weight data blocks (each weight data block has a dimension of [1,1,X], i.e., a size of 1*1*X), Y equals 9 (i.e., the product of the dimension W and the dimension H of the sub-weight data). The final accumulated result ABf_F corresponds to the sub-weight data KBf1, KBf2, ..., KBfM in dimension D. More specifically, the data in the first (D=1) WH plane is the result of the convolution operation between the input data IBf and the sub-weight data KBf1; similarly, the data in the second (D=2) and Mth (D=M) WH planes are the results of the convolution operation between the input data IBf and the sub-weight data KBf2 and KBfM, respectively. Those skilled in the art are familiar with the details of convolution operations, so they will not be elaborated upon further.

[0022] Figure 4 is a flowchart of one embodiment of the quantization operation of the present invention. Figure 5 is a data flow diagram corresponding to Figure 4. The flow of Figure 4 can be executed by the quantization circuit 222, and the corresponding steps are marked in Figure 5. The top of Figure 5 shows X tensors: tensors xBf_1, xBf_2, ..., xBf_p, ... and xBf_X (1≦p≦X), and each tensor is floating-point data. Each tensor is a data block of input data IBf (e.g., the input data block IBf_A in the upper left corner, dimension: [1,1,X]), or a piece of data (dimension: [1,1,1]) in a data block of weight data KBf (e.g., weight data blocks KBf1_A, KBf2_A, or KBfM_A, dimension: [1,1,X]). The following description is provided with reference to both Figure 4 and Figure 5.

[0023] Step S410: Decompose a tensor of floating-point data into a fixed-point part and an exponent part. As shown in Figure 5, tensor xBf_1 is decomposed into a fixed-point part Fd_1 (i.e., the mantissa part, which is a fixed-point number) and an exponent part Exp_1; tensor xBf_2 is decomposed into a fixed-point part Fd_2 and an exponent part Exp_2; tensor xBf_p is decomposed into a fixed-point part Fd_p and an exponent part Exp_p; and tensor xBf_X is decomposed into a fixed-point part Fd_X and an exponent part Exp_X.

[0024] Step S420: Find the largest among all the exponential parts Exp_1 to Exp_X to obtain the largest exponential part Exp_Max.

[0025] Step S430: Shift the fixed-point portion Fd_p to the left by a preset number of bits to obtain the intermediate data Tmp. Specifically, when the input data IBf is FP16, the preset number of bits is 4 bits. When the input data IBf is BP16, the preset number of bits is 7 bits.

[0026] Step S440: Subtract the exponential part Exp_p from the largest exponential part Exp_Max to obtain the exponential difference Exp_Dif.

[0027] Step S450: Shift the intermediate data Tmp to the right according to the exponent difference Exp_Dif to obtain the fixed-point data xBi. Specifically, for converting from floating-point FP16 or floating-point BF16 format to fixed-point INT16 format, the number of bits shifted right by the intermediate data Tmp is equal to the exponent difference Exp_Dif. For converting from floating-point FP16 or floating-point BF16 format to fixed-point INT8 format, the number of bits shifted right by the intermediate data Tmp is equal to the sum of the exponent difference Exp_Dif and 8.

[0028] Step S460: Subtract a preset value from the largest exponent portion Exp_Max to obtain the displacement ST_x. The purpose of subtracting the preset value is to correct the decimal point position. For converting from floating-point FP16 format to fixed-point INT16 format, the preset value is 29. For converting from floating-point FP16 format to fixed-point INT8 format, the preset value is 21. For converting from floating-point BF16 format to fixed-point INT16 format, the preset value is 141. For converting from floating-point FP16 format to fixed-point INT8 format, the preset value is 133. Those skilled in the art are familiar with how to determine the preset value based on the format before and after conversion, so it will not be described further.

[0029] For the input data, the fixed-point data xBi and the displacement ST_x correspond to the input data IBi and the input data displacement ST_IB in Figure 2, respectively. For the weighted data, the fixed-point data xBi and the displacement ST_x correspond to the weighted data KBi and the weighted data displacement ST_KB in Figure 2, respectively.

[0030] Please note that for the sake of simplicity, Figure 5 only shows the intermediate data Tmp and the exponential difference Exp_Dif corresponding to the tensor xBf_p. However, there are actually X intermediate data, X exponential differences, and X fixed-point data. That is, after the data block (input data block IBf_A, weight data block KBf1_A, KBf2_A, or KBfM_A) is quantized by the quantization circuit 222, it will yield X fixed-point data xBi and one displacement ST_x. In other words, each input data block (e.g., IBf_A) contains X input data IBi (size 1*1*1), and each weight data block (e.g., KBf1_A, KBf2_A, KBfM_A) contains X weight data KBi (size 1*1*1).

[0031] Please refer to Figure 3. Because the sub-weight data KBf1 (KBf2 or KBfM) slides across the input data IBf during the convolution operation, each weight data block of the sub-weight data KBf1 (KBf2 or KBfM) corresponds to L input data blocks (L is an integer greater than or equal to 1; in the example of Figure 3, L is 16, which is the product of the dimension W and dimension H of the final accumulated result ABf_F). The multiply-accumulate circuit 223 performs a multiply-accumulate operation on X input data IBi and X weight data KBi to obtain an intermediate result ABi. That is, the multiply-accumulate circuit 223 obtains an intermediate result ABi after performing a multiply-accumulate operation on the input data block IBf_A and the weight data block KBf1_A. For example, each of the 9 data blocks of the sub-weight data KBf1 corresponds to 16 intermediate results ABi (due to window sliding). Therefore, for data block ABf1_A, the order-pairing accumulation circuit 225 performs order-pairing accumulation on the nine intermediate results ABi of the nine data blocks corresponding to the sub-weight data KBf1 based on the intermediate result shift amount ST_AB to obtain data block ABf1_A. Similarly, data blocks ABf2_A and ABfM_A correspond to sub-weight data KBf2 and sub-weight data KBfM, respectively.

[0032] In some embodiments, the quantization circuit 222 simultaneously generates the input data IBi and input data displacement ST_IB of the input data block IBf_A, and the weight data KBi and weight data displacement ST_KB of the weight data blocks KBf1_A, KBf2_A, and KBfM_A in parallel processing. The displacement calculation circuit 224 adds the weight data displacement ST_KB of the weight data blocks KBf1_A, KBf2_A, and KBfM_A to the input data displacement ST_IB of the input data block IBf_A, respectively, to obtain three intermediate result displacements ST_AB corresponding to the weight data blocks KBf1_A, KBf2_A, and KBfM_A. The displacement calculation circuit 224 can be implemented using a fixed-point adder.

[0033] Following the previous section, the order accumulation circuit 225 generates a portion of the data block ABf1_A (ABf2_A or ABfM_A) (i.e., the intermediate accumulation result ABf[k]) based on the intermediate result displacement ST_AB and intermediate result ABi corresponding to the sub-weight data KBf1 (KBf2 or KBfM), and temporarily stores this portion in memory 221.

[0034] Figures 6A and 6B are flowcharts of one embodiment of the order-pair accumulation operation of the present invention. Figure 7 is a data flow diagram corresponding to Figures 6A and 6B. The processes in Figures 6A and 6B can be executed by the order-pair accumulation circuit 225, and the corresponding steps are marked in Figure 7.

[0035] Step S605: Read the previous intermediate accumulation result ABf[k] from memory 221 as a temporary intermediate accumulation result ABf[k-1].

[0036] Step S610: Obtain the intermediate fixed-point part Fxd_i and the intermediate exponent part Exp_i based on the temporary intermediate accumulation result ABf[k-1]. For example, when the temporary intermediate accumulation result ABf[k-1] is a number in FP32 format, the intermediate fixed-point part Fxd_i and the intermediate exponent part Exp_i are 24-bit and 10-bit fixed-point data, respectively.

[0037] Please note that when steps S605 and S610 are executed for the first time, memory 221 does not yet have the previous intermediate accumulation result ABf[k]. At this time, the alignment accumulation circuit 225 reads the initial value ABf[0] from memory 221. The format of the initial value is the same as that of the intermediate accumulation result ABf[k] (it can be a custom format or a standard format (e.g., FP32 format)), and it can be generated based on the bias of the convolution operation. Generating the initial value based on the bias is well known to those skilled in the art, and will not be described further.

[0038] Step S620: Compare the intermediate exponent portion Exp_i with the intermediate result displacement ST_AB to generate the larger exponent P2S_Max. That is, the larger exponent P2S_Max is the larger of the intermediate exponent portion Exp_i and the intermediate result displacement ST_AB.

[0039] Step S630: Determine the displacement SH_i based on the larger exponent P2S_Max. The displacement SH_i is the difference between the larger exponent P2S_Max and the other. For example, if the intermediate exponent Exp_i and the intermediate result displacement ST_AB are -7 and -8 respectively, then the larger exponent P2S_Max is -7, and the displacement SH_i = -7 - (-8) = 1.

[0040] Step S640: Shift one of the intermediate fixed-point number part Fxd_i and the intermediate result ABi to the right by a displacement SH_i, and shift the other to the right by 0 bits. This shift corresponds to the intermediate result ABi or the intermediate fixed-point number part Fxd_i with the smaller exponent. Continuing with the previous example (where the intermediate exponent part Exp_i and the intermediate result displacement ST_AB are -7 and -8 respectively), because the intermediate result displacement ST_AB is smaller, this step shifts the intermediate result ABi to the right by a displacement SH_i, and shifts the intermediate fixed-point number part Fxd_i to the right by 0 bits.

[0041] Step S650: Add the shifted intermediate fixed-point number part Fxd_is to the shifted intermediate result ABis to obtain the temporary fixed-point number Fxd_tmp.

[0042] Step S660: Find the leading one (i.e., the highest bit 1) of the temporary fixed-point number Fxd_tmp, and generate a displacement SH_V based on the position of the leading one. For example, if the temporary fixed-point number Fxd_tmp is 0001...0011 (48 bits in total), then the displacement SH_V is equal to 4.

[0043] Step S670: Shift the temporary fixed point number Fxd_tmp to the left based on the displacement SH_V to generate the target fixed point number Fxd_tar, that is, shift the target fixed point number Fxd_tar to the left by the displacement SH_V.

[0044] Step S680: Generate the target exponent P2S_tar based on the displacement SH_V and the larger exponent P2S_Max. More specifically, the target exponent P2S_tar is equal to the larger exponent P2S_Max minus the displacement SH_V.

[0045] Step S690: Generate an intermediate accumulation result ABf[k] based on the target fixed-point number Fxd_tar and the target exponent P2S_tar, where the target fixed-point number Fxd_tar is the fixed-point part of the intermediate accumulation result ABf[k], and the target exponent P2S_tar is the exponent part of the intermediate accumulation result ABf[k].

[0046] Figures 8A and 8B are flowcharts of one embodiment of the convolution operation of the present invention. Figures 8A and 8B can be executed by the intelligent processing unit 220 of Figure 2 and include the following steps.

[0047] Step S810: Quantization circuit 222 reads one weight data block from each of the M sub-weight data from memory 221 as the target weight data block. That is (see Figure 3), in this step, quantization circuit 222 reads one weight data block (e.g., KBf1_A, KBf2_A, ..., KBfM_A) from each of the M sub-weight data (KBf1, KBf2, ..., KBfM) to obtain M target weight data blocks.

[0048] Step S820: The quantization circuit 222 quantizes the M target weight data blocks read, and generates X weight data KBi and a weight data displacement ST_KB for each target weight data block. In some embodiments, the quantization circuit 222 processes the M target weight data blocks in parallel to substantially generate M*X weight data KBi and M weight data displacement ST_KB simultaneously.

[0049] Step S830: Quantization circuit 222 reads an input data block (e.g., IBf_A) from memory 221.

[0050] Step S840: Quantization circuit 222 quantizes the read input data block and generates X input data IBi and one input data displacement ST_IB for the input data block.

[0051] Step S850: For each of the M target weight data blocks, the multiply-accumulate circuit 223 performs a multiply-accumulate operation on the X weight data KBi and the X input data IBi to obtain an intermediate result ABi (a total of M intermediate results ABi are obtained). Taking Figure 3 as an example, when this step is executed, M intermediate results ABi are generated, corresponding to data blocks ABf1_A, ABf2_A, ..., ABfM_A respectively.

[0052] Step S860: For each of the M target weight data blocks, the displacement calculation circuit 224 adds the weight data displacement ST_KB to the input data displacement ST_IB to obtain an intermediate result displacement ST_AB. In other words, this step generates a total of M intermediate result displacements ST_AB, corresponding to the M intermediate results ABi respectively.

[0053] Step S870: Based on the displacement amount ST_AB of the M intermediate results, the alignment accumulation circuit 225 performs alignment accumulation operation on the M intermediate results ABi and their corresponding temporary intermediate accumulation results ABf[k-1] to obtain M intermediate accumulation results ABf[k]. The temporary intermediate accumulation result ABf[k-1] is the intermediate accumulation result ABf[k] of the previous alignment accumulation operation and is temporarily stored in memory 221.

[0054] Step S880: The intelligent processing unit 220 determines whether all input data blocks corresponding to the target weight data blocks (e.g., KBf1_A, KBf2_A, ..., KBfM_A) have been processed. More specifically (using Figure 3 as an example), because the sub-weight data KBf1 (KBf2 or KBfM) slides on the input data IBf during the convolution operation, each weight data block of the sub-weight data KBf1 (KBf2 or KBfM) corresponds to L input data blocks (in the example of Figure 3, L is 16). If the L input data blocks have not been processed, the process returns to step S830 to read the next input data block corresponding to the target weight data block. If the L input data blocks have been processed (meaning that the M target weight data blocks of this round have been processed), step S890 is executed.

[0055] Step S890: The intelligent processing unit 220 determines whether all weight data blocks have been processed. If not, it returns to step S810 to determine the M target weight data blocks for the next round; otherwise, it executes step S895. Taking Figure 3 as an example, since a sub-weight data contains 9 weight data blocks (i.e., the aforementioned Y values), steps S810 to S890 need to be executed 9 times (i.e., all weight data blocks will be selected as target weight data blocks in turn).

[0056] Step S895: Obtain the final accumulated result ABf_F, and then convert the data type of the final accumulated result ABf_F to generate the output data OBf.

[0057] As shown in Figures 8A and 8B, the convolution operation process of the present invention includes small loops (steps S830-S880) and large loops (steps S810-S890). For one convolution operation, the small loop is executed L times, and the large loop is executed Y times. The advantage of this design is that it avoids repeatedly reading weight data, reducing the read operation burden of the intelligent processing unit 220. Taking Figure 3 as an example, each sub-weight data (KBf1, KBf2, ..., KBfM) is only read once (that is, the weight data KBf is only read once), while the input data IBf is read at most 9 times (that is, the aforementioned Y value, which is the number of weight data blocks contained in each sub-weight data). In contrast (again, taking Figure 3 as an example), for conventional convolution operations, the input data IBf is read 9 times, and each sub-weight data is also read 9 times. Therefore, the intelligent processing unit 220 of the present invention significantly reduces the number of read operations, thereby improving the performance of the electronic device 200.

[0058] Furthermore, because the large loop of the convolution operation of the present invention (steps S810 to S890 in Figures 8A to 8B) is based on weight data blocks (instead of the convolution operation in the conventional art based on sub-weight data), the present invention is applicable to weight data KBf of various shapes (e.g., 3*3*X (as shown in the example in Figure 3), 2*4*X, 6*5*X, etc.), thereby improving the application flexibility of the electronic device 200.

[0059] Furthermore, because the multiply-accumulate circuit 223 processes fixed-point numbers (which are lower in cost and easier to implement), this invention significantly reduces the number of floating-point accumulations, thus improving both performance and cost. This advantage is more pronounced as the value of X increases, because a larger X value results in more multiply-accumulate operations performed by the multiply-accumulate circuit 223. Because this invention incorporates both fixed-point and floating-point arithmetic, its accuracy is higher than that of pure fixed-point arithmetic (Figure 1A), but its cost is lower than that of pure floating-point arithmetic (Figure 1B).

[0060] The intelligent processing unit 220 of the present invention can perform convolution operations on various types of data (including but not limited to image data or voice data). In other words, the input data IBf and the weight data KBf can be image data or voice data.

[0061] Although the embodiments of the present invention have been described above, these embodiments are not intended to limit the present invention. Those skilled in the art can make changes to the technical features of the present invention based on the explicit or implicit content of the present invention. All such changes may fall within the scope of patent protection sought by the present invention. In other words, the scope of patent protection of the present invention shall be determined by the scope of the patent application in this specification.

[0062] S110, S115, S120, S125, S130, S135, S410, S420, S430, S440, S450, S460, S605, S610, S620, S630, S640, S650, S660, S670, S680, S690, S810, S820, S830, S840, S850, S860, S870, S880, S890, S895: Steps 200: Electronic devices 210: External Memory 220: Intelligent Processing Unit 221: Memory 222: Quantization Circuit 223: Multiply-Accumulate Circuit 224: Displacement Calculation Circuit 225: Alignment Accumulator Circuit 226: Data type conversion circuit ABf[k]: Intermediate cumulative result ABf[k-1]: Temporary intermediate accumulation result ABf_F: The final accumulated result ABi: Intermediate Results D, H, W: Dimensions IBf, IBi: Input data KBf, KBi: Weighted data OBf: Output data ST_AB: Intermediate result displacement ST_IB: Input data displacement ST_KB: Weighted data displacement ABf1_A, ABf2_A, ABfM_A: Data blocks IBf_A: Input data block KBf1, KBf2, KBfM: Sub-weight data KBf1_A, KBf2_A, KBfM_A: Weighted data blocks Exp_1, Exp_2, Exp_p, Exp_X: Exponential parts Exp_Dif: Exponential Difference Exp_Max: The largest exponential portion Fd_1, Fd_2, Fd_p, Fd_X: fixed point part ST_x, SH_i, SH_V: Displacement amount Tmp: Intermediate Data xBf_1,xBf_2,xBf_p,xBf_X: tensor xBi: Fixed-point data ABis: Intermediate result after shifting Exp_i: Intermediate exponent portion Fxd_i: intermediate fixed-point part Fxd_is: The intermediate fixed-point number after shifting Fxd_tar: Number of target fixed points Fxd_tmp: Temporary fixed point number P2S_Max: A larger exponent P2S_tar: Target index

Claims

1. An intelligent processing unit, comprising: a memory for storing a first input data and a first weight data; a quantization circuit coupled to the memory for quantizing the first input data to generate a plurality of second input data and an input data displacement, and for quantizing the first weight data to generate a plurality of second weight data and a weight data displacement; a multiply-accumulate-add circuit coupled to the quantization circuit for performing a multiply-accumulate-add operation on the second input data and the second weight data to generate an intermediate result; a displacement calculation circuit coupled to the quantization circuit for generating an intermediate result displacement based on the input data displacement and the weight data displacement; a pair-order accumulation circuit coupled to the multiply-accumulate-add circuit and the displacement calculation circuit for performing a pair-order accumulation operation on the intermediate result and an intermediate accumulation result based on the intermediate result displacement to generate a final accumulation result; and a data type conversion circuit coupled to the pair-order accumulation circuit and the memory for converting the final accumulation result to generate output data; wherein... The first input data and the first weight data are floating-point data, while the second input data and the second weight data are fixed-point data.

2. As in request item 1, the intelligent processing unit, wherein, The multiplication-accumulation operation is a multiplication-accumulation operation with a fixed number of points, and the intermediate result is a fixed number of points.

3. As in request item 2, the intelligent processing unit, wherein, The intermediate accumulation result is stored in the memory. The intermediate accumulation result is floating-point data. The order-pair accumulation operation includes the following steps: Reading the previous intermediate accumulation result from the memory as a temporary intermediate accumulation result; Decomposing the temporary intermediate accumulation result into an intermediate fixed-point part and an intermediate exponent part; Comparing the intermediate exponent part with the intermediate result shift to generate a larger exponent; Determining a first shift based on the larger exponent; Shifting one of the intermediate fixed-point part and the intermediate result based on the first shift, and shifting the other to 0 bits to generate a shifted intermediate fixed-point part and a shifted intermediate result; Adding the shifted intermediate fixed-point part to the shifted intermediate result to obtain a temporary fixed-point number; Finding a leading 1 in the temporary fixed-point number and generating a second shift based on the position of the leading 1; Shifting the temporary fixed-point number based on the second shift to generate a target fixed-point number. A target index is generated based on the second displacement and the larger index; and the intermediate accumulation result is generated based on the target fixed point number and the target index.

4. As in request item 1, the intelligent processing unit, wherein, The intelligent processing unit performs a convolution operation; the first input data includes a plurality of input data blocks; each input data block includes X of the second input data; the first weight data includes a sub-weight data; the sub-weight data includes Y weight data blocks, each weight data block including X of the second weight data; in the convolution operation, each weight data block corresponds to L of the input data blocks; in one multiply-accumulate operation, the multiply-accumulate circuit performs the multiply-accumulate operation on the X of the second input data and the X of the second weight data of a target weight data block to produce the intermediate result; the target weight data block is one of the Y weight data blocks; after the multiply-accumulate circuit performs the multiply-accumulate operation L times on the target weight data block, the intelligent processing unit changes the target weight data block; X, Y, and L are all integers greater than or equal to 1.

5. As in request item 4, the intelligent processing unit, wherein, The intelligent processing unit selects one of the Y weight data blocks in turn as the target weight data block; the convolution operation reads the first input data Y times and the first weight data once.

6. As in request item 4, the intelligent processing unit, wherein, The dimension of each weighted data block is 1*1*X, the dimension of each second input data block is 1*1*1, and the dimension of each second weighted data block is 1*1*1.

7. The intelligent processing unit as described in request item 1, wherein, The first input data and the first weight data are either image data or voice data.

8. A convolution operation method, comprising: (A) quantizing a first weight data to generate a plurality of second weight data and a weight data shift; (B) quantizing a first input data to generate a plurality of second input data and an input data shift; (C) performing a multiplicative accumulation operation on the second input data and the second weight data to generate an intermediate result; (D) generating an intermediate result shift based on the input data shift and the weight data shift; (E) performing a one-order accumulation operation on the intermediate result and the intermediate accumulation result based on the intermediate result shift to generate a final accumulation result; and (F) transforming the final accumulation result to generate output data; wherein, The first input data and the first weight data are floating-point data, while the second input data and the second weight data are fixed-point data.

9. As in request item 8, wherein, The multiplication-accumulation operation is a multiplication-accumulation operation with a fixed number of points, and the intermediate result is a fixed number of points.

10. As in request item 9, wherein, The intermediate accumulation result is stored in a memory, which is floating-point data. The order-pair accumulation operation includes the following steps: (G) Reading the previous intermediate accumulation result from the memory as a temporary intermediate accumulation result; (H) Decomposing the temporary intermediate accumulation result into an intermediate fixed-point part and an intermediate exponent part; (I) Comparing the intermediate exponent part with the intermediate result shift to generate a larger exponent; (J) Determining a first shift based on the larger exponent; (K) Shifting one of the intermediate fixed-point part and the intermediate result according to the first shift, and shifting the other to 0 bits to generate a shifted intermediate fixed-point part and a shifted intermediate result; (L) Adding the shifted intermediate fixed-point part to the shifted intermediate result to obtain a temporary fixed-point number; (M) Finding a leading 1 in the temporary fixed-point number and generating a second shift based on the position of the leading 1. (N) Shift the temporary fixed point number based on the second displacement to generate a target fixed point number; (O) Generate a target exponent based on the second displacement and the larger exponent; and (P) Generate the intermediate accumulation result based on the target fixed point number and the target exponent.

11. As in request item 8, wherein, The first input data includes a plurality of input data blocks; each input data block includes X of the second input data; the first weight data includes a sub-weight data; the sub-weight data includes Y weight data blocks, each weight data block includes X of the second weight data; each weight data block corresponds to L of the input data blocks; in one multiply-accumulate operation, step (C) performs the multiply-accumulate operation on the X of the second input data and the X of the second weight data of a target weight data block to produce the intermediate result; the target weight data block is one of the Y weight data blocks; X, Y and L are all integers greater than or equal to 1; the method further includes: after step (C) performs the multiply-accumulate operation L times on the target weight data block, changing the target weight data block.

12. As in request item 11, wherein, The Y weight data blocks are used in turn as the target weight data block; the convolution operation method reads the first input data Y times and the first weight data once.

Images