Matrix multiplication device and method

Abstract

The invention provides a matrix multiplication device and method. The device comprises a data layout unit used for splitting a weight value matrix into a first sign bit plane and a plurality of first amplitude bit planes; splitting the activation value matrix into a second sign bit plane and a plurality of second amplitude bit planes; enabling the bit planes to be continuously stored in a wide word alignment mode according to a row priority rule; the bit operation unit is used for calculating different sign masks based on the first sign bit plane and the second sign bit plane, and performing bitwise AND operation on different combinations of the first amplitude bit plane and the second amplitude bit plane to obtain a plurality of common non-zero masks; the bit counting unit is used for acquiring non-zero counting values in AND operation results of the different number masks and the plurality of common non-zero masks; and the control unit is used for shifting the first amplitude bit plane and the second amplitude bit plane and performing weighted accumulation based on a non-zero count value. The method does not need a multiplier, and can reduce the operation overhead.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the field of artificial intelligence and deep learning chip design technology, in particular to a matrix multiplication operation device and a matrix multiplication operation method. BACKGROUND

[0002] With the growing parameter size of large language models (LLM), the storage and calculation of weights and activations become the bottleneck of inference performance and energy consumption. The weight is a parameter matrix learned by the model during training, which is usually a fixed number of floating-point numbers in the model, stored as a matrix or vector. The model performs linear transformation on the input data through the weight to achieve feature extraction and other functions. The activation is a matrix or vector generated after the input data is linearly transformed by the weight and processed by a nonlinear function. In the inference process of the model, a large number of matrix multiplication operations related to weights and activations are involved, resulting in huge calculation overhead. How to reduce the operation overhead and energy consumption of matrix multiplication is a technical problem that needs to be solved at present. SUMMARY

[0003] The present application provides a matrix multiplication operation device and a matrix multiplication operation method, which do not need to use a multiplier and can greatly save calculation overhead.

[0004] To solve the above technical problems, the first aspect of the present application provides a matrix multiplication operation device for performing multiplication operation of activation value matrix and weight value matrix in an artificial intelligence model, comprising: a data layout unit configured to split the weight value matrix into a first sign bit plane and a plurality of first amplitude bit planes; split the activation value matrix into a second sign bit plane and a plurality of second amplitude bit planes; make the first sign bit plane and the first amplitude bit plane in the same row in the memory, and make the second sign bit plane and the second amplitude bit plane in the same row in the memory; a bit operation unit configured to calculate an odd sign mask based on the first sign bit plane and the second sign bit plane, and perform bitwise AND operation on different combinations of the first amplitude bit plane and the second amplitude bit plane to obtain a plurality of common non-zero masks; a bit count unit configured to obtain non-zero count values in the AND operation results of the odd sign mask and the plurality of common non-zero masks, respectively; and a control unit configured to perform j+k bit shift on the jth first amplitude bit plane and the kth second amplitude bit plane, and perform weighted accumulation based on the non-zero count values to obtain the matrix product of the activation value matrix and the weight value matrix, wherein j is the bit index of the weight, k is the bit index of the activation, j=0:J-2, k=0:K-2, J and K are the total number of bits of the weight and the total number of bits of the activation, respectively.

[0005] The application provides a matrix multiplication operation method in a second aspect, which is used for performing multiplication operation of an activation value matrix and a weight value matrix in an artificial intelligence model, and the method comprises the following steps: splitting the weight value matrix into a first sign bit plane and a plurality of first amplitude bit planes; splitting the activation value matrix into a second sign bit plane and a plurality of second amplitude bit planes; arranging the first sign bit plane and the first amplitude bit planes in the same row in a memory; arranging the second sign bit plane and the second amplitude bit planes in the same row in the memory; calculating an opposite sign mask based on the first sign bit plane and the second sign bit plane; performing bit-by-bit AND operation on different combinations of the first amplitude bit planes and the second amplitude bit planes to obtain a plurality of common non-zero masks; obtaining non-zero count values in AND operation results of the opposite sign mask and the plurality of common non-zero masks respectively; and performing j+k bit shift on the jth first amplitude bit plane and the kth second amplitude bit plane, and performing weighted accumulation based on the non-zero count values to obtain a matrix product of the activation value matrix and the weight value matrix, wherein j is a bit index of the weight, k is a bit index of the activation, j = 0: J-2, k = 0: K-2, J and K are total bit numbers of the weight and the activation respectively.

[0006] The matrix multiplication operation device and method of the application split the weight value matrix and the activation value matrix into bit planes respectively, and arrange corresponding bit planes in rows, so that the weights are stored in the same row and the activations are stored in another same row. Then, the opposite sign mask is calculated in advance by the bit operation unit, so that the opposite sign mask is reused in the calculation process of the amplitude bit planes, and a plurality of common non-zero masks of different combinations of the amplitude bit planes of all weights and all activations are obtained. The non-zero count values in the AND operation results of the opposite sign mask and the plurality of common non-zero masks are obtained by the bit counting unit, and then the non-zero count values are shifted and accumulated by the control unit, so that the matrix product of the activation value matrix and the weight value matrix is obtained. The matrix multiplication operation device does not need to use a multiplier, can greatly save the calculation cost, and reduce the chip power consumption. BRIEF DESCRIPTION OF DRAWINGS

[0007] The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of this application, illustrate embodiments of the application, and together with the description serve to explain the principles of the application. In the drawings: Figure 1 is a block diagram of a matrix multiplication operation device of an embodiment of the application; Figure 2 is a schematic diagram of the principle of wide word packing of the matrix multiplication operation device of an embodiment of the application; Figure 3 is an exemplary block diagram of an operation core of an embodiment of the application, which is a plurality of parallel bit width processing units; Figure 4 is an exemplary flowchart of a matrix multiplication method according to an embodiment of the present application. DETAILED DESCRIPTION

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some examples or embodiments of the present application, and for those skilled in the art, the present application can also be applied to other similar scenarios without creative labor. Unless the context clearly indicates otherwise or otherwise stated, the same reference numbers in the drawings represent the same structures or operations.

[0009] As shown in the present application, unless the context clearly indicates otherwise or otherwise stated, the words "one", "a", "an", and / or "the" do not specifically refer to the singular, but can also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0010] Unless otherwise specifically stated, the relative arrangement of the components and steps, numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that the sizes of the various parts shown in the drawings are not drawn in proportion to the actual proportions. The technology, methods and devices known to those skilled in the relevant art can not be discussed in detail, but under appropriate circumstances, the technology, methods and devices should be considered as part of the specification. In all examples shown and discussed here, any specific value should be interpreted as merely exemplary, and not as a limitation. Therefore, other examples of exemplary embodiments can have different values. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0011] In addition, it should be noted that the use of the terms "first", "second", and the like to define parts is merely for the convenience of distinguishing the corresponding parts, and unless otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present application. In addition, although the terms used in the present application are selected from commonly known terms, some terms mentioned in the specification of the present application can be selected by the applicant according to his or her judgment, and the detailed meaning of each term is explained in the relevant part of the description. In addition, the present application is not only understood by the actual terms used, but also by the meaning implied by each term.

[0012] Flow diagrams have been used in this application to illustrate the operations performed by systems in accordance with embodiments of the application. It will be appreciated that the operations in the flow diagrams are not necessarily performed in the order shown. Rather, various steps can be performed in parallel or in a different order than shown. Also, other operations can be added to, or removed from, the processes described in these flow diagrams.

[0013] Existing low-bit (e.g., INT8, INT4) quantization usually needs to rely on integer multiply-add operations, which requires a large area of multiplier array, resulting in difficulty in further reducing chip area and power consumption. The BitNet-1.58Bit solution of Microsoft quantizes the weights to ±1 / 0, but still needs to perform multiplication on the unpacked 8-bit activations in the multiply-add stage.

[0014] This application proposes a matrix multiplication operation device and method for performing multiplication operation of activation value matrix and weight value matrix in an artificial intelligence model, relying on bit logic to complete matrix multiplication operation, without using multipliers, which can greatly save computing overhead and further improve inference energy efficiency at the hardware level. In some embodiments, the artificial intelligence model is a large language model.

[0015] Figure 1 is a block diagram of a matrix multiplication operation device according to an embodiment of the application. Referring to Figure 1 The matrix multiplication operation device 100 of this embodiment includes a data layout unit 110, a bit operation unit 120, a bit count unit 130, and a control unit 140. The data layout unit 110 is configured to split the weight value matrix into a first sign bit plane and a plurality of first amplitude bit planes, split the activation value matrix into a second sign bit plane and a plurality of second amplitude bit planes, and arrange the first sign bit plane and the first amplitude bit planes in the same row in the memory, and arrange the second sign bit plane and the second amplitude bit planes in the same row in the memory. The bit operation unit 120 is configured to calculate an opposite sign mask based on the first sign bit plane and the second sign bit plane, and perform a bitwise AND operation on different combinations of the first amplitude bit planes and the second amplitude bit planes to obtain a plurality of common non-zero masks. The bit count unit 130 is configured to obtain non-zero count values in the AND operation results of the opposite sign mask and the plurality of common non-zero masks. The control unit 140 is configured to perform j+k-bit shift accumulation on the non-zero count values to obtain the matrix product of the activation value matrix and the weight value matrix, where j is the bit index of the weight, k is the bit index of the activation, j=0:J-2, k=0:K-2, J and K are the total number of bits of the weight and the total number of bits of the activation, respectively.

[0016] According to the matrix multiplication operation device 100 of the present application, the weight value matrix and the activation value matrix are respectively split into bit planes by the data layout unit 110, and the corresponding bit planes are organized in rows, so that the weights are stored in the same row and the activations are stored in another same row. The sign mask is pre-calculated by the bit operation unit 120, so that the sign mask is reused in the calculation process of the amplitude bit plane. A plurality of common non-zero masks of different combinations of the amplitude bit planes of all weights and all activations are obtained. The non-zero count values in the AND operation results of the sign mask and the plurality of common non-zero masks are obtained by the bit count unit 130. The non-zero count values are shifted and accumulated by the control unit 140, so as to obtain the matrix product of the activation value matrix and the weight value matrix. The matrix multiplication operation device 100 does not need to use a multiplier, which can greatly save the calculation cost and reduce the chip power consumption.

[0017] The above-mentioned matrix multiplication operation device 100 is described below with an example of a bit width of 1024 bits. Bit width refers to the data bus width, register operation bit number or instruction set data width that can be processed in parallel by a processor at one time. The processor can be a CPU, GPU, DPU, etc. However, the specific implementation of the present application is not limited to 1024 bits. Here, 1024 bits specifically refer to the length of a row vector in the multiplication operation process. 1024 bits include 128 bytes, and each byte includes 8 bits.

[0018] The present application does not limit the dimensions of the weight value matrix and the dimensions of the activation value matrix. Generally, the dimensions of the weight value matrix are greater than the dimensions of the activation value matrix. For example, the dimensions of the weight value matrix are m1 n1, the dimensions of the activation value matrix are m2 n2, m1>m2, n1>n2. Each value in the matrix represents a weight value or an activation value. The weight value and the activation value are originally decimal values. In order to perform subsequent bit operations, they need to be converted into binary data. Assuming that the total number of bits of the binary converted weight and the total number of bits of the activation are both 8 bits, i.e. J=K=8. For each weight, the 7th bit is the sign bit, and the 0-6th bits are the amplitude bits. For each activation, the 7th bit is the sign bit, and the 0-6th bits are the amplitude bits. From the 0th bit to the 6th bit, the weights are 2 0 , …, 2 6 .

[0019] The data layout unit 110 can split each weight to obtain 1 first sign bit plane and 7 first amplitude bit planes. Among them, the first sign bit plane includes the sign bits of all weights, and the jth of the 7 first amplitude bit planes corresponds to the jth amplitude bit of all weights, j=0:6. Similarly, the data layout unit 110 can split each activation to obtain 1 second sign bit plane and 7 second amplitude bit planes. Among them, the second sign bit plane includes the sign bits of all activations, and the kth of the 7 second amplitude bit planes corresponds to the kth amplitude bit of all activations, k=0:6.

[0020] It should be noted that the number of bits of the weight and the number of bits of the activation can be flexibly configured according to the model accuracy or bandwidth requirement.

[0021] Here, the bit plane is explained. The bit plane refers to the independent data set corresponding to each "weight level" of binary bits after the multi-bit (Bit) data (such as image pixels, sensor data, storage unit data) is split. Bit plane is usually used for "parallel decomposition and processing by bit". For example, the pixel value (0~255) of an 8-bit grayscale image can be split into 8 bit planes (0th~7th), among which the 7th bit (the highest bit, MSB) determines the brightness of the pixel, and the 0th bit (the lowest bit, LSB) only affects the detail noise. In the specific application scenario of the present application, the 7th bit of the activation and the weight is the sign bit, and the 0th~6th bit is the amplitude bit.

[0022] The data layout unit 110 is also used to make the first sign bit plane and the first amplitude bit plane in the same row in the memory, and make the second sign bit plane and the second amplitude bit plane in the same row in the memory. The four bit planes here are collectively referred to as bit planes. The operation of the data layout unit 110 can also be called row organization. It should be noted that after row organization, each row in the first sign bit plane and the corresponding row in each first amplitude bit plane are in one row in the memory, and each row in the second sign bit plane and the corresponding row in each second amplitude bit plane are in another row in the memory. Alternatively, after row organization, each row in the first sign bit plane and the corresponding row in each first amplitude bit plane are in a weight row in the memory, and each row in the second sign bit plane and the corresponding row in each second amplitude bit plane are in an activation row in the memory, and the weight row and the activation row are in different rows.

[0023] In one embodiment, the row organization described above is stored contiguously in a wide-word aligned manner according to a row-first rule. The first sign bit plane and the first magnitude bit plane can be wide-word packed in memory according to the row-first rule of the output matrix, making the bit plane data of the same row contiguous at physical addresses. Similarly, the second sign bit plane and the second magnitude bit plane can be wide-word packed in memory according to the row-first rule of the output matrix, making the bit plane data of the same row contiguous at physical addresses. That is, each row in the first sign bit plane and its corresponding row in each first magnitude bit plane are stored contiguously as active rows in the memory; each row in the second sign bit plane and its corresponding row in each second magnitude bit plane are stored contiguously as weighted rows in the memory; wherein the rows in the memory have a preset width. The memory here can be high-bandwidth memory (HBM), dynamic random access memory (DRAM), or tile static random access memory (TRAM).

[0024] Regarding the packaging of wide characters, it can be combined with... Figure 2 Let's understand this. Assume the size of the weight matrix is ​​3. 1024, the default width is 1024. 8 bits. The preset width should be sufficient to store the first sign bit plane and all first magnitude bit planes in a single line. Figure 2 In this embodiment, the preset width is equal to 1024. 8 bits can be used to combine the first sign bit plane (1024 bits) and the entire first magnitude bit plane (1024 bits). All 7 bits are stored in one row. Figure 2 In the diagram, P7 represents the first sign bit plane, and P0~P6 represent the seven first magnitude bit planes, each of which includes 3... 1024 binary values. Pack an 8-bit bit plane into 1024. An 8-bit wide word is stored as follows: first, rows 1 of P7~P0 are stored sequentially in the first row of memory; then rows 2 of P7~P0 are stored sequentially in the second row of memory, and so on. During storage, it is assumed that the size of the weight matrix is ​​3. 512, then after storing rows 1 of P7~P0, the first row 1024 Unfilled bits in an 8-bit wide word are padded with 0s or 1s. Similarly, for the activation value matrix, the second sign bit plane and seven second magnitude bit planes are stored in other rows of memory (activation rows).

[0025] By wide word packing, it is also beneficial to align each bit plane with the bit width of the parallel bit width processing unit. The parallel bit width processing unit will be described later.

[0026] The data layout unit 110 completes the wide word aggregation operation of the bit plane at the storage layer, which can further save the physical bandwidth.

[0027] In some embodiments, the bit operation unit 120 calculates the different sign mask based on the first sign bit plane and the second sign bit plane, which can be represented by the following formula (1): S = Ws XOR As (1) Where S is the different sign mask; Ws is the sign bit of the weight row, from the first sign bit plane; As is the sign bit of the activation row, from the second sign bit plane; XOR represents exclusive or. The different sign mask S obtained by one calculation can be reused in the entire (j, k) amplitude bit plane cycle, without repeating the calculation at each clock.

[0028] The calculation of the different sign mask S can be performed by an exclusive or unit. The different sign mask S indicates that when the weight and the activation sign are the same, 0 is output, and when the weight and the activation sign are different, 1 is output. The 1 in the different sign mask S indicates the value position where the weight and the activation sign are different.

[0029] In some embodiments, the matrix multiplication operation device 100 further comprises a sign mask register, and the different sign mask S is stored in the sign mask register. In the subsequent step, when the sign mask S needs to be reused, it can be directly called from the sign mask register.

[0030] In some embodiments, the bit operation unit 120 performs a bitwise AND operation on different combinations of the first amplitude bit plane and the second amplitude bit plane to obtain a plurality of common non-zero masks, which can be represented by the following formula (2): V = Wb AND Ab (2) Where V represents the common non-zero mask; Wb represents the amplitude bit of the weight row, from the first amplitude bit plane; Ab represents the amplitude bit of the activation row, from the second amplitude bit plane; AND represents AND.

[0031] The calculation of the common non-zero mask V can be performed by an AND unit.

[0032] In the above example, the weight and the activation amplitude bit both include 7 bits, i.e. the number of the first amplitude bit plane and the second amplitude bit plane is 7, so the common non-zero mask V under 49 different combinations can be obtained.

[0033] In some embodiments, since the activation value matrix is relatively small, the second sign bit plane and the several second amplitude bit planes corresponding to the activation value matrix can be stored in a high-speed memory, and the target second amplitude bit plane can be obtained from the high-speed memory within one or more clock cycles, and the row vector of the target first amplitude bit plane of a different weight row in the memory is operated to obtain the common non-zero mask V corresponding to the combination of the target second amplitude bit plane and the target first amplitude bit plane. And the common non-zero mask V operation of the next combination is continued in the next one or more clock cycles. In this way, the common non-zero mask V operation of the 49 different combinations is completed.

[0034] In some embodiments, the bit count unit 130 obtains the non-zero count value in the result of the AND operation of the different sign mask S and the common non-zero mask V, which can be performed in the following way: For each common non-zero mask, the bit count unit 130 calculates the AND operation between the common non-zero mask V and the negation of the different sign mask S to obtain a first count value, and calculates the AND operation between the common non-zero mask V and the different sign mask S to obtain a second count value; the first count value is subtracted from the second count value to obtain the non-zero count value.

[0035] This way can be specifically expressed by the following formula (3): Δ = popcnt(V & ~S) - popcnt(V & S) (3) Where Δ is the non-zero count value; popcnt is a single bit 1 count instruction for obtaining the number of 1s in the parentheses; ~S represents the negation of S. Wherein, popcnt(V & ~S) is the first count value, and popcnt(V & S) is the second count value.

[0036] The negation of the different sign mask S, 1 in ~S indicates that the sign of the weight is the same as the activation. The result of the product with the same sign is positive, so popcnt(V & ~S) is the number of 1s in the number whose product result is positive. Similarly, popcnt(V & S) is the number of 1s in the number whose product result is negative. The former is subtracted from the latter to obtain the non-zero count value Δ, or the non-zero count difference value. That is, for a combination of a first amplitude bit plane and a second amplitude bit plane corresponding to a V, the non-zero count value Δ represents the product of the first amplitude bit plane and the second amplitude bit plane under this combination. In other words, when performing matrix multiplication of the weight value matrix and the activation value matrix, each value in the multiplication result matrix is the dot product of the two vectors, that is, the result of multiplying all elements at corresponding positions and then summing. According to the bit plane operation of the present application, the dot product operation is equivalent to the number of +1s in Δ minus the number of -1s, so as to obtain the result of the summation.

[0037] The specific calculation of the non-zero count value Δ can be performed by the Popcount unit. The bit width of the Popcount unit is not less than 1024 bits. According to these embodiments, the Popcount unit can output the non-zero count value Δ in no more than one clock cycle, and the calculation efficiency is high.

[0038] At the hardware level, the bit count unit 130 of the present application includes two Popcount units. Therefore, the bit count unit 130 of the present application can simplify the operation of “twice popcnt + subtraction” to “single popcnt directly output Δ”, that is, using 2 Popcount units in parallel to process popcnt(V & ~S) and popcnt(V & S) at the same time, which can improve the operation efficiency.

[0039] In some embodiments, the Popcount unit only obtains the first count value and the second count value, respectively. And further outputs the first count value and the second count value to the control unit 140, and the control unit 140 calculates the non-zero count value Δ.

[0040] In some embodiments, the matrix multiplication operation device 100 includes operation cores and an external control layer. Figure 1 The data layout unit 110, the bit operation unit 120 and the bit count unit 130 shown in the above are arranged in the operation core, and the control unit 140 is arranged in the external control layer. The operation core is a minimum reusable hardware unit proposed by the present application for performing bit-wise integer matrix multiplication. The operation core itself has a bit width. In the foregoing example, the bit width of the operation core is 1024 bits. The operation core can complete the tasks to be performed by the data layout unit 110, the bit operation unit 120 and the bit count unit 130. That is, the operation core itself does not perform the multiplication or unpacking operation, and the operation core can also have an output interface for outputting the first count value and the second count value to the external control layer.

[0041] According to the above embodiments, the calculation link of the operation core is only composed of AND operation, XOR operation and Popcont difference operation, the calculation throughput can be linearly enlarged with the bit width and the number of operation cores, and has good linear scalability, which can cover a larger bandwidth range.

[0042] In some embodiments, the operation core is a plurality of parallel bit width processing units (Slices). The bit width of each parallel bit width processing unit is 1024 bits.

[0043] Figure 3This is an exemplary block diagram of a processing core comprising multiple parallel bit-width processing units according to one embodiment of this application. The processing core consists of four slices, namely Slice0-Slice3, with each slice corresponding to one processing core. Each processing core can independently process data layout, as well as 49 combinations of bit operations and bit counting. According to this embodiment, the four slices can simultaneously complete the calculation of the non-zero count value Δ, thereby enabling parallel processing of multiple row vector multiplications and linearly increasing the overall computing power. Furthermore, the non-zero count value Δ can be serially output to the INT Pipeline, i.e., the pipeline structure for processing integer arithmetic instructions, which can be specifically executed by the chip's integer arithmetic unit (ALU).

[0044] The control unit 140 is located in the outer control layer, indicating that the shift accumulation step is performed in the outer control layer.

[0045] The following example illustrates the process of shift accumulation.

[0046] Assuming the current first amplitude plane is the j-th and the current second amplitude plane is the k-th, we first shift the result of multiplying the current first amplitude plane and the current second amplitude plane by Δ left by j+k positions to generate a shifted matrix. Let j=0: J-2, k=0: K-2, we can obtain 49 shifted matrices. We then sum these 49 shifted matrices to obtain the final multiplication result.

[0047] When using the external control layer for shift accumulation, a shift adder or a LUT lookup table structure can be used to perform the operation.

[0048] The following is an example pseudocode for performing shift accumulation in the outer control layer: acc = 0 for i in 0..B_w-1: # weight bit / digit index for k in 0..B_a-1: # activation bit / digit index Δ = core_popdiff(Ws[i], Wb[i], As[k], Ab[k]) # The kernel returns the difference in pop counts. acc += Δ << (i+k) # Left shift weighted sum return acc Where B_w is the number of bits in the weights, for example, 7; B_a is the number of bits in the activation, for example, 7. core_popdiff(Ws[i], Wb[i], As[k], Ab[k]) is the specific code implementation of formula (3).

[0049] In some embodiments, the output data can be in an INT format or a floating-point format.

[0050] The present application also proposes a matrix multiplication method, which can be executed by the matrix multiplication device 100 described above, and thus the above description can be used to describe the matrix multiplication method, and the repeated contents will not be described again. It can be understood that the method can also be executed by using other devices capable of implementing the method, and is not limited to the matrix multiplication device 100.

[0051] Figure 4 is an exemplary flowchart of the matrix multiplication method of an embodiment of the present application. Referring to FIG. 4, the matrix multiplication method 400 of the embodiment includes: Figure 4 Step S410: splitting the weight value matrix into a first sign bit plane and a plurality of first amplitude bit planes; Step S420: splitting the activation value matrix into a second sign bit plane and a plurality of second amplitude bit planes; Step S430: arranging the first sign bit plane and the first amplitude bit planes in the same row in the memory; Step S440: arranging the second sign bit plane and the second amplitude bit planes in the same row in the memory; Step S540: calculating a different sign mask based on the first sign bit plane and the second sign bit plane; Step S460: performing a bitwise AND operation on different combinations of the first amplitude bit planes and the second amplitude bit planes to obtain a plurality of common non-zero masks; Step S470: obtaining a non-zero count value in the AND operation result of the different sign mask and the plurality of common non-zero masks, respectively; and Step S480: performing j+k-bit shifting on the jth first amplitude bit plane and the kth second amplitude bit plane, and performing weighted accumulation based on the non-zero count value, to obtain a matrix product of the activation value matrix and the weight value matrix, where j is a bit index of the weight, k is a bit index of the activation, j=0:J-2, k=0:K-2, J and K are the total number of bits of the weight and the total number of bits of the activation, respectively.

[0052] In some embodiments, Step S470 includes: for each common non-zero mask, calculating an AND operation between the common non-zero mask and the negation of the different sign mask to obtain a first count value; calculating an AND operation between the common non-zero mask and the different sign mask to obtain a second count value; and subtracting the second count value from the first count value to obtain the non-zero count value.

[0053] In some embodiments, the matrix multiplication method 400 further includes: multiplexing the different sign mask when performing the operation for each common non-zero mask.

[0054] ​The present application also proposes a computer program product comprising machine executable instructions that, when executed by one or more processors of a machine, cause the machine to perform the above-described matrix multiplication operation method 400.

[0055] Aspects of the present application can be implemented in, completely, by hardware, completely, by software (including firmware, resident software, micro-code, etc.), or by combinations of hardware and software. The above hardware or software can be referred to as "data blocks", "modules", "engines", "units", "components", or "systems". The processor can be one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, or combinations thereof. In addition, aspects of the present application can be embodied in a computer program product located in one or more computer readable media, which includes computer readable program code. For example, the computer readable medium can include, but is not limited to, magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips...), optical disks (e.g., compact disk CD, digital versatile disk DVD...), smart cards, and flash memory devices (e.g., card, stick, key drive...).

[0056] The computer readable medium can contain a propagated data signal with computer program code embodied therein, for example, in baseband or as part of a carrier wave. Such propagated signal can take through various forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. The computer readable medium can be any computer readable medium other than a computer readable storage medium that can be connected, communicated, or transferred to an instruction execution system, apparatus, or device, so as to implement communication, propagation, or transmission of programs for use. The program code located on the computer readable medium can be propagated through any suitable medium, including, but not limited to, wireless, cable, optical fiber cable, radio frequency signals, or similar medium, or any combination of the above.

[0057] Those of ordinary skill in the art will realize and understand that all or some of the steps in the methods disclosed above and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementation, the division between the functional modules / units referred to in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable storage media, which can include computer storage media (or non-transitory media) and communication media (or transitory media).

[0058] As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable program instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Further, it is well known to those of ordinary skill in the art that communication media typically embodies computer readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. As a result, the foregoing description of computer storage media, along with communication media, applies to and fully integrates software and / or programs such as program modules, program data, and / or computer readable program instructions.

[0059] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0060] The computer program product described herein can be embodied in hardware, software, or a combination thereof. In one alternative, the computer program product is embodied in a computer storage medium. In another alternative, the computer program product is embodied in a software product, such as a software development kit (SDK), and the like.

[0061] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including a

[0062] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0063] The flow diagrams and block diagrams in the drawings are representative of the architectural, functional, and operational aspects of possible implementations of systems, methods and computer program products according to the present disclosure. In this regard, each block in the flow diagrams and block diagrams can represent a module, a segment, or a portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions ("instructions"). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may

[0064] Example embodiments have been disclosed herein and, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that features, characteristics or / and elements described in connection with a particular embodiment can be used in conjunction with other embodiments unless otherwise explicitly stated. Accordingly, it will be understood that various changes in form and details can be made without departing from the scope of the disclosure as set forth in the appended claims.

Claims

1. A matrix multiplication operation device for performing multiplication operations of activation value matrix and weight value matrix in an artificial intelligence model, characterized in that, include: A data layout unit is used to split the weight value matrix into a first sign bit plane and several first magnitude bit planes; The activation value matrix is ​​split into a second sign bit plane and several second magnitude bit planes; The first sign bit plane and the first magnitude bit plane are located in the same row in the memory, and the second sign bit plane and the second magnitude bit plane are located in the same row in the memory; The bit operation unit is used to calculate the opposite sign mask based on the first sign bit plane and the second sign bit plane, and to perform bitwise AND operation on different combinations of the first magnitude bit plane and the second magnitude bit plane to obtain multiple common non-zero masks; A bit counting unit is used to obtain the non-zero count value in the AND operation result of the different-sign masks and the multiple common non-zero masks; as well as The control unit is used to perform a j+k bit shift on the j-th first amplitude bit plane and the k-th second amplitude bit plane, and perform a weighted accumulation based on the non-zero count value to obtain the matrix product of the activation value matrix and the weight value matrix, where j is the bit index of the weight, k is the bit index of the activation, j=0:J-2, k=0:K-2, and J and K are the total number of bits of the weight and the total number of bits of the activation, respectively.

2. The matrix multiplication apparatus as described in claim 1, characterized in that, The method of placing the first sign bit plane and the first magnitude bit plane in the same row in the memory includes: storing each row in the first sign bit plane and the corresponding row in each of the first magnitude bit planes consecutively in the same row of the memory; The method of placing the second sign bit plane and the second magnitude bit plane in the same row in the memory includes: storing each row in the second sign bit plane and the corresponding row in each of the second magnitude bit planes consecutively in the same row of the memory; The rows in the memory have a preset width.

3. The matrix multiplication apparatus as described in claim 1, characterized in that, For each common non-zero mask, the bit counting unit calculates the AND operation between the inverse of the common non-zero mask and the opposite-sign mask to obtain a first count value; And calculate the AND operation between the common non-zero mask and the opposite sign mask to obtain a second count value; subtract the second count value from the first count value to obtain the non-zero count value.

4. The matrix multiplication apparatus as described in claim 1, characterized in that, For each common non-zero mask, the bit counting unit calculates the AND operation between the inverted common non-zero mask and the opposite-sign mask to obtain a first count value; and calculates the AND operation between the common non-zero mask and the opposite-sign mask to obtain a second count value; the control unit is further configured to subtract the second count value from the first count value to obtain the non-zero count value.

5. The matrix multiplication apparatus as described in claim 1, characterized in that, It also includes a symbol mask register, in which the opposite-sign mask is stored.

6. The matrix multiplication apparatus as described in claim 1, characterized in that, The matrix multiplication operation device includes an operation core and an external control layer. The data layout unit, the bit operation unit, and the bit counting unit are disposed in the operation core, and the control unit is disposed in the external control layer. The operation core also includes an output interface for outputting the non-zero count value to the external control layer.

7. A matrix multiplication method for performing multiplication of activation value matrices and weight value matrices in an artificial intelligence model, characterized in that, include: The weight value matrix is ​​split into a first sign bit plane and several first magnitude bit planes; The activation value matrix is ​​split into a second sign bit plane and several second magnitude bit planes; The first sign bit plane and the first magnitude bit plane are located in the same row in the memory; The second sign bit plane and the second magnitude bit plane are located in the same row in the memory; Calculate the opposite sign mask based on the first sign bit plane and the second sign bit plane; Perform a bitwise AND operation on different combinations of the first magnitude bit plane and the second magnitude bit plane to obtain multiple common non-zero masks; Obtain the non-zero count value from the AND operation result of the different-sign masks and the multiple common non-zero masks; as well as The j-th first amplitude bit plane and the k-th second amplitude bit plane are shifted by j+k bits, and weighted accumulation is performed based on the non-zero count values ​​to obtain the matrix product of the activation value matrix and the weight value matrix, where j is the bit index of the weight, k is the bit index of the activation, j=0:J-2, k=0:K-2, and J and K are the total number of bits of the weight and the total number of bits of the activation, respectively.

8. The matrix multiplication method as described in claim 7, characterized in that, The method of placing the first sign bit plane and the first magnitude bit plane in the same row in the memory includes: storing each row in the first sign bit plane and the corresponding row in each of the first magnitude bit planes consecutively in the same row of the memory; The method of placing the second sign bit plane and the second magnitude bit plane in the same row in the memory includes: storing each row in the second sign bit plane and the corresponding row in each of the second magnitude bit planes consecutively in the same row of the memory; The rows in the memory have a preset width.

9. The matrix multiplication method as described in claim 7, characterized in that, Obtaining the non-zero count value from the AND operation result of the different-sign masks and the multiple common non-zero masks includes: For each common non-zero mask, perform an AND operation between the inverse of the common non-zero mask and the mask with the opposite sign to obtain a first count value; perform an AND operation between the common non-zero mask and the mask with the opposite sign to obtain a second count value; and subtract the second count value from the first count value to obtain the non-zero count value.

10. The matrix multiplication method as described in claim 7, characterized in that, Also includes: When performing operations on each common non-zero mask, the mask with different signs is reused.

Images