Internal memory, chip and related electronic device

Abstract

Disclosed in the embodiments of the present application are an internal memory, a chip and a related electronic device. A target processing unit (202) of an internal memory (102) comprises a plurality of groups of primary multipliers, a plurality of adder trees and a plurality of secondary multipliers, wherein one group of primary multipliers is connected to one secondary multiplier by means of one adder tree; one group of primary multipliers comprises a plurality of pairs of first multipliers; each adder tree comprises a first adder, a second adder and a third adder; each pair of first multipliers separately receives data to perform multiplication operations, and separately inputs output results into one first adder for addition operations; each second adder receives output results of a plurality of first adders or the upper-level second adder, so as to perform addition operations, and separately inputs the output results into the lower-level second adder or one third adder; and each third adder receives output results of a plurality of second adders to perform addition operations, and inputs the output results into one secondary multiplier for multiplication operations. Implementing the embodiments of the present application can improve the processing capability of the internal memory.

Description

Memory, chip and related electronic device

[0001] The present application claims priority to the Chinese patent application No. 202411398209.8, filed on September 30, 2024, with the State Intellectual Property Office of China, and the Chinese patent application No. 202411398209.8 has the title of “Memory, chip and related electronic device”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of intelligent terminals, in particular to a memory, a chip and a related electronic device. BACKGROUND

[0003] With the development of artificial intelligence technology, generative AI has emerged, such as ChatGPT, etc. Deploying large models on the terminal side for AI inference can improve user experience and productivity, which has become one of the main competitive advantages of terminals. During the large model inference process, a large amount of data needs to be moved from the memory to the central processor, or the graphics processing unit, or the neural network processor for calculation, which causes the bottleneck of the current terminal side AI inference performance to mainly concentrate on the memory bandwidth. At present, in order to improve the terminal side AI inference performance, processing in memory (PIM) has become mainstream, that is, multiple independent processing units (PU) can be integrated in the memory, and part of the computing task can be offloaded to the memory for calculation, which can improve the computing performance of the terminal and accelerate the large model inference. In addition, using the processing in memory technology can reduce data movement, reduce terminal energy consumption, and improve terminal endurance.

[0004] With the continuous increase of the size of large models, the demand for computing and storage resources of large models also increases. In order to enable these large models to run on devices with limited resources, researchers are exploring various algorithm optimization techniques, among which quantization technology is an important direction. That is, by converting the high-precision format (such as 32-bit floating-point number FP32 or 16-bit floating-point number FP16) used in large models into a low-precision format (such as 8-bit integer INT8 or 4-bit integer INT4), the storage size of the model can be effectively reduced, which helps to reduce the demand of large models for hardware resources, so that large models can run on devices with limited computing power, while also speeding up inference and reducing power consumption. However, in this process, the accuracy of the large model may be affected, that is, the accuracy of the large model when performing tasks may decrease. In order to minimize this precision loss, researchers have begun to focus on dequantization technology. The core idea of dequantization technology is to restore or improve the precision of the quantized model through a dequantization algorithm.

[0005] However, due to the limitation of memory resources, the computing units in the memory are limited by the complexity of the formula when performing calculation, resulting in that the complex group dequantization algorithm cannot be effectively executed on these computing units. Therefore, how to improve the computing capability of the memory is a problem to be solved. SUMMARY

[0006] Embodiments of the present application provide a memory, a chip and a related electronic device to improve the computing capability of the memory.

[0007] In a first aspect, embodiments of the present application provide a memory, which includes a plurality of storage areas (Banks), a plurality of computing units, a first buffer and a first register. One Bank corresponds to one computing unit. The plurality of Banks includes a target Bank, and the target Bank corresponds to a target computing unit. The target computing unit includes a plurality of groups of first multipliers, a plurality of addition trees, a plurality of second multipliers and at least one fourth adder. One group of first multipliers is connected to one second multiplier through one addition tree. One fourth adder is connected to at least two second multipliers. One group of first multipliers includes a plurality of pairs of first multipliers. One group of first multipliers corresponds to one second multiplier. Each addition tree includes a plurality of first adders as leaf nodes, a plurality of second adders as intermediate nodes and a third adder as a root node. Each pair of first multipliers is configured to receive data stored in the target Bank and data of the first buffer respectively, perform multiplication operation, and input the output results to one connected first adder respectively to perform addition operation. Each second adder is configured to receive the output results of a plurality of connected first adders or a previous second adder, perform addition operation, and input the output results to a next second adder or the third adder connected respectively. Each third adder is configured to receive the output results of a plurality of connected second adders, perform addition operation. Each second multiplier is configured to receive the output results of the connected third adder and parameters in the first register, perform multiplication operation. The fourth adder is configured to receive the output results of the connected second multiplier, perform addition operation, and output the calculation result.

[0008] In the embodiments of the present application, the calculation circuit of the target calculation unit in the internal memory is improved, that is, the first multipliers are grouped, and each group of first multipliers is connected to a second multiplier through an addition tree, so that each group of data after grouping can be subjected to multi-stage multiplication and accumulation operation through the circuit. Specifically, a group of first multipliers includes a plurality of pairs of first multipliers, each addition tree includes a plurality of first adders as leaf nodes, a plurality of second adders as intermediate nodes, and a third adder as a root node. The internal memory can simultaneously take out a plurality of groups of to-be-calculated data from the target Bank, and can input a group of to-be-calculated data into a group of first multipliers for multiplication operation, that is, each pair of first multipliers in a group of first multipliers can respectively receive different parameters in the same group of to-be-calculated data for multiplication operation, and respectively input the output results into a connected first adder for addition operation; each second adder receives the output results of a plurality of connected first adders or the output results of the second adder of the previous stage for addition operation, and respectively inputs the output results into the second adder of the next stage or the third adder connected; each third adder receives the output results of a plurality of connected second adders for addition operation, and each second multiplier receives the output results of the connected third adder and the parameters in the first register for multiplication operation. In the prior art, the internal memory can only perform one-stage multiplication and accumulation operation on data, and cannot perform two-stage multiplication and accumulation operation on data. In the present application, a plurality of second multipliers are added, each second multiplier receives the output results of the connected third adder and the parameters in the first register for multiplication operation, and the operation results of the second multiplier are subjected to addition operation through a fourth adder, thereby realizing two-stage multiplication and accumulation operation on data and improving the calculation capability of the internal memory.

[0009] In some embodiments, each pair of first multipliers is connected to a first adder in the addition tree; the plurality of first adders, the plurality of second adders, and the third adder included in the addition tree are connected in a binary tree structure; and the third adder in the addition tree is connected to a second multiplier.

[0010] In the embodiments of the present application, each pair of first multipliers is connected with a first adder in the adder tree, so that the output of the pair of first multipliers can be input into the first adder for addition operation. The plurality of first adders, the plurality of second adders and the third adder in the adder tree are connected in a binary tree structure, so that a plurality of adders can perform addition operation in parallel at the same time, and in addition, by connecting the adders in layers, the transmission delay of data between different adders can be reduced, thereby improving the addition operation speed. The third adder in the adder tree is connected with a second-level multiplier, so that the result output by the adder tree is subjected to second-level multiplication operation by the second-level multiplier, thereby avoiding the problem that the current memory can only perform multiply-accumulate operation on data, and cannot perform respective multi-level multiply-accumulate operation on grouped data, thereby improving the computing capability of the memory.

[0011] In some embodiments, the memory further includes a first buffer, and each pair of first multipliers is specifically configured to: respectively receive data stored in the target Bank as respective first operands, and respectively receive data stored in the first buffer as respective second operands; and respectively perform multiplication operation based on the respective first operand and the respective second operand, and respectively input the output into a connected first adder for addition operation.

[0012] In the embodiments of the present application, the target Bank can be used to store weight data as the first operand of the first multiplier, and the first buffer can be used to store vector data as the second operand of the first multiplier. In the calculation, each first multiplier can receive data stored in the target Bank as the first operand, and receive data stored in the first buffer as the second operand, and perform multiplication operation based on the first operand and the second operand. Further, the first multiplier can input the output into a connected first adder for addition operation, thereby realizing multiply-accumulate operation.

[0013] In some embodiments, the target Bank stores a first sequence, the first sequence is composed of L first sub-sequences, and each first sub-sequence includes G first operands, L is an integer greater than 0; and the target computing unit is configured to receive a first sub-sequence through G first multipliers, the G first multipliers being first multipliers in one or more groups of first-level multipliers, wherein one first operand is input into one first multiplier.

[0014] In the embodiment of the present application, the target computing unit can receive a first subsequence through G first multipliers, and input a first operand into a first multiplier for calculation and dequantization. The G first multipliers can be first multipliers in a group of first multipliers, or first multipliers in multiple groups of first multipliers, so as to ensure that the target computing unit can perform multi-stage multiply-accumulate operation on the first subsequence, and to realize calculation and dequantization on the first subsequence, thereby improving the computing capability of the internal memory.

[0015] In some embodiments, when a group of first multipliers includes M first multipliers, and G is less than or equal to M, the G first multipliers are first multipliers in the group of first multipliers; when G is greater than M, the G first multipliers are first multipliers in S groups of first multipliers, and S is an integer greater than 1.

[0016] In the embodiment of the present application, when a group of first multipliers includes M first multipliers, and each first subsequence includes G first operands, since a first operand is input into a first multiplier for operation, if G is less than or equal to M, it means that the first multipliers in a group of first multipliers can process G first operands in parallel, so the first subsequence can be input into a group of first multipliers for calculation. If G is greater than M, it means that a group of first multipliers cannot completely process G first operands in parallel, so the first subsequence can be input into multiple groups of first multipliers to process G first operands in parallel, so as to ensure that the target computing unit can perform multi-stage multiply-accumulate operation on the first subsequence, thereby improving the computing capability of the internal memory.

[0017] In some embodiments, the first register stores a first parameter corresponding to each first subsequence, and each second multiplier is configured to: receive an output result of the connected third adder as a third operand, and receive the first parameter corresponding to the first subsequence sent by the first register as a fourth operand; perform multiplication operation based on the third operand and the fourth operand, and input the output result into the connected fourth adder for addition operation.

[0018] In the embodiment of the present application, after the first subsequence is input into the first multiplier and the addition tree for multiply-accumulate operation, the output result can be taken as the third operand of the second multiplier. Then, the corresponding first parameter, i.e., the dequantization factor, is taken out from the first register as the fourth operand. Further, the second multiplier performs multiplication operation based on the third operand and the fourth operand, and inputs the result of the second multiplier into the connected fourth adder for addition operation, so as to ensure that the target computing unit can perform multi-stage multiply-accumulate operation on the first subsequence, and to realize calculation and dequantization on the first subsequence, thereby improving the computing capability of the internal memory.

[0019] In some embodiments, when G is greater than M, the S secondary multipliers corresponding to the S primary multipliers receive the same first parameter as the fourth operand.

[0020] In the embodiments of the present application, if G is greater than M, a group of primary multipliers cannot completely process G first operands in a first subsequence in parallel, so the first subsequence can be input to multiple groups of primary multipliers to process G first operands in parallel. Further, a first subsequence corresponds to a first parameter, i.e., a dequantization factor, and the fourth operands of the secondary multipliers corresponding to the multiple groups of primary multipliers are the same, to ensure that the target computing unit can perform multi-level multiply-accumulate operations on the first subsequence, thereby improving the computing capability of the memory.

[0021] In some embodiments, the first instruction sent by the processing unit is received, the first sequence is read from the target Bank to the target computing unit based on the first instruction, and the first parameter corresponding to the first subsequence is read from the first register to the secondary multiplier.

[0022] In the embodiments of the present application, the first instruction can be a calculation instruction, to indicate that data is taken from the target Bank and the first sequence to be calculated is sent to the target computing unit, so that the target computing unit performs multi-level multiply-accumulate calculation on the first sequence through multiple groups of primary multipliers, multiple addition trees, and multiple secondary multipliers, such as performing calculation on the first sequence and performing grouping dequantization operation. In addition, the first module is also used to control the first register to send the first parameter corresponding to the first subsequence to the secondary multiplier, so that the calculation result of each group of first subsequence is calculated again by the multiple secondary multipliers, to realize the grouping dequantization of the grouped first subsequence.

[0023] In some embodiments, the target Bank is composed of multiple rows and multiple columns of storage units; the first parameter corresponding to each first subsequence and the first sequence are stored in the same row of storage units in the target Bank; or, the first parameter corresponding to each first subsequence and the first sequence are stored in different rows of storage units in the target Bank; or, the first parameter corresponding to each first subsequence and the first sequence are stored in different Banks.

[0024] In the embodiments of the present application, each first subsequence (i.e., weight data) and the corresponding first parameter (i.e., scale factor) can be stored in different rows of the same Bank, and each first subsequence and the corresponding first parameter can also be stored in the same row of the same Bank, so that the weight data and the corresponding scale factor can be guaranteed to be in a row of the Bank, reducing unnecessary cross-row operations. In addition, the scale factor and the quantized weight are stored in different Banks, which can concentrate the reading of the scale factor from the storage array to the global buffer, to ensure the flow of the model.

[0025] In some embodiments, the memory further includes a global buffer, and the first module is further configured to: receive the second instruction sent by the processor, read the first parameter corresponding to each first subsequence from the target Bank and store the first parameter into the first register; or read the first parameter corresponding to each first subsequence from other Banks and store the first parameter into the global buffer; and store the first parameter corresponding to each first subsequence from the global buffer into the first register.

[0026] In the embodiments of the present application, after receiving the second instruction sent by the processor, the first module can read the first parameter corresponding to each first subsequence from the target Bank and store the first parameter into the first register, so as to perform subsequent dequantization operation. Alternatively, the first module can read the first parameter corresponding to each first subsequence from other Banks and store the first parameter into the global buffer, and then store the first parameter corresponding to each first subsequence from the global buffer into the first register, so as to perform subsequent dequantization operation.

[0027] In a second aspect, the embodiments of the present application provide a control method applied to a memory, the memory including a plurality of storage areas (Banks), a plurality of calculation units, a first buffer and a first register, one Bank corresponding to one calculation unit, the plurality of Banks including a target Bank, the target Bank corresponding to a target calculation unit; the target calculation unit including a plurality of groups of first multipliers, a plurality of addition trees, a plurality of second multipliers, at least one fourth adder and the first register; wherein one group of first multipliers is connected with one second multiplier through one addition tree; one fourth adder is connected with at least two second multipliers; one group of first multipliers includes a plurality of pairs of first multipliers; one group of first multipliers corresponds to one second multiplier; each addition tree includes a plurality of first adders as leaf nodes, a plurality of second adders as intermediate nodes and a third adder as a root node; data stored in the target Bank and data of the first buffer are input into each pair of first multipliers to perform multiplication operation, and output results of each pair of first multipliers are input into one connected first adder to perform addition operation; each second adder is configured to receive output results of a plurality of connected first adders or a second adder at a previous level to perform addition operation, and input output results into a second adder at a next level or the third adder connected thereto; each third adder is configured to receive output results of a plurality of connected second adders to perform addition operation; parameters in the first register are input into each second multiplier and output results of the connected third adder to perform multiplication operation; and the fourth adder receives output results of the connected second multipliers to perform addition operation and outputs a calculation result.

[0028] In the embodiment of the present application, the calculation circuit of the target calculation unit in the memory is improved, that is, the first multipliers are grouped, and each group of first multipliers is connected to a second multiplier through an addition tree, so that each group of data after grouping can be subjected to multi-stage multiplication and accumulation operation through the circuit. Specifically, a group of first multipliers includes a plurality of pairs of first multipliers, each addition tree includes a plurality of first adders as leaf nodes, a plurality of second adders as intermediate nodes, and a third adder as a root node. The memory can simultaneously take out a plurality of groups of to-be-calculated data from the target Bank, and can input a group of to-be-calculated data into a group of first multipliers for multiplication operation, that is, each pair of first multipliers in a group of first multipliers can respectively receive different parameters in the same group of to-be-calculated data for multiplication operation, and respectively input the output results into a connected first adder for addition operation; each second adder receives the output results of a plurality of connected first adders or the output results of the second adder of the previous stage for addition operation, and respectively inputs the output results into the second adder of the next stage or the third adder connected; each third adder receives the output results of a plurality of connected second adders for addition operation, and each second multiplier receives the output results of the connected third adder and the parameters in the first register for multiplication operation. In the prior art, the memory can only perform one-stage multiplication and accumulation operation on the data, and cannot perform two-stage multiplication and accumulation operation on the data. In the present application, a plurality of second multipliers are added, each second multiplier receives the output results of the connected third adder and the parameters in the first register for multiplication operation, and the operation results of the second multiplier are subjected to addition operation through a fourth adder, thereby realizing two-stage multiplication and accumulation operation on the data and improving the calculation capability of the memory.

[0029] In some embodiments, the memory further includes a first buffer, and inputting the data stored in the target Bank into each pair of first multipliers for multiplication operation includes: inputting the data stored in the target Bank into each pair of first multipliers as first operands, respectively, and inputting the data stored in the first buffer into each pair of first multipliers as second operands, respectively; performing multiplication operation based on the respective first and second operands through each pair of first multipliers, respectively, and inputting the output results into a connected first adder for addition operation, respectively.

[0030] In the embodiments of the present application, the target Bank can be used to store weight data as the first operand of the first multiplier; the first buffer can be used to store vector data as the second operand of the first multiplier. In the calculation, each first multiplier can receive the data stored in the target Bank as the first operand and receive the data stored in the first buffer as the second operand, and perform multiplication operation based on the first operand and the second operand. Further, the first multiplier can input the output result into the connected first adder to perform addition operation, so as to realize multiply-accumulate operation.

[0031] In some embodiments, the target Bank stores a first sequence, the first sequence is composed of L first sub-sequences, and each first sub-sequence includes G first operands, L is an integer greater than 0; a group of first multipliers includes M first multipliers; when M is greater than or equal to G, the data stored in the target Bank is input into each pair of first multipliers as the first operand, including: inputting the G first operands in each first sub-sequence into the G first multipliers of a group of first multipliers to perform multiplication operation.

[0032] In the embodiments of the present application, when a group of first multipliers includes M first multipliers, and each first sub-sequence includes G first operands, since one first operand is input into one first multiplier to perform operation, if G is less than or equal to M, it means that the first multipliers in a group of first multipliers can process G first operands in parallel, so the first sub-sequence can be input into a group of first multipliers to perform calculation.

[0033] In some embodiments, the target computing unit further includes a first register, the first register stores a first parameter corresponding to each first sub-sequence, the parameter in the first register is input into each second multiplier and the output result of the connected third adder to perform multiplication operation, including: receiving the output result of the connected third adder as the third operand, and inputting the first parameter corresponding to the first sub-sequence in the first register into the second multiplier as the fourth operand; performing multiplication operation based on the third operand and the fourth operand by the second multiplier.

[0034] In the embodiments of the present application, when the first sub-sequence is input into the one-level multiplier and addition tree for multiplication accumulation operation, the output result can be taken as the third operand of the two-level multiplier. Then, the corresponding first parameter, i.e., the inverse quantization factor, is taken out from the first register as the fourth operand. Further, the two-level multiplier performs multiplication operation based on the third operand and the fourth operand, and the result of the two-level multiplier is input into the connected fourth adder for addition operation, so as to ensure that the target calculation unit can perform multi-level multiplication accumulation operation on the first sub-sequence, realize calculation on the first sub-sequence and inverse quantization, and thus improve the calculation capability of the memory.

[0035] In a third aspect, the embodiments of the present application provide a chip, which comprises the memory as described in any one of the first aspects.

[0036] In a fourth aspect, the embodiments of the present application provide an electronic device, which comprises a circuit board and the memory as described in any one of the first aspects, and the circuit board is electrically connected with the memory. BRIEF DESCRIPTION OF DRAWINGS

[0037] FIG. 1 is a schematic diagram of a chip system provided by the embodiments of the present application.

[0038] FIG. 2A is a structural schematic diagram of a memory provided by the embodiments of the present application using memory computing technology.

[0039] FIG. 2B is a structural schematic diagram of a PIM-Die provided by the embodiments of the present application.

[0040] FIG. 2C is a structural schematic diagram of a Bank provided by the embodiments of the present application.

[0041] FIG. 2D is a schematic diagram of calculation based on a weight matrix before quantization provided by the embodiments of the present application.

[0042] FIG. 2E is a schematic diagram of calculation based on a weight matrix after quantization provided by the embodiments of the present application.

[0043] FIG. 3 is a structural schematic diagram of a memory provided by the embodiments of the present application.

[0044] FIG. 4 is a structural schematic diagram of a target calculation unit provided by the embodiments of the present application.

[0045] FIG. 5 is a circuit structural schematic diagram of a target calculation unit provided by the embodiments of the present application.

[0046] FIG. 6 is a schematic diagram of the first operand and the second operand of a first multiplier provided by the embodiments of the present application.

[0047] FIG. 7 is a circuit connection schematic diagram of a fourth adder provided by the embodiments of the present application.

[0048] Figure 8 is a schematic diagram of loading data from a target Bank to a first multiplier according to an embodiment of the present application.

[0049] Figure 9 is a schematic diagram of a plurality of first multipliers according to an embodiment of the present application.

[0050] Figure 10 is a schematic diagram of a first register according to an embodiment of the present application.

[0051] Figure 11 is a schematic diagram of another first register according to an embodiment of the present application.

[0052] Figure 12 is a schematic diagram of a Bank storing each first sub-sequence and corresponding first parameter according to an embodiment of the present application.

[0053] Figure 13 is a schematic diagram of another Bank storing each first sub-sequence and corresponding first parameter according to an embodiment of the present application.

[0054] Figure 14 is a schematic diagram of a flow of quantization information configuration according to an embodiment of the present application.

[0055] Figure 15 is a schematic diagram of a flow of dequantization factor configuration instruction preparation according to an embodiment of the present application.

[0056] Figure 16 is a schematic diagram of a flow of computation according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0058] The terms "first", "second", "third", and "fourth" and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of use in either order. Furthermore, the term "comprising" or "containing" or "including" as used herein is used in the sense of "including", that is to say, in the sense of "comprising", and / or "containing", and / or "including", but not limited to, in an open, non-limiting sense.

[0059] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, or to a single alternative embodiment. It is expressly understood that any of the features, structures, or characteristics described in connection with an embodiment can be included in at least one implementation of the application. It is expressly understood that the phrase "in an embodiment" is not used to refer to or

[0060] The chip system related to the embodiments of the present application is introduced as follows:

[0061] The embodiments of the present application provide a chip system. Referring to FIG. 1, FIG. 1 is a schematic diagram of a chip system provided by the embodiments of the present application, which can include but is not limited to a system on chip (Soc) 101, an internal memory 102, and an external memory 103. The chip system can be integrated in any electronic device, such as a computer, a mobile phone, a tablet, a personal digital assistant, a smart wearable device, a smart vehicle, or a smart home appliance, and the like, and the chip system can work under necessary software driving.

[0062] The system on chip (Soc) 101 refers to a technology of integrating a complete system on a single chip and grouping all or part of necessary electronic circuits. The system on chip 101 can include a plurality of processing units and a controller 1014. The plurality of processing units can include one or more of a central processing unit (CPU), a modem processing unit, a graphics processing unit (GPU), an image signal processor (ISP), a video coding unit, a digital signal processor (DSP), a baseband processing unit, and a neural-network processing unit (NPU), etc. Optionally, the different processing units can communicate with each other through a primary bus, and each processing unit can also communicate with the controller 1014 through the primary bus.

[0063] The central processing unit 1011, which can also be referred to as a central processing unit (CPU) or a processor, can be a multi-core processor, i.e., a plurality of processor cores are integrated in one chip, or a single-core processor, which is not limited in the present application. The processor core is also referred to as a kernel, which is the most important component of the CPU. It is made of single-crystal silicon by a certain production process. All calculations, receiving instructions or storing instructions, and processing data of the CPU are performed by the processor core. The processor core can load instructions or data stored in the external memory 103 (such as a disk) into the internal memory 102 through the controller 1014, and transfer instructions or data that need to be operated from the internal memory 102 to the processor core through the controller 1014 for operation. When the operation is completed, the processor core can temporarily store the result in the internal memory 102, and store instructions or data that need to be stored for a long time in the external memory 103 through the controller 1014. The processor core can run an operating system, a file system (such as a flash file system F2FS) or an application program, etc. to control a plurality of hardware or software elements connected to the CPU, and can process various data and perform operations. Optionally, the central processing unit 1011 can include a memory, which can be a cache. The cache can include one or more of a level 1 cache (L1 Cache), a level 2 cache (L2 Cache), a level 3 cache (L3 Cache), etc. The cache can save instructions or data that have just been used or are frequently used by the central processing unit 1011. If the central processing unit 1011 needs to use the instructions or data again, it can be directly called from the cache. This avoids repeated access and reduces the waiting time of the processor core, thereby improving the efficiency of the central processing unit 1011.

[0064] The controller 1014 can be used to manage and control the communication between the central processing unit 1011 and the internal memory 102. The controller 1014 can also be used to manage and control the communication between the central processing unit 1011 and the external memory 103 (such as a disk), and to provide a standardized interface (such as a universal flash storage UFS standard) for communication between the central processing unit 1011 and the external storage device. It can also be understood that the central processing unit 1011 can send access instructions (such as read / write instructions) and control instructions to the internal memory 102 through the controller 1014, and the central processing unit 1011 can also send access instructions (such as read / write instructions) to the external memory 103 through the controller 1014.

[0065] The internal memory 102 (Memory) can be located outside the system on chip 101, and is usually a power-off volatile memory that loses the stored content when power off, and can also be referred to as a main memory. The internal memory 102 in the present application includes a readable and writable running memory, which is used to temporarily store the operation data required by the central processing unit 1011, and interacts with the external memory 103 or other external memories, and can be used as a temporary data storage medium for the operating system or other programs running.

[0066] The internal memory 102 can include one or more of a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM), etc. Among them, the DRAM includes a double data rate synchronous dynamic random access memory (DDR SDRAM) referred to as DDR, a second generation double data rate synchronous dynamic random access memory (DDR2), a third generation double data rate synchronous dynamic random access memory (DDR3), a fourth generation low power double data rate synchronous dynamic random access memory (Low Power Double Data Rate 4, LPDDR4), and a fifth generation low power double data rate synchronous dynamic random access memory (Low Power Double Data Rate 5, LPDDR5), etc.

[0067] The external memory 103 is a non-volatile memory, and the stored content will not be lost after power off. The external memory 103 can be used for long-term storage of instructions and data related to the operation of the central processing unit 1011, such as startup programs, operating systems, application programs, and data, etc. Since the central processing unit 1011 cannot directly read the instructions and data in the external memory 103, nor can it directly write instructions or data to the external memory 103, therefore, when executing a read (or load) instruction, the central processing unit 1011 actually temporarily loads the to-be-read content (including instructions and / or data) stored in the external memory 103 to the internal memory 102 through the controller 1014, and then reads it from the internal memory 102; and when executing a write (i.e. storage) instruction, the central processing unit 1011 actually temporarily writes the to-be-stored data (including instructions and / or data) to the internal memory 102, and then stores it from the internal memory 102 to the external memory 103 through the controller 1014.

[0068] The external memory 103 can include one or more of a Flash memory (e.g., a NAND Flash, a NOR Flash, etc.), a universal flash storage (UFS), an embedded Multi-Media Card (eMMC), a universal flash storage multi-chip package (uMCP) memory, an embedded Multi-Media Card multi-chip package (eMCP) memory, a solid state drive (SSD), etc.

[0069] It can be understood that the chip system in FIG. 1 is only some exemplary embodiments provided by the embodiments of the present application, and the chip system in the embodiments of the present application includes but is not limited to the above implementation.

[0070] With the development of artificial intelligence technology, generative AI is emerging, such as ChatGPT, etc. Deploying large models for AI inference on the terminal side to improve user experience and productivity has become one of the main competitiveness of the terminal. There is a large amount of data in the large model inference process that needs to be moved from the internal memory 102 to the central processing unit 1011, or the graphics processing unit 1012, or the neural network processing unit 1013 for calculation, which causes the current terminal side AI inference performance bottleneck to mainly concentrate on the memory bandwidth. At present, in order to improve the terminal side AI inference performance, processing in memory (PIM) has become the mainstream, that is, multiple independent computing units (Processing Units, PU) can be integrated in the internal memory 102, and part of the computing task is offloaded to the internal memory for calculation, which can improve the computing performance of the terminal and accelerate the large model inference. At the same time, using the processing in memory technology can reduce data movement, reduce terminal energy consumption, and improve terminal endurance. The structure of the internal memory 102 and the process of reading data from the internal memory 102 and executing a computing task in the internal memory 102 will be described in detail below in conjunction with FIGS. 2A-2E.

[0071] Referring to FIG. 2A, which is a structural diagram of a memory according to an embodiment of the present application, the memory 102 can include one or more ranks (Rank). Each Rank refers to a set of memory dies (Die), and memory dies in the same set share the same address and control signals. Each Rank is usually 64 bits wide, which means it can process 64 bits of data at the same time. The memory 102 can have one or more Ranks, such as a single Rank (1R) and a double Rank (2R). The double Rank can include two Ranks, and each Rank can be independently accessed, thereby improving the bandwidth and performance of the memory 102. For example, as shown in FIG. 2A, the Rank 0 can include a plurality of Dies, and the Rank 0 can be independently used to access data. The Rank 1 can include a plurality of PIM-Dies, and the PIM-Die is a Die with a computing unit (PU) integrated inside. The Rank 1 can be independently used to access data and perform computing tasks. It should be noted that the same structure of the Die can be included in the same Rank, or different structures of the Die can be included in the same Rank, which is not limited in the present application. The structure of the PIM-Die will be described in detail below in conjunction with FIG. 2B.

[0072] Please refer to FIG. 2B, which is a structural schematic diagram of a PIM-Die provided in an embodiment of the present application. Each PIM-Die can include a plurality of storage areas (Banks), such as Banks A-J shown in FIG. 2B. A Bank is a basic unit for storing and managing data. Each Bank can be regarded as a storage area and can process data requests independently. Different Banks allow parallel access and operation, which means that a memory controller can access multiple Banks simultaneously, thereby improving data read efficiency. For example, a PIM-Die includes 4 Banks, and data is stored in the 4 Banks. When data is accessed, the memory controller can activate data in the 4 Banks simultaneously. A plurality of computing units (PUs) are also integrated in each PIM-Die, such as PUs 1-5 shown in FIG. 2B. A PU is used to perform a computing task to achieve direct computing in the memory 102, instead of transferring data to an external system-on-chip 101 for processing, thereby improving computing efficiency. Each PU can correspond to one or more Banks. For example, PU 1 shown in FIG. 2B can correspond to Banks A and F, that is, data read from Banks A and F can be directly transferred to PU 1 for computing. A plurality of mode registers (MRs) can also be integrated in each PIM-Die, such as mode register 1 and mode register 2 shown in FIG. 2B. An MR can be used to configure the operation mode and functions of the memory 102, for example, to set parameters such as data width, delay, refresh rate, and the like. The presence of an MR enables the memory controller to configure and optimize the memory 102 according to the needs of the system, thereby improving performance and compatibility. By programming these registers, the system can adjust the behavior of the memory 102 to adapt to different application scenarios and performance requirements. A global buffer can also be included in each PIM-Die, which can be used as a cache to store frequently accessed data to reduce access delay and improve system response speed. The structure of a Bank will be described in detail below in conjunction with FIG. 2C.

[0073] Please refer to FIG.2C, which is a structural diagram of a Bank according to an embodiment of the present application. The Bank can include a storage array and a row buffer. The storage array is responsible for actual data storage and is composed of a plurality of storage units, which are usually organized in the form of rows and columns, i.e., the storage array can be composed of M rows * N columns of storage units, allowing efficient data access. The row buffer is used to store the row data of a certain row read from the storage array. Since data is stored in the storage array of the Bank by row and column, and in order to improve data reading efficiency, when reading data from the memory 102, a row of data in the Bank needs to be activated first, which can also be referred to as row activation. The purpose of row activation is to load the data of a specific row in the Bank to the temporary storage area of the row buffer, so that the data in the row can be quickly accessed. Once the row is activated, the row data will be loaded into the row buffer of the corresponding Bank, and then the CPU can send a compute instruction (COMPUTE) to the memory 102, which includes a column address for specifying the specific column data of the row data stored in the row buffer, and the column data is transferred to the PU corresponding to the Bank for calculation.

[0074] With the continuous increase in the size of large models, the demand for computing and storage resources of large models also increases. In order to enable these large models to run on devices with limited resources, researchers are exploring various algorithm optimization techniques, among which the grouping quantization technique is an important direction. By converting the high-precision format (such as 32-bit floating-point number FP32 or 16-bit floating-point number FP16) used in large models into a low-precision format (such as 8-bit integer INT8 or 4-bit integer INT4), the storage size of the model can be effectively reduced, which helps to reduce the demand of large models for hardware resources, so that large models can run on devices with limited computing power, while also speeding up inference and reducing power consumption.

[0075] It should be noted that the weight matrix before quantization can be represented by W, and the weight matrix after quantization can be represented by Q. In the linear grouping quantization process, a certain continuous amount of high-bit values is converted into low-bit values, while a set of quantization factors S and zero points Z can be obtained, which can achieve a larger model parameter compression ratio with less precision loss. By combining the quantized value Q, the quantization factor S, and the zero point Z, the value before quantization can be approximately obtained, W≈W’=S*(Q-Z). When Z is 0, W≈W’=S*Q, which is called dequantization. The grouping quantization can be performed according to the matrix (per tensor), or according to a row or a column of the matrix (per tokn), or according to 32 / 64 / 128 continuous values.

[0076] Assuming that the dimension of the weight matrix W is 16 rows * 64 columns, the 32 consecutive values can be quantized and grouped as a group, that is, the first 32 values of each row are grouped as a group, and the last 32 values are grouped as another group. The current weight matrix W can be divided into 32 groups, and each group can correspond to a quantization factor S.

[0077] Referring to FIG. 2D, FIG. 2D is a schematic diagram of calculation based on the quantized weight matrix according to an embodiment of the present application. When W*V needs to be calculated, the rows of the matrix W and the columns of the vector V are multiplied, and the calculation result is

[0078] Referring to FIG. 2E, FIG. 2E is a schematic diagram of calculation based on the quantized weight matrix according to an embodiment of the present application. When W*V needs to be calculated, the stored weight matrix Q is read out from the memory 102, and then the rows of the weight matrix Q and the columns of the vector V are multiplied and dequantized. The calculation result is The calculation result is approximately equal to the result of calculation based on W, so as to maintain the accuracy of the model.

[0079] However, due to the limitation of the memory resource, the calculation units PU in the memory 102 are limited by the complexity of the formula when performing calculation, which leads to the fact that the complex grouping dequantization algorithm cannot be effectively executed on these calculation units. Therefore, in the present application, part or all of the calculation units in the memory 102 are improved to improve the calculation ability of the calculation units, so that the calculation units can efficiently implement the grouping dequantization algorithm when performing calculation, so as to maintain the accuracy of the model and improve the memory calculation performance. Details of how to improve the calculation units will be described later, and will not be described here.

[0080] Next, the memory involved in the present application will be introduced in combination with the drawings. As shown in FIG. 3, FIG. 3 is a structural schematic diagram of a memory according to an embodiment of the present application. The memory 102 can include a plurality of storage areas Bank, and the plurality of storage areas Bank include a target Bank 201. The target Bank 201 can store data, which can be low-precision data after quantization. The memory 102 can also include a plurality of calculation units, and the plurality of calculation units include a target calculation unit 202. The target Bank 201 corresponds to the target calculation unit 202, that is, the data taken out of the target Bank 201 can be sent to the target calculation unit 202 for calculation. It should be noted that one Bank can only correspond to one calculation unit, but one calculation unit can correspond to one or more Banks. In FIG. 3, only the case where one Bank corresponds to one calculation unit is shown.

[0081] Please refer to Fig. 4 as well, which is a structural schematic diagram of a target calculation unit provided in an embodiment of the present application. The target calculation unit 202 includes multiple groups of first multipliers, multiple addition trees and multiple second multipliers. One group of first multipliers is connected with one addition tree and one second multiplier.

[0082] Specifically, one group of first multipliers in the multiple groups of first multipliers includes multiple first multipliers. Every two first multipliers in the multiple first multipliers can constitute a pair of first multipliers. Therefore, one group of first multipliers includes multiple pairs of first multipliers. Each first multiplier can receive at least two input data and perform multiplication operation on the input data to output a calculation result. In addition, each addition tree includes multiple first adders as leaf nodes, multiple second adders as intermediate nodes and a third adder as a root node. The first adder, the second adder and the third adder can receive at least two input data and perform addition operation on the input data to output a calculation result. The first adder is connected with a first multiplier, the second adder is connected with a second adder of a previous stage or a first adder, and the third adder is connected with a second multiplier.

[0083] It should be noted that the structure of the multiple calculation units included in the internal memory 102 can be the same as or different from the structure of the target calculation unit 202, which is not specifically limited in the present application.

[0084] Each pair of first multipliers is configured to receive data stored in the target Bank 201 to perform multiplication operation and input the output results into a connected first adder to perform addition operation, respectively.

[0085] Specifically, one pair of first multipliers can include two first multipliers. Each pair of first multipliers can be connected with a first adder in an addition tree. Each pair of first multipliers can receive data stored in the target Bank 201 to perform multiplication operation and input the output results into a connected first adder to perform addition operation, respectively. That is, the output results of one pair of first multipliers can be added by a first adder.

[0086] Each second adder is configured to receive output results of multiple connected first adders or second adders of a previous stage to perform addition operation and input the output results into a connected second adder of a next stage or a third adder, respectively.

[0087] Specifically, each second adder can be connected with a first adder and a second adder at a previous stage, that is, the input of each second adder can be a first adder or a second adder at a previous stage. Each second adder receives the output results of multiple connected first adders or second adders at a previous stage for addition operation, and respectively inputs the output results to a second adder at a next stage or a third adder.

[0088] Each third adder is configured to receive the output results of multiple connected second adders for addition operation, and input the output results to a two-stage multiplier connected thereto for multiplication operation.

[0089] Specifically, the third adder is connected with the two-stage multiplier, each third adder can receive the output results of multiple connected second adders for addition operation, and input the output results to a two-stage multiplier connected thereto for multiplication operation, so as to realize multiplication and accumulation of data and then two-stage multiplication operation.

[0090] In the present application, the calculation circuit of the target calculation unit 202 in the internal memory 102 is improved, that is, the first multipliers are grouped, and each group of first multipliers is connected with an adder tree and a two-stage multiplier, so that each group of data after grouping can perform multi-stage multiplication and accumulation operation through the circuit. Specifically, a group of first multipliers includes multiple pairs of first multipliers, each adder tree includes multiple first adders as leaf nodes, multiple second adders as intermediate nodes, and a third adder as a root node. The internal memory 102 can simultaneously take out multiple groups of to-be-calculated data from the target Bank 201, and can input a group of to-be-calculated data to a group of first multipliers for multiplication operation, that is, each pair of first multipliers in a group of first multipliers can respectively receive different parameters in the same group of to-be-calculated data for multiplication operation, and respectively input the output results to a connected first adder for addition operation; each second adder receives the output results of multiple connected first adders or second adders at a previous stage for addition operation, and respectively inputs the output results to a second adder at a next stage or a third adder; each third adder receives the output results of multiple connected second adders for addition operation, and inputs the output results to a two-stage multiplier connected thereto for multiplication operation, avoiding the problem that the internal memory 102 can only perform multiplication and accumulation operation on data, and cannot perform multi-stage multiplication and accumulation operation on grouped data, thereby improving the calculation capability of the internal memory 102.

[0091] In some embodiments, referring to FIG. 5, which is a schematic diagram of a target calculation unit circuit structure according to an embodiment of the present application, each pair of first multipliers is connected to a first adder in an adder tree; the first adders, the second adders and the third adder included in the adder tree are connected in a binary tree structure; and the third adder in the adder tree is connected to a second-level multiplier.

[0092] Specifically, the output end of each pair of first multipliers can be connected to the input end of a first adder in the adder tree, so that the output result of a pair of first multipliers can be input to a first adder for addition operation. The first adders, the second adders and the third adder included in the adder tree are connected in a binary tree structure, so that multiple adders can perform addition operation in parallel at the same time, and in addition, by connecting the adders in layers, the transmission delay of data between different adders can be reduced, thereby improving the addition operation speed. The third adder in the adder tree is connected to a second-level multiplier, so that the result output by the adder tree is subjected to second-level multiplication operation by the second-level multiplier, thereby avoiding the problem that the memory 102 can only perform multiply-accumulate operation on data and cannot perform respective multi-level multiply-accumulate operation on grouped data, thereby improving the calculation capability of the memory 102.

[0093] In some embodiments, referring to FIG. 6, which is a schematic diagram of the first operand and the second operand of a first multiplier according to an embodiment of the present application, the memory 102 further includes a first buffer, and each pair of first multipliers is specifically configured to: respectively receive data stored in the target Bank 201 as respective first operands, and respectively receive data stored in the first buffer as respective second operands; and respectively perform multiplication operation based on the respective first operand and the respective second operand, and respectively input the output result to a connected first adder for addition operation.

[0094] Specifically, the first buffer can be a global buffer (Global Buffer) accessible by all Banks in the memory 102, or a buffer inside the PU. The first buffer can be used to store vector data as the second operand of the first multiplier. The target Bank can be used to store weight data as the first operand of the first multiplier. During calculation, each first multiplier can receive data stored in the target Bank 201 as the first operand, and receive data stored in the first buffer as the second operand, and perform multiplication operation based on the first operand and the second operand. Further, the first multiplier can input the output result to a connected first adder for addition operation, thereby realizing multiply-accumulate operation.

[0095] In some embodiments, referring to FIG. 7, which is a circuit connection diagram of a fourth adder according to an embodiment of the present application, the plurality of second multipliers are connected to one or more fourth adders, one second multiplier is connected to one fourth adder, and one fourth adder is connected to at least two second multipliers. The one or more fourth adders are configured to receive the output results of the plurality of second multipliers and perform addition operation, and output the calculation results.

[0096] Specifically, the output ends of the plurality of second multipliers are connected to one or more fourth adders, one second multiplier is connected to one fourth adder, and one fourth adder is connected to at least two second multipliers. The fourth adder can receive the calculation results of the second multipliers and perform addition operation. If a plurality of groups of first multipliers in the target calculation unit 202 receive a plurality of groups of data to perform multiplication operation, the calculation results are input into the corresponding addition trees, and the addition operation is performed through the corresponding addition trees. The output results of the plurality of addition trees are input into the second multipliers to perform multiplication operation. The output results of the second multipliers can be accumulated through the fourth adder, and the calculation results are output to enable the target calculation unit of the internal memory 102 to perform multi-level multiplication and accumulation operation, thereby improving the calculation capability of the internal memory 102.

[0097] Optionally, the target calculation unit 202 further includes a second register (REG) and a fifth adder. One input end of the fifth adder is connected to the output end of the fourth adder, and the other input end of the fifth adder is connected to the second register.

[0098] Optionally, the output end of the fourth adder can be connected to the second register.

[0099] Optionally, when the first-level multiplier, the second-level multiplier, and the fourth adder are used to perform multi-round calculation in one calculation task, in the first round of calculation, the fourth adder can input the calculation results into the second register for storage. When the next round of calculation is performed, the fourth adder can input the calculation results into the fifth adder, and input the calculation results of the previous round stored in the second register into the fifth adder for accumulation. The results can be stored in the second register again. After the last round of calculation is completed, the target calculation result can be determined from the second register.

[0100] In some embodiments, referring to FIG. 8, FIG. 8 is a schematic diagram of loading data from a target bank to a first multiplier according to an embodiment of the present application. The target bank 201 is configured to send a first sequence to a target computing unit 202, the first sequence is composed of L first sub-sequences, and each first sub-sequence includes G first operands, L is an integer greater than 0; the target computing unit 202 is configured to receive a first sub-sequence through G first multipliers, the G first multipliers are first multipliers in one or more groups of first multipliers, and one first operand is input into one first multiplier.

[0101] Specifically, the first sequence can be quantized low-precision data. The target bank 201 can be composed of a storage array and a row buffer, and the storage array includes a plurality of rows and a plurality of columns of storage units. When reading data from the target bank 201, a row of data needs to be loaded into the row buffer first, and then a column (col.) of data, such as Ci0 column of data, i.e., data of one storage unit (i.e., the first sequence) is input into the target computing unit 202 for calculation. Generally, the data taken from one storage unit is 256 bits, i.e., the length of the first sequence can be 256 bits. The first sequence can be composed of L first sub-sequences, and each first sub-sequence includes G first operands. When the first sequence is quantized low-precision data, the length of each first operand in each first sub-sequence can be the quantization precision corresponding to the operand, such as 4 bits. The target computing unit 202 can receive a first sub-sequence through G first multipliers, and input one first operand into one first multiplier for calculation and dequantization. The G first multipliers can be first multipliers in one group of first multipliers, or first multipliers in multiple groups of first multipliers, so as to ensure that the target computing unit 202 can perform multi-level multiply-accumulate operations on the first sub-sequences, realize calculation on the first sub-sequences and dequantization, thereby improving the computing capability of the memory 102.

[0102] In some embodiments, referring to FIG. 9, FIG. 9 is a schematic diagram of multiple groups of first multipliers according to an embodiment of the present application. When one group of first multipliers includes M first multipliers, and G is less than or equal to M, the G first multipliers are first multipliers in one group of first multipliers; when G is greater than M, the G first multipliers are first multipliers in S groups of first multipliers, S is an integer greater than 1.

[0103] Specifically, when a group of first multipliers includes M first multipliers, and each first sub-sequence includes G first operands, since one first operand is input into one first multiplier for operation, if G is less than or equal to M, it means that the first multipliers in the group of first multipliers can process G first operands in parallel, and thus the first sub-sequence can be input into the group of first multipliers for calculation. If G is greater than M, it means that the group of first multipliers cannot completely process G first operands in parallel, and thus the first sub-sequence can be input into multiple groups of first multipliers for processing G first operands in parallel, so as to ensure that the target computing unit 202 can perform multi-level multiplication and accumulation operation on the first sub-sequence, thereby improving the computing capability of the memory 102.

[0104] In some embodiments, referring to FIG. 10, which is a schematic diagram of a first register provided in an embodiment of the present application, the target computing unit 202 further includes a first register, which stores a first parameter corresponding to each first sub-sequence. Each second multiplier is configured to: receive an output result of a connected third adder as a third operand, and receive a first parameter corresponding to the first sub-sequence sent by the first register as a fourth operand; perform multiplication operation based on the third operand and the fourth operand, and input an output result into a connected fourth adder for addition operation.

[0105] Specifically, the first register can also be referred to as a scale register. When the G first operands included in the first sub-sequence are data quantized based on a quantization factor, the first parameter can be a scale factor corresponding to the first sub-sequence, and the quantization factor and the scale factor can be inverses. After the first sub-sequence is input into the first multiplier and the adder tree for multiplication and accumulation operation, the output result can be taken as the third operand of the second multiplier. Then, the corresponding first parameter, i.e., the scale factor, is taken from the first register as the fourth operand. Further, the second multiplier performs multiplication operation based on the third operand and the fourth operand, and inputs a result of the second multiplier into a connected fourth adder for addition operation, so as to ensure that the target computing unit 202 can perform multi-level multiplication and accumulation operation on the first sub-sequence, implement calculation on the first sub-sequence and dequantization, thereby improving the computing capability of the memory 102.

[0106] It should be noted that when one first sub-sequence is input into a group of first multipliers for calculation, each first sub-sequence corresponds to a first parameter, i.e., a scale factor, and the fourth operand of the second multiplier corresponding to each first sub-sequence is different, such as S0 for the fourth operand of the second multiplier corresponding to the first group of first multipliers, and S1 for the fourth operand of the second multiplier corresponding to the second group of first multipliers.

[0107] In some embodiments, referring to FIG. 11, which is a schematic diagram of another first register provided by the embodiments of the present application, when G is greater than M, the S groups of first multipliers correspond to S groups of second multipliers, and the S groups of second multipliers receive the same first parameter as the fourth operand.

[0108] Specifically, when G is greater than M, the G first operands in a first subsequence cannot be completely processed in parallel by one group of first multipliers, and thus the first subsequence can be input into multiple groups of first multipliers to process the G first operands in parallel. Further, the first subsequence corresponds to one first parameter, i.e., a dequantization factor, and the fourth operands of the second multipliers corresponding to the multiple groups of first multipliers are the same, so as to ensure that the target computing unit 202 can perform multi-level multiply-accumulate operations on the first subsequence, thereby improving the computing capability of the memory 102. Assuming that the first subsequence is input into the first group of first multipliers and the second group of first multipliers to perform multiply-accumulate operations, the fourth operand of the second multiplier corresponding to the first group of first multipliers is the same as the fourth operand of the second multiplier corresponding to the second group of first multipliers, such as S0.

[0109] In some embodiments, the memory 102 further includes a first module, which is configured to receive a first instruction sent by a processor, control the target Bank 201 to send a first sequence to the target computing unit 202 based on the first instruction, and control the first register to send a first parameter corresponding to a first subsequence to the second multiplier.

[0110] Specifically, the first module can be an instruction receiving module or a control module in the memory 102, and can be configured to receive an instruction sent by a processor. The first instruction can be a computing instruction, which indicates to take data from the target Bank 201 and send a first sequence to be computed to the target computing unit 202, so as to perform multi-level multiply-accumulate computation on the first sequence (including multiple first subsequences) by the multiple groups of first multipliers, the multiple addition trees, and the multiple second multipliers of the target computing unit 202, such as computing the first sequence and performing a grouping dequantization operation. In addition, the first module is further configured to control the first register to send a first parameter corresponding to a first subsequence to the second multiplier, so as to perform multiply-accumulate computation on the computing result of each first subsequence by the multiple second multipliers, so as to realize grouping dequantization on the grouped first subsequence.

[0111] In some embodiments, the target Bank 201 is composed of multiple rows and multiple columns of storage units. As shown in FIG. 12, which is a schematic diagram of a Bank for storing each first subsequence and a corresponding first parameter provided by the embodiments of the present application, assuming that the target Bank 201 is Bank A, the first parameter corresponding to each first subsequence and the first sequence are stored in the same row of storage units of the target Bank 201.

[0112] Specifically, since one Bank can include multiple columns (cols), such as 64 cols, one placement strategy is to ensure that the scale factor and its corresponding quantization weight (i.e., the first subsequence) are in one row. One col places multiple scale factors, and then 15 cols place the weight matrix corresponding to the scale factor. In this way, each 16 cols form a group, so that the weight data and its corresponding scale factor are in one row of the Bank, reducing unnecessary cross-row operations.

[0113] In some embodiments, the first parameter corresponding to each first subsequence and the first sequence are stored in different rows of the target Bank 201. Specifically, since one Bank can include multiple cols, such as 64 cols, one placement strategy is to place the weight data in the same row and the scale factor in an independent row. For example, the scale factor corresponding to Row i Col 0-15 is placed in Row j Col 3.

[0114] In some embodiments, please refer to FIG. 13, which is a schematic diagram of another Bank for storing each first subsequence and the corresponding first parameter according to an embodiment of the present application. The first parameter corresponding to each first subsequence and the first sequence are stored in different Banks.

[0115] Specifically, the first parameter (such as the Scale value) is stored in one Bank (such as Bank B), and each first subsequence (such as the quantization weight) is stored in the remaining Banks (such as Bank A). When calculating, a certain amount of Scale (COL integer multiple) is first read from Bank B to the first register of the global buffer. When doing multiply-accumulate, the data is read from Bank A to the corresponding target computing unit 202, the input vector is read from the Global Buffer to the target computing unit 202, and then the Scale value is read from the first register to the target computing unit 202. The scale factor and the quantization weight are stored in different Banks, which can be concentrated to read the scale factor from the storage array to the global buffer, ensuring the flow of the model.

[0116] It should be noted that the first buffer and the first register can be located inside the global buffer as shown in FIG. 13, or can be located in the storage space in the computing unit. The specific setting method needs to be selected according to the actual product design needs, which will not be described here.

[0117] In some embodiments, the memory further includes a global buffer, and the first module is further configured to receive the second instruction sent by the processor, read the first parameter corresponding to each first sub-sequence from the target Bank and store the first parameter into the first register, or read the first parameter corresponding to each first sub-sequence from other Banks and store the first parameter into the global buffer, and store the first parameter corresponding to each first sub-sequence from the global buffer into the first register.

[0118] Specifically, the second instruction can be an instruction for reading the first parameter from the Bank and storing the first parameter into the corresponding first register. After receiving the second instruction sent by the processor, the first module can read the first parameter corresponding to each first sub-sequence from the target Bank and store the first parameter into the first register, so as to perform the inverse quantization operation subsequently. Alternatively, the first module can read the first parameter corresponding to each first sub-sequence from other Banks and store the first parameter into the global buffer, and then store the first parameter corresponding to each first sub-sequence from the global buffer into the first register, so as to perform the inverse quantization operation subsequently.

[0119] Suppose that the target computing unit 202 includes N groups of first multipliers, N addition trees, and N second multipliers; each group of first multipliers includes M first multipliers, each addition tree includes M / 2 first adders as leaf nodes, (M / 2-2) second adders as intermediate nodes, and one third adder as a root node, and M and N are integers greater than 0. The process of the target computing unit 202 for computing and inverse quantizing the weight data (Q) and the vector data (V) can include a quantization information configuration process, an input vector transmission process, and a computing process.

[0120] Referring to FIG. 14, which is a flowchart of a quantization information configuration process according to an embodiment of the present application, first, an XPU (such as a CPU) side prepares an inverse quantization factor configuration instruction related to an inverse quantization factor. The inverse quantization factor configuration instruction includes multiple parameters for informing the memory 102 how to correctly use the read Scale data to complete the computation when performing the computation. When sending the inverse quantization factor configuration related information to the memory 102, the memory 102 receives the inverse quantization factor configuration related information and writes the information into a related register, thereby completing the configuration. Then, the XPU can send a computation enable Scale instruction for informing the memory 102 to correctly read the Scale value and participate in the computation when performing the computation.

[0121] Please refer to FIG. 15, which is a flowchart of a preparation process of a dequantization factor configuration instruction according to an embodiment of the present application. The processor can receive input parameters, which can include model parameters and hardware information. The model parameters can include a quantization group size G, a number of bits s occupied by a Scale value, a number of Scale values contained in a storage unit (i.e., a col) scaleNum, etc. Generally, scaleNum = A / s, where A is the amount of data stored in a storage unit, for example, A = 256 bits.

[0122] The processor can calculate output parameters based on the input parameters. The parameter para1 is used to indicate how many Scale values are read from the first register (i.e., the Scale register) at a time during the calculation. Generally, para1 = M*N / G. The parameter para2 is used to indicate the number of times each Scale value is used after being read out. Generally, para2 = G / M. Finally, the processor can send the parameters para1, para2, s, and scaleNum, etc. to the memory 102.

[0123] Suppose the size of the weight data in the calculation task is a 16-row*64-column matrix data, the size of each quantization group is 32, i.e., G = 32, so one row of data can be divided into two quantization groups, each of which includes 32 weight parameters. The weight data is divided into 32 groups of data for calculation and dequantization. The number of bits s occupied by a Scale value is 8, A = 256 bits, and the number of Scale values contained in a storage unit (i.e., a col) scaleNum is 32. The number of groups of first multipliers N is 2, and the number of first multipliers included in each group of first multipliers M is 32. In this example, para1 is 2, i.e., two Scale values need to be read from the first register at a time for calculation, and para2 is 1, i.e., each Scale value is used once.

[0124] Suppose the size of the weight data in the calculation task is a matrix data of 16 rows * 64 columns, the size of each quantization group is 64, that is, G = 64, so one row of data can be divided into one quantization group, and 64 weight parameters are included in each group. The weight data is divided into 16 groups of data for calculation and dequantization. The number of bits s occupied by one Scale value is 16, A = 256 bits, and the number of Scale values included in one storage unit (that is, one col) is scaleNum = 16. The number of groups N of the first multiplier is 2; the number of the first multiplier included in each group of the first multiplier is M = 32. In this example, para1 is 1, that is, one Scale value needs to be taken out from the first register to participate in the calculation each time; para2 is 2, that is, each Scale value is used twice.

[0125] Referring to FIG. 16, FIG. 16 is a calculation flow diagram provided by an embodiment of the present application, and the details are as follows:

[0126] S301: reading Scale data from a storage array to a first register.

[0127] Specifically, the Scale data includes one or more Scale values, which can be Scale values required in a calculation task. When performing the calculation task, all Scale values required for calculation can be read from the storage array at a time, or the Scale values required for calculation can be read from the storage array in multiple times, which is not limited in the present application.

[0128] S302: loading an input vector (that is, a second operand) into the first multiplier and loading a weight parameter (that is, a first operand) into the first multiplier to obtain a first calculation result.

[0129] Specifically, when K groups of quantization group data (the size of each quantization group is G) need to be calculated and dequantized in the calculation task, the target calculation unit only includes N groups of first multipliers (one group includes M first multipliers), and G is less than or equal to M, then one group of quantization group data can be input into one group of first multipliers for calculation, and the target calculation unit can simultaneously calculate N groups of quantization group data. When K is greater than N, the target calculation unit reads other quantization group data to continue the calculation after calculating N groups of quantization group data, until K groups of dequantization group data are calculated.

[0130] S303: taking out para1 Scale values from the first register, and inputting each Scale value into para2 corresponding second multipliers.

[0131] Specifically, for example, para1 is 2 and para2 is 1, 2 Scale values are read from the first register each time, and each Scale value is input into a corresponding second multiplier for calculation.

[0132] S304: Each second multiplier performs multiplication operation on the received first calculation result and the Scale value, and inputs the calculation result into a fourth adder for addition operation.

[0133] S305: The calculation result is output through the fourth adder.

[0134] Optionally, in the first round of calculation, the calculation result output by the fourth adder can be stored in a second register; in the next round of calculation, the calculation result of the fourth adder and the calculation result of the last round of calculation can be input into a fifth adder for addition operation, until the target calculation result is calculated by the fifth adder, and the target calculation result can be stored in the second register and output through the second register.

[0135] In some embodiments, it can be determined whether there is a Scale value in the first register that has not been used.

[0136] Specifically, the determination of whether there is a Scale value in the first register that has not been used can be made by counting, that is, the number of Scale values stored in the first register can be recorded first, that is, scaleNum, and then the number of Scale values stored in the first register is updated each time para1 Scale values are read, that is, scaleNum-para1.

[0137] In some embodiments, if there is a Scale value in the first register that has not been used, S302 is continued; if there is no Scale value in the first register that has not been used, it is further determined whether all the Scale data stored in the storage array have been read out, if not, S301 is continued; if there is no Scale value in the first register that has not been used and all the Scale data stored in the storage array have been read out, the target calculation result can be output through the second register.

[0138] The embodiment of the present application provides a control method, which is applied to an internal memory, the internal memory comprises a plurality of storage areas (Banks) and a plurality of calculation units, one Bank corresponds to one calculation unit, the plurality of Banks comprise a target Bank, and the target Bank corresponds to a target calculation unit; the target calculation unit comprises a plurality of groups of first multipliers, a plurality of addition trees, a plurality of second multipliers, at least one fourth adder and a first register; wherein one group of first multipliers is connected with one second multiplier through one addition tree; one second multiplier is further connected with the first register; one fourth adder is connected with at least two second multipliers; one group of first multipliers comprises a plurality of pairs of first multipliers; each addition tree comprises a plurality of first adders as leaf nodes, a plurality of second adders as intermediate nodes and a third adder as a root node; data stored in the target Bank is input into each pair of first multipliers to perform multiplication operation, and output results of each pair of first multipliers are respectively input into one connected first adder to perform addition operation; each second adder is used for receiving output results of a plurality of connected first adders or a second adder at a previous stage to perform addition operation, and the output results are respectively input into a second adder at a next stage or the third adder connected; each third adder is used for receiving output results of a plurality of connected second adders to perform addition operation; parameters in the first register are input into each second multiplier and output results of the third adder connected to perform multiplication operation; output results of the connected second multipliers are received by the fourth adder to perform addition operation, and a calculation result is output.

[0139] In the embodiment of the present application, the calculation circuit of the target calculation unit in the memory is improved, that is, the first multipliers are grouped, and each group of first multipliers is connected to a second multiplier through an addition tree, so that each group of data after grouping can be subjected to multi-stage multiplication and accumulation operation through the circuit. Specifically, a group of first multipliers includes a plurality of pairs of first multipliers, each addition tree includes a plurality of first adders as leaf nodes, a plurality of second adders as intermediate nodes, and a third adder as a root node. The memory can simultaneously take out a plurality of groups of to-be-calculated data from the target Bank, and can input a group of to-be-calculated data into a group of first multipliers for multiplication operation, that is, each pair of first multipliers in a group of first multipliers can respectively receive different parameters in the same group of to-be-calculated data for multiplication operation, and respectively input the output results into a connected first adder for addition operation; each second adder receives the output results of a plurality of connected first adders or the output results of the second adder of the previous stage for addition operation, and respectively inputs the output results into the second adder of the next stage or the third adder connected; each third adder receives the output results of a plurality of connected second adders for addition operation, and each second multiplier receives the output results of the connected third adder and the parameters in the first register for multiplication operation. In the prior art, the memory can only perform one-stage multiplication and accumulation operation on the data, and cannot perform two-stage multiplication and accumulation operation on the data. In the present application, a plurality of second multipliers are added, each second multiplier receives the output results of the connected third adder and the parameters in the first register for multiplication operation, and the operation results of the second multiplier are subjected to addition operation through a fourth adder, thereby realizing two-stage multiplication and accumulation operation on the data and improving the calculation capability of the memory.

[0140] In some embodiments, the memory further includes a first buffer, and inputting the data stored in the target Bank into each pair of first multipliers for multiplication operation includes: inputting the data stored in the target Bank into each pair of first multipliers as first operands, respectively, and inputting the data stored in the first buffer into each pair of first multipliers as second operands, respectively; performing multiplication operation based on the respective first and second operands through each pair of first multipliers, respectively, and inputting the output results into a connected first adder for addition operation, respectively.

[0141] In the embodiments of the present application, the target Bank can be used to store weight data as the first operand of the first multiplier; the first buffer can be used to store vector data as the second operand of the first multiplier. In the calculation, each first multiplier can receive the data stored in the target Bank as the first operand and receive the data stored in the first buffer as the second operand, and perform multiplication operation based on the first operand and the second operand. Further, the first multiplier can input the output result into the connected first adder to perform addition operation, so as to realize multiply-accumulate operation.

[0142] In some embodiments, the target Bank stores a first sequence, the first sequence is composed of L first sub-sequences, and each first sub-sequence includes G first operands, L is an integer greater than 0; a group of first multipliers includes M first multipliers; when M is greater than or equal to G, the data stored in the target Bank is input into each pair of first multipliers as the first operand, including: inputting the G first operands in each first sub-sequence into the G first multipliers of a group of first multipliers to perform multiplication operation.

[0143] In the embodiments of the present application, when a group of first multipliers includes M first multipliers, and each first sub-sequence includes G first operands, since one first operand is input into one first multiplier to perform operation, if G is less than or equal to M, it means that the first multipliers in a group of first multipliers can process G first operands in parallel, so the first sub-sequence can be input into a group of first multipliers to perform calculation.

[0144] In some embodiments, the target computing unit further includes a first register, the first register stores a first parameter corresponding to each first sub-sequence, the parameter in the first register is input into each second multiplier and the output result of the connected third adder to perform multiplication operation, including: receiving the output result of the connected third adder as the third operand, and inputting the first parameter corresponding to the first sub-sequence in the first register into the second multiplier as the fourth operand; performing multiplication operation based on the third operand and the fourth operand by the second multiplier.

[0145] In the embodiment of the present application, when the first sub-sequence is input into the one-stage multiplier and addition tree for multiply-accumulate operation, the output result can be taken as the third operand of the two-stage multiplier. Then, the corresponding first parameter, i.e., the inverse quantization factor, is taken from the first register as the fourth operand. Further, the two-stage multiplier performs multiplication operation based on the third operand and the fourth operand, and the result of the two-stage multiplier is input into the connected fourth adder for addition operation, so as to ensure that the target computing unit can perform multi-stage multiply-accumulate operation on the first sub-sequence, realize computation on the first sub-sequence and inverse quantization, and thus improve the computing capability of the memory.

[0146] The embodiment of the present application also provides a chip, wherein the chip comprises the circuit and the memory as mentioned in the embodiment shown in FIG. 3-FIG. 14 or the memory in the above embodiment, and the chip can be composed of a single chip or a plurality of chips.

[0147] The embodiment of the present application provides an electronic device comprising a circuit board and the memory as mentioned in the embodiment shown in FIG. 3-FIG. 13 or the memory in the above embodiment, and the circuit board is electrically connected with the memory.

[0148] In the above embodiments, the description of each embodiment has its own focus, and the part not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0149] It should be noted that, for each method embodiment described above, in order to simply describe, each method embodiment is described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, some steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0150] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented by other manners. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical or other forms.

[0151] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0152] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0153] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, and specifically can be a processor in the computer device) to execute all or part of the steps of the above-mentioned method of each embodiment of the present application. Among them, the storage medium mentioned above can include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (Read-Only Memory, abbreviated as: ROM) or random access memory (Random Access Memory, abbreviated as: RAM) and various program code storage media.

[0154] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An in-memory storage, characterized by, The internal memory comprises a plurality of memory banks, a plurality of computing units, a first buffer and a first register, one bank corresponding to one computing unit, the plurality of banks comprising a target bank, the target bank corresponding to a target computing unit; The target computing unit comprises a plurality of groups of first multipliers, a plurality of addition trees, a plurality of second multipliers and at least one fourth adder; wherein one group of first multipliers is connected with one addition tree and one second multiplier; one fourth adder is connected with at least two second multipliers; one group of first multipliers comprises a plurality of pairs of first multipliers; one group of first multipliers corresponds to one second multiplier; Each addition tree comprises a plurality of first adders as leaf nodes, a plurality of second adders as intermediate nodes and a third adder as a root node; Each first multiplier is configured to receive data stored in the target bank and data of the first buffer respectively, perform multiplication operation, and input an output result into one connected first adder to perform addition operation; Each second adder is configured to receive an output result of a connected first adder or a second adder at a previous level to perform addition operation, and input an output result into a second adder at a next level or the third adder respectively; Each third adder is configured to receive an output result of a connected second adder to perform addition operation; Each second multiplier is configured to receive an output result of a connected third adder, perform multiplication operation on the output result of the connected third adder and a parameter in the first register, and output a result. The fourth adder is configured to receive an output result of a connected second multiplier to perform addition operation, and output a result.

2. The memory of claim 1, wherein, Each pair of first multipliers is connected with one first adder in the addition tree; the plurality of first adders, the plurality of second adders and the third adder in the addition tree are connected in a binary tree structure; the third adder in the addition tree is connected with one second multiplier.

3. The internal memory according to claim 1 or 2, characterized in that, Each pair of first multipliers is specifically configured to: receive data stored in the target bank as a respective first operand, and receive data stored in the first buffer as a respective second operand; perform multiplication operation based on the respective first operand and the respective second operand respectively, and input an output result into one connected first adder to perform addition operation.

4. The memory of claim 3, wherein, The target bank stores a first sequence, the first sequence is composed of L first sub-sequences, and each first sub-sequence comprises G first operands, L being an integer greater than 0; The target computing unit is configured to receive one first sub-sequence through G first multipliers, the G first multipliers being first multipliers in one or more groups of first multipliers, wherein one first operand is input into one first multiplier.

5. The memory of claim 4, wherein, When one group of first multipliers comprises M first multipliers, and G is less than or equal to M, the G first multipliers are first multipliers in one group of first multipliers; When G is greater than M, the G first multipliers are first multipliers in a plurality of groups of first multipliers.

6. The internal memory according to claim 4 or 5, characterized in that The first register stores a first parameter corresponding to each of the first sub-sequences, and each of the second multipliers is specifically configured to: receive an output result of a connected third adder as a third operand, and receive the first parameter corresponding to the first sub-sequence sent by the first register as a fourth operand; perform multiplication operation based on the third operand and the fourth operand, and input an output result to a connected fourth adder to perform addition operation.

7. The memory of claim 6, wherein, When G is greater than M, the fourth operands of the plurality of second multipliers corresponding to the plurality of groups of first multipliers are the same.

8. The internal memory according to any one of claims 4-7, characterized in that, The memory further includes a first module, and the first module is configured to: receive a first instruction sent by a processor, read the first sequence from the target Bank to the first multipliers based on the first instruction, and control reading the first parameter corresponding to the first sub-sequence from the first register to the second multipliers.

9. The internal memory according to any one of claims 4-8, characterized in that, The target Bank is composed of a plurality of rows and a plurality of columns of storage units; the first parameter corresponding to each of the first sub-sequences and the first sequence are stored in the same row of storage units in the target Bank; or the first parameter corresponding to each of the first sub-sequences and the first sequence are stored in different rows of storage units in the target Bank. Or, the first parameter corresponding to each of the first sub-sequences and the first sequence are stored in different Banks.

10. The memory of claim 9, wherein, The memory further includes a global buffer, and the first module is further configured to: receive a second instruction sent by the processor, read the first parameter corresponding to each of the first sub-sequences from the target Bank, and store the first parameter in the first register; or read the first parameter corresponding to each of the first sub-sequences from other Banks, and store the first parameter in the global buffer; and store the first parameter corresponding to each of the first sub-sequences from the global buffer to the first register.

11. A chip, characterized by The memory includes any one of the memories according to claims 1-10.

12. An electronic device, comprising: The memory includes a circuit board and any one of the memories according to claims 1-10, and the circuit board is electrically connected with the memory.

Images