Storage and calculation integration-based big language model reasoning device, reasoning system and electronic equipment

Abstract

The invention discloses a big language model reasoning device and system based on storage and calculation integration and electronic equipment. The reasoning device comprises a storage layer at least used for storage; the calculation layer is at least used for calculation, and the calculation layer and the storage layer are stacked in a mixed bonding mode; the calculation layer comprises a neural network accelerator based on storage and calculation integration, the neural network accelerator comprises an in-storage calculation matrix, and the in-storage calculation matrix is used for performing neural network calculation on the input feature data and the weight from the storage layer; the calculation layer is also used for being electrically connected with a main control chip for controlling the reasoning device, and the calculation layer is also used for performing pre-filling processing of large language model reasoning and transmitting the pre-filled data to the main control chip to perform decoding processing of large language model reasoning, so that the pre-filling processing and the decoding processing are separated. The reasoning device provided by the invention supports high bandwidth, is large in computing power and low in power consumption, and can also solve the heat dissipation problem of the existing LLM reasoning device.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to an LLM reasoning device, reasoning system and electronic device based on storage and computing integration. Background Art

[0002] AI large language model (LLM) inference devices are a major direction for the implementation of future AI applications. In particular, the end-side AI large language model inference chip is an important device for the implementation of future AI applications. It has a very broad development space in the fields of AI laptops, AI mobile phones, and smart robots.

[0003] The big language model is a natural language processing model based on deep learning technology. With the rapid development of the inference device of the AI ​​big language model and the rapid growth of social demand for it, the deployment of the inference device of the end-side AI big language model faces many difficulties and challenges. For example, with the rapid development of the end-side big model, its parameter quantity will continue to increase with iterations, its demand for bandwidth will continue to increase, and its demand for computing power will become greater and greater. When it processes a large amount of data, it will require a lot of power consumption and generate a lot of heat. Correspondingly, its demand for low power consumption is increasing and it is urgent to solve the problem of heat dissipation so that the inference device of the end-side AI big language model can adapt to people's growing demand for various functions of the inference device of the end-side AI big language model.

[0004] In short, with the development of artificial intelligence, the inference devices of AI large language models increasingly require high bandwidth, high computing power, low power consumption and good heat dissipation. Summary of the invention

[0005] The purpose of the present invention is to provide a large language model reasoning device, reasoning system and electronic device based on integrated storage and computing to solve some or all of the above-mentioned technical problems.

[0006] To achieve the above-mentioned purpose, the large language model inference device based on storage and computing integration provided by the present invention includes: a storage layer at least for storage; a computing layer at least for computing, the computing layer is stacked with the storage layer by a hybrid bonding method; the computing layer includes a neural network accelerator based on storage and computing integration, the neural network accelerator includes an in-memory computing matrix, the in-memory computing matrix is ​​used to perform neural network calculations on input feature data and weights from the storage layer; the computing layer is also used to be electrically connected to the main control chip of the control inference device, the computing layer is also used to perform pre-filling processing of large language model inference and transmit the pre-filled data to the main control chip for decoding processing of large language model inference, so that the pre-filling processing and the decoding processing are separated.

[0007] In a preferred embodiment, the in-memory calculation matrix includes: N columns of storage operator modules, each column of the storage operator module includes M rows of storage and calculation units for storing and calculating the stored data with the input feature data.

[0008] In a preferred embodiment, each storage and computing unit includes: an SRAM memory for storing weights from the storage layer, and a logic unit for computing arranged adjacent to the SRAM memory.

[0009] In a preferred embodiment, the in-memory calculation matrix includes: a weight storage array, a bit multiplier, a storage readout circuit and a logic operation unit; the weight storage array is used to store weights; the bit multiplier is used to receive a weight read enable signal and input feature data, and when the weight read enable signal is enabled, the weight in the weight storage array is selected according to the weight read enable signal, and the bit of the input feature data is multiplied by the selected weight; the storage readout circuit is used to read the multiplied product to the logic operation unit when the bit of the input feature data is 1, and does not perform the readout operation when the bit of the input feature data is 0; the logic operation unit is used to accumulate the product read out by the storage readout circuit when the bit of the input feature data is 1, so as to realize the multiplication and accumulation of the input feature data and the weight.

[0010] In a preferred embodiment, the bit multiplier includes N AND gates respectively connected to the weight storage array; one input end of the AND gate is used to access the input feature data, the other input end is used to access the weight read enable signal, and the output end is used to output a control signal.

[0011] In a preferred embodiment, the storage layer is also used for calculation to implement the first level calculation of the neural network calculation, and the calculation layer completes the neural network calculation according to the result of the first level calculation.

[0012] In a preferred embodiment, the storage readout circuit also includes an AND gate; one input end of the AND gate is used to access the input feature data, the other input end is used to access the weight read enable signal, and the output end is used to output the first control signal.

[0013] In a preferred embodiment, the storage layer includes at least two layers of DRAM memory, and the at least two layers of DRAM memory are connected via TSV.

[0014] The present invention also provides an inference system, which includes the large language model inference device described in any one of the above items, and a main control chip electrically connected to the large language model inference device, wherein the main control chip is used to perform large language model inference decoding processing on the data after pre-filling processing of the computing layer in the large language model inference device.

[0015] The present invention also provides an electronic device, comprising the large language model inference device described in any one of the above items.

[0016] The large language model inference device based on storage and computing integration provided by the present invention has a storage layer and a computing layer including a neural network accelerator based on storage and computing integration stacked in a hybrid bonding manner, thereby reducing data movement, reducing energy loss in signal transmission, and reducing time delay in signal propagation. It can reduce power consumption, generate less heat, and have high computing power. Moreover, it performs the pre-filling processing and decoding processing in LLM inference in the above-mentioned computing layer and the main control chip respectively, which can make full use of the neural network accelerator with high computing power in the present invention, so that the pre-filling processing that requires high computing power is executed in the above-mentioned computing layer, thereby supporting a large language model inference device that requires high bandwidth, and also improving the efficiency of large language model inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a structural diagram of a large language model inference device and a large language model inference system provided by one embodiment of the present invention; Figure 2 is a module schematic diagram of a neural network accelerator provided by an embodiment of the present invention; Figure 3 It is a module schematic diagram of a column storage operator module in an in-memory computing matrix in one embodiment of the present invention; Figure 4 It is a schematic diagram of the vertical structure of a column of storage operator modules in an in-memory computing matrix in one embodiment of the present invention; Figure 5 It is a schematic diagram of the principle of performing calculations using a calculation matrix stored in memory in one embodiment of the present invention; Figure 6 is a schematic diagram of a module for calculating a matrix in memory in one embodiment of the present invention; Figure 7 It is a schematic diagram of the structure of an in-memory computing matrix according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application are clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application. In the absence of conflict, the following embodiments and their technical features can be combined with each other.

[0020] In order to make the purpose, technical solutions and advantages of the present invention clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present invention are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, devices, etc. that are consistent with some aspects of the present disclosure as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present invention.

[0021] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", etc. indicate the orientation or positional relationship based on the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. The terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "multiple" means two or more. The terms "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0022] Reasoning of large language models is mainly divided into two stages. The first stage is the prefill phase, which processes input tokens in parallel. The second stage is the decode phase, which produces the next token one by one. These two steps are repeated until the EOS (end-of-sequence) token is generated or the stop condition set by the user is reached. That is, prefilling is the process of providing the user's input as a prompt (initial information) to the model. This prompt can be regarded as the initial information of the model, which is used to guide subsequent generation. Since the prefill phase can be processed in parallel, it means that multiple inputs can be processed at the same time, thereby improving the efficiency of reasoning. Through parallel computing, the prefill operation can be completed faster and the waiting time can be reduced.

[0023] The inventors of the present invention found that: the amount of calculation in the pre-filling stage of large language model reasoning is large, and the computing power requirement is greater than the bandwidth requirement; the computing power requirement in the decoding stage of large language model reasoning is greatly reduced, but its bandwidth requirement is greatly increased. Take a typical end-side application scenario as an example: LIama3-8B model, int8 quantization, first token time = 1s, decoding stage 100 tokens / s. The computing power and bandwidth requirements at different stages are shown in Table 1: Table 1: Computing power and bandwidth requirements of the LIama3-8B model (int8) at different stages As can be seen from Table 1, in the application of the large language model (LLM) inference chip on the end, the prefill stage has a large amount of calculation, and the demand for computing power is greater than the demand for bandwidth; on the contrary, in the decode stage, the demand for computing power is greatly reduced, but the demand for bandwidth is greatly increased to 800GB / s. This is because the input in the decode stage becomes a vector, and the calculation changes from matrix multiplication to vector-matrix multiplication. At the same time, since more than 90% of the time of the large model inference on the end is executed in the decode stage, it is memory-intensive, and its inference efficiency is limited by bandwidth rather than computing power. Due to the huge demand for bandwidth in the decode stage, the existing GPU and LPDDR-based end-side chips cannot meet the bandwidth requirements of the deployment of large models (LLM) on the end. In addition, the large language model inference device needs to infer more and more, and the corresponding power consumption is increasing. In the prior art, the components used for calculation in the LLM inference device are far away from the radiator, making it difficult to effectively dissipate the heat generated by the above-mentioned calculation components, which makes the LLM inference device face the challenge of heat dissipation. In the prior art, the heat dissipation capacity of the radiator is usually improved, but the heat dissipation improvement effect is still poor.

[0024] The large language model inference device based on integrated storage and computing provided by the present invention can solve the problems related to computing power, bandwidth, power consumption and heat dissipation.

[0025] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below, which only shows the parts related to the embodiment of the present invention.

[0026] Reference Figure 1 As shown, the large language model inference device based on storage and computing integration provided by the present invention includes: a stacked storage layer (illustrated by DRAM Layers in the figure) and a computing layer (i.e., the bottom layer in the example shown in the figure). The storage layer is at least used for storage, which can be used to store parameters of a large model, such as storing 8GB of large model parameters and storing weights used for neural network calculations in the computing layer; the storage layer can also be used to store output feature data after neural network calculations. The computing layer is at least used for calculations, and the computing layer is stacked with the storage layer by hybrid bonding, and the computing layer is electrically connected to the storage layer to realize data interaction between the storage layer and the computing layer. The computing layer includes a neural network accelerator based on storage and computing integration (i.e., the NPU acceleration chip in the figure) for neural network calculations. The computing layer may include multiple sub-NPU clusters. The neural network accelerator includes an in-memory computing matrix, which is used to perform neural network calculations on input feature data and weights from the storage layer. The computing layer is also used to be electrically connected to the main control chip that controls the inference device. The computing layer is also used to perform pre-filling processing of large language model inference and transmit the data after the pre-filling processing to the main control chip for decoding processing of the large language model inference, so that the pre-filling processing and the decoding processing are separated.

[0027] The storage layer and computing layer provided by the present invention may be in various forms, for example, they may be chips or wafers.

[0028] The neural network accelerator included in the computing layer of the present invention realizes storage and computing integration through the in-memory computing matrix, which can realize the neural network calculation of the input feature data and the weights from the storage layer. Storage and computing integration is an innovative architecture that aims to overcome the "memory wall" problem by integrating storage and computing functions on the same chip. This technology embeds computing power in the memory and adopts a new computing architecture to realize the multiplication and accumulation of matrix vectors, so it can reduce the frequent data movement between the memory and the computing chip in the prior art, thereby greatly reducing the time, power consumption and heat generation of the movement, and improving the parallel processing efficiency of data; it realizes in-situ computing, eliminates bandwidth limitations and data movement costs. In this way, it can essentially eliminate unnecessary data movement delays and power consumption, improve artificial intelligence computing efficiency by hundreds or thousands of times, reduce costs, and break the "storage wall" and "power consumption wall". It can make the computing layer and the large language model inference device have greater computing power, consume less power, and generate less heat than the devices that perform calculations under the same conditions in the prior art, so as to solve the existing heat dissipation problem from the source of heat.

[0029] Moreover, the present invention realizes a 3D stacked structure of the storage layer and the computing layer in the LLM reasoning device by hybrid bonding. Hybrid bonding can realize high-density and high-performance interconnection between different chips. Through hybrid bonding, the metal layers (usually copper layers) of two or more chips can be precisely aligned and directly pressed together to form direct electrical contact. It can achieve submicron or even nanometer interconnection spacing, allowing more connection points to be placed on a smaller area, greatly increasing the data communication bandwidth between chips. Its interconnection density (contact density) can reach 10K-1MM / mm2. Hybrid bonding eliminates intermediate media such as solder and other materials, and the direct copper-to-copper connection has lower resistance, reduces the energy loss of signal transmission, and also reduces the time delay of signal propagation; the compact structure and direct conductive path formed by hybrid bonding help improve thermal management and reduce heating problems, so it can achieve higher integration, faster data transmission speed, larger bandwidth, and lower power consumption.

[0030] Furthermore, the inference device provided by the present invention combines the advantage of high computing power based on the integrated storage and computing computing layer, and separates prefill from decode, that is, it completes the prefill that requires high computing power in the computing layer, and transmits the data processed by prefill in the computing layer to the main control chip electrically connected to it, and the main control chip decodes the pre-filled data. Therefore, on the one hand, the demand for the bandwidth of the inference device is reduced, so that the inference device can perform the reasoning work of the large language model faster and better, and also enables the large language model inference device under the same computing power to support a larger bandwidth to perform the reasoning work of the large language model.

[0031] In summary, the inference device provided by the present invention can solve the existing bandwidth bottleneck to support large bandwidth and large computing power, and the entire inference device reduces power consumption in many aspects and can solve the existing heat dissipation problem.

[0032] Reference Figure 2 In the embodiment shown, the neural network accelerator includes: a pre-processing module, an in-memory computing matrix and a vector processing module electrically connected in sequence, and a shared memory electrically connected to the pre-processing module, the in-memory computing matrix and the vector processing module respectively; and also includes a chip controller electrically connected to the shared memory. Among them, the NPU is also electrically connected to the external memory. The above-mentioned storage layer in the present invention is Figure 2The external memory shown. The data stored in the external memory is moved to the shared memory of the NPU in batches. For example, weights are read from the external memory to the shared memory in batches. One of the working methods is briefly described as follows: the input feature data is obtained from the shared memory through the pre-processing module, the input feature data is processed into matrix data in the format of an in-memory calculation matrix, and the matrix data is input into the in-memory calculation matrix; wherein, if the input feature data includes at least two input feature maps, and the at least two input feature maps are stored in the shared memory in a jump address manner, then the pre-processing module obtains the at least two input feature maps from the shared memory in an address sequential reading manner; or, if the input feature data includes at least two input feature maps, and the at least two input feature maps are stored in the shared memory in a continuous storage manner, then the pre-processing module obtains the at least two input feature maps from the shared memory in a jump address reading manner; weight data is obtained through the in-memory calculation matrix, through the The in-memory calculation matrix performs convolution calculation according to the matrix data and the weight data to obtain a calculation result, and inputs the calculation result into the vector processing module; the calculation result is vector processed by the vector processing module to obtain output feature map data, and the obtained output feature map data is written into the shared memory and / or the external memory; wherein, if the output feature map data is an intermediate calculation result, the output feature map data is used as input data for the calculation result of the next level; before the pre-processing module obtains the input feature data, and / or, after the calculation result is processed by the vector processing module to obtain the output feature map data, the data interaction between the shared memory and the external memory is controlled by the chip controller, wherein the data for interaction includes the input feature data and / or the output feature map data.

[0033] In a preferred embodiment of the present invention, the in-memory computing matrix includes: N columns of storage operator modules, each column of the storage operator module includes: M rows of storage and computing units for storing and calculating the stored data with the input feature data, and capacitors electrically connected to the storage and computing units, and the charges output by the capacitors in the M rows are gathered together to realize charge accumulation.

[0034] See also Figure 3The module schematic diagram of a column of storage operator modules in the in-memory calculation matrix in one embodiment of the present invention is shown. In this preferred embodiment, the storage unit can store and multiply the weights and input feature data, and the corresponding product is charged and accumulated through the capacitor connected to the storage unit to realize the multiplication and accumulation (MAC) calculation in the neural network calculation. That is, the matrix calculation can be realized inside the in-memory calculation matrix, which can avoid the frequent data transfer between two different and far-away memories and computing chips in the prior art. The power consumption of multiplication and accumulation calculation usually occupies most of the power consumption of the entire calculation, for example, in most cases, it will occupy 50-70% of the power consumption. The present invention is based on the storage and calculation integrated architecture, and a neural network accelerator with an in-memory calculation matrix is ​​set in the calculation layer to realize two-dimensional and three-dimensional matrix multiplication / addition operations through a new calculation architecture, thereby greatly reducing the power consumption of neural network calculation, and can make the area of ​​the neural network accelerator smaller, and can also make it possible to support greater computing power and generate less heat under the same area and power consumption to solve the heat dissipation problem.

[0035] See also Figure 4 and Figure 5 As shown, Figure 4 The vertical structure diagram of a column of storage computing submodules in the in-memory computing matrix in one embodiment of the present invention is shown. Each storage computing unit includes: an SRAM memory (i.e., Figure 4 The digital logic is the above-mentioned logic unit, which can be used to perform multiplication and addition calculations on the weights in the SRAM memory and the input feature data. Figure 4The typical architecture block diagram of the integrated digital memory and computing (CIMD) is also displayed. Specifically, it is based on SRAM memory, and the model weights are stored in multiple SRAM Memory Banks on the chip. The multiplication and accumulation (MAC) calculation logic circuit required by CIMD is deployed near each SRAM Memory Bank to greatly reduce the delay and power consumption caused by the weight transfer. Compared with the traditional von Neumann architecture artificial intelligence (AI) chip, the integrated digital memory and computing (CIMD) greatly reduces the power consumption of data transfer, and at the same time greatly improves the speed of parallel multiplication and accumulation (MAC) calculation, effectively alleviating the two major problems of "storage wall" and "power consumption wall". The SRAM memory and the logic unit used for calculation are set up in close proximity by stacking, which can make the structure of the calculation matrix in the memory more compact, the corresponding computing power is greater, the power consumption is lower, and the combination of digital logic units can make its calculation speed faster and the calculation accuracy higher. As shown in Table 2, at the 12nm process node, the CIM in-memory computing unit's matrix multiplication energy efficiency can reach 24.7Tops / W, which is an order of magnitude higher than the traditional digital architecture (i.e., general-purpose computing units, such as GPUs and TPUs). Computing-in-memory (CIMD) significantly reduces the power consumption of data movement, while significantly increasing the speed of parallel multiplication-accumulation (MAC) calculations. Thanks to the innovation of this underlying computing unit architecture, the in-memory computing (CIMD) technology can significantly improve the energy efficiency of the edge AI large model (LLM) inference chip, that is, the computing power remains unchanged and its power consumption requirements are greatly reduced.

[0036] Table 2 Figure 5 The schematic diagram is a principle diagram of the calculation of the in-memory calculation matrix in one embodiment of the present invention. The storage calculation unit performs multiplication calculation on the input feature data and the weight, and then accumulates and sums the product through a multi-level addition tree to obtain a multiplication-accumulation result.

[0037] See also Figure 6 In this preferred embodiment, the in-memory calculation matrix includes a weight storage array 10 (also called a weight parameter storage array), a bit multiplier 12, a storage readout circuit 21 and a logic operation unit 22. One input end of the bit multiplier 12 is connected to a weight read enable signal, and the other input end is connected to input feature data. The weight acquisition end is connected to the weight storage array 10; the input end of the storage readout circuit 21 is connected to the weight storage array 10, and the output end is connected to the logic operation unit 22; the output end of the logic operation unit 22 is used to output the convolution operation result. The weight storage array 10 is used to store weights.

[0038] The bit multiplier 12 is used to receive a weight read enable signal and input feature data (in a convolutional neural network, it is usually necessary to process the input feature map). When the weight read enable signal is enabled, the weight in the weight storage array is selected according to the weight read enable signal, and the bit of the input feature data is multiplied by the selected weight. The input feature data is used to characterize the data output by the previous layer of operation during the convolution operation, and may include at least one bit, wherein the bit value is 1 or 0; for the input feature data, its bit is 1 to represent that the input feature data is 1, and its bit is 0 to represent that the input feature data is 0. For example, when the bit of the input feature data is 1, the product of the bit of the input feature data and the corresponding weight is the corresponding weight, and at this time, the storage read circuit 21 reads the corresponding weight from the weight storage array 10 to realize the corresponding product read; when the bit of the input feature data is 0, the storage read circuit 21 does not perform the read operation at this time, so the storage read circuit 21 can directly perform the read operation to the weight storage array 10 to realize the corresponding product read. The storage readout circuit 21 is used to read the multiplied product to the logic operation unit 22 when the bit of the input feature data is 1, and does not perform the readout operation when the bit of the input feature data is 0, so that the storage readout circuit 21 does not generate power consumption when the bit of the input feature data is 0, thereby reducing the power consumption of the storage readout circuit 21. Specifically, the storage readout circuit 21 can read the corresponding weight to the logic operation unit 22 when the bit of the input feature data is 1. Specifically, the storage medium of the storage readout circuit 21 includes any one of SRAM, DRAM and RRAM, and the storage medium can be a volatile storage medium or a non-volatile storage medium. The logic operation unit 22 is used to accumulate the product read out by the storage readout circuit 21 when the bit of the input feature data is 1, so as to realize the multiplication and accumulation of the input feature data and the weight. The above-mentioned logic operation unit 22 only needs to output the final convolution operation result, without transmitting the intermediate data in the convolution operation process, which can reduce the requirements for data output and data transmission bandwidth. That is, it makes the storage readout circuit generate no power consumption when the bit of the input feature data is 0, thereby reducing the power consumption of the storage readout circuit; the output data of the logic operation unit is the accumulation result, and there is no need to transmit the intermediate data in the convolution operation process, and the data bit width of the output data is reduced. If a higher accumulation operation is implemented in the logic operation unit, the output data bit width can be further compressed. It can be seen that the present application can reduce the power consumption of a large number of concurrent data readings and reduce the requirements for data output and data transmission bandwidth.

[0039] For further preference, see Figure 7As shown, the bit multiplier includes N AND gates respectively connected to the weight storage array; one input end of the AND gate is used to access the input feature data fmi, the other input end is used to access the weight read enable signal, and the output end is used to output a control signal. The weight storage array 10 includes K rows and N columns of parameter storage units. Each parameter storage unit is used to store a weight or an electrical signal used to characterize the corresponding weight. The parameter storage unit in the hth row and jth column stores the weight W (h, j), 1≤h≤K, 1≤j≤N, K is the number of rows of weights, and N is the number of columns of weights. The K rows of parameter storage units are correspondingly arranged on the K rows of bit lines; the hth row of bit lines is recorded as bitline,h. The above-mentioned weight read enable signal is used to enable a column of parameter storage units in the N columns of parameter storage units, that is, a weight read enable signal corresponds to a column of parameter storage units, such as the weight read enable signal corresponding to the jth column of parameter storage units can be recorded as wordline,j. When the corresponding weight read enable signal wordline,j is 1, the parameter storage unit of the jth column outputs the electrical signal representing the weight to the corresponding bit line, so that the storage read circuit 21 can read the corresponding weight signal through each row of bit lines. For example, the probability that an N-bit input vector is all 0 is much lower than the probability that some bits in an N-dimensional input vector are 0. The bit-level sparsity of data is very high. However, the smaller the granularity, the more difficult it is to use the data sparsity, which is mainly reflected in the need to perform real-time calculations and judgments on each bit of the input data. Through this preferred method, the present invention can solve the problem of data sparsity. For the case where the data sparse matrix contains a large number of zero values, it can greatly reduce resource waste, reduce unnecessary calculations, and improve storage space utilization.

[0040] In a preferred embodiment of the present invention, the storage readout circuit further includes an AND gate; one input end of the AND gate is used to access the input feature data, the other input end is used to access the weight read enable signal, and the output end is used to output the first control signal; for the weight read enable signal wordline,j and the input feature data fmi,j corresponding to the jth column, when the weight read enable signal wordline,j and the input feature data fmi,j are both 1, the first control signal is 1; when at least one of the weight read enable signal wordline,j and the input feature data fmi,j is 0, the first control signal is 0. The circuit is simple and can accurately control the read operation, which can reduce the power consumption of the storage readout circuit and reduce the power consumption of the large language model inference device.

[0041] In a preferred embodiment of the present invention, the storage layer is also used for calculation to realize the first level calculation of the neural network calculation, and the calculation layer completes the neural network calculation according to the result of the first level calculation. That is, some simple calculations can also be done in the storage layer, such as the first multiplication of the multiplication parameter, that is, the first level calculation, and the result after the first level calculation is transmitted to the NPU for cumulative calculation to complete the neural network calculation. It can make full use of the storage layer, reduce the amount of calculation of the calculation layer, and thus reduce the power consumption of the calculation layer. Moreover, after the first level calculation processing, the data of the storage layer can be reduced to the calculation layer and the demand for storage and high bandwidth can be reduced accordingly. The storage layer preferably includes DRAM memory, which has low cost and high density. Further preferably, the storage layer includes at least two layers of DRAM memory, and the at least two layers of DRAM memory are connected by TSV (Through Silicon Vias, silicon through hole), which can stack multiple DRAM chips vertically, significantly improve memory bandwidth, and reduce power consumption; realize large capacity and high bandwidth storage, and meet the stringent requirements of high performance computing, artificial intelligence and other fields on memory.

[0042] As shown in Table 3, the 3D stacked DRAM with integrated Hybrid Bonding provided by the present invention has a bandwidth of 1-60TB, which is a significant improvement compared to other most advanced GDDR6 and HBM3E technologies. At the same time, the energy requirement for transmitting a single bit of data is also greatly reduced to 0.5pJ / bit, which greatly reduces the power consumption of data movement. It can solve the bandwidth bottleneck problem encountered in the deployment of edge-side AI large model (LLM) chips, and also greatly reduces the power consumption of data movement.

[0043] Table 3 In summary, the present invention provides a new LLM inference device, which combines 3D stacked DRAM, HybridBonding technology, and high energy efficiency ratio SRAM storage and computing integrated technologies, which can solve the bottleneck problems of bandwidth, power consumption, computing power and heat dissipation currently encountered by the end-side AI large model (LLM) inference device (the inference device can be in various forms such as chips or modules). It can significantly improve memory bandwidth, reduce power consumption, achieve large capacity, high bandwidth storage, and meet the stringent requirements of high-performance computing, artificial intelligence and other fields on memory. It can be widely used in various end sides, such as mobile phones, computers, robots, etc.

[0044] The present invention also provides an inference system (see Figure 1As shown), it includes any of the above-mentioned large language model inference devices, and a main control chip electrically connected to the large language model inference device, and the main control chip is used to decode the data after the pre-filling processing of the computing layer in the large language model inference device. Among them, the structure of the NPU included in the computing layer is also Figure 1 The inference system provided by the present invention allows the pre-filling process and the decoding process to be performed separately in different devices, and the pre-filling process that requires large computing power can be processed in a storage-computing integrated structure based on SRAM, while the main control chip, which has relatively small computing power, performs decoding, which enables the inference system of the large language model to greatly solve the existing bandwidth bottleneck problem, greatly reduce its power consumption and alleviate the heat dissipation problem. The main control chip can be a variety of types of chips, such as GPU, SOC, etc.

[0045] The present invention also provides an electronic device, which includes the above-mentioned large language model reasoning device or reasoning system. It can greatly reduce the power consumption and cost of electronic devices and improve the efficiency and experience of AI reasoning. It can be widely used in smart phones, tablet computers, wearable electronic equipment, smart home electronic products, etc.

[0046] The above description is only the preferred embodiment of the present invention. It is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the protection scope of the present invention.

Claims

1. A large language model reasoning device based on storage and computing integration, characterized in that: include: A storage layer for at least storage; a computing layer at least used for computing, the computing layer being stacked with the storage layer in a hybrid bonding manner, and the computing layer being electrically connected to the storage layer for data exchange; The computing layer includes a neural network accelerator based on storage and computing integration, and the neural network accelerator includes an in-memory computing matrix, and the in-memory computing matrix is ​​used to perform neural network computing on input feature data and weights from the storage layer; The computing layer is also used to be electrically connected to the main control chip that controls the inference device. The computing layer is also used to perform pre-filling processing of large language model inference and transmit the data after the pre-filling processing to the main control chip for decoding processing of the large language model inference, so that the pre-filling processing and the decoding processing are separated.

2. The large language model inference device according to claim 1, characterized in that: The in-memory calculation matrix includes: N columns of storage operator modules, each column of the storage operator module includes M rows of storage and calculation units for storing and calculating the stored data and the input feature data.

3. The large language model inference device according to claim 2, characterized in that: Each storage and calculation unit includes: an SRAM memory for storing weights from the storage layer, and a logic unit for calculation arranged adjacent to the SRAM memory.

4. The large language model inference device according to claim 1, characterized in that: The in-memory calculation matrix includes: a weight storage array for storing weights, a bit multiplier, a storage readout circuit and a logic operation unit; the bit multiplier is used to receive a weight read enable signal and input feature data, and when the weight read enable signal is enabled, the weight in the weight storage array is selected according to the weight read enable signal, and the bit of the input feature data is multiplied by the selected weight; the storage readout circuit is used to read the multiplied product to the logic operation unit when the bit of the input feature data is 1, and does not perform the readout operation when the bit of the input feature data is 0; the logic operation unit is used to accumulate the product read out by the storage readout circuit when the bit of the input feature data is 1, so as to realize the multiplication and accumulation of the input feature data and the weight.

5. The large language model inference device according to claim 4, characterized in that: The bit multiplier includes N AND gates respectively connected to the weight storage array; one input end of the AND gate is used to access the input feature data, the other input end is used to access the weight read enable signal, and the output end is used to output a control signal.

6. The large language model inference device according to claim 4, characterized in that: The storage readout circuit also includes an AND gate; one input end of the AND gate is used to access the input feature data, the other input end is used to access the weight read enable signal, and the output end is used to output the first control signal.

7. The large language model inference device according to any one of claims 1 to 5, characterized in that: The storage layer is also used for calculation to implement the first level calculation of the neural network calculation, and the calculation layer completes the neural network calculation according to the result of the first level calculation.

8. The large language model inference device according to any one of claims 1 to 5, characterized in that: The storage layer includes at least two layers of DRAM memory, and the at least two layers of DRAM memory are connected via TSV.

9. An inference system, characterized in that It comprises a large language model inference device as described in any one of claims 1 to 8, and a main control chip electrically connected to the large language model inference device, wherein the main control chip is used to perform large language model inference decoding processing on the data after pre-filling processing of the computing layer in the large language model inference device.

10. An electronic device, characterized in that: A large language model inference device comprising any one of claims 1-8 or an inference system comprising claim 9.

Images