Storage and calculation integration-based big language model reasoning device, reasoning system and electronic equipment

Abstract

The utility model discloses a big language model reasoning device based on storage and calculation integration, a reasoning system and electronic equipment. The reasoning device comprises a storage layer at least used for storage; the calculation layer is at least used for calculation, and the calculation layer and the storage layer are stacked in a mixed bonding mode; the calculation layer comprises a neural network accelerator based on storage and calculation integration, the neural network accelerator comprises an in-storage calculation matrix, and the in-storage calculation matrix is used for performing neural network calculation on the input feature data and the weight from the storage layer; the calculation layer is also used for being electrically connected with a main control chip for controlling the reasoning device, and the calculation layer is also used for performing pre-filling processing of large language model reasoning and transmitting the pre-filled data to the main control chip to perform decoding processing of large language model reasoning, so that the pre-filling processing and the decoding processing are separated. The reasoning device provided by the utility model supports high bandwidth, is large in computing power and low in power consumption, and can also solve the heat dissipation problem of the existing LLM reasoning device.

Description

Technical Field

[0001] This utility model relates to the field of artificial intelligence technology, specifically to an LLM inference device, inference system, and electronic device based on in-memory computing. Background Technology

[0002] Large Language Model (LLM) inference devices are a major direction for the future application of AI, especially edge-side large language model inference chips, which are important devices for the future application of AI and have a very broad development prospect in fields such as AI laptops, AI mobile phones, and intelligent robots.

[0003] Large language models are natural language processing models based on deep learning technology. With the rapid development of AI large language model inference devices and the increasing societal demand for them, the deployment of edge AI large language model inference devices faces numerous difficulties and challenges. For example, as edge large models rapidly develop, their parameter count increases with iteration, leading to a continuous increase in bandwidth and computing power requirements. Furthermore, processing large amounts of data generates significant power consumption and heat, necessitating increasingly lower power consumption and urgent solutions to heat dissipation issues. This ensures that edge AI large language model inference devices can meet the growing functional demands of users.

[0004] In summary, with the development of artificial intelligence, the inference devices of AI large language models increasingly require high bandwidth, high computing power, low power consumption, and good heat dissipation. Utility Model Content

[0005] The purpose of this invention is to provide a large language model reasoning device, reasoning system, and electronic device based on in-memory computing, so as to solve some or all of the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention provides a large language model inference device based on in-memory computing, comprising: at least a storage layer for storage, the storage layer including a DRAM memory; at least a computation layer for computation, the computation layer being stacked with the storage layer via hybrid bonding; the computation layer including an in-memory computing-based neural network accelerator, the neural network accelerator including an in-memory computation matrix, the in-memory computation matrix being used to perform neural network computation on input feature data and weights from the storage layer; the computation layer is also used to be electrically connected to a main control chip controlling the inference device, and the computation layer is also used to perform pre-filling processing for large language model inference and to transmit the pre-filled data to the main control chip for decoding processing of large language model inference, so as to separate the pre-filling processing and the decoding processing.

[0007] In a preferred embodiment, the in-memory computation matrix includes: N column storage operator modules, each column storage operator module including M rows of storage units for storing and performing computations on the stored data and input feature data.

[0008] In a preferred embodiment, each computing unit includes: an SRAM memory for storing weights from the storage layer, and a logic unit for computing disposed adjacent to the SRAM memory.

[0009] In a preferred embodiment, the in-memory computation matrix includes: a weight storage array, a bit multiplier, a storage readout circuit, and a logic operation unit; the weight storage array is used to store weights; the bit multiplier is used to receive a weight readout enable signal and input feature data, and when the weight readout enable signal is enabled, it selects a weight in the weight storage array according to the weight readout enable signal, and multiplies the bit of the input feature data with the selected weight; the storage readout circuit is used to read the product of the multiplication to the logic operation unit when the bit of the input feature data is 1, and does not perform a readout operation when the bit of the input feature data is 0; the logic operation unit is used to accumulate the product read out by the storage readout circuit when the bit of the input feature data is 1, so as to realize the multiplication and accumulation of the input feature data and the weight.

[0010] In a preferred embodiment, the bit multiplier includes N AND gates that are respectively connected to the weight storage array; one input of each AND gate is used to receive the input feature data, another input is used to receive the weight read enable signal, and the output is used to output a control signal.

[0011] In a preferred embodiment, the storage layer is further used for computation to implement the first-level computation of the neural network computation, and the computation layer completes the neural network computation based on the result of the first-level computation.

[0012] In a preferred embodiment, the storage readout circuit further includes an AND gate; one input terminal of the AND gate is used to receive the input feature data, the other input terminal is used to receive the weight readout enable signal, and the output terminal is used to output a first control signal.

[0013] In a preferred embodiment, the storage layer includes at least two DRAM memory layers, and the at least two DRAM memory layers are connected via TSV.

[0014] This utility model also provides a reasoning system, which includes the large language model reasoning device described in any of the above claims, and a main control chip electrically connected to the large language model reasoning device. The main control chip is used to perform large language model reasoning decoding processing on the data after the computation layer pre-filling processing in the large language model reasoning device.

[0015] This invention also provides an electronic device, including any of the above-described large language model reasoning devices.

[0016] The large language model inference device based on in-memory computing provided by this invention stacks its storage layer and computation layer, which includes an in-memory computing-based neural network accelerator, using a hybrid bonding method. This reduces data movement, energy loss during signal transmission, and time delay in signal propagation, resulting in lower power consumption, less heat generation, and higher computing power. Furthermore, it performs pre-filling and decoding processes in LLM inference separately in the aforementioned computation layer and the main control chip. This fully utilizes the high-performance neural network accelerator of this invention, allowing the computationally intensive pre-filling process to be executed in the computation layer. This supports large language model inference devices requiring high bandwidth and improves the efficiency of large language model inference. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of the large language model reasoning device and the large language model reasoning system provided in an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of a neural network accelerator module provided in one embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of a column of stored operator modules in an in-memory calculation matrix in one embodiment of this utility model;

[0021] Figure 4 This is a schematic diagram of the vertical structure of a column of stored operator modules in an in-memory calculation matrix in one embodiment of this utility model;

[0022] Figure 5 This is a schematic diagram illustrating the principle of calculation using an in-memory calculation matrix in one embodiment of this utility model;

[0023] Figure 6This is a schematic diagram of the in-memory calculation matrix in one embodiment of the present invention;

[0024] Figure 7 This is a schematic diagram of the structure of an in-memory computation matrix according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. In the absence of conflict, the following embodiments and their technical features can be combined with each other.

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, various exemplary embodiments described below will be referenced to the accompanying drawings, which form part of the exemplary embodiments, illustrating various exemplary embodiments that may be used to implement the invention. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. It should be understood that they are merely examples of processes, methods, and apparatuses consistent with some aspects of this invention disclosed as detailed in the appended claims, and other embodiments may be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and spirit of this invention.

[0027] In the description of this utility model, it should be understood that the terms "center," "longitudinal," "lateral," etc., indicate the orientation or positional relationship based on the accompanying drawings, and are only for the convenience of describing this utility model and simplifying the description, and do not indicate or imply that the referred element must have a specific orientation, or be constructed and operated in a specific orientation. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. The term "multiple" means two or more. The terms "connected" and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, integral connections, mechanical connections, electrical connections, communication connections, direct connections, indirect connections through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more of the related listed items. Those skilled in the art can understand the specific meaning of the above terms in this utility model according to the specific circumstances.

[0028] Inference in a large language model mainly consists of two phases: the prefill phase, which processes input tokens in parallel; and the decoding phase, which generates the next token sequentially. These two phases are repeated until an end-of-sequence (EOS) token is generated or a user-defined stopping condition is met. In other words, prefilling provides the model with the user's input as a prompt (initial information). This prompt can be seen as the model's initial information, guiding subsequent generation. Because the prefill phase can be processed in parallel, multiple inputs can be processed simultaneously, thus improving inference efficiency. Parallel computation allows for faster prefilling operations, reducing waiting time.

[0029] The inventors of this invention discovered that the pre-filling stage in large language model inference has a large computational load, with the computational power requirement exceeding the bandwidth requirement; while the computational power requirement of the decoding stage in large language model inference is significantly reduced, its bandwidth requirement is significantly increased. Taking a typical edge application scenario as an example: the LIama3-8B model, int8 quantization, first token time = 1s, decoding stage 100 tokens / s. The computational power and bandwidth requirements of its different stages are shown in Table 1.

[0030]

[0031] Table 1: Computing power and bandwidth requirements of the LIama3-8B model (int8) at different stages

[0032] As shown in Table 1, in the application of edge-side Large Language Model (LLM) inference chips, the prefill stage has a large computational load, with the computational power requirement exceeding the bandwidth requirement. Conversely, in the decode stage, the computational power requirement decreases significantly, but the bandwidth requirement increases dramatically to 800GB / s. This is because the input in the decode stage becomes a vector, and the calculation changes from matrix multiplication to vector-matrix multiplication. Furthermore, since edge-side LLM inference spends over 90% of its time in the decode stage, it is memory-intensive, and its inference efficiency is limited by bandwidth rather than computational power. Due to the huge bandwidth requirement of the decode stage, existing GPUs and LPDDR-based edge chips cannot meet the deployment bandwidth requirements of edge-side LLM. Moreover, the amount of data that LLM inference devices need to infer is increasing, leading to higher power consumption. In existing technologies, the computational components in LLM inference devices are far from the heat sink, making it difficult to effectively dissipate the heat generated by these components. This presents a challenge for LLM inference devices in terms of heat dissipation. Existing technologies typically improve the heat dissipation capacity of the heat sink, but the improvement effect is still unsatisfactory.

[0033] The present invention provides a large language model inference device based on in-memory computing, which can solve the problems of computing power, bandwidth, power consumption and heat dissipation.

[0034] To illustrate the technical solution described in this utility model, specific embodiments are described below, which only show the parts related to the embodiments of this utility model.

[0035] Reference Figure 1As shown, the large language model inference device based on in-memory computing provided by this utility model includes: a stacked storage layer (illustrated as DRAM Layers in the figure) and a computation layer (i.e., the bottom layer in the example shown in the figure). The storage layer is used for storage at least, and can be used to store parameters of a large model, such as storing 8GB of large model parameters, and storing weights used for neural network computation in the computation layer; the storage layer can also be used to store output feature data after neural network computation, and the storage layer includes DRAM memory. The computation layer is used for computation at least, and is stacked with the storage layer via hybrid bonding, and the computation layer is electrically connected to the storage layer to realize data interaction between the storage layer and the computation layer. The computation layer includes an in-memory computing-based neural network accelerator (i.e., the NPU acceleration chip in the figure) for neural network computation. The computation layer can include multiple sub-NPU clusters. The neural network accelerator includes an in-memory computation matrix, which is used to perform neural network computation on the input feature data and the weights from the storage layer. The computing layer is also used to electrically connect to the main control chip that controls the inference device. The computing layer is also used to perform pre-filling processing for large language model inference and to transmit the pre-filled data to the main control chip for decoding processing of large language model inference, so that the pre-filling processing and the decoding processing are separated.

[0036] The storage layer and computing layer provided by this invention can take many forms, such as a chip or a wafer.

[0037] The neural network accelerator included in the computing layer of this invention achieves in-memory computing through an in-memory computing matrix, enabling neural network computation by combining input feature data and weights from the storage layer. In-memory computing is an innovative architecture designed to overcome the "memory wall" problem by integrating storage and computing functions onto a single chip. This technology embeds computing power into memory and employs a new operational architecture to perform matrix-vector multiplication and accumulation operations, thus reducing the frequent data transfer between memory and computing chips as in existing technologies. This significantly reduces transfer time, power consumption, and heat generation, and improves the efficiency of parallel data processing; it achieves in-situ computation, eliminates bandwidth limitations, and reduces data movement costs. This fundamentally eliminates unnecessary data transfer latency and power consumption, improving AI computing efficiency by hundreds or thousands of times, reducing costs, and breaking down the "memory wall" and "power wall." Compared to existing devices performing computations under the same conditions, the computing layer and the large language model inference device can have greater computing power, consume less power, and generate less heat, thus addressing existing heat dissipation problems at their source.

[0038] Furthermore, this invention achieves a 3D stacked structure of the storage and compute layers in an LLM inference device through hybrid bonding. Hybrid bonding enables high-density, high-performance interconnects between different chips. Through hybrid bonding, the metal layers (usually copper layers) of two or more chips can be precisely aligned and directly pressed together to form direct electrical contacts. It can achieve sub-micron or even nanometer-level interconnect spacing, allowing more connection points to be placed in a smaller area, significantly increasing the data communication bandwidth between chips. Its interconnect density can reach 10K-1MM / mm². Because hybrid bonding eliminates intermediate media such as solder, the direct copper-to-copper connection has lower resistance, reducing energy loss during signal transmission and also reducing signal propagation time delay. The compact structure and direct conductive path formed by hybrid bonding help improve thermal management and reduce heat generation, thus enabling higher integration, faster data transmission speeds, greater bandwidth, and lower power consumption.

[0039] Furthermore, the inference device provided by this utility model combines the advantage of its high computing power based on in-memory computing layer, and separates prefill and decoding. That is, the prefill that requires high computing power is completed in the computing layer, and the data processed by the computing layer through prefill is transmitted to the main control chip electrically connected to it. The main control chip decodes the prefilled data, thereby reducing the bandwidth requirement of the inference device, enabling the inference device to perform large language model inference work faster and better, and enabling the large language model inference device with the same computing power to support greater bandwidth for large language model inference work.

[0040] In summary, the inference device provided by this utility model can not only solve the existing bandwidth bottleneck to support large bandwidth and large computing power, but also reduce power consumption in many ways and solve the existing heat dissipation problem.

[0041] Reference Figure 2 The illustrated embodiment of the neural network accelerator includes: a preprocessing module, an in-memory computation matrix, and a vector processing module, all electrically connected in sequence; and a shared memory electrically connected to the preprocessing module, the in-memory computation matrix, and the vector processing module, respectively; it also includes a chip controller electrically connected to the shared memory. The NPU is also electrically connected to an external memory, and in this invention, the aforementioned storage layer is... Figure 2The external memory is shown. Data stored in the external memory is transferred to the shared memory of the NPU in batches. For example, weights are read from the external memory in batches to the shared memory. One working mode is briefly described as follows: The preprocessing module obtains input feature data from the shared memory, processes the input feature data into matrix data in the format of an in-memory computation matrix, and inputs the matrix data into the in-memory computation matrix; wherein, if the input feature data includes at least two input feature maps, and the at least two input feature maps are stored in the shared memory in a skip address manner, then the preprocessing module obtains the at least two input feature maps from the shared memory in address-order reading; or, if the input feature data includes at least two input feature maps, and the at least two input feature maps are stored in the shared memory in a contiguous storage manner, then the preprocessing module obtains the at least two input feature maps from the shared memory in a skip address reading manner; weight data is obtained through the in-memory computation matrix, and through the... The in-memory computation matrix performs convolution calculations based on the matrix data and the weight data to obtain the computation result, and inputs the computation result into the vector processing module; the vector processing module performs vector processing on the computation result to obtain output feature map data, and writes the obtained output feature map data into the shared memory and / or external memory; wherein, if the output feature map data is an intermediate computation result, the output feature map data is used as input data for the next level of computation result; before the preprocessing module obtains the input feature data, and / or after the vector processing module processes the computation result to obtain the output feature map data, the chip controller controls the data interaction between the shared memory and the external memory, wherein the data interacted includes the input feature data and / or the output feature map data.

[0042] In a preferred embodiment of this utility model, the in-memory calculation matrix includes: N columns of storage operator modules, each column of storage operator module includes: M rows of storage units for storing and calculating the stored data with the input feature data, and capacitors electrically connected to the storage units, the charges output by the capacitors in the M rows are pooled together to achieve charge accumulation.

[0043] See Figure 3The diagram shown illustrates a module of a column of in-memory operator submodules within an in-memory computation matrix in one embodiment of this invention. In this preferred embodiment, the in-memory computation unit can store and perform multiplication calculations on weights and input feature data. The corresponding product is accumulated through a capacitor connected to the in-memory computation unit to achieve multiply-accumulate (MAC) calculations in neural network computation. That is, matrix calculations can be performed within the in-memory computation matrix, avoiding the frequent data transfer between two different and geographically distant memories and computing chips as in existing technologies. The power consumption of multiply-accumulate calculations typically accounts for a large portion of the total computation power consumption, often ranging from 50% to 70%. This invention, based on an integrated in-memory computing architecture, sets up a neural network accelerator with an in-memory computation matrix in the computation layer to achieve two-dimensional and three-dimensional matrix multiplication / addition operations through a new computational architecture. This significantly reduces the power consumption of neural network computation, allows for a smaller neural network accelerator area, and enables greater computing power with less heat generation within the same area and power consumption, thus solving the heat dissipation problem.

[0044] See Figure 4 and Figure 5 As shown, Figure 4 The diagram shown is a vertical structural schematic of a column of stored-operator submodules in an in-memory computation matrix according to an embodiment of the present invention. Each stored-operator unit includes: an SRAM memory for storing weights from the storage layer (i.e., Figure 4 The system includes an SRAM memory and adjacent logic units for computation. Multiple computation units are stacked to form a column, and several columns of computation units form a computation matrix. The digital logic units are the aforementioned logic units, which can be used to perform multiplication and addition calculations on the weights and input feature data in the SRAM memory. Figure 4The diagram showcases a typical architecture of Integrated In-Memory Computing (CIMD). Specifically, it is based on SRAM memory, with model weights stored in multiple on-chip SRAM memory banks. The multiply-accumulate (MAC) computation logic required by CIMD is deployed near each SRAM memory bank to significantly reduce latency and power consumption caused by weight movement. Compared to traditional von Neumann architecture AI chips, CIMD significantly reduces power consumption during data movement while greatly increasing the speed of parallel multiply-accumulate (MAC) computation, effectively alleviating the two major challenges of the 'memory wall' and the 'power wall'. The SRAM memory and the logic units used for computation are stacked adjacent to each other, which allows for a more compact structure of the in-memory computation matrix, greater computing power, lower power consumption, and, combined with digital logic units, faster and more accurate computation. As shown in Table 2, at the 12nm process node, the matrix multiplication energy efficiency of the CIM in-memory computing unit can reach 24.7 TOPS / W, which is an order of magnitude improvement compared to traditional digital architectures (i.e., general-purpose computing units, such as GPUs and TPUs). In-memory computing (CIMD) significantly reduces the power consumption of data transfer while greatly increasing the speed of parallel multiply-accumulate (MAC) computation. Thanks to this innovative underlying computing unit architecture, in-memory computing (CIMD) technology can significantly improve the energy efficiency of edge AI large model (LLM) inference chips, meaning that while maintaining the same computing power, its power consumption requirements are greatly reduced.

[0045]

[0046]

[0047] Table 2

[0048] Figure 5 This is a schematic diagram illustrating the principle of in-memory computation matrix calculation in one embodiment of this utility model. The in-memory computation unit performs multiplication calculations on the input feature data and weights, and then accumulates the product through a multi-level addition tree to obtain the result of multiplication and accumulation.

[0049] See Figure 6In this preferred embodiment, the in-memory computation matrix includes a weight storage array 10 (also called a weight parameter storage array), a bit multiplier 12, a storage readout circuit 21, and a logic operation unit 22. One input of the bit multiplier 12 is connected to a weight readout enable signal, and the other input is connected to input feature data; the weight acquisition end is connected to the weight storage array 10. The input of the storage readout circuit 21 is connected to the weight storage array 10, and its output is connected to the logic operation unit 22. The output of the logic operation unit 22 is used to output the convolution operation result. The weight storage array 10 is used to store weights.

[0050] The bit multiplier 12 receives a weight read-enable signal and input feature data (in convolutional neural networks, the input feature map is typically processed). When the weight read-enable signal is enabled, the multiplier selects a weight from the weight storage array and multiplies the bits of the input feature data with the selected weight. The input feature data represents the output data of the previous layer during the convolution operation and may include at least one bit, where the bit value is 1 or 0. For input feature data, a bit value of 1 indicates that the input feature data is 1, and a bit value of 0 indicates that the input feature data is 0. For example, when a bit of the input feature data is 1, the product of the bit and the corresponding weight is the corresponding weight. In this case, the storage read-out circuit 21 reads the corresponding weight from the weight storage array 10 to achieve the corresponding product read-out. When a bit of the input feature data is 0, the storage read-out circuit 21 does not perform a read-out operation, and therefore, the storage read-out circuit 21 can directly read from the weight storage array 10 to achieve the corresponding product read-out. The storage readout circuit 21 is used to read the product of the multiplication to the logic operation unit 22 when the bit of the input feature data is 1, and not to perform the readout operation when the bit of the input feature data is 0, so that the storage readout circuit 21 does not generate power consumption when the bit of the input feature data is 0, thereby reducing the power consumption of the storage readout circuit 21. Specifically, the storage readout circuit 21 can read the corresponding weight to the logic operation unit 22 when the bit of the input feature data is 1. Specifically, the storage medium of the storage readout circuit 21 includes any one of SRAM, DRAM and RRAM, and the storage medium can be a volatile storage medium or a non-volatile storage medium. The logic operation unit 22 is used to accumulate the product read out by the storage readout circuit 21 when the bit of the input feature data is 1, so as to realize the multiplication and accumulation of the input feature data and weight. The above-mentioned logic operation unit 22 only needs to output the final convolution operation result, without transmitting intermediate data in the convolution operation process, which can reduce the requirements of data output volume and data transmission bandwidth. In other words, it ensures that the storage read circuit generates no power when the input feature data bits are 0, thereby reducing the power consumption of the storage read circuit. The output data of the logic operation unit is the accumulation result, eliminating the need to transmit intermediate data during the convolution operation process, thus reducing the data bit width of the output data. If more complex accumulation operations are implemented in the logic operation unit, the output data bit width can be further compressed. Therefore, this application can reduce the power consumption of reading large amounts of concurrent data and lower the requirements for data output volume and data transmission bandwidth.

[0051] For further optimization, see Figure 7As shown, the bit multiplier includes N AND gates respectively connected to the weight storage array; one input of each AND gate is used to receive the input feature data fmi, the other input is used to receive the weight read enable signal, and the output is used to output a control signal. The weight storage array 10 includes K rows and N columns of parameter storage units. Each parameter storage unit is used to store a weight or an electrical signal representing the corresponding weight. The h-th row and j-th column parameter storage unit stores the weight W(h, j), 1 ≤ h ≤ K, 1 ≤ j ≤ N, where K is the number of rows of the weight and N is the number of columns of the weight. The K rows of parameter storage units are correspondingly set on the K row bit lines; the h-th row bit line is denoted as bitline, h. The aforementioned weight read enable signal is used to enable one column of parameter storage units in the N columns of parameter storage units, that is, one weight read enable signal corresponds to one column of parameter storage units. For example, the weight read enable signal corresponding to the j-th column of parameter storage units can be denoted as wordline, j. When the corresponding weight read enable signal wordline,j is 1, the j-th column parameter storage unit outputs the electrical signal representing the weight to the corresponding bit line, so that the storage read circuit 21 can read the corresponding weight signal through each row bit line. For example, the probability of an N-bit input vector being all zeros is much lower than the probability of some bits in an N-dimensional input vector being zero. The bit-level sparsity of data is very high. However, the smaller the granularity of data sparsity, the more difficult it is to use, mainly because it requires real-time calculation and judgment on each bit of the input data. Through this preferred method, this utility model can solve the problem of data sparsity. For cases where the data sparse matrix contains a large number of zero values, it can greatly reduce resource waste, reduce unnecessary computation, and improve storage space utilization.

[0052] In a preferred embodiment of this invention, the storage readout circuit further includes an AND gate; one input terminal of the AND gate is used to receive input feature data, the other input terminal is used to receive a weight readout enable signal, and the output terminal is used to output a first control signal; for the weight readout enable signal wordline,j and the input feature data fmi,j corresponding to the j-th column, when both the weight readout enable signal wordline,j and the input feature data fmi,j are 1, the first control signal is 1; when at least one of the weight readout enable signal wordline,j and the input feature data fmi,j is 0, the first control signal is 0. Its circuit is simple and can accurately control the read operation, reducing the power consumption of the storage readout circuit and the power consumption of the large language model inference device.

[0053] In a preferred embodiment of this invention, the storage layer is also used for computation to perform the first-level computation of the neural network. The computation layer completes the neural network computation based on the result of the first-level computation. That is, simple computations can also be performed in the storage layer, such as the first multiplication based on the multiplication parameters, i.e., the first-level computation. The result of the first-level computation is transmitted to the NPU for accumulation computation to complete the neural network computation. This fully utilizes the storage layer, reduces the computational load of the computation layer, and thus reduces the power consumption of the computation layer. Moreover, after the first-level computation, the data transfer from the storage layer to the computation layer can be reduced, correspondingly reducing the demand for storage capacity and high bandwidth. The storage layer preferably includes DRAM memory, which has low cost and high density. More preferably, the storage layer includes at least two layers of DRAM memory, and the at least two layers of DRAM memory are connected via TSV (Through Silicon Vias), which allows multiple DRAM chips to be vertically stacked, significantly improving memory bandwidth and reducing power consumption; achieving large-capacity, high-bandwidth storage to meet the stringent memory requirements of high-performance computing, artificial intelligence, and other fields.

[0054] As shown in Table 3, the 3D stacked DRAM with integrated Hybrid Bonding provided by this invention can achieve a bandwidth of 1-60TB, which is a significant improvement compared to other advanced GDDR6 and HBM3E technologies. At the same time, its energy requirement for transmitting single bit of data is also greatly reduced to 0.5pJ / bit, which greatly reduces the power consumption of data transfer. It can solve the bandwidth bottleneck problem encountered in the deployment of edge AI large model (LLM) chips, and also significantly reduce the power consumption of data transfer.

[0055]

[0056]

[0057] Table 3

[0058] In summary, this invention provides a novel LLM inference device that combines 3D stacked DRAM, hybrid bonding technology, and high-efficiency SRAM in-memory computing, among other technologies. It addresses the current bottlenecks in bandwidth, power consumption, computing power, and heat dissipation encountered by edge AI large model (LLM) inference devices (which can take various forms such as chips or modules). It significantly improves memory bandwidth, reduces power consumption, and achieves large-capacity, high-bandwidth storage, meeting the stringent memory requirements of high-performance computing, artificial intelligence, and other fields. It can be widely applied to various edge devices, such as mobile phones, computers, and robots.

[0059] This utility model also provides a reasoning system (see [reference]). Figure 1 As shown), it includes any of the aforementioned large language model inference devices and a main control chip electrically connected to the large language model inference device. The main control chip is used to perform large language model inference decoding processing on the data pre-filled by the computation layer in the large language model inference device. The computation layer includes an NPU structure... Figure 1 The diagram provides a brief illustration. The inference system provided by this invention separates pre-filling and decoding processes into different devices. Furthermore, it allows the computationally intensive pre-filling process to be handled within an SRAM-based in-memory computing architecture, while the main control chip and other devices with relatively lower computing power perform the decoding. This significantly addresses the bandwidth bottleneck problem in large language model inference systems, greatly reducing power consumption and alleviating heat dissipation issues. The main control chip can be various types of chips, such as GPUs and SOCs.

[0060] This invention also provides an electronic device comprising the aforementioned large language model inference device or inference system. It can significantly reduce the power consumption and cost of electronic devices, while improving the efficiency and experience of AI inference. It can be widely used in smartphones, tablets, wearable electronic devices, smart home electronic products, and so on.

[0061] The above description is merely a preferred embodiment of the present utility model. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present utility model. Furthermore, under the teachings of the present utility model, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present utility model. Therefore, the present utility model is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present utility model.

Claims

1. A large language model reasoning device based on in-memory computing, characterized in that, include: The system includes at least a storage layer for storage and at least a computing layer for computation, wherein the computing layer is stacked with the storage layer via hybrid bonding and is electrically connected to the storage layer for data interaction. The computing layer includes a memory-based neural network accelerator, which includes an in-memory computing matrix. The in-memory computing matrix is ​​used to perform neural network calculations on the input feature data and the weights from the storage layer. The storage layer includes a DRAM memory. The computing layer is also used to be electrically connected to the main control chip that controls the inference device. The computing layer is also used to perform pre-filling processing for large language model inference and to transmit the pre-filled data to the main control chip for decoding processing of large language model inference, so that the pre-filling processing and the decoding processing are separated.

2. The large language model reasoning device according to claim 1, characterized in that, The in-memory computation matrix includes: N columns of storage operator modules, each column of storage operator module including M rows of storage units for storing and performing computations on the stored data and input feature data.

3. The large language model reasoning device according to claim 2, characterized in that, Each storage unit includes: an SRAM memory for storing weights from the storage layer, and a logic unit for calculation disposed adjacent to the SRAM memory.

4. The large language model reasoning device according to claim 1, characterized in that, The in-memory computation matrix includes: a weight storage array for storing weights, a bit multiplier, a storage readout circuit, and a logic operation unit. The bit multiplier receives a weight readout enable signal and input feature data. When the weight readout enable signal is enabled, it selects a weight from the weight storage array according to the weight readout enable signal and multiplies the bit of the input feature data with the selected weight. The storage readout circuit reads the product of the multiplication to the logic operation unit when the bit of the input feature data is 1, and does not perform a readout operation when the bit of the input feature data is 0. The logic operation unit accumulates the product read out by the storage readout circuit when the bit of the input feature data is 1, so as to realize the multiplication and accumulation of the input feature data and the weight.

5. The large language model reasoning device according to claim 4, characterized in that, The bit multiplier includes N AND gates that are respectively connected to the weight storage array; one input of each AND gate is used to receive the input feature data, the other input is used to receive the weight read enable signal, and the output is used to output a control signal.

6. The large language model reasoning device according to claim 4, characterized in that, The storage readout circuit further includes an AND gate; one input terminal of the AND gate is used to receive the input feature data, the other input terminal is used to receive the weight readout enable signal, and the output terminal is used to output the first control signal.

7. The large language model reasoning device according to any one of claims 1 to 5, characterized in that, The storage layer is also used for computation to implement the first-level computation of the neural network, and the computation layer completes the neural network computation based on the result of the first-level computation.

8. The large language model reasoning device according to any one of claims 1 to 5, characterized in that, The storage layer includes at least two DRAM memory layers, and the at least two DRAM memory layers are connected via TSV.

9. A reasoning system, characterized in that, The device includes a large language model inference apparatus as described in any one of claims 1-8, and a main control chip electrically connected to the large language model inference apparatus. The main control chip is used to perform large language model inference decoding processing on the data after the computational layer pre-filling processing in the large language model inference apparatus.

10. An electronic device, characterized in that, The large language model reasoning device includes any one of claims 1-8 or the reasoning system includes the one described in claim 9.

Images