Edge end large language model reasoning acceleration method based on core particle and near memory calculation

Abstract

The invention provides an edge-end large language model reasoning acceleration method and device based on core particle and near memory calculation, and the method comprises the steps: storing a model weight in Flash, and completing the operation of all matrix vector multiplication in the whole reasoning process through the cooperation of the Flash and NPU. And in the calculation part in the Flash, the NPU needs to send the input vector to the Flash, the weight matrix stored in the Flash and the input vector are used for matrix vector multiplication, and a result is obtained and sent back to the NPU. And the calculation part in the NPU is as follows: an input vector is stored in the NPU, a weight matrix needs to be read block by block from the Flash, and the operation is completed in the NPU. And the operation result is stored in the NPU. The attention operation is independently completed by the NPU. The special function calculation is independently completed by the NPU. And the NPU and the Flash respectively bear the task proportions, so that the NPU and the Flash can better collaboratively complete the reasoning task of the large language model.

Description

Technical Field

[0001] The present invention relates to the technical fields of in-memory computing, large language model accelerators and robot accelerators, and in particular to an edge-end large language model reasoning acceleration method based on core particles and near-memory computing. Background Art

[0002] Existing large language model (LLM) reasoning is mainly performed by NPU / GPU. During reasoning, a storage medium is required to store the weights of the large language model. Currently, the technology for storing model weights in traditional architectures can be divided into two categories.

[0003] Traditional architecture 1: Place the large language model entirely in DRAM. In this approach, the model weights are loaded into DRAM before inference begins. When inference is in progress, the model weights are moved from DRAM to the chip in blocks for computation. This technique is commonly used in modern GPUs and mobile phone SOCs.

[0004] Traditional architecture 2: Put the large language model in SSD. This method is to put the model weights in SSD with larger capacity instead of DRAM before the inference starts, so as to support the inference of larger and more complex models. When the inference is in progress, the model weights are first moved to DRAM in blocks, and then moved from DRAM to the chip in blocks for calculation. This method takes advantage of the high storage density of SSD to store huge model weights.

[0005] In addition to the above two traditional architectures, there are also some more advanced SSD-based near-memory computing technologies, which reduce memory access bottlenecks and improve computing efficiency by moving computing units closer to storage.

[0006] The near-memory computing architecture deploys the operator inside the Flash die to relieve the pressure on the Flash channel. This architecture can make the most of the parallelism of the components inside the Flash, break the bottleneck of the Flash channel, and maximize the exposure of the internal bandwidth to the operator.

[0007] The two traditional architectures and two near-memory computing architectures mentioned above have their own defects and are difficult to support the reasoning of large language models with large parameters on edge devices. Specifically, the disadvantage of traditional architecture 1 is that it cannot support large language models with large parameters. The DRAM capacity of edge devices such as mobile phones is usually 8-12GB, which is not enough to store all the parameters of large language models, limiting the capabilities of large language models. The disadvantage of traditional architecture 2 is that the reasoning speed is too slow. Traditional architecture 2 completely unloads the weights of large language models to SSDs based on Flash flash memory, so the reasoning speed is limited by the bandwidth of SSDs. Specifically, the bandwidth of SSDs is usually 4-8GB / s, and the theoretical maximum speed of reasoning INT8 quantized 70B large models is only 0.11 token / s, which is far below the standard of smooth interaction (usually 3-10 token / s). The near-memory architecture based on Flash has not yet been officially applied to large language model reasoning tasks, and it also has certain shortcomings. Specifically, the space inside the Flash die of the near-memory computing architecture is very small and cannot accommodate special function computing logic. Moreover, the near-memory computing architecture does not consider the error correction problem. Transferring the calculation to the Flash die will make the original error correction module ineffective. The high error rate of Flash may seriously affect the reasoning accuracy of large language models. Summary of the invention

[0008] The purpose of the present invention is to solve the problems of the above-mentioned various existing technologies. Traditional architecture 1 cannot provide sufficient storage space, and traditional architecture 2 cannot provide sufficient bandwidth. The near-memory architecture can alleviate the bandwidth pressure to a certain extent, but it cannot calculate special functions well and cannot provide error correction capabilities, resulting in a significant reduction in reasoning accuracy.

[0009] The present invention proposes a collaborative architecture Cambricon-LLM of NPU and Flash, including an NPU and a Flash chip with in-die computing capability. The high storage density of Flash is used to store the parameters of a large language model. The NPU can read data directly from the Flash without passing through DRAM. The Flash of Cambricon-LLM has the ability of in-die computing, which is used to calculate a part of the matrix vector multiplication to reduce the amount of data transmission on the Flash channel. Cambricon-LLM distributes all matrix operations and multiplications in large model single batch reasoning to NPU and Flash for collaborative operation, so as to make full use of all operation units to improve the reasoning efficiency. All special function operations in the large model single batch reasoning task are completed by the NPU alone, because the space in the Flash die is very limited and cannot accommodate complex special function operators. In addition, all operations related to KV cache are completed independently by the NPU. This is because the total volume of the KV cache of the large language model single batch reasoning is not large, and it needs to be written and read frequently, which is suitable for being placed in DRAM. Only the NPU can interact directly with DRAM.

[0010] In order to make the computing components on the NPU and Flash work together to improve the efficiency of large language model reasoning, the inventor proposed a hardware-aware matrix clipping and task allocation strategy. This strategy determines the matrix clipping method and the task ratio of the NPU and Flash based on the hardware parameters of the NPU and Flash, so that the two can better cooperate to complete the reasoning task of the large language model.

[0011] In order to ensure that the inference accuracy of large language models is not affected by Flash errors, the inventors proposed a lightweight in-die error correction algorithm, a matching hardware execution module and an error correction code to ensure the accuracy of large language model inference on this new architecture.

[0012] Specifically, in view of the shortcomings of existing technologies, such as Fig. 9 As shown, the present invention proposes an edge-end large language model reasoning acceleration method based on core particles and near-memory computing, which includes:

[0013] In the initial step of reasoning, a neural network model accelerator including NPU, Flash and DRAM is constructed. The Flash is located inside the NPU and the NPU and the Flash are packaged on the same substrate. A computing module is provided inside the die of the Flash, and the computing module is used to complete the reasoning calculation of the neural network model; DRAM is used to store the KV cache generated in the large model reasoning;

[0014] Near memory calculation step: all weights in the neural network model to be inferred are stored in the Flash, and all matrix-vector multiplication operations of the neural network model to be inferred are completed by the Flash and the NPU in collaboration; the part calculated in the Flash: the NPU sends the input vector to the Flash, uses the weight matrix stored in the Flash and the input vector to perform matrix-vector multiplication, and sends the matrix multiplication result back to the NPU after obtaining it; the part calculated in the NPU: the NPU reads the weight matrix from the Flash, combines the input vector in the NPU, completes the matrix-vector multiplication in the NPU, and the matrix multiplication result is stored in the NPU;

[0015] Attention calculation step: the NPU reads the KV cache from the DRAM to complete the attention calculation and obtain the attention result;

[0016] Special function calculation steps: The special function is completed independently by the NPU to obtain the special function calculation result.

[0017] In the inference end step, the NPU obtains the inference result of the neural network model to be inferred based on the matrix multiplication result, the attention result and the special function operation result.

[0018] The edge-end large language model reasoning acceleration method based on core particle and near-memory computing, wherein the computing module is provided with a computing core Compute Core for matrix-vector multiplication operations; the matrix data of the matrix-vector multiplication operation comes from the Flash, and after being read, the matrix data enters the data register and the cache register in sequence and is used by the computing core ComputeCore.

[0019] The edge-end large language model reasoning acceleration method based on core particles and near-memory computing, wherein the near-memory computing step includes: a segmentation step and an allocation step, so as to determine the proportion of matrix multiplication tasks allocated on the NPU and the Flash according to the hardware information of the NPU and the Flash;

[0020] The segmentation step includes:

[0021] The weight matrix (H_weight×W_weight) is divided into matrix blocks (H_req×W_req);

[0022] H_req*=√(ccore_num×pagesize);

[0023] W_req*=channel_num×√(ccore_num×pagesize);

[0024] Where H_weight and W_weight are the length and width of the weight matrix, H_req and W_req are the length and width of the matrix block, ccore_num is the total number of computing cores connected to each Flash channel, pagesize is the page size, and channel_num is the total number of Flash channels;

[0025] The allocation steps include:

[0026] According to the execution time of the read calculation request and the read request, the proportion α of the matrix multiplication tasks allocated to the NPU and the Flash is determined respectively, and the matrix blocks equal to the proportion α of the matrix multiplication tasks are delivered to the Flash to complete the calculation of the matrix-vector multiplication;

[0027] α=t_r / (t_r+t_rc);

[0028] t_rc=t_R+W_req / (channel_num×bw_channel);

[0029] t_r=pagesize / ((1-rate_rc)×bw_channel);

[0030] rate_rc=(H_req+W_req / channel_num) / (t_R×bw_channel);

[0031] Where the execution time t_rc is the sum of the input data transfer time through the flash channel and the read operation delay from the flash storage array, t_R is the read operation time, bw_channel is the channel bandwidth, rate_rc is the channel utilization of the read calculation request to the Flash, and t_r is the execution time of the read request.

[0032] The edge-end large language model reasoning acceleration method based on core particles and near-memory computing, wherein the matrix data passes through the error correction module after passing through the cache register and then is used by the computing core Compute Core;

[0033] The error correction module includes:

[0034] The data values ​​and positions that exceed the threshold value stored in the Flash data are recorded and backed up to obtain an error correction code; for each piece of data from the Flash, determine whether the current input data belongs to the protected range according to the position stored in the error correction code; if so, multiple backups stored in the error correction code and the input data itself vote together to obtain the output result; otherwise, determine whether the input data exceeds the threshold value. If not, the input data will be directly output. If so, the input data is determined to be abnormal, discarded, and directly output 0.

[0035] like Fig.10 As shown, the present invention also proposes an edge-end large language model reasoning acceleration device based on core particles and near-memory computing, which includes:

[0036] The inference initialization module builds a neural network model accelerator including NPU, Flash and DRAM. The Flash is located in the NPU and the NPU and the Flash are packaged on the same substrate. A computing module is provided inside the die of the Flash, and the computing module is used to complete the inference calculation of the neural network model; DRAM is used to store the KV cache generated in the large model inference;

[0037] The near-storage computing module stores all weights in the neural network model to be inferred into the Flash, and all matrix-vector multiplication operations of the neural network model to be inferred are completed by the Flash and the NPU in collaboration; the part calculated in the Flash: the NPU sends the input vector to the Flash, uses the weight matrix stored in the Flash and the input vector to perform matrix-vector multiplication, and sends the matrix multiplication result back to the NPU after obtaining it; the part calculated in the NPU: the NPU reads the weight matrix from the Flash, combines it with the input vector in the NPU, completes the matrix-vector multiplication in the NPU, and the matrix multiplication result is stored in the NPU;

[0038] Attention calculation module, the NPU reads the KV cache from the DRAM to complete the attention calculation and obtain the attention result;

[0039] Special function calculation module: the special function is completed independently by the NPU to obtain the special function calculation result.

[0040] The inference end module, the NPU obtains the inference result of the neural network model to be inferred according to the matrix multiplication result, the attention result and the special function operation result.

[0041] The edge-end large language model inference acceleration device based on core particle and near-memory computing, wherein the computing module is provided with a computing core Compute Core for matrix-vector multiplication operation; the matrix data of the matrix-vector multiplication operation comes from the Flash, and after being read, the matrix data enters the data register and the cache register in sequence and is used by the computing core ComputeCore.

[0042] The edge-end large language model reasoning acceleration device based on core particle and near memory computing, wherein the near memory computing module includes: a segmentation module and an allocation module to determine the proportion of matrix multiplication tasks allocated to the NPU and the Flash according to the hardware information of the NPU and the Flash;

[0043] The segmentation module includes:

[0044] The weight matrix (H_weight×W_weight) is divided into matrix blocks (H_req×W_req);

[0045] H_req*=√(ccore_num×pagesize);

[0046] W_req*=channel_num×√(ccore_num×pagesize);

[0047] Where H_weight and W_weight are the length and width of the weight matrix, H_req and W_req are the length and width of the matrix block, ccore_num is the total number of computing cores connected to each Flash channel, pagesize is the page size, and channel_num is the total number of Flash channels;

[0048] The allocation module includes:

[0049] According to the execution time of the read calculation request and the read request, the proportion α of the matrix multiplication tasks allocated to the NPU and the Flash is determined respectively, and the matrix blocks equal to the proportion α of the matrix multiplication tasks are delivered to the Flash to complete the calculation of the matrix-vector multiplication;

[0050] α=t_r / (t_r+t_rc);

[0051] t_rc=t_R+W_req / (channel_num×bw_channel);

[0052] t_r=pagesize / ((1-rate_rc)×bw_channel);

[0053] rate_rc=(H_req+W_req / channel_num) / (t_R×bw_channel);

[0054] Where the execution time t_rc is the sum of the input data transfer time through the flash channel and the read operation delay from the flash storage array, t_R is the read operation time, bw_channel is the channel bandwidth, rate_rc is the channel utilization rate of the read calculation request to the Flash, and t_r is the execution time of the read request;

[0055] After passing through the cache register, the matrix data first passes through the error correction module and then is used by the Compute Core;

[0056] The error correction module includes:

[0057] The data values ​​and positions that exceed the threshold value stored in the Flash data are recorded and backed up to obtain an error correction code; for each piece of data from the Flash, determine whether the current input data belongs to the protected range according to the position stored in the error correction code; if so, multiple backups stored in the error correction code and the input data itself vote together to obtain the output result; otherwise, determine whether the input data exceeds the threshold value. If not, the input data will be directly output. If so, the input data is determined to be abnormal, discarded, and directly output 0.

[0058] The present invention also proposes an electronic device, which includes the aforementioned edge-end large language model inference acceleration device, and the electronic device is connected to an information display device, which is used to display the inference result according to display parameters and attributes set by the user or through an artificial intelligence model.

[0059] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the edge large language model reasoning acceleration method are implemented.

[0060] The present invention also proposes a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the edge large language model reasoning acceleration method are implemented.

[0061] It can be seen from the above scheme that the advantages of the present invention are:

[0062] The present invention proposes a collaborative architecture of NPU and Flash, Cambricon-LLM, which includes an NPU and a Flash chip with in-die computing capability. Compared with traditional architecture 1 and traditional architecture 2, the reasoning efficiency of Cambricon-LLM is improved by 22.1 times. The experiment provides three Cambrion-LLM architectures with different resource configurations as shown in Table 1, two traditional architectures 1 (Flexgen-DRAM, MLC-LLM) as shown in Table 2, and one traditional architecture 2 (Flexgen-SSD) as shown in Table 2.

[0063] Table 1 Specific configurations of three different Cambricon-LLM architectures

[0064]

[0065] Table 2 Specific configurations of two traditional architectures

[0066]

[0067] The calculation unit and error correction module introduced by Cambricon-LLM do not bring about a large area and power consumption overhead. As shown in Table 4, the area and power consumption overhead of Cambricon-LLM are evaluated under the 65nm process, and the area overhead is controlled at 1.2% and the power consumption overhead is controlled at 4.5%.

[0068] The Cambricon-LLM architecture reduces the amount of data moved by 9.7-11.6 times and reduces the energy consumption to 67% of the traditional architecture2. Figure 2 shown.

[0069] Table 3 Area and power consumption of Cambricon-LLM

[0070] <![CDATA[Area(um 2 )]]> Power(uw) Error Correction Unit 496.4 0.4 PEs 562.0 343.6 Input B buffer and Output B buffer 58755.1 1591.7 Total Compute Core 39813.5 1935.6 Overhead 1.2％ 4.5％

[0071] The present invention uses a hardware-aware matrix clipping and task allocation strategy to determine the clipping method during matrix-vector multiplication operations and the task ratio of NPU and Flash according to the performance parameters of NPU and Flash, so that the two can better cooperate to complete the reasoning task of large language models. Compared with letting Flash alone be responsible for all matrix-vector multiplication operations, this collaborative technology can increase the reasoning speed by 1.3-1.4 times and increase the Flash channel utilization by 76.2%. Figure 2 shown.

[0072] The present invention uses an error correction algorithm for near-memory computing scenarios on Flash and its corresponding hardware implementation. The error correction algorithm can eliminate the reading errors of the Flash medium to a certain extent and ensure the reasoning accuracy of the large language model. Compared with the Flash in-memory computing architecture without any error correction, this technology can increase the tolerance of the large language model to the error rate of the Flash medium by 2.3 times, and the error rate of the large language model can be reduced by 2x 10 -4 The accuracy is still 92%-95% under the error rate of Figure 3 shown.

[0073] Through experimental analysis, Cambricon-LLM can achieve 22.1 times acceleration in large language model edge inference tasks compared to traditional architectures. Compared to in-memory computing architectures, Cambricon-LLM can fully support all operations of large language model inference and provide 2.3 times the error protection capability. Figure 1 and Figure 4 shown. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a performance diagram of Cambricon-LLM;

[0075] Figure 2Schematic diagram of data handling and energy consumption of Cambricon-LLM compared with traditional architecture;

[0076] Figure 3 Schematic diagram of the impact of matrix pruning and task allocation strategies on Cambricon-LLM performance;

[0077] Figure 4 Schematic diagram of the error correction effect of Cambricon-LLM on three different data sets;

[0078] Figure 5 This is the architecture diagram of Cambricon-LLM;

[0079] Figure 6 The computational flow for large language model inference and the mapping diagram on Cambricon-LLM hardware;

[0080] Figure 7 Schematic diagram of hardware-aware matrix pruning and task allocation strategy;

[0081] Figure 8 Design diagrams for the error correction code and hardware units of Cambricon-LLM;

[0082] Fig. 9 is a flow chart of the method of the present invention;

[0083] Fig.10 It is a module diagram of the device of the present invention;

[0084] Fig.11 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0085] Fig.12 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0086] Fig.13 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0087] Reference numerals:

[0088] A-First electronic device;

[0089] B-Edge large language model reasoning acceleration device;

[0090] C-data acquisition equipment;

[0091] D-information display device;

[0092] 1000 - second electronic device;

[0093] Ⅰ-computational unit;

[0094] II-ROM;

[0095] III-RAM;

[0096] IV-bus;

[0097] V-interface;

[0098] VI - input unit;

[0099] VII-output unit;

[0100] VIII- Storage medium;

[0101] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0102] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0103] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0104] The processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0105] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0106] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0107] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0108] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0109] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0110] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0111] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0112] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0113] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0116] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0117] When the inventor was conducting research on the deployment of large language model reasoning at the edge, he found that the existing technology and architecture could not support this task well. The reason for this result is that the large language model has a large number of parameters and requires a large storage space. And when performing single batch reasoning at the edge, the computing access memory ratio is very low, resulting in a large bandwidth requirement for this task. Traditional architecture 1 cannot provide enough storage space, and traditional architecture 2 cannot provide enough bandwidth. The near memory architecture can alleviate the bandwidth problem to a certain extent, but it cannot calculate special functions well and cannot provide error correction capabilities, resulting in a significant reduction in reasoning accuracy. Therefore, traditional technologies and architectures cannot support this task well, and a completely new architecture needs to be designed.

[0118] After analyzing the single batch reasoning of a large language model, the inventor found that a collaborative architecture of NPU and Flash can be designed to solve this problem. The two chips are packaged on the same substrate through core particle technology. Collaborative computing is performed. The high storage density of Flash is used to store the parameters of the large language model. The NPU can read data directly from the Flash without going through DRAM. This design based on core particle technology has the characteristics of high integration and low latency, and at the same time, the Flash bandwidth is no longer limited by the PCIE protocol, which is suitable for edge LLM reasoning scenarios. The Flash of the new architecture of the present invention has the ability to calculate within the die, which is used to calculate a part of the matrix vector multiplication to reduce the amount of data transmission on the Flash channel. The NPU is connected to an external 200MB DRAM to store the key-value kv cache.

[0119] In order to enable the computing components on the NPU and Flash to work together to improve the efficiency of large language model reasoning, the present invention proposes a hardware-aware matrix clipping and task allocation strategy. This strategy determines the matrix clipping method and the task ratio of the NPU and Flash based on the hardware parameters of the NPU and Flash, so that the two can better collaborate to complete the reasoning task of the large language model.

[0120] In order to ensure that the inference accuracy of large language models is not affected by Flash errors, the inventors proposed a lightweight in-die error correction algorithm, a matching hardware execution module and an error correction code to ensure the accuracy of large language model inference on this new architecture.

[0121] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0122] The present invention proposes an accelerator Cambricon-LLM for edge reasoning tasks of large language models, whose architecture is as follows Figure 5 As shown. The main structure includes NPU, Flash and DRAM. The NPU integrates Flash controller and DRAM controller, which can communicate directly with Flash and DRAM. The computing modules (PE) are distributed inside the die of Flash on the NPU, and work together to complete the calculations required for LLM reasoning. There is a large enough buffer on the NPU to buffer data from DRAM and Flash and store the intermediate results of LLM reasoning. The NPU is also equipped with a Special Function Unit (SFU) for special function operations. The intermediate variables involved in special function operations are small and can all be saved in the on-chip cache of the NPU. The calculation results will continue to be used by subsequent operations and will not be stored in the memory DRAM and flash Flash. In addition, Cambricon-LLM's Flash has on-die processing capabilities. In addition to conventional read requests, it also supports read calculation requests. Read calculation requests can complete matrix-vector multiplication calculations inside the Flash die. In actual use, the weight matrix is ​​stored in the Flash medium, the NPU sends the input vector to the die, completes the matrix-vector multiplication on the compute core in the die, and transmits the result vector to the NPU.

[0123] The present invention designs the detailed structure of the computing core to support both computing functions and normal reading and writing functions. Figure 5 (b) is shown. Compute Core can complete the operation of matrix multiplication by vector, where the matrix data comes from the Flash medium, and after being read, it enters the data register Data Register and the cache register Cache Register in turn and is then used by the Compute Core. The function of the register is to cache data, and the entire process of reading data from Flash is divided into multiple stages to form a pipeline. Improve the overall reading efficiency. The first stage: read from the Flash medium to the Data Register; the second stage: read from the Data Register to the Cache Register; the third stage: read from the Cache Register to the outside world.

[0124] Since the data of NAND Flash MemoryArray is read in pages, the matrix size of each calculation is equal to the page size. The vector data is transmitted from the outside through the Flash channel and will be temporarily stored in the input buffer until the calculation is completed. The result vector of each matrix-vector multiplication calculation will be temporarily stored in the output buffer until the calculation is completed and then transmitted along the Flash channel.

[0125] In order to prevent the Flash medium from affecting the LLM reasoning results due to errors, the present invention also designs a lightweight die error correction module. The data read from the Cache Register will first pass through the error correction module before entering the PE calculation. The computing power of the PE in each ComputeCore needs to match the reading speed of the Flash. For example, the time tR=20us required for a Flash to read 16KB of int8 weights. In order to make the calculation speed match the reading speed and avoid path waiting due to insufficient computing power, the PE is required to complete 32KOPS operations within 20us. The corresponding PE computing power is 1.6TOPS. Approximately 2 multipliers are required.

[0126] When performing large model edge inference tasks, the model weight matrix requires a large space, so it is stored in Flash. The edge inference batch is 1, the total size of the KV cache is not large, and frequent data exchange with the NPU is required, so it is stored in DRAM. The three types of operations designed in the inference process are as follows: Figure 6 As shown in the figure, first, the matrix-vector multiplication operations involving weights are completed by the NPU and Flash in collaboration, including the calculation of QKV vectors, the calculation of the projection layer, the calculation of the fully connected layer, etc. Figure 6As shown. Secondly, the attention operation that needs to read the KV cache is completely completed by the NPU. Because the KV cache is stored in DRAM, the NPU can directly read data from DRAM to complete the calculation. Furthermore, the operation of special functions is completely completed by the NPU, because only the NPU has a special function processing module. Special functions are verb functions that all GPUs / CPUs / NPUs must support, such as sin, cos, exponential operations, and logarithmic operations.

[0127] The present invention designs a hardware-aware matrix clipping and task allocation strategy, which is specifically divided into two steps: 1. determining the most appropriate matrix segmentation shape according to hardware information; 2. determining the proportion of matrix multiplication vector tasks allocated on NPU and Flash according to hardware information.

[0128] 1. Determine the matrix segmentation shape.

[0129] During LLM inference, the entire weight matrix (H_weight×W_weight) is split into smaller blocks (H_req×W_req). ccore_num is the total number of compute cores connected to each channel. The total data transfer through the flash channel is equal to the size of the input vector plus the size of the result vector. Assuming there are channel_num×ccore_num compute cores on the flash, the data transfer is:

[0130] Trans=ccore_num×W_req+channel_num×H_req

[0131] Where channel_num is the number of channels contained in the flash memory.

[0132] However, the four cores on the same channel (e.g. core 0 / 1 / 2 / 3) share the same input vector. Therefore, a broadcast operation can be used to send the input vector to all computing cores on the same channel at the same time. This reduces the total amount of transmitted data to:

[0133] Trans=W_req+channel_num×H_req

[0134] Given that the size of each atomic block matches the page size, the following relationship is established under INT8 quantization:

[0135] H_req×W_req=channel_num×ccore_num×pagesize

[0136] Where pagesize is the page size, which is the smallest unit of single read of flash memory. Common page sizes are: 4 / 8 / 16 / 32KB, etc.

[0137] Therefore, the formula for Trans represents a standard arithmetic-geometric mean inequality that allows us to calculate the minimum data transfer size:

[0138] min{Trans}=2×√(W_req×H_req×channel_num)

[0139] =2×channel_num×√(ccore_num×pagesize)

[0140] The length and width of the corresponding matrix segmentation are:

[0141] H_req*=√(ccore_num×pagesize)

[0142] W_req*=channel_num×√(ccore_num×pagesize)

[0143] In the formula, √ is the square root.

[0144] 2. Determine the task allocation ratio.

[0145] The main goal of determining the task allocation ratio is to balance the execution time of the flash memory and the NPU. To achieve this goal, we first estimate the execution time of a single read compute request and a single read request, and then determine the ratio that can balance the total execution time of all read compute requests and read requests. For a read compute request, ignoring the pipeline startup delay, the execution time t_rc is the sum of the input data transfer time through the flash channel and the read operation delay from the flash storage array:

[0146] t_rc=t_R+W_req / (channel_num×bw_channel)

[0147] Where t_R is the read operation time, that is, the time required to read a page from the Flash medium to the Data Register, W_req is the block width, channel_num is the number of channels, and bw_channel is the channel bandwidth.

[0148] The Flash channel utilization of read calculation requests is:

[0149] rate_rc=(H_req+W_req / channel_num) / (t_R×bw_channel)

[0150] The period during which the read computation request does not occupy the channel can be used to transmit the read request. The execution time (t_r) of the read request can be estimated as:

[0151] t_r=pagesize / ((1-rate_rc)×bw_channel)

[0152] The optimal workload distribution is achieved when the execution time of read and read-compute requests is equal. Let α be the proportion of read-compute workload, then:

[0153] α=t_r / (t_r+t_rc)

[0154] α determines the proportion of tasks assigned to Flash, and its value range is [0,1]. For example, if there are 20 blocks in total and alpha is 0.4, then 8 blocks will be assigned to NPU. Figure 7 In (a), a total of 8 blocks are assigned to Flash, and the remaining weight matrices are transferred through the Flash channel and processed on the NPU.

[0155] The present invention designs a lightweight error correction algorithm, an error correction code and a corresponding hardware unit. The error correction algorithm is mainly implemented in two aspects: 1. Preventing the original large value from being changed due to a Flash error. 2. Preventing the original small value from being flipped into a large value due to a Flash error. For the first aspect, we will find the top 1% of the large values ​​in the entire page and record their position coordinates and values ​​in the error correction code to achieve protection. The value will be recorded in N copies (N is an even number) in the error correction code. When decoding, the recorded N copies of the value will be voted bit by bit together with the original data, and the majority is the final decoded value. For example, the value of a certain position is 66, that is, 0100_0010. When N=2, the position of the value and the original value twice will be recorded in the error correction code. For example, when the three values ​​are 0110_0010, 0100_0011, and 0100_0010, the final decoded value is 0100_0010, thereby achieving data protection. For the second aspect, we will record the largest value of the first 1% of the entire page and the smallest of these large values ​​as the threshold. If there is data larger than the threshold but not recorded as a large value by the error correction code during decoding, it means that this "large value" is caused by a single bit flip error in the Flash. In order to prevent these pseudo-large values ​​from affecting the accuracy of the large model, we will truncate these values ​​to 0 after decoding.

[0156] The present invention provides a specific organizational format of the error correction code such as Figure 8(a) is shown. The header of the entire error correction code is used to store the threshold. This value has only one copy (8 bits) for the entire page and is very important, so multiple copies (for example, 9 copies) are stored to ensure its correctness. The large value information to be protected in the entire page is stored after the threshold, and the number is 1% of the entire page. For example, for a 16KB page, when the weight format is INT8, a page has 16384 data, so a total of 163 large values ​​are protected. For each protected large value, its address is stored and two copies of the data are copied. When each page has 16384 elements, a total of 14 bits of address are required to describe the address of each element. Since the address itself may also be wrong, each address is equipped with a 5-bit error correction code using the Hamming coding format. This coding format has low hardware decoding overhead and can be efficiently completed in the die. When a 1-bit error occurs in the address, it will be corrected by the decoder on the die. When a 2-bit error occurs in the address, the large value will be abandoned. The total size of each page error correction code is 8x9+(14+5+8x2)*163=722B, which can be stored in the free area of ​​the Flash.

[0157] The present invention provides a specific logic design of the die error correction module as follows: Figure 8 (b) As shown. For each piece of data from the Flash medium, our in-die error correction module will determine whether the current input data belongs to the protected range based on the address stored in the error correction code. If it is found to be protected data, the two backups stored in the error correction code and the input data itself will vote together to obtain the output result. If it is found that the data is not protected, it will be determined whether the input data exceeds the maximum value threshold stored in the error correction code. If it does not exceed, the input data will be output directly. If it exceeds, it can be determined that the input data is an abnormal value caused by a single-bit flip error in the Flash, and it will be discarded and directly output as 0.

[0158] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0159] like Fig.10 As shown, the present invention also proposes an edge-end large language model reasoning acceleration device based on core particles and near-memory computing, which includes:

[0160] The inference initialization module builds a neural network model accelerator including NPU, Flash and DRAM. The Flash is located in the NPU and the NPU and the Flash are packaged on the same substrate. A computing module is provided inside the die of the Flash, and the computing module is used to complete the inference calculation of the neural network model; DRAM is used to store the KV cache generated in the large model inference;

[0161] The near-storage computing module stores all weights in the neural network model to be inferred into the Flash, and all matrix-vector multiplication operations of the neural network model to be inferred are completed by the Flash and the NPU in collaboration; the part calculated in the Flash: the NPU sends the input vector to the Flash, uses the weight matrix stored in the Flash and the input vector to perform matrix-vector multiplication, and sends the matrix multiplication result back to the NPU after obtaining it; the part calculated in the NPU: the NPU reads the weight matrix from the Flash, combines it with the input vector in the NPU, completes the matrix-vector multiplication in the NPU, and the matrix multiplication result is stored in the NPU;

[0162] Attention calculation module, the NPU reads the KV cache from the DRAM to complete the attention calculation and obtain the attention result;

[0163] Special function calculation module: the special function is completed independently by the NPU to obtain the special function calculation result.

[0164] The inference end module, the NPU obtains the inference result of the neural network model to be inferred according to the matrix multiplication result, the attention result and the special function operation result.

[0165] The edge-end large language model inference acceleration device based on core particle and near-memory computing, wherein the computing module is provided with a computing core Compute Core for matrix-vector multiplication operation; the matrix data of the matrix-vector multiplication operation comes from the Flash, and after being read, the matrix data enters the data register and the cache register in sequence and is used by the computing core ComputeCore.

[0166] The edge-end large language model reasoning acceleration device based on core particle and near memory computing, wherein the near memory computing module includes: a segmentation module and an allocation module to determine the proportion of matrix multiplication tasks allocated to the NPU and the Flash according to the hardware information of the NPU and the Flash;

[0167] The segmentation module includes:

[0168] The weight matrix (H_weight×W_weight) is divided into matrix blocks (H_req×W_req);

[0169] H_req*=√(ccore_num×pagesize);

[0170] W_req*=channel_num×√(ccore_num×pagesize);

[0171] Where H_weight and W_weight are the length and width of the weight matrix, H_req and W_req are the length and width of the matrix block, ccore_num is the total number of computing cores connected to each Flash channel, pagesize is the page size, and channel_num is the total number of Flash channels;

[0172] The allocation module includes:

[0173] According to the execution time of the read calculation request and the read request, the proportion α of the matrix multiplication tasks allocated to the NPU and the Flash is determined respectively, and the matrix blocks equal to the proportion α of the matrix multiplication tasks are delivered to the Flash to complete the calculation of the matrix-vector multiplication;

[0174] α=t_r / (t_r+t_rc);

[0175] t_rc=t_R+W_req / (channel_num×bw_channel);

[0176] t_r=pagesize / ((1-rate_rc)×bw_channel);

[0177] rate_rc=(H_req+W_req / channel_num) / (t_R×bw_channel);

[0178] Where the execution time t_rc is the sum of the input data transfer time through the flash channel and the read operation delay from the flash storage array, t_R is the read operation time, bw_channel is the channel bandwidth, rate_rc is the channel utilization rate of the read calculation request to the Flash, and t_r is the execution time of the read request;

[0179] After passing through the cache register, the matrix data first passes through the error correction module and then is used by the Compute Core;

[0180] The error correction module includes:

[0181] The data values ​​and positions that exceed the threshold value stored in the Flash data are recorded and backed up to obtain an error correction code; for each piece of data from the Flash, determine whether the current input data belongs to the protected range according to the position stored in the error correction code; if so, multiple backups stored in the error correction code and the input data itself vote together to obtain the output result; otherwise, determine whether the input data exceeds the threshold value. If not, the input data will be directly output. If so, the input data is determined to be abnormal, discarded, and directly output 0.

[0182] like Fig.11 As shown, the present invention further proposes a first electronic device A in another embodiment, which includes the aforementioned edge-end large language model inference acceleration device.

[0183] like Fig.12 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect the neural network model to be inferred or the neural network calculation task, and the information display device D is used to display the reasoning results obtained by the analysis of the present invention.

[0184] The information display device D can sort and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, such as the key information in the reasoning result, etc. The user can understand this information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0185] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the edge large language model reasoning acceleration method provided by the above methods.

[0186] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the edge-end large language model reasoning acceleration method. It should be understood that the storage medium in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0187] Fig.13 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same or different from the first electronic device A.

[0188] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0189] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0190] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S5. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or a communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0191] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A method for accelerating reasoning of large language models at the edge based on core particles and near-memory computing, characterized in that: include: In the initial step of reasoning, a neural network model accelerator including NPU, Flash and DRAM is constructed. The Flash is located inside the NPU and the NPU and the Flash are packaged on the same substrate. A computing module is provided inside the die of the Flash, and the computing module is used to complete the reasoning calculation of the neural network model; DRAM is used to store the KV cache generated in the large model reasoning; In the near-storage computing step, all weights in the neural network model to be inferred are stored in the Flash, and all matrix-vector multiplication operations of the neural network model to be inferred are completed by the Flash and the NPU in collaboration; Part calculated in Flash: The NPU sends the input vector to the Flash, uses the weight matrix stored in the Flash and the input vector to perform matrix-vector multiplication, and sends the matrix multiplication result back to the NPU after obtaining it; Part calculated in the NPU: The NPU reads the weight matrix from the Flash, combines it with the input vector in the NPU, completes the matrix-vector multiplication in the NPU, and the matrix multiplication result is stored in the NPU; Attention calculation step: the NPU reads the KV cache from the DRAM to complete the attention calculation and obtain the attention result; Special function calculation steps: The special function is completed independently by the NPU to obtain the special function calculation result. In the inference end step, the NPU obtains the inference result of the neural network model to be inferred based on the matrix multiplication result, the attention result and the special function operation result.

2. The edge-end large language model reasoning acceleration method based on core particles and near-memory computing as claimed in claim 1, characterized in that: The computing module is provided with a computing core Compute Core for matrix-vector multiplication operation; the matrix data of the matrix-vector multiplication operation comes from the Flash, and after being read, the matrix data enters the data register and the cache register in sequence and is used by the computing core Compute Core.

3. The edge-end large language model reasoning acceleration method based on core particles and near-memory computing as claimed in claim 2, characterized in that: The near memory calculation step includes: a segmentation step and an allocation step, so as to determine the proportion of matrix multiplication tasks allocated to the NPU and the Flash according to the hardware information of the NPU and the Flash; The segmentation step includes: The weight matrix (H_weight×W_weight) is divided into matrix blocks (H_req×W_req); H_req*=√(ccore_num×pagesize); W_req*=channel_num×√(ccore_num×pagesize); Where H_weight and W_weight are the length and width of the weight matrix, H_req and W_req are the length and width of the matrix block, ccore_num is the total number of computing cores connected to each Flash channel, pagesize is the page size, and channel_num is the total number of Flash channels; The allocation steps include: According to the execution time of the read calculation request and the read request, the proportion α of the matrix multiplication tasks allocated to the NPU and the Flash is determined respectively, and the matrix blocks equal to the proportion α of the matrix multiplication tasks are delivered to the Flash to complete the calculation of the matrix-vector multiplication; α=t_r / (t_r+t_rc); t_rc=t_R+W_req / (channel_num×bw_channel); t_r=pagesize / ((1-rate_rc)×bw_channel); rate_rc=(H_req+W_req / channel_num) / (t_R×bw_channel); Where the execution time t_rc is the sum of the input data transfer time through the flash channel and the read operation delay from the flash storage array, t_R is the read operation time, bw_channel is the channel bandwidth, rate_rc is the channel utilization of the read calculation request to the Flash, and t_r is the execution time of the read request.

4. The edge-end large language model reasoning acceleration method based on core particles and near-memory computing as claimed in claim 2, characterized in that: After passing through the cache register, the matrix data first passes through the error correction module and then is used by the Compute Core; The error correction module includes: Record and back up the data value and position exceeding the threshold value in the Flash data to obtain an error correction code; For each piece of data from Flash, determine whether the current input data belongs to the protected range according to the position stored in the error correction code; if so, multiple backups stored in the error correction code and the input data itself vote together to obtain the output result; otherwise, determine whether the input data exceeds the threshold. If not, the input data will be output directly. If so, the input data is determined to be abnormal, discarded, and directly output 0.

5. An edge-end large language model reasoning acceleration device based on core particles and near-memory computing, characterized in that: include: The inference initialization module builds a neural network model accelerator including NPU, Flash and DRAM. The Flash is located in the NPU and the NPU and the Flash are packaged on the same substrate. A computing module is provided inside the die of the Flash, and the computing module is used to complete the inference calculation of the neural network model; DRAM is used to store the KV cache generated in the large model inference; The near-storage computing module stores all weights in the neural network model to be inferred into the Flash, and all matrix-vector multiplication operations of the neural network model to be inferred are completed by the Flash and the NPU in collaboration; Part calculated in Flash: The NPU sends the input vector to the Flash, uses the weight matrix stored in the Flash and the input vector to perform matrix-vector multiplication, and sends the matrix multiplication result back to the NPU after obtaining it; Part calculated in the NPU: The NPU reads the weight matrix from the Flash, combines it with the input vector in the NPU, completes the matrix-vector multiplication in the NPU, and the matrix multiplication result is stored in the NPU; Attention calculation module, the NPU reads the KV cache from the DRAM to complete the attention calculation and obtain the attention result; Special function calculation module: the special function is completed independently by the NPU to obtain the special function calculation result. The inference end module, the NPU obtains the inference result of the neural network model to be inferred according to the matrix multiplication result, the attention result and the special function operation result.

6. The edge-end large language model reasoning acceleration device based on core particles and near-memory computing as claimed in claim 5, characterized in that: The computing module is provided with a computing core Compute Core for matrix-vector multiplication operation; the matrix data of the matrix-vector multiplication operation comes from the Flash, and after being read, the matrix data enters the data register and the cache register in sequence and is used by the computing core Compute Core.

7. The edge-end large language model reasoning acceleration device based on core particles and near-memory computing as claimed in claim 6, characterized in that: The near memory computing module includes: a segmentation module and an allocation module to determine the proportion of matrix multiplication tasks allocated to the NPU and the Flash according to the hardware information of the NPU and the Flash; The segmentation module includes: The weight matrix (H_weight×W_weight) is divided into matrix blocks (H_req×W_req); H_req*=√(ccore_num×pagesize); W_req*=channel_num×√(ccore_num×pagesize); Where H_weight and W_weight are the length and width of the weight matrix, H_req and W_req are the length and width of the matrix block, ccore_num is the total number of computing cores connected to each Flash channel, pagesize is the page size, and channel_num is the total number of Flash channels; The allocation module includes: According to the execution time of the read calculation request and the read request, the proportion α of the matrix multiplication tasks allocated to the NPU and the Flash is determined respectively, and the matrix blocks equal to the proportion α of the matrix multiplication tasks are delivered to the Flash to complete the calculation of the matrix-vector multiplication; α=t_r / (t_r+t_rc); t_rc=t_R+W_req / (channel_num×bw_channel); t_r=pagesize / ((1-rate_rc)×bw_channel); rate_rc=(H_req+W_req / channel_num) / (t_R×bw_channel); Where the execution time t_rc is the sum of the input data transfer time through the flash channel and the read operation delay from the flash storage array, t_R is the read operation time, bw_channel is the channel bandwidth, rate_rc is the channel utilization rate of the read calculation request to the Flash, and t_r is the execution time of the read request; After passing through the cache register, the matrix data first passes through the error correction module and then is used by the Compute Core; The error correction module includes: The data values ​​and positions that exceed the threshold value stored in the Flash data are recorded and backed up to obtain an error correction code; for each piece of data from the Flash, determine whether the current input data belongs to the protected range according to the position stored in the error correction code; if so, multiple backups stored in the error correction code and the input data itself vote together to obtain the output result; otherwise, determine whether the input data exceeds the threshold value. If not, the input data will be directly output. If so, the input data is determined to be abnormal, discarded, and directly output 0.

8. An electronic device, characterized in that: It includes an edge-end large language model reasoning acceleration device as described in claims 5-7, and the electronic device is connected to an information display device, and the information display device is used to display the reasoning result according to the display parameters and attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the edge-end large language model reasoning acceleration method described in any one of claims 1-4.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the edge large language model reasoning acceleration method described in any one of claims 1-4 are implemented.

Images