Mobile computing device for acceleration of large language model, and operating method thereof

Abstract

A computing device for a personal computing device for acceleration of a large language model, according to one embodiment of the present invention, comprises: a flash memory including at least one plane including a flash memory cell array, a page buffer, and an in-memory computing unit; a main memory; a parallel processing processor for processing operations for the large language model; and a central processing device for executing a control program. The in-memory computing unit processes a matrix multiplication operation on weight data read from the flash memory cell array, and then received from the page buffer, and on input data received from the main memory, and outputs an operation result thereof to the main memory.

Description

Mobile computing device for accelerating large-scale language models and method of operation thereof

[0001] The present invention relates to a mobile computing device for accelerating large-scale language models and a method of operation thereof.

[0002] Recently, with the rapid advancement of AI technology, generative AI models and Large Language Models (LLMs) are being implemented to a significant level and applied to various services. Furthermore, these AI models are now running on personal computers and smartphones.

[0003] Figure 1 illustrates the LLM operation process used in a conventional computing device.

[0004] Conventional computing devices have a system architecture that includes NAND flash memory, DRAM, and NPUs / GPUs. Weight data stored in NAND flash memory is transferred to parallel processing units, such as NPUs / GPUs, via DRAM. In particular, fully connected (FC) layer operations of LLMs are processed more efficiently by NPUs / GPUs rather than general-purpose CPUs.

[0005] However, in conventional mobile systems such as smartphones or PCs, there is a limitation in that the capacity of DRAM is limited, making it impossible to store a large amount of weights for the FC layer operations of LLM entirely in DRAM. Therefore, weight data must be transferred from NAND flash memory to DRAM, then read back into a computing device such as a GPU or NPU, and then FC layer operations must be performed.

[0006] However, NAND flash memory has significantly lower memory bandwidth compared to DRAM, which consequently increases latency and makes it difficult to run LLM in mobile systems. In particular, the decoding stage, which sequentially generates tokens in LLM, requires reading weights multiple times compared to the prefill stage; this results in significantly higher memory bandwidth requirements, posing a problem that severely degrades system performance.

[0007] The present invention proposes a mobile computing device with a new structure capable of solving these conventional technical problems.

[0008] (Patent Document 1) U.S. Published Patent No. 2024-0378019 (Title of Invention: Flash-based AI Accelerator)

[0009] The present invention aims to solve the problems of the aforementioned prior art by proposing a flash memory including an in-memory processing unit internally and a processing unit for a personal computing device using the same.

[0010] However, the technical problem that this embodiment aims to solve is not limited to the technical problem described above, and other technical problems may exist.

[0011] As a technical means for achieving the technical problem described above, an arithmetic device for a personal computing device for accelerating a large-scale language model according to the first aspect of the present invention comprises: a flash memory cell array, a page buffer, and at least one plane including an in-memory arithmetic unit; a main memory; a parallel processing processor for processing operations on a large-scale language model; and a central processing unit for executing a control program. At this time, the in-memory arithmetic unit processes a matrix multiplication operation on weight data received from the page buffer after being read from the flash memory cell array and input data received from the main memory, and outputs the result of the operation to the main memory.

[0012] In addition, a flash memory having a computation function according to the second aspect of the present invention comprises at least one plane including a flash memory cell array, a page buffer, and an in-memory computation unit, and the in-memory computation unit processes a matrix multiplication operation on weight data received from the page buffer after being read from the flash memory cell array and input data received from an external main memory, and outputs the result of the operation to the main memory.

[0013] According to the configuration of the present invention, acceleration functions for large-scale language models can be implemented even on personal computing devices. Large-scale language models, such as deep learning models, require tens to hundreds of GB of parameter values, and in particular, in a personal computing device environment, these parameters are stored on a Solid State Disk (SSD). In the prefill stage, the memory bandwidth requirement is relatively low, so computation can be performed on an NPU / GPU, but in the decode stage, the memory bandwidth requirement is high, making it inefficient to perform computation on an NPU / GPU.

[0014] The present invention enables efficient computation of weights and input data through a flash memory having an in-memory computation function. In particular, while conventional methods faced the problem of consuming significant time and energy to move weight data stored in flash memory, the present invention improves computational efficiency by processing this through a computation unit within the flash memory.

[0015] Figure 1 illustrates the LLM operation process used in a conventional computing device.

[0016] FIG. 2 illustrates a computing device for a personal computing device according to one embodiment of the present invention.

[0017] FIGS. 3 and FIGS. 4 illustrate the detailed configuration of a computational device according to one embodiment of the present invention.

[0018] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0019] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" with other components interposed between them. Furthermore, when a part is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0020] Throughout this specification, when a component is described as being located "on" another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.

[0021] Preferred embodiments of the present invention will be described in detail below with reference to the attached drawings and the contents described below. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Throughout the specification, the same reference numerals indicate the same components.

[0022] FIG. 2 illustrates a computing device for a personal computing device according to one embodiment of the present invention.

[0023] The illustrated computing device (10) is for accelerating a large-scale language model and can be implemented for various types of personal computing devices such as smartphones, tablet PCs, laptops, mobile computers, or desktop PCs.

[0024] The computing device (10) of the present invention includes a flash memory (100), a main memory (200), a central processing unit (300), and a parallel processing processor (400).

[0025] Furthermore, the flash memory (100) includes a memory cell that stores weights and an in-memory operation unit that performs operations within it. The in-memory operation unit processes a matrix multiplication operation on weight data read from the flash memory cell array and input data received from the main memory, and transmits the result of the operation to the main memory (200) to directly process the operation on the weights. This operation is an operation for the FC layer among the major operations performed in the LLM module, and since this operation is processed directly in the flash memory (100), the operation efficiency of the entire operation device (10) can be improved.

[0026] The main memory (200) is implemented as DRAM and exchanges various data with the flash memory (100). In particular, the present invention transmits input data to the flash memory (100), receives the matrix multiplication result of the weight data calculated in the flash memory (100) and the input data, and transmits it to a parallel processing processor (400) that processes the operation for LLM. In addition, the main memory (200) can store a key and a value that are processed for an attention operation.

[0027] The parallel processing processor (300) is such as a GPU or NPU and performs various parallel processing operations for driving the LLM. In the past, the parallel processing processor (300) performed operations for the FC layer, but in the present invention, the operations are performed in the flash memory (100). Meanwhile, the parallel processing processor (300) can perform attention operations using a key and a value.

[0028] FIGS. 3 and FIGS. 4 illustrate the detailed configuration of a computational device according to one embodiment of the present invention.

[0029] First, referring to FIG. 3, in a conventional computer architecture composed of DRAM, CPU, NPU / GPU, and SSD, a MAC calculator (111), a MAC control unit (112) that controls the operation of the MAC calculator, and a PIM control firmware (140) that controls the PIM (Processing-in-memory) may be included inside the NAND flash memory (100). For reference, in the present invention, PIM refers to an in-memory calculator and may include a MAC calculator (111) or a MAC control unit (112).

[0030] Referring to FIG. 4, as illustrated in (a), a plurality of dies may be included in the NAND flash memory (100), and each die may include a plurality of planes. Additionally, each plane includes a flash memory cell array (130), a page buffer (132), a column decoder (134), and an in-memory operation unit (110).

[0031] The in-memory operation unit (110) processes a matrix multiplication operation on the weight data received from the page buffer (132) after being read from the flash memory cell array (130) and the input data received from the main memory (200), and transmits the result of the operation to the main memory (200).

[0032] To this end, the in-memory operation unit (110) includes a first multiplexer (113) for receiving input data, a MAC operator (111) for performing matrix multiplication operations, and a second multiplexer (115) for outputting the output of the MAC operator (111) to the outside.

[0033] The first multiplexer (113) selectively transmits the data to the MAC calculator (111) or the column decoder (134) depending on the type of data transmitted from the outside. For example, when weight data is input, the weight data is stored directly in the flash memory cell array (130) through the column decoder (134) and the page buffer (132) by means of a control signal applied to the first multiplexer (113). In contrast, when input data to be computed with the weight data is input, the input data is transmitted to the MAC calculator (111) instead of the flash memory cell array (130).

[0034] Likewise, the second muxer transmits the output of the MAC operator (111) or the output of the column decoder (134) to the outside. The output of the MAC operator (111) is the result of the matrix multiplication operation, and the output of the column decoder (134) may be weight data for which the matrix multiplication operation has not been performed.

[0035] Next, looking at the detailed configuration of the MAC calculator (111) illustrated in (b), the MAC calculator (111) includes an input buffer (120) in which input data is stored, a weight buffer (124) in which weight data is stored, and an output buffer (127) in which the output of the calculator is stored.

[0036] Additionally, the MAC operator (111) includes a multiplier (125) that performs a multiplication operation on input data output from the input buffer (120) and weight data output from the weight buffer (124), and an adder (126) that accumulates and sums the output of the multiplier (125) and the output value stored in the output buffer (127). Additionally, the MAC operator (111) may include a third multiplexer (117) that receives weight data transmitted from the flash memory cell array (130) via the page buffer (132) and the column decoder (134). The third multiplexer (117) performs the function of selectively transmitting only a portion of the weight data received from the column decoder (134) to the weight buffer (124) because the size of the weight buffer (124) is smaller than the number of bits of the weight data received from the column decoder (134).

[0037] Through this configuration, the multiplier (125) transmits the result of multiplying the input data and the weight data to the adder (126). Then, the adder (126) adds the output of the multiplier (125) and the value already stored in the output buffer (127) and stores it back in the output buffer (127). Accordingly, the output of the multiplier (125) is added again to the value previously stored in the output buffer (127), and as this process is repeated, a MAC operation in which multiplication and addition are sequentially performed can be implemented.

[0038] Meanwhile, the input buffer of the present invention uses a configuration that enables sequential processing of input data. To this end, the input buffer (120) includes a first register (121) and a second register (122) that sequentially store input data, and a third register (123) that stores input data stored in the first register (121) and the second register (122) and outputs the stored input data to an arithmetic unit. At this time, the first input data and the second input data, which are input sequentially in time, are stored separately in the first register (121) and the second register (122), respectively, and when the first input data is output to the third register (123), the third input data, which is input sequentially after the second input data, is stored in the first register (121). Subsequently, when the second input data is output again to the third register (123), the fourth input data, which is input sequentially after the third input data, is stored in the second register (122). Depending on the structure of such a register, consecutive input data can be applied sequentially to the multiplication operation.

[0039] One embodiment of the present invention may also be implemented in the form of a non-transient recording medium comprising computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include all computer storage media. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0040] Although the method and system of the present invention have been described in relation to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

[0041] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0042] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention.

Claims

1. In a computing device for a personal computing device for accelerating a large-scale language model, A flash memory comprising at least one plane including a flash memory cell array, a page buffer, and an in-memory operation unit; Main memory; A parallel processing processor that handles operations on a large-scale language model; and Includes a central processing unit that executes a control program, The above-described in-memory operation unit processes a matrix multiplication operation on weight data received from the page buffer after being read from the flash memory cell array and input data received from the main memory, and outputs the operation result to the main memory.

2. In Paragraph 1, The above in-memory operation unit An arithmetic device comprising a first multiplexer receiving the above input data, a MAC operator performing the matrix multiplication operation, and a second multiplexer outputting the output of the MAC operator to the outside.

3. In Paragraph 1, The above in-memory operation unit includes a MAC operator that performs the matrix multiplication operation, and The above MAC calculator An input buffer where the above input data is stored, A weight buffer in which the above weight data is stored, A third mux that selectively transmits weight data received from the page buffer to the weight buffer, An output buffer that stores the output of the above MAC operator, A multiplier that performs a multiplication operation on input data output from the input buffer and weight data output from the weight buffer; and An arithmetic device comprising an adder that accumulates and sums the output of the multiplier and the output value stored in the output buffer.

4. In Paragraph 3, The above input buffer comprises a first register and a second register that sequentially store the input data, and It includes a third register that stores input data stored in the first and second registers and outputs the stored input data to the MAC operator, An arithmetic device wherein a first input data and a second input data that are input sequentially in time are stored in the first register and the second register, respectively, and when the first input data is output to the third register, a third input data that is input sequentially after the second input data is stored in the first register.

5. In Paragraph 1, A computing device in which, by the above control program, the operation of the FC layer performed during the execution of the large-scale language model is processed in the in-memory computing unit, and the attention operation performed during the execution of the large-scale language model is executed in the parallel processing processor.

6. In a flash memory with built-in computational functions, It includes at least one plane comprising a flash memory cell array, a page buffer, and an in-memory operation unit, and The flash memory, wherein the in-memory operation unit processes a matrix multiplication operation on weight data received from the page buffer after being read from the flash memory cell array and input data received from an external main memory, and outputs the result of the operation to the main memory.

7. In Paragraph 6, The above in-memory operation unit A flash memory comprising a first multiplexer receiving the above input data, a MAC operator performing the matrix multiplication operation, and a second multiplexer outputting the output of the MAC operator to the outside.

8. In Paragraph 6, The above in-memory operation unit includes a MAC operator that performs the matrix multiplication operation, and The above MAC calculator An input buffer where the above input data is stored, A weight buffer in which the above weight data is stored, A third mux that selectively transmits weight data received from the page buffer to the weight buffer. An output buffer that stores the output of the above MAC operator, A multiplier that performs a multiplication operation on input data output from the input buffer and weight data output from the weight buffer; and A flash memory comprising an adder that accumulates and sums the output of the multiplier and the output value stored in the output buffer.

9. In Paragraph 8, The above input buffer comprises a first register and a second register that sequentially store the input data, and It includes a third register that stores input data stored in the first and second registers and outputs the stored input data to the arithmetic unit, A flash memory in which a first input data and a second input data, which are input sequentially in time, are stored in the first register and the second register, respectively, and when the first input data is output to the third register, a third input data, which is input sequentially after the second input data, is stored in the first register.

Images