Computation and storage method and system suitable for sparse reasoning of large language models

Abstract

The present invention provides a computing and storage method, system and device suitable for sparse reasoning of a large language model, including: step S1: dividing the reasoning calculation process of the large language model into multiple fine-grained computing tasks according to the calculation graph of the large language model reasoning; step S2: arranging the storage format of the large language model parameters; step S3: caching the parameters of the large language model, and issuing an I / O request to read the corresponding model parameters into the memory; step S4: using a computing queue and an I / O queue to respectively record the computing tasks of the large language model parameters that are already cached and the computing tasks that are not cached; executing the computing tasks in the computing queue; step S5: removing the completed computing tasks from the computing queue. The present invention relates to a computing and storage framework suitable for sparse reasoning of a large language model, which maximizes the utilization of the device processor and storage transmission through the process of collaborative computing and storage loading, thereby improving the reasoning performance of the large language model.

Description

Technical Field

[0001] The present invention belongs to the field of data acquisition and processing technology, and specifically, relates to a computing and storage method and system suitable for sparse reasoning of large language models. More specifically, it relates to a computing and storage framework suitable for sparse reasoning of large language models, that is, a computing and storage collaborative scheduling framework suitable for accelerating sparse reasoning of large language models, a method, system and device for implementing the framework. Background Art

[0002] Large language models are artificial intelligence models designed to understand and generate human language. They are usually trained on large amounts of text data and use deep learning architectures to learn complex patterns in language data. Large language models can perform natural language processing tasks such as text classification, question answering, conversation, text summarization, translation, and sentiment analysis. In other words, large language models are autoregressive models, meaning that the next output of the model is based on the model's previous output. For example, in a text generation task, a large language model generates one word at a time. The generated word is added to the model input to generate the next word. This process is also called model inference. Large language models are very large, containing billions or even hundreds of billions of parameters, which enables them to handle complex language structures and semantics. Large language models are one of the important developments in the field of artificial intelligence, providing a foundation for achieving more advanced natural language understanding and generation capabilities.

[0003] Sparse reasoning is an important technology for accelerating model reasoning. Sparse reasoning uses only some important model parameters for each reasoning task based on the importance distribution of model parameters, which can significantly reduce the cost of reasoning resources while ensuring the quality of reasoning tasks. Sparsity is a common feature in large language models, which is suitable for acceleration using sparse reasoning. In addition, introducing hybrid expert technology in large language models can further improve the sparsity of the model itself.

[0004] However, when performing sparse inference, it is necessary to statically trim the model according to the importance of model parameters in advance, or dynamically decide which model parameters to use at runtime. The above two technical solutions will face the following problems:

[0005] 1. Static pruning cannot predict the actual input of the reasoning task in advance, so it will inevitably delete model parameters that may be very important to the task, resulting in a decrease in the quality of model reasoning;

[0006] 2. Dynamic sparse inference retains all model parameters, so the performance of loading model parameters will become a bottleneck in the inference process, resulting in low inference efficiency;

[0007] 3. The traditional model storage format is designed for non-sparse reasoning and is suitable for high-speed sequential reading. However, sparse reasoning is usually random reading, which makes it impossible for sparse reasoning to fully utilize the transmission capacity of the storage medium.

[0008] Patent document CN116150048A discloses a memory optimization method, device, equipment and medium. When performing model reasoning through the on-chip distributed computing core, the solution splits the associated model parameters on the target computing graph set into equal amounts and stores them affinity-wise, aiming to improve the computing power of the chip but unable to reduce the resource usage of reasoning. Summary of the invention

[0009] In view of the defects in the prior art, the object of the present invention is to provide a computing and storage method and system suitable for sparse reasoning of large language models.

[0010] A computing and storage method suitable for sparse reasoning of a large language model provided by the present invention includes:

[0011] Step S1: According to the calculation graph of the large language model reasoning, the reasoning calculation process of the large language model is divided into multiple fine-grained calculation tasks;

[0012] Step S2: Arrange the storage format of the large language model parameters based on the fine-grained computing tasks;

[0013] Step S3: Based on the storage format, cache the parameters of the large language model, and issue an I / O request to read the corresponding model parameters into the memory;

[0014] Step S4: using the computing queue and the I / O queue to respectively record the computing tasks of the large language model parameters that are already cached and the computing tasks that are not cached; based on the computing queue and the I / O queue, executing the computing tasks in the computing queue;

[0015] Step S5: Remove the completed computing task from the computing queue.

[0016] Preferably, in step S1:

[0017] The calculation task includes: the calculation process and the model parameters required in the calculation;

[0018] The computing task is activated and executed based on the sparsity of the large language model.

[0019] The process division includes: division within a single operator of the computation graph and division after fusion of multiple operators of the computation graph.

[0020] Preferably, in step S2: the storage format, that is, the large language model parameters of the same computing task are placed continuously in the file.

[0021] Preferably, in step S3: the model parameters required by the activated computing task are cached first;

[0022] In step S4:

[0023] The computing queue executes computing tasks in sequence;

[0024] The I / O queue issues I / O requests for all computing tasks in the queue to read parameters, and moves computing tasks from the I / O queue to the computing queue;

[0025] I / O queue, based on asynchronous I / O;

[0026] In step S5: the model parameters corresponding to the computing tasks entering the computing queue are not swapped out by the parameter cache until the corresponding computing tasks are completed and removed from the computing queue.

[0027] A computing and storage system suitable for sparse reasoning of a large language model provided by the present invention includes:

[0028] Module M1: divides the reasoning calculation process of the large language model into multiple fine-grained computing tasks according to the calculation graph of the large language model reasoning;

[0029] Module M2: Arranges the storage format of large language model parameters based on fine-grained computing tasks;

[0030] Module M3: Based on the storage format, cache the parameters of the large language model, and issue I / O requests to read the corresponding model parameters into the memory;

[0031] Module M4: Use the computing queue and the I / O queue to respectively record the computing tasks of the large language model parameters that are already cached and the computing tasks that are not cached; based on the computing queue and the I / O queue, execute the computing tasks in the computing queue;

[0032] Module M5: Remove the completed computing tasks from the computing queue.

[0033] Preferably, in the module M1:

[0034] The calculation task includes: the calculation process and the model parameters required in the calculation;

[0035] The computing task is activated and executed based on the sparsity of the large language model.

[0036] The process division includes: division within a single operator of the computation graph and division after fusion of multiple operators of the computation graph.

[0037] Preferably, in the module M2: the storage format, that is, the large language model parameters of the same computing task are placed continuously in the file.

[0038] Preferably, in the module M3: the model parameters required by the activated computing task are cached first;

[0039] In the module M4:

[0040] The computing queue executes computing tasks in sequence;

[0041] The I / O queue issues I / O requests for all computing tasks in the queue to read parameters, and moves computing tasks from the I / O queue to the computing queue;

[0042] I / O queue, based on asynchronous I / O;

[0043] In the module M5: the model parameters corresponding to the computing tasks entering the computing queue are not swapped out by the parameter cache until the corresponding computing tasks are completed and removed from the computing queue.

[0044] According to the present invention, a computing and storage device suitable for sparse reasoning of a large language model is provided, which can run a computer program. When the computer program is executed, the steps of the computing and storage method suitable for sparse reasoning of a large language model are implemented.

[0045] Preferably, it includes: a model conversion component, a calculation execution component, an I / O processing component, a parameter cache component and a physical device;

[0046] The model conversion component is used to divide the original model calculation graph and reorganize the storage format of the model parameters;

[0047] The computing execution component is used to execute computing tasks on the processor;

[0048] The I / O processing component is used to send and process an I / O request for reading model parameters;

[0049] The parameter cache component is used to save the most recently used model parameters in memory;

[0050] The physical device includes: a processor, memory and storage device required to perform large language model reasoning.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The present invention adopts a method of dynamically determining whether a computing task is activated at runtime, thereby avoiding the problem of reduced model output quality caused by static clipping.

[0053] 2. The present invention adopts a method of retaining only part of the parameters in the parameter cache, which greatly reduces the memory usage of large language model reasoning.

[0054] 3. The present invention adopts a fine-grained computing task division method, so that I / O reading and computing can be highly parallelized, reducing the end-to-end delay of the reasoning process.

[0055] 4. The present invention adopts a model parameter storage format that is coordinated with the division of computing tasks, which reduces the overhead caused by fragmented random reading due to sparse reasoning and improves the utilization of storage transmission bandwidth. Compared with the prior art, the present invention has the characteristics of saving resources, high efficiency, and generality, and is very suitable for scenarios where large language models are deployed and run locally. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0057] Figure 1 A schematic diagram of the workflow provided by the present invention;

[0058] Figure 2 This is a schematic diagram of the structure of the software and hardware device provided by the present invention. DETAILED DESCRIPTION

[0059] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0060] The present invention provides a computing and storage framework suitable for sparse reasoning of large language models. The framework first divides the reasoning calculation process of the large language model into multiple fine-grained sub-computing tasks, each of which involves only a small part of the model parameters; using the sparsity of the model, the framework can dynamically determine whether each computing task needs to be executed; at runtime, the framework only retains a part of the most recently used model parameters in the memory as a parameter cache, and the remaining model parameters are exchanged from the model storage to the parameter cache as needed; the framework maximizes the utilization of the processor and storage transmission bandwidth by reasonably scheduling computing tasks and storage I / O (Input / Output) requests. In the process of using the framework for model reasoning, the sparsity of the model is fully utilized to reduce the computing power and memory requirements of reasoning, and the output quality of the model is not affected.

[0061] That is, a computing and storage method suitable for sparse reasoning of a large language model provided by the present invention includes:

[0062] Step S1: According to the computational graph of the large language model inference, the computation is divided into several independent fine-grained computational tasks. Each computational task includes a computational process and the model parameters required in the computation. For example, the computational process is a vector dot product, and the model parameter required is a row in the original model parameter matrix that participates in the dot product operation. The computational task can be activated and executed based on the sparsity of the model;

[0063] Step S2: Rearrange the storage format of the model parameters according to the division of the computing tasks, and place the model parameters belonging to the same computing task continuously in the file;

[0064] Step S3: During the inference process, the model parameters required by the activated computing task will first be exchanged into the parameter cache. In order to complete the exchange, the inference framework needs to send an I / O read request to the storage to read the model parameters corresponding to the computing task into the memory;

[0065] Step S4: Use the computing queue and I / O queue to record the computing tasks whose parameters are already in the cache and those whose parameters are not in the parameter cache. For the computing queue, the computing tasks will be executed in sequence; for the I / O queue, I / O requests for reading parameters for all computing tasks in the queue will be issued at the same time;

[0066] Step S5: When the read I / O request is completed, the computing task will be moved from the I / O queue to the computing queue; when the computing task is executed, the computing task will be removed from the computing queue.

[0067] In the step S1, the partitioning of the computation graph includes the partitioning within a single operator of the computation graph and the partitioning after merging multiple operators.

[0068] In step S2, the model parameters of each computing task can be divided into two parts: preload and on-demand loading; the preload part of all computing tasks is concentrated in a part of the model storage and read into the memory at one time before reasoning; the on-demand loading part has appropriate padding alignment and rearrangement to meet the requirements of high-speed I / O reading. The representation of each parameter in the storage remains unchanged.

[0069] In step S3, the parameter cache uses any cache replacement algorithm to select the model parameters to be swapped out, such as LRU, i.e., Least Recently Used, LFU, i.e., Least Frequently Used, etc.

[0070] In step S4, the computing queue uses priority scheduling and does not necessarily need to be executed strictly in queue order; the I / O queue can be implemented using any asynchronous I / O technology, including but not limited to AIO, i.e. Asynchronous I / O, and IOCP, i.e. I / O Completion Ports, input and output completion ports.

[0071] In step S5, the model parameters corresponding to the computing tasks entering the computing queue will not be swapped out by the parameter cache until the computing tasks are completed and removed from the computing queue.

[0072] The present invention also provides a computing and storage system suitable for sparse reasoning of a large language model. The computing and storage system suitable for sparse reasoning of a large language model can be implemented by executing the process steps of the computing and storage method suitable for sparse reasoning of a large language model, that is, those skilled in the art can understand the computing and storage method suitable for sparse reasoning of a large language model as a preferred implementation of the computing and storage system suitable for sparse reasoning of a large language model.

[0073] A computing and storage system suitable for sparse reasoning of a large language model provided by the present invention includes:

[0074] Module M1: divides the reasoning calculation process of the large language model into multiple fine-grained computing tasks according to the calculation graph of the large language model reasoning;

[0075] Module M2: Arranges the storage format of large language model parameters based on fine-grained computing tasks;

[0076] Module M3: Based on the storage format, cache the parameters of the large language model, and issue I / O requests to read the corresponding model parameters into the memory;

[0077] Module M4: Use the computing queue and the I / O queue to respectively record the computing tasks of the large language model parameters that are already cached and the computing tasks that are not cached; based on the computing queue and the I / O queue, execute the computing tasks in the computing queue;

[0078] Module M5: Remove the completed computing tasks from the computing queue.

[0079] Specifically, in the module M1:

[0080] The calculation task includes: the calculation process and the model parameters required in the calculation;

[0081] The computing task is activated and executed based on the sparsity of the large language model.

[0082] The process division includes: division within a single operator of the computation graph and division after fusion of multiple operators of the computation graph.

[0083] Specifically, in the module M2: the storage format, that is, the large language model parameters of the same computing task are placed continuously in the file.

[0084] Specifically, in the module M3: the model parameters required by the activated computing task are cached first;

[0085] In the module M4:

[0086] The computing queue executes computing tasks in sequence;

[0087] The I / O queue issues I / O requests for all computing tasks in the queue to read parameters, and moves computing tasks from the I / O queue to the computing queue;

[0088] I / O queue, based on asynchronous I / O;

[0089] In the module M5: the model parameters corresponding to the computing tasks entering the computing queue are not swapped out by the parameter cache until the corresponding computing tasks are completed and removed from the computing queue.

[0090] The present invention will be further described below in conjunction with the accompanying drawings.

[0091] like Figure 1 As shown in the figure, the specific process of sparse reasoning of the large language model of the present invention is as follows. Taking a single reasoning task as an example, Figure 1 The following model inference steps are described in detail:

[0092] In step 101, the reasoning framework determines whether the computing queue and the I / O queue are both empty. If both are empty, the reasoning ends; otherwise, step 102 is executed;

[0093] In other words, if the result is yes, the reasoning ends; if the result is no, step 102 is executed;

[0094] In step 102, the I / O processing component determines whether there is a read request for a computing task in the I / O queue that has not been issued. If so, execute step 103; if not, execute step 104;

[0095] In other words, if the result is yes, then execute step 103; if the result is no, then execute step 104;

[0096] In step 103, the I / O processing component sends a read request corresponding to the computing task to the storage layer, and step 101 is executed;

[0097] In step 104, the I / O processing component determines whether there is a read request that has been completed but not yet processed. If so, step 105 is executed; otherwise, step 106 is executed;

[0098] In step 105, the I / O processing component exchanges the model parameter data obtained by the completed read request into the parameter cache, migrates the corresponding computing task from the I / O queue to the computing queue, and executes step 101;

[0099] In step 106, the calculation execution component determines whether the calculation queue is not empty. If it is not empty, step 107 is executed, otherwise step 108 is executed; in other words, if the result is yes, step 107 is executed, and if the result is no, step 108 is executed;

[0100] In step 107, the computing execution component selects a computing task from the computing queue to execute, and then executes step 101;

[0101] In step 108 , the I / O processing component waits for at least one read request to be completed and executes step 101 .

[0102] like Figure 2 As shown, the present invention uses a model conversion component to generate a model file suitable for sparse reasoning of a large language model. The model file is loaded into the parameter cache by the I / O processing component. The calculation processing component reads the required model parameters from the parameter cache and completes the calculation task, finally obtaining the output of the model and providing it to the user.

[0103] According to the present invention, a computing and storage device suitable for sparse reasoning of a large language model is provided, which can run a computer program. When the computer program is executed, the steps of the computing and storage method suitable for sparse reasoning of a large language model are implemented.

[0104] Specifically, it includes: model conversion component, calculation execution component, I / O processing component, parameter cache component and physical device;

[0105] The model conversion component is used to divide the original model calculation graph and reorganize the storage format of the model parameters;

[0106] The computing execution component is used to execute computing tasks on the processor;

[0107] The I / O processing component is used to send and process an I / O request for reading model parameters;

[0108] The parameter cache component is used to save the most recently used model parameters in memory;

[0109] The physical device includes: a processor, memory and storage device required to perform large language model reasoning.

[0110] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0111] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A computing and storage method suitable for sparse reasoning of large language models, characterized in that: include: Step S1: According to the calculation graph of the large language model reasoning, the reasoning calculation process of the large language model is divided into multiple fine-grained calculation tasks; Step S2: Arranging the storage format of the large language model parameters based on the fine-grained computing tasks; the storage format is that the large language model parameters of the same computing task are placed continuously in the file; Step S3: Based on the storage format, cache the parameters of the large language model, and issue an I / O request to read the corresponding model parameters into the memory; Step S4: using the computing queue and the I / O queue to respectively record the computing tasks of the large language model parameters that are already cached and the computing tasks that are not cached; based on the computing queue and the I / O queue, executing the computing tasks in the computing queue; Step S5: remove the completed computing task from the computing queue; The computing task is activated and executed based on the sparsity of the large language model.

2. The computing and storage method for sparse reasoning of a large language model according to claim 1, characterized in that: In step S1: The calculation task includes: the calculation process and the model parameters required in the calculation; The process division includes: division within a single operator of the computation graph and division after fusion of multiple operators of the computation graph.

3. The computing and storage method for sparse reasoning of a large language model according to claim 2, characterized in that: In the step S3: the model parameters required by the activated computing task are cached first; In step S4: The computing queue executes computing tasks in sequence; The I / O queue issues I / O requests for all computing tasks in the queue to read parameters, and moves computing tasks from the I / O queue to the computing queue; I / O queue, based on asynchronous I / O; In step S5: the model parameters corresponding to the computing tasks entering the computing queue are not swapped out by the parameter cache until the corresponding computing tasks are completed and removed from the computing queue.

4. A computing and storage system suitable for sparse reasoning of large language models, characterized in that: include: Module M1: divides the reasoning calculation process of the large language model into multiple fine-grained computing tasks according to the calculation graph of the large language model reasoning; Module M2: Arrange the storage format of large language model parameters based on fine-grained computing tasks; the storage format is that the large language model parameters of the same computing task are placed continuously in the file; Module M3: Based on the storage format, cache the parameters of the large language model, and issue I / O requests to read the corresponding model parameters into the memory; Module M4: Use the computing queue and the I / O queue to respectively record the computing tasks of the large language model parameters that are already cached and the computing tasks that are not cached; based on the computing queue and the I / O queue, execute the computing tasks in the computing queue; Module M5: removes the completed computing tasks from the computing queue; The computing task is activated and executed based on the sparsity of the large language model.

5. The computing and storage system for sparse reasoning of a large language model according to claim 4, characterized in that: In the module M1: The calculation task includes: the calculation process and the model parameters required in the calculation; The process division includes: division within a single operator of the computation graph and division after fusion of multiple operators of the computation graph.

6. The computing and storage system for sparse reasoning of a large language model according to claim 5, characterized in that: In the module M3: the model parameters required by the activated computing task are cached first; In the module M4: The computing queue executes computing tasks in sequence; The I / O queue issues I / O requests for all computing tasks in the queue to read parameters, and moves computing tasks from the I / O queue to the computing queue; I / O queue, based on asynchronous I / O; In the module M5: the model parameters corresponding to the computing tasks entering the computing queue are not swapped out by the parameter cache until the corresponding computing tasks are completed and removed from the computing queue.

7. A computing and storage device suitable for sparse reasoning of a large language model, capable of running a computer program, characterized in that: When the computer program is executed, the steps of the calculation and storage method applicable to sparse reasoning of a large language model according to any one of claims 1 to 3 are implemented.

8. The computing and storage device for sparse reasoning of a large language model according to claim 7, characterized in that: include: Model conversion components, computation execution components, I / O processing components, parameter cache components, and physical devices; The model conversion component is used to divide the original model calculation graph and reorganize the storage format of the model parameters; The computing execution component is used to execute computing tasks on the processor; The I / O processing component is used to send and process an I / O request for reading model parameters; The parameter cache component is used to save the most recently used model parameters in memory; The physical device includes: a processor, memory and storage device required to perform large language model reasoning.

Images