Method and device for realizing large language model deployment

Abstract

The invention discloses a method and device for realizing deployment of a large language model, and the method comprises the steps: storing a part of model parameters of which the occupied space is greater than a preset condition in a large language model in SLC and TLC regions of a flash memory, and dynamically loading the model parameters into a DRAM according to the needs, thereby reducing the demands of the large language model for the DRAM capacity. The embodiment of the invention is particularly suitable for mobile terminal equipment with limited DRAM capacity, and high-efficiency operation of the large language model is realized. In the embodiment of the invention, based on the sparsity characteristic of the large language model, the model parameters are reasonably distributed and stored in the SLC and TLC areas of the flash memory, and the fast read-write performance of the flash memory in the SLC mode and the large capacity advantage of the flash memory in the TLC mode are fully utilized.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the technical field of storage systems, and specifically to a method and device for implementing large language model deployment. Background Art

[0002] Large Language Model (LLM) is a natural language processing model based on deep learning. It usually has hundreds of millions or even hundreds of billions of parameters and can demonstrate strong generation and understanding capabilities in a variety of language tasks. This type of model uses self-attention mechanisms (such as the Transformer architecture) to process large-scale text data. After learning from a large amount of training data, it can effectively complete various natural language processing tasks, such as text generation, question answering, translation, and dialogue systems.

[0003] LLMs are at the core of modern natural language processing and are able to provide excellent performance in a variety of tasks. However, these models have very high requirements for computing power and memory, especially on mobile terminal devices with limited dynamic random access memory (DRAM) capacity, which faces huge challenges. According to the scaling law of large models, the larger the model parameters, the stronger its capabilities, but it also means that more memory and computing resources are needed to support the reasoning process. In a typical LLM reasoning phase, the parameters of the model usually need to be fully loaded into DRAM. Taking a half-precision LLM with 7 billion parameters as an example, the storage space required to fully load it into DRAM is more than 14GB. However, considering that the DRAM capacity of mainstream mobile terminal devices is generally small, it is very difficult or even impossible to directly load such a large-scale model on these devices. Therefore, how to efficiently deploy large language models on mobile devices with limited DRAM capacity has become a problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a method and device for implementing large language model deployment, which can solve the problem of loading LLM that exceeds the available capacity of DRAM.

[0005] An embodiment of the present invention provides a method for implementing large language model deployment, including: Segmenting model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in a flash memory array; Store the segmented model parameters into flash memory.

[0006] In an exemplary embodiment, the model parameters that occupy a space greater than a preset condition include: model weight information that requires a large amount of storage space.

[0007] In an exemplary embodiment, the structure of the large language model is a Transformer model; The model weight information that requires a large amount of storage space includes: the first weight parameter matrix that needs to be learned in the feedforward neural network FNN calculation and the second weight parameter matrix .

[0008] In an exemplary embodiment, segmenting the model parameters of the large language model whose occupied space is greater than a preset condition includes: Calculate the amount of data obtained from the flash array of the flash memory in one concurrent read operation ; Calculate the number of neurons that store weight parameter matrices continuously ,in, Represents the storage space required for the weight parameters corresponding to each neuron; Based on the number of neurons that can be read simultaneously , will be continuous The neurons are marked as a group, and the size of each group is neurons; Based on the size of the flash array, the first weight parameter matrix and the second weight parameter matrix Segmented storage, each segment as a group, corresponding to continuous The weight parameters required for each neuron.

[0009] In an exemplary embodiment, the amount of data The size of the page depends on the storage space allocation information of the flash memory array, and the storage space allocation information includes the number of channels, the number of storage areas, and the size of each page.

[0010] In an exemplary embodiment, storing the segmented model parameters in a flash memory includes: According to usage requirements, the segmented model parameters are stored in the SLC block area and the TLC block area of ​​the flash memory.

[0011] In an exemplary embodiment, storing the segmented model parameters in a flash memory includes: Writing the segmented weight parameters into the TLC block area of ​​the flash memory in groups, and recording in the DRAM the mapping relationship between the data segment number LCA accessed by the system application and the actual physical address PPA of each segmented weight parameter group in the storage array; According to a preset strategy, the corresponding weight parameters in the TLC block area are simultaneously stored in the SLC block area of ​​the flash memory.

[0012] In an exemplary embodiment, the preset strategy is: storing a preset number of weight parameters of a specified storage space size in the SLC block area in the order in which they are stored in the TLC block area; or, The weight parameters frequently used in the FNN operation are stored in the SLC block area.

[0013] In an exemplary embodiment, writing the segmented weight parameters into the TLC block area of ​​the flash memory in groups includes: The segmented weight parameters are written into the TLC block area in groups. When writing, the data segment number is the group number, and the mapping relationship between the LCA accessed by the system application and the actual physical address of the segmented weight parameter set in the storage array is recorded in the DRAM; the process of writing the segmented weight parameters into the TLC block area in groups is repeated until all the weight parameters are written into the TLC block area.

[0014] An embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the above methods for implementing large language model deployment.

[0015] An embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores the following instructions executable by the processor: used to execute the steps of any of the above-mentioned methods for implementing large language model deployment.

[0016] The embodiment of the present application further provides a device for implementing large language model deployment, including a segmentation module and a storage module; wherein: A segmentation module, for segmenting model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in a flash memory array; The storage module is used to store the segmented model parameters in the flash memory.

[0017] In an exemplary embodiment, the model parameters that occupy a space greater than a preset condition include: model weight information that requires a large amount of storage space.

[0018] In an exemplary embodiment, the storage module is used to store the segmented model parameters in the SLC block area and the TLC block area of ​​the flash memory according to usage requirements.

[0019] The method for implementing the deployment of a large language model provided by the embodiment of the present application stores some model parameters (such as model weight information that needs to be trained) in the large language model that occupy more space than the preset conditions in the SLC and TLC areas of the flash memory, and dynamically loads them into the DRAM as needed, thereby reducing the demand for DRAM capacity by the large language model. The embodiment of the present application is particularly suitable for mobile terminal devices with limited DRAM capacity, and realizes the efficient operation of the large language model. In the embodiment of the present application, based on the sparsity characteristics of the large language model, the model parameters are reasonably allocated and stored in the SLC and TLC areas of the flash memory, making full use of the fast read and write performance of the flash memory in the SLC mode and the large capacity advantage in the TLC mode.

[0020] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0022] Figure 1 A flowchart of a method for implementing large language model deployment in an embodiment of the present application; Figure 2 This is an example schematic diagram of the segmentation of the weight parameter matrix that needs to be trained for a large language model in an embodiment of the present application; Figure 3 This is an example schematic diagram of storing segmented model weight information into SLC blocks and TLC blocks in an embodiment of the present application; Figure 4 This is a schematic diagram of a system embodiment for optimizing the calculation of FNN based on the method for implementing large language model deployment provided in the embodiment of the present application; Figure 5 A schematic diagram of the composition structure of an apparatus for implementing large language model deployment in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

[0024] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0026] It is understood that the terms "first" and "second" used in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0027] It can be understood that the “connection” in the following embodiments should be understood as “electrical connection”, “communication connection”, etc. if the connected circuits, modules, units, etc. have electrical signals or data transmission between each other.

[0028] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0029] When deploying LLM on a mobile terminal device, it may be difficult to fully load into the device's memory due to its high demand for DRAM capacity, especially for large models (such as 7B parameter models). The inventors of this application have found through research on related technologies that the root cause of this problem lies in the contradiction between the parameter scale of large language models and memory requirements. Specifically, the inventors of this application have found that the related technology did not consider the physical read, write and erase restrictions of flash memory during design, and the model parameters were not segmented according to the block size or page size in the flash array. It did not consider the performance enhancement of the read parallelism of multiple flash memories, did not fully utilize the differences in read and write performance under different modes of flash memory blocks, such as single-level cell (SLC) and multi-level cell (TLC) modes, and did not optimize for high-throughput random reads.

[0030] To this end, an embodiment of the present application provides a method for implementing large language model deployment, which is used to solve the problem of loading LLM that exceeds the available capacity of DRAM. Figure 1 This is a flow chart of a method for implementing large language model deployment in an embodiment of the present application. Figure 1 As shown, this may include:

[0031] Step 100: segment model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in a flash memory array.

[0032] In an exemplary instance, model parameters that occupy more space than preset conditions may include, but are not limited to, model weight information that requires a large amount of storage space. In one embodiment, the preset condition may be that the storage requirement of the model reaches a preset threshold, such as 100G, or 50G, or 200G, etc. In this case, model parameters that exceed the threshold are considered to be model parameters that occupy more space than the preset condition. It should be noted that the preset threshold can be adjusted according to the actual application scenario in order to free up storage space in DRAM to better suit scenarios where the DRAM capacity of mainstream mobile terminal devices is generally small.

[0033] As a new type of storage device, flash memory has been widely used in consumer PCs and enterprise servers because of its significant advantages over traditional mechanical hard disks in terms of capacity, reliability, read and write speed, shock resistance, power consumption, and noise. A flash memory array is mainly composed of multiple Nand Flash Dies. Multiple Dies form multiple channels and storage areas, which can perform read and write operations concurrently. Each Die consists of multiple blocks, each block includes multiple pages, and the physical size of each page is 16KB.

[0034] In an exemplary embodiment, the channel, bank, block and page in the flash memory array can be regarded as a storage space allocation method in the flash memory array. These elements constitute the hierarchical structure of the flash memory array and determine the specific storage location of data at the physical level.

[0035] In an exemplary embodiment, the structure of the large language model can be a Transformer model. The model weight information of the large language model in step 100 that requires a large amount of storage space includes: the first weight parameter matrix that needs to be learned in the feedforward neural network (FNN) calculation and the second weight parameter matrix Step 100 may include:

[0036] First, calculate the amount of data retrieved from the flash array in a concurrent read operation. In one embodiment, the amount of data The storage space allocation information depends on the structure of the flash memory array. The storage space allocation information may include the number of channels, the number of banks, and the size of each page. In one embodiment, the data volume .

[0037] Next, calculate the number of neurons that continuously store the weight parameter matrix ,in, Represents the storage space required for the weight parameters corresponding to each neuron, It is also the number of neurons that can be read simultaneously.

[0038] Then, based on the number of neurons that can be read simultaneously , will be continuous The neurons are marked as a group, and the size of each group is neurons, which ensures that in each read operation, The weight parameters of neurons. For example: The neurons are numbered from 1 to ; The neurons are numbered from arrive .

[0039] Finally, based on the size of the flash array, the first weight parameter matrix and the second weight parameter matrix Segmented storage, each segment as a group, corresponding to continuous The weight parameters required by each neuron. In this way, each read can be performed by group, so that enough parameters can be obtained for calculation in one operation, reducing the performance overhead caused by multiple accesses. The first weight parameter matrix and the second weight parameter matrix Segment according to the size of the flash array, such as Figure 2 As shown, each group corresponds to a continuous The weight parameters required for neuron calculation.

[0040] Through the segmented storage method in step 100, the system can significantly improve the efficiency of reading model weights from the flash memory array. Especially in the reasoning process of large models, this method reduces the frequency of I / O operations and improves the throughput of data access. At the same time, due to the rationality of group division, it is ensured that the weight data obtained in a concurrent read operation can meet the computing needs to the greatest extent, thereby reducing the situation of redundant reading. The storage design of weight segmentation is carried out for the efficient reasoning scenario of large models in the embodiment of the present application, which improves the overall performance and reduces the burden on the storage device.

[0041] Take the structure of a large language model based on the Transformer model as an example. According to different task requirements, the Transformer architecture mainly includes three types: decoder-only, encoder-only, and encoder-decoder. The model trained based on the decoder-only architecture can handle a wide range of text generation tasks, and can also understand the text content through unsupervised learning without the need for special training and adjustment for each task, showing great flexibility and wide application potential. In related technologies, large language models basically choose the decoder-only architecture.

[0042] Taking the Transformer model based on the Decoder-only architecture as an example, it consists of multiple identical layers, and each layer is mainly divided into two parts: self-attention mechanism (Self-Attention) and feedforward neural network (FNN).

[0043] Transformer's self-attention mechanism allows the model to consider all other elements in the sequence and give different elements importance when processing any element in the sequence. Through the self-attention mechanism, the model is better able to handle complex patterns and long-distance dependencies. Transformer's self-attention mechanism gives different importance to each element of the sequence by considering other elements at the same time. The specific process is as follows:

[0044] Input representation: Assume that there is a text input of length n, that is, the input sequence has n words, and each word in the text is represented by a vector of length d. Then the input text is represented as a matrix X, which is a matrix with n rows and d columns.

[0045] For each word, the model generates three matrices - a query matrix Q representing the query vector for each word in the text, a key matrix K representing the key vector for each word in the text, and a value matrix V representing the value vector for each word in the text. These matrices are a function of the input X and the parameter matrix to be learned. , , Performing matrix multiplication yields: , , ,in, , , is the parameter matrix that needs to be learned;

[0046] Calculate the similarity score between each word in the input text and other words to get the weight distribution of each word in the text. Here, we can calculate the dot product of the query vector Q and the key vector K, and convert the similarity into a weight distribution through the softmax function to calculate the self-attention output ,in, is the number of columns of the query matrix Q.

[0047] The Transformer's FNN layer consists of two linear transformations and , a nonlinear activation function is inserted in the middle . Transformer's nonlinear activation function Using the ReLU activation function, the FNN formula is as follows: ;in, and is the weight parameter matrix that needs to be learned, for matrix, for Matrix. Simply put, FFN can be seen as a Key-Value memory that stores a large amount of specific knowledge. The first layer of FFN is the Key layer, which is used to identify the language or knowledge pattern in the input; the second layer is the Value layer, which is used to store the specific knowledge corresponding to the Key layer. The input layer of FFN is the overall information of the entire input text obtained through Self-Attention. The forward propagation process of FFN can be seen as detecting the knowledge pattern through the Key layer, then taking out the corresponding Value, and reflecting it in the output.

[0048] The parameters involved in the Transformer's Self-Attention and FNN calculation processes are the main components of the Transformer's model parameters. Among them, the Self-Attention part accounts for about one-third of the total parameters, and the FNN part parameters account for about two-thirds of the total parameters. In the embodiment of the present application, the weight parameters of the Attention layer are stored in DRAM, and this part does not involve dynamic loading of parameters. The characteristic of the ReLU activation function is that it outputs a non-negative value. When the input is less than 0, the output is 0, which results in more than 90% of the intermediate outputs in the FNN layer being 0, resulting in a high degree of sparsity. This means that when passing through the FNN layer, only neurons with inputs greater than 0 will be activated, and the outputs of most neurons will be 0, resulting in a high degree of sparsity in the FNN layer. The sparsity of the FNN layer depends on the current input, that is, at each inference, only a small number of neurons are actually activated, and most neurons do not participate in the inference operation. Therefore, for the weight parameters of the FNN layer, only the parameters corresponding to the active neurons need to be loaded, and the parameters of the entire model do not need to be loaded, which greatly reduces the load on DRAM. ‌

[0049] In an embodiment of the present application, the sparsity of FNN is utilized, and during the inference process, only the weight parameters corresponding to the activated neurons are loaded into DRAM. This dynamic loading method avoids unnecessary memory usage and optimizes the efficiency of DRAM use. Moreover, due to the sparsity of the FNN layer, most of the weight parameters do not need to participate in the calculation during the inference process. By only loading the weight parameters of the active part, the demand for DRAM capacity is greatly reduced, so that large models can be inferred with limited hardware resources. The embodiment of the present application not only maintains the computational efficiency of the model, but also optimizes the use of hardware resources. It is a very effective strategy for deploying the Transformer model in a resource-constrained environment.

[0050] The FNN formula is The first weight parameter matrix in and the second weight parameter matrix It needs to be learned, and its size accounts for an important proportion of the size of the entire large model parameters; for the first weight parameter matrix and the second weight parameter matrix , the weight parameters of each neuron correspond to the first weight parameter matrix and the second weight parameter matrix Column, The weight parameters of neurons correspond to their When the When a neuron is activated, the first weight parameter matrix needs to be read from the flash array at the same time and the second weight parameter matrix Therefore, in order to improve the reading speed, in the embodiment of the present application, the first weight parameter matrix and the second weight parameter matrix The corresponding column elements of are connected and stored together, and the storage space size is S, so that the data of these two parts can be read at one time when accessed. This storage method reduces the read latency and makes full use of the parallel read and write capabilities of the flash array.

[0051] Step 101: Store the segmented model parameters in a flash memory.

[0052] In the flash array, the data reading and writing process follows a specific order. First, the channel and bank of the flash array are determined, and then the block and page locations are determined. When the system application writes data to the flash array, the system application operates by accessing the data segment number (LCA, logical cluster address) in the logical space of the flash array, and the flash array needs to translate the LCA accessed by the system application into the actual physical address (PPA, Physical Page Address) of the storage array, that is, the specific location of the channel, bank, block and page where the data is stored. The one-to-one mapping relationship between LCA and PPA constitutes a mapping table from the logical space data segment address to the physical address of the flash array. When the system application reads data from the flash array, it searches the mapping table through LCA, finds the corresponding PPA, retrieves the data from the actual physical location of the flash storage array and returns it to the upper-level application. Through this mapping mechanism between logical addresses and physical addresses, the flash array can effectively manage the storage and access of data. At the same time, due to the concurrent processing capabilities of different channels and banks, the overall system read and write performance is greatly improved.

[0053] In an exemplary embodiment, step 101 may include: storing the segmented model parameters such as model weight information in an SLC block area (SLC block) and a TLC block area (TLC block) of the flash memory according to usage requirements.

[0054] The block where the flash memory stores data can be switched to an SLC block or a TLC block according to usage requirements. The capacity of an SLC block is only one-third of a TLC block, but the data read and write speeds of an SLC block are faster than those of a TLC block. Figure 3 As shown, the flash memory array is divided into an SLC block area and a TLC block area, and is divided into two banks, each bank including multiple channels (such as Channel 0 and Channel 1). Among them, for the SLC Block, each cell stores 1 bit of data, has a faster read and write speed, higher durability, but a smaller capacity (only one-third of the capacity of the TLC block); for the TLC Block, each cell stores 3 bits of data, and the read and write speeds are slower than the SLC block, but it provides a larger storage capacity. In one embodiment, in order to optimize the storage of model parameters, the SLC block can be used to store parameters with higher access frequency and higher speed requirements because it has a faster read and write speed; while the TLC block is suitable for storing parameters that are less frequently accessed or have larger capacity. Although it is slower, it can maximize the storage capacity.

[0055] In the embodiment of the present application, the flash memory controller dynamically manages the allocation of blocks according to usage requirements. For example, some blocks can switch between SLC and TLC modes according to usage to balance performance (speed) and storage capacity. This hybrid storage strategy makes full use of the high speed of SLC and the high capacity of TLC, ensuring that the model parameters can be efficiently stored in the flash memory array.

[0056] In an exemplary embodiment, step 101 may include: The segmented weight parameters are written into the TLC block area of ​​the flash memory in groups, and the mapping relationship between the LCA accessed by the system application and the actual physical address of each segmented weight parameter group in the storage array is recorded in the DRAM; According to a preset strategy, the corresponding weight parameters in the TLC block area are simultaneously stored in the SLC block area of ​​the flash memory.

[0057] In one embodiment, the preset strategy may be: storing a preset number of weight parameters of a specified storage space size in the SLC block area in the order of storing in the TLC block area. For example: storing the first 1G weight parameter and the second 1G weight parameter in the SLC block area at the same time.

[0058] In one embodiment, the preset strategy may be: storing the weight parameters frequently used in the FNN operation in the SLC block area. For example, the weight parameters that are easy to hit may be stored in the SLC block area at the same time according to historical statistics.

[0059] In an exemplary embodiment, writing the segmented weight parameters into the TLC block area of ​​the flash memory in groups may include: The weight parameters after segmentation are divided into groups. Written in TLC Block, when writing, the data segment number for , and record in DRAM the mapping relationship between the LCA accessed by the system application and the segmented weight parameter set in the actual physical address of the storage array, that is, < , >; Repeat the segmented weight parameter set The process of writing to the TLC Block continues until all weight parameter sets group are written to the TLC Block of the storage array. In this way, a set of mapping relationships will be stored in DRAM. , >, ....< , >.

[0060] The method for implementing the deployment of a large language model provided in an embodiment of the present application reduces the demand for DRAM capacity by the large language model by storing some model parameters (such as model weight information that needs to be trained) that occupy more space than a preset condition in the SLC and TLC areas of the flash memory, and dynamically loading them into DRAM as needed. The embodiment of the present application is particularly suitable for mobile terminal devices with limited DRAM capacity, and realizes the efficient operation of the large language model. In the embodiment of the present application, based on the sparsity characteristics of the large language model, the model parameters are reasonably allocated and stored in the SLC and TLC areas of the flash memory, making full use of the fast read and write performance of the flash memory in the SLC mode and the large capacity advantage in the TLC mode.

[0061] In an exemplary embodiment, the present application also includes: The model parameters are re-encoded and stored on the same page of different flash memories, which facilitates parallel reading of multiple flash memories and improves the model parameter loading performance.

[0062] The following is a description of a method for implementing a large language model deployment based on an embodiment of the present application, and optimizing the calculation process of FNN, such as Figure 4 As shown, the system includes a control center, DRAM and Nand Flash.

[0063] First, the control center determines which neurons are in an activated state based on the input of the FNN, and determines the group to which they belong based on the number of the activated neurons. For example, if neurons Ni, Nm, and Nj are in an activated state, the control center will determine the group number to which they belong based on the number (for example, the group number to which neuron Ni belongs is ). Each time the FNN input is processed, the control center also records the number of activated neurons in each group C[ ];

[0064] Next, the control center queries the cache management table T in DRAM. The cache management table T contains the physical storage address of the group number and its corresponding weight parameter in DRAM, that is,<Group,physical address> If the query results show that The corresponding weight parameters have been cached in DRAM. The system can directly obtain the weight parameters from DRAM and participate in the calculation of FNN. If they are not cached in DRAM, the control center will The number is used as LCA to query the mapping table in DRAM<LCA,PPA> , get The corresponding weight parameters are stored in the storage location PPA of the flash array, and the corresponding data is accessed from NandFlash;

[0065] When the system starts, The corresponding weight parameters are in the TLC Block area. The control center will read them from the TLCBlock The corresponding weight parameters are loaded into DRAM to participate in the calculation of FNN. In this process, the cache management table T needs to be updated to and the position of its weight parameters in DRAM are added to the cache management table T. In order to ensure that the DRAM cache area is not overloaded, the control center will record the number of activated neurons C[ ], delete the groups with the least activated neurons to make room for new groups, so that the control center can The DRAM address of the corresponding weight parameter is added to the position of the deleted group in the cache management table T;

[0066] After completing the above operations, the control center will , as well as The weight parameters are read from the TLC Block and written to the SLC Block with higher performance. At the same time, the mapping table in the DRAM is updated to reflect the new storage location of these weight parameters in the SLC Block. , >,< , > and < , >Update the mapping relationship stored in DRAM; If the control center needs to update a certain The control center first queries the cache management table T to determine Is it in DRAM? If In DRAM, the new weight parameters are written to The location of the weight parameters in DRAM; if If it is not in DRAM, The new weight relationship parameters are written into the SLC Block and a new mapping relationship is generated. , >, and < , > Synchronously update the mapping table in DRAM. In subsequent calculations, the control center reads When the weight parameter is obtained, if the cache management table T is hit, the weight parameter is read from DRAM to participate in the operation in FNN, otherwise the mapping relationship is queried to obtain ,according to Read the weight parameters from the SLC block to participate in the calculation in the FNN.

[0067] Through the method for implementing large language model deployment provided in the embodiment of the present application, the calculation process of FNN realizes the intelligent scheduling of weight parameters between DRAM and Nand Flash, ensuring the high efficiency of the feedforward neural network during calculation and the timeliness of data access.

[0068] The present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the above-described methods for implementing large language model deployment.

[0069] The present disclosure embodiment further provides a computer device, including a memory; and a processor connected to the memory, wherein the processor executes the steps of the method for implementing large language model deployment as described in any of the preceding items based on instructions stored in the memory.

[0070] Figure 5 Schematic diagram of the composition structure of the device for implementing large language model deployment in the embodiment of the present application, such as Figure 5 As shown, it may include: a segmentation module and a storage module; wherein, A segmentation module, for segmenting model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in a flash memory array; The storage module is used to store the segmented model parameters in a flash memory.

[0071] In an exemplary embodiment, the model parameters that occupy a space greater than a preset condition may include, but are not limited to, model weight information that requires a large amount of storage space.

[0072] In an exemplary embodiment, the model weight information of the large language model that requires a large amount of storage space includes: a first weight parameter matrix and the second weight parameter matrix ; The segmentation module can be used for: Calculate the amount of data retrieved from the flash array in one concurrent read operation ; Calculate the number of neurons that continuously store the weight parameter matrix ,in, Represents the storage space required for the weight parameters corresponding to each neuron, is also the number of neurons that can be read simultaneously; based on the number of neurons that can be read simultaneously , will be continuous The neurons are marked as a group, and the size of each group is neurons; based on the size of the flash array, the first weight parameter matrix and the second weight parameter matrix Segmented storage, each segment as a group, corresponding to continuous The weight parameters required for each neuron.

[0073] In one exemplary embodiment, the storage module may be used to: According to usage requirements, the segmented model parameters are stored in the SLC block area and the TLC block area of ​​the flash memory.

[0074] In an exemplary embodiment, the storage module is specifically used for: The segmented weight parameters are written into the TLC block area of ​​the flash memory in groups, and the mapping relationship between the LCA accessed by the system application and the actual physical address of each segmented weight parameter group in the storage array is recorded in the DRAM; According to a preset strategy, the corresponding weight parameters in the TLC block area are simultaneously stored in the SLC block area of ​​the flash memory.

[0075] In one embodiment, the preset strategy may be: storing a preset number of weight parameters of a specified storage space size in the SLC block area in the order in which they are stored in the TLC block area. Alternatively, storing weight parameters frequently used in FNN calculations in the SLC block area.

[0076] In an exemplary embodiment, writing the segmented weight parameters into the TLC block area of ​​the flash memory in groups in the storage module may include: The segmented weight parameters are written into the TLC Block in groups. When writing, the data segment number The group number is , and record in DRAM the mapping relationship between the LCA accessed by the system application and the segmented weight parameter set in the actual physical address of the storage array, that is, < , > Repeat the process of writing the segmented weight parameters into the TLCBlock in groups until all weight parameters are written into the TLC Block of the flash array.

[0077] In an exemplary embodiment, the storage module is further configured to: The model parameters are re-encoded and stored on the same page of different flash memories, which facilitates parallel reading of multiple flash memories and improves the model parameter loading performance.

[0078] The device for implementing the deployment of a large language model provided in an embodiment of the present application reduces the demand for DRAM capacity of a large language model by storing some model parameters (such as model weight information that needs to be trained) in the large language model that occupy more space than a preset condition in the SLC and TLC areas of the flash memory, and dynamically loading them into the DRAM as needed. The embodiment of the present application is particularly suitable for mobile terminal devices with limited DRAM capacity, and realizes the efficient operation of a large language model. In the embodiment of the present application, based on the sparsity characteristics of the large language model, the model parameters are reasonably allocated and stored in the SLC and TLC areas of the flash memory, making full use of the fast read and write performance of the flash memory in the SLC mode and the large capacity advantage in the TLC mode.

[0079] Although the embodiments disclosed in this application are as above, the contents described are only embodiments adopted to facilitate understanding of this application and are not intended to limit this application. Any technician in the field to which this application belongs can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in this application, but the scope of patent protection of this application shall still be based on the scope defined in the attached claims.

Claims

1. A method for implementing large language model deployment, characterized in that: include: Segmenting model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in a flash memory array; Store the segmented model parameters into flash memory.

2. The method according to claim 1, wherein: The model parameters that occupy a space greater than a preset condition include: model weight information that requires a large amount of storage space.

3. The method according to claim 2, wherein: The structure of the large language model is a Transformer model; The model weight information that requires a large amount of storage space includes: the first weight parameter matrix that needs to be learned in the feedforward neural network FNN calculation and the second weight parameter matrix .

4. The method according to claim 3, wherein: The segmenting of the model parameters of the large language model whose occupied space is greater than a preset condition includes: Calculate the amount of data obtained from the flash array of the flash memory in one concurrent read operation ; Calculate the number of neurons that store weight parameter matrices continuously ,in, Represents the storage space required for the weight parameters corresponding to each neuron; Based on the number of neurons that can be read simultaneously , will be continuous The neurons are marked as a group, and the size of each group is neurons; Based on the size of the flash array, the first weight parameter matrix and the second weight parameter matrix Segmented storage, each segment as a group, corresponding to continuous The weight parameters required for each neuron.

5. The method according to claim 4, wherein: The amount of data The size of the page depends on the storage space allocation information of the flash memory array, and the storage space allocation information includes the number of channels, the number of storage areas, and the size of each page.

6. The method according to claim 1, wherein: The step of storing the segmented model parameters in the flash memory comprises: According to usage requirements, the segmented model parameters are stored in the SLC block area and the TLC block area of ​​the flash memory.

7. The method according to claim 4, wherein: The step of storing the segmented model parameters in the flash memory comprises: Writing the segmented weight parameters into the TLC block area of ​​the flash memory in groups, and recording in the DRAM the mapping relationship between the data segment number LCA accessed by the system application and the actual physical address PPA of each segmented weight parameter group in the storage array; According to a preset strategy, the corresponding weight parameters in the TLC block area are simultaneously stored in the SLC block area of ​​the flash memory.

8. The method according to claim 7, wherein: The preset strategy is: storing a preset number of weight parameters of a specified storage space size into the SLC block area in the order in which they are stored in the TLC block area; or, The weight parameters frequently used in the FNN operation are stored in the SLC block area.

9. The method according to claim 7, wherein: The step of writing the segmented weight parameters into the TLC block area of ​​the flash memory according to the groups includes: The segmented weight parameters are written into the TLC block area in groups. When writing, the data segment number is the group number, and the mapping relationship between the LCA accessed by the system application and the actual physical address of the segmented weight parameter set in the storage array is recorded in the DRAM; the process of writing the segmented weight parameters into the TLC block area in groups is repeated until all the weight parameters are written into the TLC block area.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method for implementing large language model deployment as described in any one of claims 1-9.

11. A computer device comprising a memory and a processor, wherein: The memory stores the following instructions executable by the processor: used to execute the steps of the method for implementing large language model deployment as described in any one of claims 1-9.

12. A device for implementing large language model deployment, characterized in that: It includes a segmentation module and a storage module; wherein, A segmentation module, for segmenting model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in a flash memory array; The storage module is used to store the segmented model parameters in the flash memory.

13. The device according to claim 12, wherein: The model parameters that occupy a space greater than a preset condition include: model weight information that requires a large amount of storage space.

14. The device according to claim 12, wherein: The storage module is used to store the segmented model parameters in the SLC block area and the TLC block area of ​​the flash memory according to usage requirements.

Images