A method and device for implementing large language model deployment

Abstract

The present application discloses a method and apparatus for implementing the deployment of a large language model, by storing some model parameters of the large language model that occupy more space than a preset condition in the SLC and TLC areas of the flash memory, and dynamically loading them into the DRAM as needed, thereby reducing the demand for DRAM capacity by the large language model. The embodiment of the present application is particularly suitable for mobile terminal devices with limited DRAM capacity, and realizes the efficient operation of the large language model. In the embodiment of the present application, based on the sparsity characteristics of the large language model, the model parameters are reasonably allocated and stored in the SLC and TLC areas of the flash memory, making full use of the fast read and write performance of the flash memory in the SLC mode and the large capacity advantage in the TLC mode.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the field of storage system technology, and specifically to a method and apparatus for implementing large language model deployment. Background Art

[0002] Large Language Models (LLMs) are natural language processing models based on deep learning. They typically have hundreds of millions or even hundreds of billions of parameters and demonstrate powerful generation and comprehension capabilities across a wide range of language tasks. These models use self-attention mechanisms (such as the Transformer architecture) to process large amounts of text data. By learning from extensive training data, they can effectively perform a variety of natural language processing tasks, such as text generation, question answering, translation, and dialogue systems.

[0003] LLMs are at the heart of modern natural language processing, delivering exceptional performance across a wide range of tasks. However, these models have high computational and memory requirements, posing significant challenges on mobile devices with limited dynamic random access memory (DRAM). According to the scaling law for large models, larger model parameters increase performance, but this also requires more memory and computational resources to support inference. During a typical LLM inference phase, all model parameters must be loaded into DRAM. For example, a half-precision LLM with 7 billion parameters would require over 14GB of storage space to fully load into DRAM. However, given the generally limited DRAM capacity of mainstream mobile devices, directly loading such large models onto these devices is extremely difficult or even impossible. Therefore, efficiently deploying large language models on mobile devices with limited DRAM has become a pressing challenge. Summary of the Invention

[0004] The present application provides a method and apparatus for implementing large language model deployment, which can solve the problem of loading LLM that exceeds the available DRAM capacity.

[0005] An embodiment of the present invention provides a method for implementing large language model deployment, including:

[0006] Segmenting model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in a flash memory array;

[0007] Store the segmented model parameters into flash memory.

[0008] In an exemplary embodiment, the model parameters that occupy a space greater than a preset condition include model weight information that requires a large amount of storage space.

[0009] In an exemplary embodiment, the structure of the large language model is a Transformer model;

[0010] The model weight information that requires a large amount of storage space includes: the first weight parameter matrix that needs to be learned in the feedforward neural network FNN calculation and the second weight parameter matrix .

[0011] In an exemplary embodiment, segmenting the model parameters of the large language model whose occupied space is larger than a preset condition includes:

[0012] Calculate the amount of data obtained from the flash array of the flash memory in a concurrent read operation ;

[0013] Calculate the number of neurons that continuously store weight parameter matrices ,in, Represents the storage space required for the weight parameters corresponding to each neuron;

[0014] Based on the number of neurons that can be read simultaneously , will be continuous neurons are marked as a group, and the size of each group is neurons;

[0015] Based on the size of the flash memory array, the first weight parameter matrix and the second weight parameter matrix Segmented storage, each segment as a group, corresponding to continuous The weight parameters required for each neuron.

[0016] In an exemplary embodiment, the amount of data The size of the page depends on the storage space allocation information of the flash memory array, and the storage space allocation information includes the number of channels, the number of storage areas, and the size of each page.

[0017] In an exemplary embodiment, storing the segmented model parameters in a flash memory includes:

[0018] According to usage requirements, the segmented model parameters are stored in the SLC block area and the TLC block area of the flash memory.

[0019] In an exemplary embodiment, storing the segmented model parameters in a flash memory includes:

[0020] Writing the segmented weight parameters into the TLC block area of the flash memory in groups, and recording in the DRAM the mapping relationship between the data segment number LCA accessed by the system application and the actual physical address PPA of each segmented weight parameter group in the storage array;

[0021] According to a preset strategy, the corresponding weight parameters in the TLC block area are simultaneously stored in the SLC block area of the flash memory.

[0022] In an exemplary embodiment, the preset strategy is: storing a preset number of weight parameters of a specified storage space size in the SLC block area in the order in which they are stored in the TLC block area; or

[0023] The weight parameters frequently used in the FNN operation are stored in the SLC block area.

[0024] In an exemplary embodiment, writing the segmented weight parameters into the TLC block area of the flash memory in groups includes:

[0025] The segmented weight parameters are written into the TLC block area in groups. When writing, the data segment number is the group number, and the mapping relationship between the LCA accessed by the system application and the actual physical address of the segmented weight parameter set in the storage array is recorded in the DRAM; the process of writing the segmented weight parameters into the TLC block area in groups is repeated until all the weight parameters are written into the TLC block area.

[0026] An embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the above methods for implementing large language model deployment.

[0027] An embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores the following instructions that can be executed by the processor: used to execute the steps of any of the above-mentioned methods for implementing large language model deployment.

[0028] The embodiment of the present application further provides a device for implementing large language model deployment, including a segmentation module and a storage module; wherein,

[0029] A segmentation module, configured to segment model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in the flash memory array;

[0030] The storage module is used to store the segmented model parameters in a flash memory.

[0031] In an exemplary embodiment, the model parameters that occupy a space greater than a preset condition include model weight information that requires a large amount of storage space.

[0032] In an exemplary embodiment, the storage module is used to store the segmented model parameters in the SLC block area and the TLC block area of the flash memory according to usage requirements.

[0033] The method for implementing large language model deployment provided by the embodiment of the present application stores some model parameters (such as model weight information that needs to be trained) in the large language model that occupy more space than the preset conditions in the SLC and TLC areas of the flash memory, and dynamically loads them into the DRAM as needed. In this way, the demand for DRAM capacity by the large language model is reduced. The embodiment of the present application is particularly suitable for mobile terminal devices with limited DRAM capacity, and realizes the efficient operation of the large language model. In the embodiment of the present application, based on the sparsity characteristics of the large language model, the model parameters are reasonably allocated and stored in the SLC and TLC areas of the flash memory, making full use of the fast read and write performance of the flash memory in the SLC mode and the large capacity advantage in the TLC mode.

[0034] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0036] Figure 1 A flowchart of a method for implementing large language model deployment in an embodiment of the present application;

[0037] Figure 2 This is an example schematic diagram of segmenting a weight parameter matrix that needs to be trained for a large language model in an embodiment of the present application;

[0038] Figure 3 This is an example diagram of storing the segmented model weight information into the SLC block and the TLC block in an embodiment of the present application;

[0039] Figure 4 This is a schematic diagram of a system embodiment for optimizing FNN calculations based on the method for implementing large language model deployment provided in an embodiment of the present application;

[0040] Figure 5Schematic diagram of the composition structure of the device for implementing large language model deployment in an embodiment of the present application. DETAILED DESCRIPTION

[0041] To make the purpose, technical solutions and advantages of this application more clear, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other in any way.

[0042] To facilitate understanding of the present application, the present application will be described more fully below with reference to the accompanying drawings. The accompanying drawings provide embodiments of the present application. However, the present application may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to make the disclosure of the present application more thorough and comprehensive.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0044] It should be understood that the terms "first" and "second" used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0045] It can be understood that the “connection” in the following embodiments should be understood as “electrical connection”, “communication connection”, etc. if there is transmission of electrical signals or data between the connected circuits, modules, units, etc.

[0046] As used herein, the singular forms "a," "an," and "the" may also include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include," "comprising," "having," and the like specify the presence of stated features, integers, steps, operations, components, parts, or combinations thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, components, parts, or combinations thereof. Furthermore, the term "and / or" as used in this specification includes any and all combinations of the relevant listed items.

[0047] When deploying LLM on a mobile terminal device, due to its high demand for DRAM capacity, large models (such as 7B parameter models) may be difficult to fully load into the device's memory. Through research on related technologies, the inventors of this application found that the root of this problem lies in the contradiction between the parameter scale of large language models and memory requirements. Specifically, the inventors of this application found that the design of related technologies did not take into account the physical read, write and erase limitations of flash memory. The model parameters were not segmented according to the block size or page size in the flash memory array. The performance enhancement of read parallelism of multiple flash blocks was not considered. The performance difference between read and write performance in different flash block modes, such as single-level cell (SLC) and multi-level cell (TLC) modes, was not fully utilized. There was no optimization for high-throughput random reads.

[0048] To this end, an embodiment of the present application provides a method for implementing large language model deployment to solve the problem of loading LLM that exceeds the available DRAM capacity. Figure 1 This is a flow chart of a method for implementing large language model deployment in an embodiment of the present application. Figure 1 As shown, this may include:

[0049] Step 100: Segment model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in a flash memory array.

[0050] In an exemplary instance, model parameters that occupy more space than the preset conditions may include, but are not limited to, model weight information that requires a large amount of storage space. In one embodiment, the preset condition may be that the storage requirement of the model reaches a preset threshold, such as 100G, or 50G, or 200G, etc. In this case, model parameters that exceed the threshold are considered to be model parameters that occupy more space than the preset conditions. It should be noted that the preset threshold can be adjusted according to the actual application scenario in order to free up storage space in DRAM to better suit scenarios where the DRAM capacity of mainstream mobile terminal devices is generally small.

[0051] Flash memory, a new type of storage device, has been widely adopted in consumer PCs and enterprise servers due to its significant advantages over traditional mechanical hard drives in terms of capacity, reliability, read / write speed, shock and drop resistance, power consumption, and noise. A flash memory array primarily consists of multiple Nand Flash dies. These dies form multiple channels and banks, enabling concurrent read and write operations. Each die consists of multiple blocks, each of which contains multiple pages. Each page is physically sized, for example, 16KB.

[0052] In one exemplary embodiment, the channels, banks, blocks, and pages in a flash memory array can be considered as the way the storage space in the flash memory array is allocated. These elements constitute the hierarchical structure of the flash memory array and determine the specific storage location of data at the physical level.

[0053] In an exemplary embodiment, the structure of the large language model can be a Transformer model. The model weight information of the large language model in step 100 that requires a large amount of storage space includes: the first weight parameter matrix that needs to be learned in the feedforward neural network (FNN) calculation and the second weight parameter matrix Step 100 may include:

[0054] First, calculate the amount of data retrieved from the flash array in a concurrent read operation. In one embodiment, the amount of data The storage space allocation information depends on the structure of the flash memory array. The storage space allocation information may include the number of channels, the number of banks, and the size of each page. In one embodiment, the data volume .

[0055] Next, calculate the number of neurons that continuously store the weight parameter matrix ,in, Represents the storage space required for the weight parameters corresponding to each neuron, It is also the number of neurons that can be read simultaneously.

[0056] Then, based on the number of neurons that can be read simultaneously , will be continuous The neurons are marked as a group, and the size of each group is neurons, which ensures that in each read operation, The weight parameters of each neuron. For example: The neurons are numbered from 1 to ; The neurons are numbered from arrive .

[0057] Finally, based on the size of the flash array, the first weight parameter matrix and the second weight parameter matrix Segmented storage, each segment as a group, corresponding to continuous The weight parameters required by each neuron. In this way, each read can be performed in groups, so that enough parameters can be obtained for calculation in one operation, reducing the performance overhead caused by multiple accesses. The first weight parameter matrix and the second weight parameter matrix Segment according to the size of the flash array, such as Figure 2 As shown, each group corresponds to a continuous The weight parameters required for neuron calculation.

[0058] Through the segmented storage method in step 100, the system can significantly improve the efficiency of reading model weights from the flash memory array. Especially in the reasoning process of large models, this method reduces the frequency of I / O operations and improves the throughput of data access. At the same time, due to the rationality of group division, it is ensured that the weight data obtained in a concurrent read operation can meet the computing requirements to the greatest extent, thereby reducing the situation of redundant reading. The embodiment of the present application performs a weight segmented storage design for the efficient reasoning scenario of large models, improves the overall performance, and reduces the burden on the storage device.

[0059] Take, for example, the Transformer-based architecture of large language models. Depending on the task requirements, the Transformer architecture primarily consists of three types: decoder-only, encoder-only, and encoder-decoder. Models trained using the decoder-only architecture can handle a wide range of text generation tasks and can also understand text content through unsupervised learning, eliminating the need for specialized training and tuning for each task. This demonstrates great flexibility and broad application potential. In related technologies, the decoder-only architecture is generally used for large language models.

[0060] Taking the Transformer model based on the decoder-only architecture as an example, it consists of multiple identical layers, and each layer is mainly divided into two parts: the self-attention mechanism (Self-Attention) and the feedforward neural network (FNN).

[0061] The Transformer's self-attention mechanism allows the model to consider all other elements in the sequence when processing any element in the sequence and assign different levels of importance to different elements. Through the self-attention mechanism, the model is able to better handle complex patterns and long-range dependencies. The Transformer's self-attention mechanism assigns different levels of importance to each element in the sequence by considering all other elements simultaneously. The specific process is as follows:

[0062] Input representation: Assume there is a text input of length n, that is, the input sequence has n words. Each word in the text is represented by a vector of length d. Then the input text is represented as a matrix X, which is a matrix with n rows and d columns.

[0063] For each word, the model generates three matrices: a query matrix Q representing the query vector of each word in the text, a key matrix K representing the key vector of each word in the text, and a value matrix V representing the value vector of each word in the text. These matrices are composed of the input X and the parameter matrix to be learned. 、 、 Performing matrix multiplication yields: 、 、 ,in, 、 、 is the parameter matrix that needs to be learned;

[0064] Calculate the similarity score between each word in the input text and other words to obtain the weight distribution of each word in the text. Here, we can calculate the dot product of the query vector Q and the key vector K, and convert the similarity into a weight distribution through the softmax function to calculate the self-attention output ,in, is the number of columns of the query matrix Q.

[0065] The Transformer's FNN layer consists of two linear transformations and , a nonlinear activation function is inserted in the middle . Transformer's nonlinear activation function Using the ReLU activation function, the FNN formula is as follows: ;in, and is the weight parameter matrix that needs to be learned, for matrix, for Matrix. Simply put, FFN can be thought of as a key-value memory that stores a large amount of specific knowledge. The first layer of FFN is the key layer, which is used to identify language or knowledge patterns in the input; the second layer is the value layer, which is used to store the specific knowledge corresponding to the key layer. The input layer of FFN is the overall information of the entire input text obtained through self-attention. The forward propagation process of FFN can be thought of as detecting knowledge patterns through the key layer, extracting the corresponding value, and reflecting it in the output.

[0066] The parameters involved in the Transformer's Self-Attention and FNN calculation processes are the main components of the Transformer model parameters. Among them, the Self-Attention part accounts for approximately one-third of the total parameters, and the FNN part parameters account for approximately two-thirds of the total parameters. In the embodiment of the present application, the weight parameters of the Attention layer are stored in DRAM, and this part does not involve dynamic loading of parameters. The characteristic of the ReLU activation function is that it outputs non-negative values. When the input is less than 0, the output is 0. This results in more than 90% of the intermediate outputs in the FNN layer being 0, resulting in a high degree of sparsity. This means that when passing through the FNN layer, only neurons with inputs greater than 0 will be activated, while the outputs of most neurons will be 0, resulting in a high degree of sparsity in the FNN layer. The sparsity of the FNN layer depends on the current input, that is, during each inference, only a small number of neurons are actually activated, and most neurons do not participate in the inference operation. Therefore, for the weight parameters of the FNN layer, only the parameters corresponding to the active neurons need to be loaded, and there is no need to load the parameters of the entire model, which greatly reduces the load on DRAM. ‌

[0067] In an embodiment of the present application, the sparsity of FNN is utilized, and during the inference process, only the weight parameters corresponding to the activated neurons are loaded into DRAM. This dynamic loading method avoids unnecessary memory usage and optimizes the efficiency of DRAM use. Moreover, due to the sparsity of the FNN layer, most weight parameters do not need to participate in the calculation during the inference process. By only loading the weight parameters of the active part, the demand for DRAM capacity is greatly reduced, making it possible to infer large models with limited hardware resources. The embodiment of the present application not only maintains the computational efficiency of the model, but also optimizes the use of hardware resources. It is a very effective strategy for deploying Transformer models in resource-constrained environments.

[0068] The FNN formula is The first weight parameter matrix in and the second weight parameter matrix It needs to be learned, and its size accounts for an important proportion of the size of the entire large model parameters; for the first weight parameter matrix and the second weight parameter matrix , the weight parameters of each neuron correspond to the first weight parameter matrix and the second weight parameter matrix A column, The weight parameters of neurons correspond to their When the When a neuron is activated, the first weight parameter matrix needs to be read from the flash array at the same time and the second weight parameter matrix Therefore, in order to improve the reading speed, in the embodiment of the present application, the first weight parameter matrix and the second weight parameter matrix The corresponding column elements of are concatenated and stored together, with a storage space of size S. This allows both parts of data to be read at once during access. This storage method reduces read latency and fully utilizes the parallel read and write capabilities of the flash array.

[0069] Step 101: Store the segmented model parameters in a flash memory.

[0070] In a flash array, data reads and writes follow a specific sequence, first determining the flash array's channel and bank, then determining the block and page locations. When a system application writes data to the flash array, it accesses the logical cluster address (LCA) in the array's logical space. The flash array then translates the LCA into the physical page address (PPA) of the storage array—the specific channel, bank, block, and page where the data is stored. The one-to-one mapping between LCAs and PPAs forms the flash array's logical data cluster address-to-physical address mapping table. When reading data from the flash array, the system application uses the LCA to look up the mapping table, finds the corresponding PPA, retrieves the data from the physical location in the flash array, and returns it to the upper-layer application. This logical-to-physical address mapping mechanism enables the flash array to effectively manage data storage and access. Furthermore, the concurrent processing capabilities of different channels and banks significantly improve overall system read and write performance.

[0071] In an exemplary embodiment, step 101 may include: storing the segmented model parameters, such as model weight information, in an SLC block area (SLC block) and a TLC block area (TLC block) of a flash memory according to usage requirements.

[0072] The block where the flash memory stores data can be switched to an SLC block or a TLC block according to usage requirements. The capacity of an SLC block is only one-third of a TLC block, but the data read and write speeds of an SLC block are faster than those of a TLC block. Figure 3 As shown, the flash memory array is divided into an SLC block area and a TLC block area, and is divided into two banks, each bank including multiple channels (such as Channel 0 and Channel 1). For the SLC block, each cell stores 1 bit of data, has faster read and write speeds, higher endurance, but smaller capacity (only one-third of the capacity of the TLC block); for the TLC block, each cell stores 3 bits of data, has slower read and write speeds than the SLC block, but provides larger storage capacity. In one embodiment, in order to optimize the storage of model parameters, the SLC block can be used to store parameters that are accessed more frequently and have higher speed requirements because it has faster read and write speeds; while the TLC block is suitable for storing parameters that are accessed less frequently or have larger capacity. Although it is slower, it can maximize storage capacity.

[0073] In this embodiment, the flash memory controller dynamically manages block allocation based on usage. For example, certain blocks can switch between SLC and TLC modes based on usage to balance performance (speed) and storage capacity. This hybrid storage strategy leverages the high speed of SLC and the high capacity of TLC, ensuring efficient storage of model parameters in the flash memory array.

[0074] In an exemplary embodiment, step 101 may include:

[0075] The segmented weight parameters are written into the TLC block area of the flash memory in groups, and the mapping relationship between the LCA accessed by the system application and the actual physical address of each segmented weight parameter group in the storage array is recorded in DRAM;

[0076] According to a preset strategy, the corresponding weight parameters in the TLC block area are simultaneously stored in the SLC block area of the flash memory.

[0077] In one embodiment, the preset strategy may be to store a preset number of weight parameters of a specified storage space size in the SLC block area in the order in which they are stored in the TLC block area. For example, the weight parameter of the first 1G and the weight parameter of the second 1G are stored in the SLC block area at the same time.

[0078] In one embodiment, the preset strategy may be to store weight parameters frequently used in FNN operations in the SLC block area. For example, weight parameters that are easily hit may be stored in the SLC block area based on historical statistics.

[0079] In an exemplary embodiment, writing the segmented weight parameters into the TLC block area of the flash memory in groups may include:

[0080] The weight parameters after segmentation are divided into groups. When writing to the TLC Block, the data segment number for , and record the mapping relationship between the LCA accessed by the system application and the actual physical address of the segmented weight parameter set in the storage array in DRAM, that is, < , >; Repeat the segmented weight parameter set The process of writing to the TLC Block continues until all weight parameter sets are written to the TLC Block of the storage array. In this way, a set of mapping relationships will be stored in DRAM. , >,....< , >.

[0081] The method for implementing the deployment of a large language model provided in an embodiment of the present application reduces the demand for DRAM capacity for the large language model by storing some model parameters (such as model weight information that needs to be trained) in the SLC and TLC areas of the flash memory that occupy more space than a preset condition in the large language model, and dynamically loading them into the DRAM as needed. The embodiment of the present application is particularly suitable for mobile terminal devices with limited DRAM capacity, and realizes the efficient operation of the large language model. In the embodiment of the present application, based on the sparsity characteristics of the large language model, the model parameters are reasonably allocated and stored in the SLC and TLC areas of the flash memory, making full use of the fast read and write performance of the flash memory in the SLC mode and the large capacity advantage in the TLC mode.

[0082] In an exemplary embodiment, the present application also includes:

[0083] The model parameters are re-encoded and stored on the same page of different flash memories, which facilitates parallel reading of multiple flash memories and improves the model parameter loading performance.

[0084] The following is an example of an embodiment to illustrate the method for implementing a large language model deployment based on the embodiment of the present application, and to optimize the calculation process of the FNN. Figure 4 As shown, the system includes a control center, DRAM and Nand Flash.

[0085] First, the control center determines which neurons are in the active state based on the input of the FNN, and determines the group to which they belong based on the number of the activated neurons. For example, if neurons Ni, Nm, and Nj are in the active state, the control center will determine the group number to which they belong based on the number (for example, the group number of neuron Ni is ). Each time the FNN input is processed, the control center also records the number of activated neurons in each group C[ ];

[0086] Next, the control center queries the cache management table T in DRAM. The cache management table T contains the physical storage address of the group number and its corresponding weight parameter in DRAM, that is,<Group,physical address> If the query result shows, The corresponding weight parameters have been cached in DRAM, and the system can directly obtain the weight parameters from DRAM and participate in the calculation of FNN; if they are not cached in DRAM, the control center will The number is used as LCA to query the mapping table in DRAM<LCA,PPA> , get The corresponding weight parameters are stored in the PPA of the flash array, and the corresponding data is accessed from NandFlash;

[0087] When the system starts, The corresponding weight parameters are in the TLC Block area. The control center will read them from the TLCBlock The corresponding weight parameters are loaded into DRAM to participate in the calculation of FNN. In this process, the cache management table T needs to be updated to The location of its weight parameters in DRAM is added to the cache management table T. In order to ensure that the DRAM cache area is not overloaded, the control center will record the number of activated neurons C[ ], delete the groups with the least activated neurons to make room for new groups, so that the control center can The DRAM address of the corresponding weight parameter is added to the position of the deleted group in the cache management table T;

[0088] After completing the above operations, the control center will 、 as well as The weight parameters are read from the TLC Block and written to the SLC Block with higher performance. At the same time, the mapping table in the DRAM is updated to reflect the new storage location of these weight parameters in the SLC Block. , >,< , > and < , >Update the mapping relationship stored in DRAM;

[0089] If the control center needs to update a certain The control center first queries the cache management table T to determine the weight parameter Is it in DRAM? If In DRAM, the new weight parameters are written to The location of the weight parameters in DRAM; if If it is not in DRAM, The new weight relationship parameters are written into the SLC Block and a new mapping relationship is generated. , >, and < , > Synchronously update the mapping table in DRAM. In subsequent calculations, the control center reads When the weight parameter is obtained, if the cache management table T is hit, the weight parameter is read from DRAM to participate in the operation in FNN, otherwise the mapping relationship is queried to obtain ,according to Read the weight parameters from the SLC block to participate in the calculation in the FNN.

[0090] Through the method for implementing large-scale language model deployment provided in the embodiment of the present application, the calculation process of the FNN realizes the intelligent scheduling of weight parameters between DRAM and Nand Flash, ensuring the high efficiency of the feedforward neural network during calculation and the timeliness of data access.

[0091] The present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute any of the above-described methods for implementing large-scale language model deployment.

[0092] An embodiment of the present disclosure further provides a computer device, including a memory; and a processor connected to the memory, wherein the processor executes the steps of the method for implementing large language model deployment as described in any of the preceding items based on instructions stored in the memory.

[0093] Figure 5 This is a schematic diagram of the structure of the device for implementing large language model deployment in an embodiment of the present application. Figure 5 As shown, it may include: a segmentation module, a storage module; wherein,

[0094] A segmentation module, configured to segment model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in the flash memory array;

[0095] The storage module is used to store the segmented model parameters in a flash memory.

[0096] In an exemplary embodiment, the model parameters that occupy a space greater than a preset condition may include, but are not limited to, model weight information that requires a large amount of storage space.

[0097] In an exemplary embodiment, the model weight information of the large language model that requires a large amount of storage space includes: a first weight parameter matrix and the second weight parameter matrix ; The segmentation module can be used to:

[0098] Calculate the amount of data retrieved from the flash array in a concurrent read operation ; Calculate the number of neurons that continuously store weight parameter matrices ,in, Represents the storage space required for the weight parameters corresponding to each neuron, It is also the number of neurons that can be read simultaneously; based on the number of neurons that can be read simultaneously , will be continuous The neurons are marked as a group, and the size of each group is neurons; based on the size of the flash array, the first weight parameter matrix and the second weight parameter matrix Segmented storage, each segment as a group, corresponding to continuous The weight parameters required for each neuron.

[0099] In one exemplary embodiment, the storage module may be used to:

[0100] According to usage requirements, the segmented model parameters are stored in the SLC block area and TLC block area of the flash memory.

[0101] In an exemplary embodiment, the storage module is specifically configured to:

[0102] The segmented weight parameters are written into the TLC block area of the flash memory in groups, and the mapping relationship between the LCA accessed by the system application and the actual physical address of each segmented weight parameter group in the storage array is recorded in DRAM;

[0103] According to a preset strategy, the corresponding weight parameters in the TLC block area are simultaneously stored in the SLC block area of the flash memory.

[0104] In one embodiment, the preset strategy may be to store a preset number of weight parameters of a specified storage space size in the SLC block area in the order in which they were stored in the TLC block area. Alternatively, the weight parameters frequently used in FNN calculations may be stored in the SLC block area.

[0105] In an exemplary embodiment, writing the segmented weight parameters into the TLC block area of the flash memory in groups in the storage module may include:

[0106] The segmented weight parameters are written into the TLC Block in groups. When writing, the data segment number The group number is , and record the mapping relationship between the LCA accessed by the system application and the actual physical address of the segmented weight parameter set in the storage array in DRAM, that is, < , > Repeat the process of writing the segmented weight parameters into the TLCBlock in groups until all weight parameters are written into the TLC Block of the flash array.

[0107] In an exemplary embodiment, the storage module is further configured to:

[0108] The model parameters are re-encoded and stored on the same page of different flash memories. This facilitates parallel reading of multiple flash memories, thereby improving the model parameter loading performance.

[0109] The device for implementing the deployment of a large language model provided in an embodiment of the present application stores some model parameters (such as model weight information that needs to be trained) in the large language model that occupy more space than a preset condition in the SLC and TLC areas of the flash memory, and dynamically loads them into DRAM as needed, thereby reducing the demand for DRAM capacity by the large language model. The embodiment of the present application is particularly suitable for mobile terminal devices with limited DRAM capacity, and realizes the efficient operation of the large language model. In the embodiment of the present application, based on the sparsity characteristics of the large language model, the model parameters are reasonably allocated and stored in the SLC and TLC areas of the flash memory, making full use of the fast read and write performance of the flash memory in SLC mode and the large capacity advantage in TLC mode.

[0110] Although the embodiments disclosed in this application are as described above, the contents described are merely embodiments adopted to facilitate understanding of this application and are not intended to limit this application. Any person skilled in the art to which this application belongs may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in this application. However, the scope of patent protection of this application shall still be based on the scope defined by the attached claims.

Claims

1. A method for implementing large language model deployment, characterized in that: include: Segmenting model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in a flash memory array; The model parameters that occupy a space greater than the preset condition include: model weight information that requires a large amount of storage space; the model weight information that requires a large amount of storage space includes: a first weight parameter matrix W1 and a second weight parameter matrix W2 that need to be learned in the calculation of the feedforward neural network FNN; the segmenting of the model parameters of the large language model that occupy a space greater than the preset condition includes: calculating the amount of data T obtained from the flash memory array of the flash memory in a concurrent read operation; calculating the number of neurons K=T / S for continuously storing the weight parameter matrix, where S represents the storage space required for the weight parameters corresponding to each neuron; based on the number of neurons K that can be read simultaneously, marking K consecutive neurons as a group, and the size of each group is K neurons; based on the size of the flash memory array, storing the first weight parameter matrix W1 and the second weight parameter matrix W2 in segments, with each segment as a group corresponding to the weight parameters required for K consecutive neurons; The segmented model parameters are stored in the flash memory, including: writing the segmented weight parameters into the TLC block area of the flash memory in groups, and recording the mapping relationship between the data segment number LCA accessed by the system application and the actual physical address PPA of each segmented weight parameter group in the storage array in the DRAM; according to a pre-set strategy, the corresponding weight parameters in the TLC block area are simultaneously stored in the SLC block area of the flash memory.

2. The method according to claim 1, wherein The structure of the large language model is a Transformer model.

3. The method according to claim 2, wherein: The size of the data volume T depends on the storage space allocation information of the flash memory array, and the storage space allocation information includes the number of channels, the number of storage areas, and the size of each page.

4. The method according to claim 1, wherein The preset strategy is: storing a preset number of weight parameters of a specified storage space size into the SLC block area in the order in which they are stored in the TLC block area; or, The weight parameters frequently used in the FNN operation are stored in the SLC block area.

5. The method according to claim 1, wherein Writing the segmented weight parameters into the TLC block area of the flash memory according to groups includes: The segmented weight parameters are written into the TLC block area in groups. When writing, the data segment number is the group number, and the mapping relationship between the LCA accessed by the system application and the actual physical address of the segmented weight parameter set in the storage array is recorded in the DRAM; the process of writing the segmented weight parameters into the TLC block area in groups is repeated until all the weight parameters are written into the TLC block area.

6. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method for implementing large language model deployment as described in any one of claims 1-5.

7. A computer device comprising a memory and a processor, wherein: The memory stores the following instructions that can be executed by the processor: used to execute the steps of the method for implementing large language model deployment as described in any one of claims 1-5.

8. A device for implementing large language model deployment, characterized in that: It includes segmentation module and storage module; wherein, A segmentation module is configured to segment the model parameters of a large language model whose occupied space is greater than a preset condition according to storage space allocation information in a flash memory array; the model parameters whose occupied space is greater than the preset condition include: model weight information requiring a large amount of storage space; the model weight information requiring a large amount of storage space includes: a first weight parameter matrix W1 and a second weight parameter matrix W2 required to be learned in a feedforward neural network FNN calculation; segmenting the model parameters of a large language model whose occupied space is greater than the preset condition includes: calculating the amount of data T obtained from the flash memory array of the flash memory in a concurrent read operation; calculating the number of neurons K=T / S for continuously storing the weight parameter matrix, where S represents the storage space required for the weight parameter corresponding to each neuron; based on the number of neurons K that can be read simultaneously, marking K consecutive neurons as a group, with the size of each group being K neurons; based on the size of the flash memory array, storing the first weight parameter matrix W1 and the second weight parameter matrix W2 in segments, with each segment being a group corresponding to the weight parameters required for K consecutive neurons; A storage module is used to store the segmented model parameters in the flash memory, including: writing the segmented weight parameters into the TLC block area of the flash memory in groups, and recording in the DRAM the mapping relationship between the data segment number LCA accessed by the system application and the actual physical address PPA of each segmented weight parameter group in the storage array; according to a pre-set strategy, the corresponding weight parameters in the TLC block area are simultaneously stored in the SLC block area of the flash memory.

Images