Data storage method, medium and computer program product for large models

Abstract

The application provides a data storage method, medium and computer program product for a large model. The data storage method for the large model comprises: in an inference stage of the large model, organizing quantized data to be stored and corresponding quantized parameters in a same storage tensor, wherein the storage tensor has multiple dimensions, and from an outermost layer to an innermost layer dimension, the dimensions comprise: a block dimension corresponding to a physical storage block in a paging management mechanism, each physical storage block having a continuous physical address range; an attention head dimension corresponding to an attention head defined for the large model; a word element dimension corresponding to a word element generated in the inference stage; a quantization grouping dimension corresponding to a quantization grouping of each word element on a feature dimension of each attention head; and a data dimension organizing quantized data and quantized parameters of each quantization grouping; and based on a logical storage structure of the storage tensor, storing the quantized data and the quantized parameters in a physical storage block of a cache.

Description

Technical Field

[0001] This disclosure relates to data storage methods for large models, non-transitory computer-readable media, and computer program products. Background Technology

[0002] In recent years, the application of large-scale model inference has rapidly become widespread. During the inference phase of large models, especially in the autoregressive generation or decoding phase, the model generates output sequences on a token-by-token basis. In each generation step used to generate new tokens, it is necessary to read all key and value tensors corresponding to the historical context or tokens stored in the key cache and value cache (collectively referred to as the Key / Value Cache, KVCache), and update the KVCache to add the newly generated key and value tensors in the current generation step. Therefore, the KVCache is frequently read and updated. With the increase in output sequence length and concurrent processing requests, the storage space occupied by the KVCache and the memory access bandwidth overhead associated with accessing the KVCache increase dramatically. This often limits the inference system to input / output (I / O) or bandwidth, becoming a key bottleneck affecting system performance such as throughput and latency. Summary of the Invention

[0003] In one aspect, this application discloses a data storage method for a large model, characterized by comprising: during the inference phase of the large model, organizing quantized data to be stored and corresponding quantization parameters within the same storage tensor, wherein the storage tensor has multiple dimensions, the multiple dimensions from the outermost dimension to the innermost dimension sequentially including: a block dimension, corresponding to a physical storage block in a paging management mechanism, wherein the paging management mechanism is used to manage the cache to which the quantized data and the quantization parameters are to be stored, the cache including multiple physical storage blocks, and each physical storage block having a continuous physical address range; an attention head dimension, corresponding to an attention head defined for the large model; a lexical dimension, corresponding to a lexical generated during the inference phase; a quantization grouping dimension, corresponding to the quantization grouping of each lexical on the feature dimension of each attention head; and a data dimension, used to organize the quantized data and the quantization parameters of each quantization group; and storing the quantized data and the quantization parameters in the physical storage blocks of the cache based on the logical storage structure of the storage tensor.

[0004] In other respects, this application discloses a non-transitory computer-readable medium storing instructions and a computer program product including instructions. These instructions, when executed by one or more processors, cause the processors to perform the methods described in this application. Attached Figure Description

[0005] When read in conjunction with the accompanying drawings, various aspects of this disclosure are best understood through the following detailed description. It should be noted that, in accordance with standard practice in the art, the features are not drawn to scale. In fact, for clarity of discussion, the dimensions of the features may be arbitrarily increased or decreased.

[0006] Figure 1 A schematic diagram of a data storage method for a large model according to an embodiment of this application is shown.

[0007] Figure 2A and Figure 2B A schematic diagram of the layout of data stored in a physical storage block using a data storage method according to an embodiment of this application is shown.

[0008] Figure 3 A schematic diagram of a computing device that can implement embodiments of this application is shown. Detailed Implementation

[0009] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this disclosure. Of course, these are merely examples and not limiting.

[0010] To reduce the storage cost and data access overhead of KVCache, existing technologies typically employ methods such as quantization or sparsification to process key and value tensors before storing them in the KVCache. Specifically, quantization techniques such as KIVI convert the original high-precision (e.g., FP16, BF16, etc.) key and value tensors into low-bit-width (e.g., INT8, INT4, etc.) representations, sacrificing a small amount of precision in exchange for significant savings in storage space and access bandwidth.

[0011] Quantization techniques include symmetric quantization and asymmetric quantization. A quantized tensor obtained by symmetric quantization of a data tensor typically consists of at least two parts: the quantized data tensor and a quantization parameter tensor, where the quantization parameter tensor includes a scaling factor tensor. Asymmetric quantization, by introducing a zero-point offset parameter on top of symmetric quantization, can more flexibly adapt to the data distribution of the tensor, thus accommodating a wider range of dynamic data and achieving lower quantization errors within a given bit width. Therefore, a quantized tensor obtained by asymmetric quantization of a data tensor typically consists of at least two parts: the quantized data tensor and a quantization parameter tensor, where the quantization parameter tensor includes a scaling factor tensor and a zero-point offset tensor.

[0012] Current mass-produced technologies can strike a balance between compression ratio and accuracy, theoretically significantly reducing the storage capacity and access bandwidth requirements of KVCache. However, current mass-produced technologies for KVCache face significant challenges in practical engineering and system integration, mainly manifested in the following two aspects:

[0013] First, it is incompatible with the paginated KVCache management mechanisms used by mainstream inference frameworks. For example, the paginated KVCache management mechanisms (e.g., PageAttention) in mainstream inference frameworks for large models (e.g., vLLM) employ paginated / blocked KVCache organization and indexing to support high-throughput concurrent inference and dynamic scheduling. However, existing quantization techniques (e.g., KIVI) typically use specific data packaging and grouping layouts, resulting in quantized data and corresponding quantization parameters (e.g., scaling factors, zero-point offsets, etc.) being stored separately (e.g., in independent caches or buffers) or organized across block boundaries defined by the paginated management mechanism. This makes it difficult to directly map quantized data and quantization parameters to the paginated / blocked data structures of inference frameworks such as vLLM, often requiring additional data organization, rearrangement, aggregation, or copying operations, increasing implementation complexity and offsetting some of the performance gains from quantization.

[0014] Second, the minimum read / write granularity and startup overhead of dedicated training and inference devices are not adequately adapted. On many dedicated training and inference devices, a single access to on-chip or off-chip storage has a fixed minimum read / write granularity (e.g., 64 bytes), and each read / write operation incurs inherent startup overhead. If the layout of quantized data and quantization parameters is discontinuous or fragmented at this minimum read / write granularity, it generates a large number of short and discrete I / O requests, leading to increased startup overhead and decreased effective bandwidth utilization. This significantly prolongs inference time during the inference phase (e.g., autoregressive generation, decoding, etc.).

[0015] Therefore, the shortcomings of existing technologies lie in their inability to achieve the storage compression and accuracy advantages brought by quantization technology while seamlessly integrating with the paged KVCache management mechanism (e.g., PageAttention mechanism) widely used in inference frameworks (e.g., vLLM), and to perform continuous and efficient read and write operations at the smallest hardware read and write granularity. How to design a quantization data and quantization parameter storage method that can maintain the benefits of quantization, seamlessly integrate into inference frameworks, and meet the requirements of efficient hardware access has become an urgent technical problem to be solved.

[0016] To address the aforementioned issues, this paper discloses various implementation methods for data storage of large models. More specifically, it provides a data storage method for quantized data and quantization parameters of key and value tensors in large models that is compatible with a paged KVCache management mechanism. This method redesigns the storage layout of the quantized data and quantization parameters of the key and value tensors to be stored in the KVCache, enabling them to be arranged contiguously and supporting merged access at the smallest read / write granularity. Therefore, while meeting the requirements of quantization techniques such as KIVI, this method is compatible with the paged KVCache management mechanism of inference frameworks for large models, such as vLLM, and adapts to the smallest read / write granularity of the hardware. It effectively reduces short and discrete I / O requests, significantly reduces KVCache access overhead during the inference phase (e.g., autoregressive generation phase, decoding phase, etc.), thereby improving overall inference performance.

[0017] Figure 1 A schematic diagram of a data storage method 100 for a large model according to an embodiment of this application is shown. Figure 1 As shown, the method 100 may include steps S102 and S104.

[0018] In step S102, during the inference phase of the large model, the quantized data to be stored and the corresponding quantization parameters are organized within the same storage tensor. This storage tensor has multiple dimensions, which, from the outermost to the innermost dimension, can include: block dimension, attention head dimension, lexical dimension, quantization grouping dimension, and data dimension. The block dimension corresponds to the physical storage blocks in the paging management mechanism, which manages the cache where the quantized data and quantization parameters are stored. This cache includes multiple physical storage blocks, each with a contiguous range of physical addresses. The attention head dimension corresponds to the attention heads defined for the large model. The lexical dimension corresponds to the lexical units generated during the inference phase. The quantization grouping dimension corresponds to the quantization grouping of each lexical unit on the feature dimension of each attention head. The data dimension is used to organize the quantized data and quantization parameters for each quantization group.

[0019] In some embodiments, the large model mentioned in step S102 may refer to a model that uses an autoregressive generation approach for inference during the inference phase. Since autoregressive generation requires calculating the current output based on all historical contexts, such models need to maintain a cache (e.g., KVCache) or buffer during the inference phase to store all key tensors and value tensors corresponding to all historical lexical units, or their quantized data and quantization parameters (e.g., scaling factor, zero-point offset, etc.). As an example, the large model may include, but is not limited to, autoregressive generation models, such as autoregressive generation models based on the Transformer architecture. As an example, autoregressive generation models may include, but are not limited to, large language models (LLMs). Those skilled in the art will understand that the above description of the types of large models described in method 100 is for illustrative purposes only and is not intended to be limiting; the large models described in this application may also include any other suitable types of models depending on actual needs.

[0020] In some embodiments, prior to step S102, method 100 may further include: quantizing the key tensors and value tensors of the tokens generated during the inference phase of the large model, respectively, to generate corresponding quantized data and quantization parameters. As an example, quantization may include symmetric quantization, and correspondingly, the quantization parameters may include a scaling factor. As another example, quantization may include asymmetric quantization, and correspondingly, the quantization parameters may include a scaling factor and a zero-point offset. Those skilled in the art will understand that appropriate types of quantization techniques (including but not limited to symmetric quantization techniques and asymmetric quantization techniques such as KIVI) can be selected to generate the quantized data to be stored and the corresponding quantization parameters, depending on actual requirements such as accuracy and compression ratio.

[0021] In a further embodiment, quantization can be performed in batches to accommodate the word-by-word appending characteristic of autoregressive inference. Specifically, the quantization of the key tensors and value tensors of words can be performed in batches according to a predetermined quantization batch size, wherein the predetermined quantization batch size indicates the number of words included in each batch of quantization. When the accumulated number of words to be quantized reaches the predetermined quantization batch size, the key tensors and value tensors of these accumulated words are quantized together to generate low-bit-width quantized data (e.g., INT8, INT4, etc.) and the corresponding quantization parameters. When the accumulated number of words to be quantized does not fill a complete batch (i.e., less than the predetermined quantization batch size), the key tensors and value tensors of these accumulated words can first be kept in full precision, i.e., the original high precision (e.g., BF16, FP16, etc.), and then quantized together when the number of subsequently accumulated words to be quantized reaches the predetermined quantization batch size.

[0022] As an example, the predetermined quantization batch size can be set equal to the capacity of each physical storage block in the cache, i.e., the number of tokens that each physical storage block can hold. For example, if each physical storage block is designed to hold 64 tokens, the quantization batch size can be set accordingly to 64. This method of aligning the quantization batch size with the block capacity helps to efficiently and neatly store the quantized data and parameters generated in the same batch in a complete physical storage block after quantization, thereby reducing storage fragmentation and improving storage efficiency. However, those skilled in the art will understand that any appropriate quantization batch size can be used for quantization according to actual needs, and this application does not impose any limitations on this.

[0023] In some embodiments, the preset shape of the storage tensor described in step S102 can be represented as: [Block_ID, Head_ID, Token_ID, Group_Count, Data], where:

[0024] Block_ID indicates the outermost dimension of the stored tensor, i.e., the block dimension, corresponding to the physical storage block in the paging management mechanism (e.g., PageAttention mechanism) used by the inference framework of large models (e.g., vLLM). As an example, the value of Block_ID can satisfy the following formula: 0 ≤ Block_ID ≤ block_num - 1, where block_num represents the total number of physical storage blocks allocated in the cache for storing quantized data and quantization parameters in the paging management mechanism, and block_num is a positive integer. Those skilled in the art will understand that block_num can be a predetermined fixed value or a value that dynamically changes with the length of the output sequence; this application does not impose any limitations on this.

[0025] Head_ID indicates the dimension of the attention head storing the tensor, corresponding to the attention heads defined for the large model. As an example, the value of Head_ID can satisfy the following formula: 0 ≤ Head_ID ≤ head_num - 1, where head_num represents the number of attention heads defined for the large model, and head_num is a positive integer. Those skilled in the art will understand that head_num is typically a fixed value predetermined before the inference phase of the large model begins.

[0026] Token_ID indicates the token dimension of the storage tensor, corresponding to the tokens generated during the inference phase of the large model. As an example, the value of Token_ID can satisfy the following formula: 0 ≤ Token_ID ≤ block_size - 1, where block_size represents the capacity of each physical storage block, i.e., the total number of tokens contained in each physical storage block, and block_size is a positive integer. Those skilled in the art will understand that block_size can be a fixed value predetermined according to actual needs, such as 64 or 128, and this application does not impose any limitations on this.

[0027] Group_Count indicates the quantization grouping dimension of the storage tensor, corresponding to the quantization grouping of each term in each attention head's feature dimension. As an example, Group_Count can satisfy the following formula: 0 ≤ Group_Count ≤ M-1, where M represents the number of quantization groups for each term in each attention head's feature dimension, and M is a positive integer. The number of feature dimensions for each attention head, head_dim, is usually a fixed value predetermined before the inference phase of the large model begins; for example, head_dim is 128. Quantization of the key and value tensors for each term can be performed by dividing them into M groups (e.g., 2 groups) in each attention head's feature dimension. The number of features contained in each key and value tensor within each quantization group is head_dim / M. For example, with head_dim = 128 and M = 2, the number of features contained in each key and value tensor within each quantization group is 128 / 2 = 64. Those skilled in the art will understand that M can be a fixed value predetermined based on actual quantization accuracy and storage efficiency requirements, and this application does not impose any limitations on this.

[0028] The Data dimension indicates the innermost dimension of the storage tensor, i.e., the data dimension, which is used to organize the quantization data and quantization parameters for each quantization group.

[0029] Regarding quantized data, as an example, the quantized data of each quantization group can be represented in INT4 (i.e., occupying 4 bits) format in each feature dimension. Since data is usually addressed byte-wise when actually stored, two INT4 data can be packaged into one INT8 (i.e., occupying 8 bits or 1 byte) storage unit. Since the number of features contained in the quantized data of each quantization group is head_dim / M, when each feature is represented in INT4 format, the size of the quantized data of each quantization group in the data dimension is head_dim / M INT4 features. For example, taking head_dim as 128 and M=2, i.e., the number of features contained in the quantized data of each quantization group is 128 / 2=64, as an example, when each feature is represented in INT4 format, the size of the quantized data of each quantization group in the data dimension is 64 4-bit units, or 32 bytes. Those skilled in the art will understand that the above description of the format and size of the quantized data is for illustrative purposes only and is not intended to be limiting. Other appropriate formats and sizes can be selected according to actual needs, and this application does not impose any limitations in this regard.

[0030] Regarding quantization parameters, as an example, each quantization parameter (e.g., scaling factor, zero offset, etc.) for each quantization group can be represented in FP32 (i.e., occupying 32 bits, 4 bytes) format, which can be represented in the storage tensor view as follows:

[0031] FP32 view: [Block_ID, Head_ID, Token_ID, Group_Count, 1]

[0032] INT8 view: [Block_ID, Head_ID, Token_ID, Group_Count, 4]

[0033] Therefore, the size of each quantization parameter in each quantization group within the data dimension is 4 bytes. In the case of symmetric quantization, the quantization parameter includes one quantization parameter, namely the scaling factor, so the size of the quantization parameter in each quantization group within the data dimension is 4 bytes. In the case of asymmetric quantization, the quantization parameter includes two quantization parameters, namely the scaling factor and the zero-point offset, so the size of the quantization parameter in each quantization group within the data dimension is 8 bytes. Those skilled in the art will understand that the above description regarding the format and size of the quantization parameters is for illustrative purposes only and is not intended to be limiting. Other appropriate formats and sizes can be selected according to actual needs, and this application does not impose any limitations in this regard.

[0034] As an example, in the case of symmetric quantization, the data dimension of each quantization group corresponds to a predetermined number of bytes in the physical storage block of the cache. The first part of this predetermined number of bytes is used to store the quantized data, and the second part of this predetermined number of bytes is used to store the scaling factor. Taking the size of the quantized data of each quantization group in the data dimension as 32 bytes and the size of the quantization parameter as 4 bytes as an example, the predetermined number of bytes is 36 bytes, the first part is the first 32 bytes of the 36 bytes, and the second part is the last 4 bytes of the 36 bytes.

[0035] As another example, in the case of asymmetric quantization, the data dimension of each quantization group corresponds to a predetermined number of bytes in the cached physical storage block. A first portion of this predetermined number of bytes is used to store the quantized data, and a second portion is used to store the scaling factor and zero-point offset. Taking an example where the size of the quantized data for each quantization group in the data dimension is 32 bytes and the size of the quantization parameters is 8 bytes, the predetermined number of bytes is 40 bytes. The first portion is the first 32 bytes of these 40 bytes, and the second portion is the last 8 bytes of these 40 bytes.

[0036] In step S104, based on the logical storage structure of the storage tensor, the quantization data and quantization parameters are stored in the physical storage block of the cache.

[0037] In some embodiments, the data size for each attention head in each physical storage block can be expressed as: block_size × M × (size of quantized data for each quantization group + size of quantization parameters), where, as mentioned above, block_size represents the capacity of each physical storage block, i.e., the total number of tokens contained in each physical storage block, and M represents the number of quantization groups for each token in each attention head's feature dimension. Further, the size of each physical storage block can be expressed as: head_num × block_size × M × (size of quantized data for each quantization group + size of quantization parameters), where, as mentioned above, head_num represents the number of attention heads defined for a large model.

[0038] As an example, with symmetric quantization, assuming block_size=64, M=2, and head_num=2, the size of each physical memory block can be 2×64×2×(32+4)=9216 bytes. As another example, with asymmetric quantization, assuming block_size=64, M=2, and head_num=2, the size of each physical memory block can be 2×64×2×(32+4+4)=10240 bytes.

[0039] In some embodiments, the cache includes a key cache and a value cache, wherein the quantized data and quantization parameters of the key tensor are stored in physical storage blocks in the key cache, and the quantized data and quantization parameters of the value tensor are stored in physical storage blocks in the value cache.

[0040] In some embodiments, when batching is performed, the cache can be divided into a quantization segment and a residual segment. As mentioned earlier, when the number of accumulated unquantized terms reaches a predetermined quantization batch size, the key tensors and value tensors of these accumulated terms are quantized together to generate low-bit-width quantized data (e.g., INT8, INT4, etc.) and corresponding quantization parameters. The quantization segment can store this quantized data and quantization parameters. When the number of accumulated unquantized terms fails to fill a complete batch (i.e., less than the predetermined quantization batch size), the key tensors and value tensors of these accumulated terms can first remain in full precision, i.e., the original high precision (e.g., BF16, FP16, etc.), and then be quantized together when the number of subsequently accumulated unquantized terms reaches the predetermined quantization batch size. The residual segment can store this unquantized original full-precision data. Thus, a mixed-precision cache organization is formed.

[0041] In some embodiments, within each physical storage block, the quantization data and quantization parameters of each quantization group are stored contiguously, such as... Figure 2A As shown. In Figure 2A For ease of illustration and explanation, the layout of the data stored in a single physical storage block, Block_0, is shown. Those skilled in the art will understand that the data stored in other physical storage blocks within the cache used to store quantization data and quantization parameters can have a similar layout to Block_0. Figure 2A In the physical storage block Block_0 shown: data for P attention heads (equal to the number of attention heads in the large model, head_num) are stored sequentially and continuously; in each attention head, data for N words (equal to the number of words included in Block_0, block_size) are stored sequentially and continuously; each word has M quantization groups on the feature dimension of each attention head; within each quantization group, the quantization data and the corresponding quantization parameters are stored continuously.

[0042] As an example, in the case of symmetric quantization, the quantization data and scaling factor of each quantization group are stored contiguously within each physical memory block. Taking an example where the size of the quantization data of each quantization group in the data dimension is 32 bytes and the size of the quantization parameter is 4 bytes, the quantization data and quantization parameter of each quantization group occupy a total of 36 bytes in each physical memory block. These 36 bytes are stored contiguously with a continuous physical address range, where the quantization data corresponds to the first 32 bytes of these 36 bytes and the scaling factor corresponds to the last 4 bytes of these 36 bytes.

[0043] As another example, in the case of asymmetric quantization, the quantization data, scaling factor, and zero offset of each quantization group are stored contiguously in each physical memory block. Taking an example where the size of the quantization data for each quantization group in the data dimension is 32 bytes and the size of the quantization parameter is 8 bytes, the quantization data and quantization parameter of each quantization group occupy 40 bytes in each physical memory block. These 40 bytes are stored contiguously with a continuous physical address range, where the quantization data corresponds to the first 32 bytes of these 40 bytes, and the scaling factor and zero offset correspond to the last 8 bytes. Those skilled in the art will understand that the above description of the byte size of the quantization data and quantization parameters is for illustrative purposes only and is not intended to be limiting; the quantization data and quantization parameters may have other suitable byte sizes, and this application does not impose any limitations on this.

[0044] In some embodiments, within each physical storage block, quantization data corresponding to all tokens in that physical storage block is stored contiguously and centrally, and quantization parameters corresponding to all tokens in that physical storage block are also stored contiguously and centrally, such as... Figure 2B As shown. In Figure 2B For ease of illustration and explanation, the layout of the data stored in a single physical storage block, Block_0, is shown. Those skilled in the art will understand that the data stored in other physical storage blocks within the cache used to store quantization data and quantization parameters can have a similar layout to Block_0. Figure 2BIn the physical storage block Block_0 shown, quantized data corresponding to all terms in Block_0 is stored consecutively first, followed by quantized parameters corresponding to all terms in Block_0. That is, Block_0 is divided into a quantized data segment and a quantized parameter segment. In the quantized data segment of Block_0, quantized data for P attention heads (equal to the number of attention heads in the large model, head_num) are stored consecutively. Within each attention head, quantized data for N terms (equal to the number of terms in each physical storage block, block_size) are stored consecutively. Each term has M quantized groups on the feature dimension of each attention head. Within each quantized group, the quantized data is stored consecutively. In the quantized parameter segment, quantized parameters for P attention heads (equal to the number of attention heads in the large model, head_num) are stored consecutively. Within each attention head, quantized parameters for N terms (equal to the number of terms in each physical storage block, block_size) are stored consecutively. Each term has M quantized groups on the feature dimension of each attention head. Within each quantized group, the quantized parameters are stored consecutively. For ease of explanation, Figure 2B The diagram illustrates a quantization parameter segment using a single quantization parameter as an example. It can be understood that when there are two or more quantization parameters, the quantization parameter segment can include two or more consecutive sub-segments, each corresponding to one quantization parameter.

[0045] As an example, in the case of symmetric quantization, within each physical memory block, the quantized data corresponding to all terms in that physical memory block is stored contiguously, and the scaling factor corresponding to all terms in that physical memory block is also stored contiguously. Taking block_size=64, M=2, head_num=2, the size of the quantized data for each quantization group in the data dimension is 32 bytes, and the size of the quantization parameter is 4 bytes as an example: the quantized data corresponding to all 64 terms in that physical memory block occupies 2 × 64 × 2 × 32 = 8192 bytes, which are stored contiguously with a continuous physical address range; and the scaling factor corresponding to all 64 terms in that physical memory block occupies 2 × 64 × 2 × 4 = 1024 bytes, which are stored contiguously (e.g., immediately following the 8192 bytes of quantized data corresponding to all 64 terms in that physical memory block) with a continuous physical address range.

[0046] As another example, in the case of asymmetric quantization, within each physical memory block, the quantized data corresponding to all terms in that physical memory block is stored contiguously, the scaling factor corresponding to all terms in that physical memory block is stored contiguously, and the zero offset corresponding to all terms in that physical memory block is stored contiguously. Taking block_size=64, M=2, head_num=2, the size of the quantized data for each quantization group in the data dimension is 32 bytes, the scaling factor is 4 bytes, and the zero offset is 4 bytes as an example: the quantized data corresponding to all 64 terms in that physical memory block occupies 2 × 64 × 2 × 32 = 8192 bytes, which are stored contiguously with a continuous physical address range; the scaling factor corresponding to all 64 terms in that physical memory block occupies 2 × 64 × 2 × 4 bytes. =1024 bytes, which are stored contiguously (e.g., immediately following the 8192 bytes of quantization data corresponding to all 64 terms in this physical memory block), having a continuous physical address range; and the zero-point offset corresponding to all 64 terms in this physical memory block occupies a size of 2×64×2×4=1024 bytes, which are stored contiguously (e.g., immediately following the 1024 bytes of scaling factors corresponding to all 64 terms in this physical memory block), having a continuous physical address range.

[0047] Embodiments of this application may include a non-transitory computer-readable medium. The medium stores instructions that, when executed by one or more processors, cause the processors to perform the methods described herein.

[0048] Embodiments of this application may also include a computer program product. This computer program product includes instructions that, when executed by one or more processors, cause the processors to perform the methods described herein.

[0049] Unlike existing quantization techniques that store quantized data and quantization parameters separately or in different caches or buffers, the data storage method for large models according to embodiments of this application organizes and arranges the quantized data to be stored and the corresponding quantization parameters in the same storage tensor according to the innermost dimension. This is compatible with the paginated / block-based index access mode of the paginated KVCache management mechanism (e.g., PageAttention mechanism) of the inference framework of large models, and can adapt to the smallest read / write granularity of the hardware. Specifically, by using the data storage method for large models according to embodiments of this application, quantized data and quantization parameters can be directly mapped to the "block-based management, attention-based and term indexing" access mode of the paginated KVCache management mechanism, and the quantization parameters are included in the innermost continuous dimension of the same storage tensor, so that the quantized data and its corresponding quantization parameters can be read within the same continuous address range of the physical storage block. This avoids additional address jumps and multi-segment loading to obtain quantization parameters, and avoids additional data organization, rearrangement, aggregation, or copying operations caused by the quantized data and quantization parameters being located in separate caches or buffers. Furthermore, by using the data storage method for large models according to embodiments of this application, for dedicated training and inference devices with a fixed minimum read / write granularity (e.g., 64 bytes) and startup overhead, it is easier to form merged or sequential accesses, reduce the number of short and discrete I / O requests and startup overhead, meet the efficiency requirements of the minimum read / write granularity of the hardware, and thus effectively reduce I / O or inference time during the inference phase.

[0050] Figure 3 A schematic diagram is shown of a computing device 300 that can implement embodiments according to this application. This computing device 300 can be used to perform the various methods described above in conjunction with embodiments of this application, for example, in conjunction with... Figure 1 The method described. For example... Figure 3 As shown, computing device 300 may include bus 302 or other communication mechanism for transmitting information, and one or more processors 304 coupled to bus 302 for processing information. The one or more processors 304 may include, for example, one or more general-purpose microprocessors.

[0051] like Figure 3As shown, in some embodiments, computing device 300 may further include main memory 306 coupled to bus 302, which is used to store information and instructions executed by one or more processors 304. For example, main memory 306 includes, but is not limited to, random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 306 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 304. When these instructions are stored in a storage medium accessible to one or more processors 304, they can cause computing device 300 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 308 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical disks or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0052] like Figure 3 As shown, in some embodiments, computing device 300 may further include one or more communication interfaces or network interfaces 310 coupled to bus 302. Network interface 310 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 310 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0053] The various processes, methods, and algorithms described in the preceding sections can be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by these code modules. The processes and algorithms can be implemented, partially or fully, in dedicated circuit systems.

[0054] When the functions disclosed herein are implemented as software functional units and sold or used as standalone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes instructions to cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0055] Some embodiments further provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0056] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to as the “service system”) that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device capable of installing platform applications.

[0057] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations should be considered to fall within the scope of this disclosure. Additionally, certain methods or processes may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and the blocks or states associated with them may be executed in other suitable orders. For example, the described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0058] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). The algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly turn the computer into an executable function but may learn from training data to produce a predictive model of the executable function.

[0059] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementations that operate to perform one or more of the operations or functions described herein.

[0060] Similarly, the methods described herein can be implemented at least in part by a processor, where one or more specific processors are instances of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also operate to support the execution of related operations in a “cloud computing” environment or as the execution of related operations in a “Software as a Service” (SaaS) context. For example, at least some operations can be performed by a group of computers (as an example of a machine containing processors), where these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0061] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In some embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0062] Throughout this specification, multiple examples may be implemented as components, operations, or structures of a single example. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configurations may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as separate components. These and other variations, modifications, additions, and improvements also fall within the scope of this document.

[0063] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0064] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0065] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0066] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, “implementation” is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A data storage method for large models, characterized in that, include: During the inference phase of the large model, the quantized data to be stored and the corresponding quantization parameters are organized within the same storage tensor. This storage tensor has multiple dimensions, which, from the outermost to the innermost dimension, include: The block dimension corresponds to the physical storage block in the paging management mechanism, wherein the paging management mechanism is used to manage the cache where the quantized data and the quantized parameters are to be stored, the cache includes multiple physical storage blocks, and each physical storage block has a continuous physical address range; Attention head dimension, corresponding to the attention head defined for the large model; Lexical dimension, corresponding to the lexical units generated during the reasoning phase; The quantization grouping dimension corresponds to the quantization grouping of each word unit on the feature dimension of each attention head; and Data dimensions, used to organize the quantization data and quantization parameters for each quantization group; and Based on the logical storage structure of the storage tensor, the quantized data and the quantization parameters are stored in the physical storage blocks of the cache, wherein the quantized data and the quantization parameters of each quantization group are stored consecutively in each physical storage block.

2. The method according to claim 1, characterized in that, Also includes: The key tensors and value tensors of the lexical units generated during the inference phase are quantized to generate the corresponding quantized data and quantization parameters. The cache includes a key cache and a value cache. The quantization data and quantization parameters of the key tensor are stored in the physical storage block of the key cache, and the quantization data and quantization parameters of the value tensor are stored in the physical storage block of the value cache.

3. The method according to claim 2, characterized in that, in, The quantization includes symmetric quantization, and the quantization parameters include a scaling factor.

4. The method according to claim 3, characterized in that, in, The data dimension of each quantization group corresponds to a predetermined number of bytes in the physical storage block of the cache, a first portion of which is used to store the quantized data, and a second portion of which is used to store the scaling factor.

5. The method according to claim 4, characterized in that, in, The predetermined number of bytes is 36 bytes, the first part is the first 32 bytes of the 36 bytes, and the second part is the last 4 bytes of the 36 bytes.

6. The method according to claim 2, characterized in that, in, The quantization includes asymmetric quantization, and the quantization parameters include a scaling factor and a zero offset.

7. The method according to claim 6, characterized in that, in, The data dimension of each quantization group corresponds to a predetermined number of bytes in the physical storage block of the cache, a first portion of which is used to store the quantization data, and a second portion of which is used to store the scaling factor and the zero offset.

8. The method according to claim 7, characterized in that, in, The predetermined number of bytes is 40 bytes, the first part is the first 32 bytes of the 40 bytes, and the second part is the last 8 bytes of the 40 bytes.

9. The method according to claim 1, characterized in that, in, The pagination management mechanism includes the PageAttention mechanism.

10. The method according to claim 1, characterized in that, in, The large model includes autoregressive generative models.

11. The method according to claim 10, characterized in that, in, The autoregressive generative model includes a large language model.

12. A non-transitory computer-readable medium storing instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-11.

13. A computer program product, comprising instructions, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-11.

Images