Storage management methods, storage management devices, electronic devices, and storage media

Abstract

Embodiments of this disclosure provide a storage management method, storage management device, electronic device, and storage medium. The method includes: dividing storage space into multiple physical blocks of fixed size; dividing inter-operator transfer tensors during model inference into one or more first logical blocks, each first logical block including at most a first number of inter-operator transfer sub-tensors corresponding to terms; dividing key-value tensors generated during model inference into one or more second logical blocks, each second logical block including at most a second number of key-value sub-tensors corresponding to terms; allocating physical blocks to each of the first and second logical blocks, and managing the mapping relationship between logical blocks and physical blocks based on a block table. This method can effectively avoid inference latency caused by temporary allocation of video memory resources during inference, as well as video memory fragmentation generated by splitting and merging physical blocks, thereby improving video memory utilization.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of artificial intelligence, and more specifically to a storage management method, storage management device, electronic device, and storage medium. Background Technology

[0002] With the rapid development of Artificial Intelligence (AI) technology, the number of parameters in large-scale models is growing exponentially. In practical deployments, the rapid expansion of model size, high memory usage, and memory bandwidth limitations in the autoregressive generation process lead to high computational latency and high resource consumption during inference. Therefore, achieving efficient and low-cost model inference has become one of the key technical challenges that need to be addressed. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a storage management method, the method comprising: dividing a storage space into multiple physical blocks of fixed size; dividing an inter-operator transit tensor in a model inference process into one or more first logical blocks, each first logical block including at most a first number of inter-operator transit sub-tensors corresponding to terms; dividing a key-value tensor generated in the model inference process into one or more second logical blocks, each second logical block including at most a second number of key-value sub-tensors corresponding to terms; allocating physical blocks to each of the first and second logical blocks, and managing the mapping relationship between logical blocks and physical blocks based on a block table.

[0004] In a storage management method provided in at least one embodiment of this disclosure, dividing the storage space into multiple physical blocks of fixed size includes: dividing the storage space into multiple first physical blocks and multiple second physical blocks, wherein the first physical block has a first size, the second physical block has a second size, the first size is determined based on the size of the inter-operator transfer tensor and the first quantity, and the second size is determined based on the size of the key-value tensor and the second quantity.

[0005] In the storage management method provided in at least one embodiment of this disclosure, during the model inference process, the input sequence is split into multiple consecutive subsequences based on a preset block size, and the inter-operator transfer tensor is generated during the forward computation of processing a single subsequence, wherein the first quantity is equal to the number of tokens in the single subsequence.

[0006] In a storage management method provided in at least one embodiment of this disclosure, allocating physical blocks for the first logical block includes: for each inference request, determining the first physical block with the highest historical usage frequency of the current inference request as the target first physical block; and preferentially allocating the target first physical block to the first logical blocks corresponding to each subsequence of the current inference request.

[0007] In a storage management method provided in at least one embodiment of this disclosure, when the first size and the second size are integer multiples of each other, allocating physical blocks for each of the first and second logical blocks includes: during the process of allocating physical blocks for the first logical block, in response to the absence of a free first physical block, allocating one or more free second physical blocks for the first logical block; and during the process of allocating physical blocks for the second logical block, in response to the absence of a free second physical block, allocating one or more free first physical blocks for the second logical block.

[0008] In a storage management method provided in at least one embodiment of this disclosure, when the first size is an integer multiple of the second size, allocating physical blocks for each of the first and second logical blocks includes: during the process of allocating physical blocks for the first logical block, in response to the absence of a free first physical block, allocating one or more free second physical blocks for the first logical block; and during the process of allocating physical blocks for the second logical block, in response to the absence of a free second physical block, allocating one free first physical block for one or more second logical blocks.

[0009] In a storage management method provided in at least one embodiment of this disclosure, when the second size is an integer multiple of the first size, allocating physical blocks for each of the first and second logical blocks includes: during the process of allocating physical blocks for the first logical block, in response to the absence of a free first physical block, allocating a free second physical block for one or more of the first logical blocks; and during the process of allocating physical blocks for the second logical block, in response to the absence of a free second physical block, allocating one or more free first physical blocks for the second logical block.

[0010] In a storage management method provided in at least one embodiment of this disclosure, when the first size and the second size are integer multiples of each other, the step of managing the mapping relationship between logical blocks and physical blocks based on block tables includes: managing the mapping relationship between the first logical block and the second logical block and the first physical block based on a first block table; and managing the mapping relationship between the first logical block and the second logical block and the second physical block based on a second block table.

[0011] In a storage management method provided in at least one embodiment of this disclosure, when the first size and the second size are the same, the step of managing the mapping relationship between logical blocks and physical blocks based on a block table includes: for the first logical block and the second logical block, managing the mapping relationship between logical blocks and physical blocks based on the same block table.

[0012] The storage management method provided in at least one embodiment of this disclosure further includes: in response to receiving a data read request, searching for the target physical block mapped to the target logical block corresponding to the data read request based on the block table, and reading the data corresponding to the target logical block from the target physical block.

[0013] In the storage management method provided in at least one embodiment of this disclosure, the block table maintains the status information of at least one physical block being occupied by an operator based on a counter.

[0014] At least one embodiment of this disclosure provides a storage management apparatus, comprising: a physical partitioning module configured to partition storage space into a plurality of physical blocks of fixed size; a first logical partitioning module configured to partition inter-operator transitive tensors in a model inference process into one or more first logical blocks, each first logical block including at most a first number of inter-operator transitive sub-tensors corresponding to terms; a second logical partitioning module configured to partition key-value tensors generated in the model inference process into one or more second logical blocks, each second logical block including at most a second number of key-value sub-tensors corresponding to terms; an allocation module configured to allocate physical blocks to each of the first and second logical blocks; and a management module configured to manage the mapping relationship between logical blocks and physical blocks based on a block table.

[0015] At least one embodiment of this disclosure provides an electronic device, including at least one processor and at least one memory, wherein the at least one memory stores program code that, when executed by the at least one processor, causes the at least one processor to perform a storage management method according to at least one embodiment of this disclosure.

[0016] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by at least one processor, cause the at least one processor to perform a storage management method according to at least one embodiment of this disclosure.

[0017] In the storage management method, storage management device, electronic device, and storage medium provided in at least one embodiment of this disclosure, since the storage space is divided into multiple physical blocks of fixed size before inference begins, and the physical blocks are allocated to the key-value cache and operator-transfer tensors on demand, it is only necessary to select free physical blocks and establish a mapping relationship through a block table. There is no need to temporarily call the video memory allocation interface to request new storage resources during inference, thereby effectively avoiding inference latency caused by embedding storage resource request operations into the inference execution process. Furthermore, since the storage space is allocated using fixed-size physical blocks as the basic unit, there is no need to perform splitting and merging operations on the physical blocks, which can effectively avoid the generation of video memory fragmentation and improve video memory utilization. Further, the above method can achieve unified storage management and allocation of the key-value cache and operator-transfer tensors, and the actual storage space of the two supports dynamic sharing and on-demand allocation, thus possessing high resource scheduling flexibility. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0019] Figure 1 This is a schematic flowchart illustrating a storage management method provided for at least one embodiment of the present disclosure.

[0020] Figure 2 This is a schematic diagram of a storage management method provided in at least one embodiment of the present disclosure.

[0021] Figure 3 This is a schematic block diagram of a storage management device provided for at least one embodiment of the present disclosure.

[0022] Figure 4 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0023] Figure 5 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0024] Figure 6 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0026] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0027] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numerals in each drawing.

[0028] With the development of artificial intelligence technology, the application of large-scale models is becoming increasingly widespread. Large-scale models are artificial intelligence models with a large number of parameters built by artificial intelligence networks. Examples include large language models (LLM), large-scale visual models, or multimodal large-scale models, which are content generation models based on prompt words. Examples include models based on the Transformer architecture or models built based on attention mechanisms.

[0029] Traditional model inference architectures typically treat the entire inference process as a single unit, making it difficult to fully utilize hardware resources and easily leading to performance bottlenecks and a degraded user experience. With the rapid increase in the size of model parameters, large models have significantly increased demands on computing resources, GPU memory capacity, and memory bandwidth during the inference phase, resulting in high inference latency, low throughput, and high deployment costs, severely limiting their availability and scalability in real-world scenarios.

[0030] To address the aforementioned issues, the prefill-decode (PD) separation technique was proposed and applied to the inference process of large models. This technique divides the inference process into two logically independent stages: the prefill stage and the decode stage, and performs resource scheduling and system optimization according to their respective computational characteristics, thereby improving the overall inference efficiency.

[0031] The pre-filling stage handles the user input prompts, completes preprocessing, and generates the first output token and the corresponding key-value cache. In natural language processing, a token is the basic processing unit after text segmentation; it can be a character, word, or sub-word fragment. A complete sentence typically consists of multiple tokens and is the fundamental unit for model understanding and language generation. Since the input usually contains a long context sequence, the pre-filling stage requires a complete forward propagation computation, involving a large number of matrix operations, making it a computationally intensive task. Specifically, after the input text is converted into a token sequence by the tokenizer, the model performs a complete forward propagation, generating the first output token. Simultaneously, it calculates and caches the corresponding key (K) matrix and value (V) matrix for the attention layer in each Transformer block, forming a key-value cache for reuse in the decoding stage.

[0032] Here, a lexical unit is the smallest unit into which text data is segmented. It can be a single character, an entire word, or a phrase. A complete sentence consists of multiple lexical units. In natural language processing, lexical units are the basic units that models use to process and understand text.

[0033] The decoding phase employs an autoregressive approach to generate output tokens one by one. This phase processes only a single token per step, resulting in relatively low computational cost, but it requires frequent access to and updates to the key-value cache, placing high demands on memory bandwidth; thus, it is a memory-intensive task. In each generation step, the model performs a single-step forward computation based solely on the tokens and key-value cache from the previous step. It avoids repeatedly calculating attention keys from historical contexts, instead directly reading the stored K and V matrices from the key-value cache and appending the K and V matrices corresponding to the newly generated tokens to the cache. By storing and reusing the K and V matrices of historical tokens, redundant computations in the self-attention mechanism are avoided, significantly improving inference efficiency.

[0034] Key-value caching (KV-Cache) is an optimization technique used in the autoregressive inference process of the Transformer architecture. During autoregressive inference, the model generates the next token step-by-step based on user-provided prompts, with each step relying on previously generated tokens. In natural language processing, a token is the basic processing unit after text is segmented by a tokenizer. It can be a character, word, or sub-word fragment. A complete sentence typically consists of multiple tokens and is the fundamental unit for the model to understand and generate language. In the model's self-attention mechanism, three vectors are calculated for each token: query (Q), key (K), and value (V). Each time a new token is generated, the model performs a dot product operation between the current query vector and the key vectors of all previous tokens to calculate an attention score, and then performs a weighted sum of the value vectors accordingly. This calculation requires frequent access to previous token information. To avoid repeatedly calculating the key and value vectors of all historical tokens when generating new tokens, the inference system can store these already calculated key-value data. In generating the next lexical unit, it is only necessary to calculate the Query, Key, and Value of the current lexical unit, and then perform attention calculations with the historical Key-Value data in the cache. This approach significantly reduces the computational complexity of inference from O(N²) to O(N), greatly improving inference performance.

[0035] Some large model inference frameworks rely on Torch's memory management mechanism to reuse the GPU memory of operator input and output tensors. The core logic involves performing a dummy run on the model to detect peak GPU memory usage, calling the GPU memory allocation interface to request multiple memory pools for the model, and dynamically splitting memory blocks from these pools to store tensors according to operator requirements. Because the length of the input sequence changes dynamically, frequent splitting and merging of memory blocks is necessary.

[0036] However, this memory management scheme has some problems: First, during dynamic shape inference, frequent memory block splitting operations cause free memory areas to be scattered among allocated blocks. Since the free memory areas are not adjacent, they are difficult to merge and reuse, resulting in memory fragmentation. Second, when the requested memory exceeds the current memory block capacity (for example, when the shape of the input data is significantly larger than the shape of the data that the system has previously inferred), the system needs to temporarily call the GPU memory allocation interface to request a new memory pool during the inference process. This dynamic memory allocation behavior at runtime will severely block the computation flow, thereby significantly increasing inference latency and reducing the overall throughput performance of the service.

[0037] This disclosure provides at least one embodiment of a storage management method, a storage management device, an electronic device, and a storage medium.

[0038] The storage management method provided in at least one embodiment of this disclosure includes: dividing storage space into multiple physical blocks of fixed size; dividing inter-operator transit tensors in the model inference process into one or more first logical blocks, each first logical block including at most a first number of inter-operator transit sub-tensors corresponding to terms; dividing key-value tensors generated in the model inference process into one or more second logical blocks, each second logical block including at most a second number of key-value sub-tensors corresponding to terms; allocating physical blocks to each logical block in the first and second logical blocks, and managing the mapping relationship between logical blocks and physical blocks based on a block table.

[0039] In the storage management method provided in at least one embodiment of this disclosure, since the storage space is divided into multiple physical blocks of fixed size before inference begins, and the physical blocks are allocated on demand to the key-value cache and operator-to-operator tensors, only free physical blocks need to be selected and a mapping relationship established through the block table. There is no need to temporarily call the video memory allocation interface to request new video memory resources during inference, thus effectively avoiding inference latency caused by embedding video memory resource request operations into the inference execution process. Furthermore, since the storage space is allocated using fixed-size physical blocks as the basic unit, there is no need to perform splitting and merging operations on the physical blocks, effectively avoiding the generation of video memory fragmentation and improving video memory utilization. Further, the above method can achieve unified storage management and allocation for the key-value cache and operator-to-operator tensors, and their actual storage space supports dynamic sharing and on-demand allocation, thus possessing high resource scheduling flexibility.

[0040] Figure 1 This is a schematic flowchart illustrating a storage management method provided for at least one embodiment of the present disclosure.

[0041] For example, such as Figure 1 As shown, at least one embodiment of the storage management method provided in this disclosure includes steps S101 to S104.

[0042] Step S101: Divide the storage space into multiple physical blocks of fixed size.

[0043] Step S102: Divide the inter-operator transit tensor in the model inference process into one or more first logic blocks, each first logic block including at most a first number of inter-operator transit sub-tensors corresponding to tokens.

[0044] Step S103: Divide the key-value tensor generated during the model inference process into one or more second logical blocks, each second logical block including at most a second number of key-value sub-tensors corresponding to words.

[0045] Step S104: Allocate physical blocks for each logical block in the first logical block and the second logical block, and manage the mapping relationship between logical blocks and physical blocks based on the block table.

[0046] For example, in step S101, the storage space can be located in the on-chip or off-chip storage medium of various artificial intelligence acceleration devices. Acceleration devices may include: Graphics Processing Units (GPUs), General-purpose Graphics Processing Units (GPGPUs), Tensor Processing Units (TPUs), Deep Learning Processing Units (DPUs), Accelerated Processing Units (APUs), Neural Network Processing Units (NPUs), etc. Storage media may include Video Random Access Memory (VRAM), High Bandwidth Memory (HBM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), etc.

[0047] For example, after the inference service framework is initialized or started, the system can obtain the total amount of available storage resources. After deducting the space occupied by model weights and other basic runtime libraries, the remaining storage resources are determined as dedicated storage space for tensor and key-value cache transfer between operators, which is the storage space mentioned in step S101.

[0048] For example, for a distributed inference system that includes multiple GPUs, the total available video memory resources of each GPU can be obtained. After deducting the space occupied by model weights and other basic runtime libraries, the remaining video memory is determined as dedicated video memory space for passing tensors and key-value caches between operators.

[0049] In some examples, the remaining storage resources can be allocated in a specified proportion as dedicated storage space for tensor and key-value buffers for inter-operator transfer. For example, a configuration parameter of 0.9 can be set to indicate that 90% of the remaining storage resources are allocated to tensor and key-value buffers for inter-operator transfer. The configuration parameter can be set according to actual needs, and this disclosure does not impose any limitations on it.

[0050] For example, in step S101, the storage space can be divided into multiple physical blocks of fixed size, and these multiple physical blocks can be indexed and numbered (e.g., numbered from physical block 0 to physical block n). The index number can serve as a unique physical address identifier for each physical block in the underlying storage space.

[0051] For example, the aforementioned physical blocks can be managed in a unified manner to form an idle physical block pool, which is used to record and manage physical blocks that are currently in an unallocated state.

[0052] A neural network model can be viewed as a directed acyclic graph (DAG) composed of multiple nodes, where each node corresponds to an operator. Operators in a neural network typically refer to the basic mathematical operations or processes used in the network layers. These operators are used to construct the various layers and components of the network, enabling data transfer, transformation, and computation. Operators are the fundamental building blocks of a network model, defining its structure and computational flow, including inputs, outputs, and intermediate computations. In the model, the connections between operators form a directed graph, reflecting the order of computation for different operations. By combining these operators, complex and powerful neural network models can be constructed to handle various complex tasks and data. For example, operators can include embedding operators, attention operators, normalization operators, hybrid expert operators, addition operators, language model head operators, sampling operators, etc.

[0053] For example, the operator input tensor can represent the set of data entering the current operator for processing; the operator output tensor can represent the result data generated by the current operator after performing a specified operation on the input data. In a cascaded operator chain, based on the continuity of data flow, the preceding operator O... i The output tensor of the operator is its successor operator O. i+1 The operator input tensor.

[0054] For example, the operator-transfer tensor can represent the tensor passed between operators along the data flow path during the forward propagation process of the model. For instance, when describing data interaction between operators, the operator-transfer tensor can refer to the operator output tensor generated by the preceding operator, or it can refer to the operator input tensor received by the subsequent operator; both refer to the same data object in physical entity.

[0055] For example, the interoperator transitive subtensor can represent the subunits obtained by segmenting the interoperator transitive tensor along the sequence dimension, which includes the feature data of a single word in all hidden layer dimensions.

[0056] For example, in step S102, assuming the tensor shape of the tensor passed between operators is [N×S, hidden_size], then the tensor shape of the sub-tensor passed between operators can be [1, hidden_size], and the shape of each first logical block can be [N1, hidden_size]. Here, N represents the number of concurrent requests during model inference (N≥1); S represents the length of the input sequence for a single request during model inference (in terms of the number of tokens). In the pre-filling stage, S represents the length of the prompt word; in the decoding stage, S=1; hidden_size is the dimension of the hidden layer size, representing the dimension of each token represented as a vector within the model; N1 is the first quantity, i.e., the maximum number of tokens corresponding to each first logical block.

[0057] In other words, each first logical block corresponds to at most N1 tokens in the inter-operator transit tensor, and N1 inter-operator transit sub-tensors can form a first logical block. By dividing the inter-operator transit tensor into multiple first logical blocks along the sequence dimension (e.g., the N×S dimension mentioned above), block management of the inter-operator transit tensor can be achieved.

[0058] For example, one embodiment of step S102 may include: dividing at least one operator input tensor in the model inference process into one or more first logical blocks, each first logical block including input sub-tensors corresponding to at most a first number of terms. For example, the input sub-tensors may represent sub-units obtained by segmenting the operator input tensor in the sequence dimension, which include feature data of a single term in all hidden layer dimensions.

[0059] For example, another example of step S102 may include: dividing at least one operator output tensor in the model inference process into one or more first logical blocks, each first logical block including output sub-tensors corresponding to at most a first number of terms. For example, the output sub-tensor may represent a sub-unit obtained by segmenting the operator output tensor in the sequence dimension, which includes feature data of a single term in all hidden layer dimensions.

[0060] In some examples, the partitioned operator input tensor or operator output tensor can be tensors passed between operators along the data flow path during the forward propagation process of the model.

[0061] In neural network models, the tensors passed between operators along the data flow path can be represented as hidden state tensors, which essentially correspond to the activation values ​​calculated during the forward propagation of the neural network. For example, a hidden state tensor can include the output tensor generated after the first operator (e.g., an embedding layer operator) performs an initial mapping on the original input data, or the input tensor received by the last operator (e.g., a sampling layer operator) before performing the final prediction, as well as the input and output tensors of each intermediate layer operator. The hidden state tensor essentially constitutes the intermediate layer representation of the input data after the model performs successive feature transformations.

[0062] For example, one embodiment of step S102 may include: dividing the hidden state tensor in the model inference process into one or more first logical blocks, each first logical block including hidden state sub-tensors corresponding to at most a first number of words. For example, a hidden state sub-tensor may represent a sub-unit obtained by segmenting the hidden state tensor in the sequence dimension, which includes feature data of a single word in all hidden layer dimensions.

[0063] For example, in step S103, the key-value tensor can be key-value cache (KV Cache) data, which is essentially a set of tensors composed of key tensors and value tensors; the key-value tensor can also be only a key tensor or only a value tensor. The tensor shape of the key tensor and value tensor can be, for example, [N×S, num_heads, head_dim], where num_heads represents the number of attention heads, and head_dim represents the vector dimension of each attention head. For example, the key-value sub-tensor can represent the sub-unit obtained by splitting the key-value tensor along the sequence dimension.

[0064] For example, in step S103, assuming the tensor shape of the divided key-value tensor is [N×S, num_heads, head_dim], then the tensor shape of the key-value sub-tensor can be [1, num_heads, head_dim], and the shape of each second logical block can be [N2, num_heads, head_dim]. N2 is the second quantity, that is, the maximum number of tokens corresponding to each second logical block.

[0065] In other words, each second logical block corresponds to at most N2 tokens in the key-value tensor, and N2 key-value sub-tensors can form a second logical block. By dividing the key-value tensor into multiple second logical blocks along the sequence dimension, block-based management of the key-value cache can be achieved.

[0066] This disclosure does not limit the specific values ​​of the first quantity and the second quantity. The first quantity and the second quantity can be the same or different, and can be set according to actual needs.

[0067] It should be noted that, since the number of tokens corresponding to the inter-operator transitive tensor is not necessarily an integer multiple of the first number, a certain first logic block (e.g., the last first logic block) may also contain inter-operator transitive sub-tensors corresponding to fewer than the first number of tokens. Similarly, since the number of tokens corresponding to the key-value tensor is not necessarily an integer multiple of the second number, a certain second logic block (e.g., the last second logic block) may also contain key-value sub-tensors corresponding to fewer than the second number of tokens.

[0068] For example, in step S104, the first logical block and the second logical block can be collectively referred to as logical blocks. Physical blocks can be allocated to the first and second logical blocks respectively, so that each first logical block and each second logical block can be mapped to contiguous or non-contiguous physical blocks in physical storage, thereby supporting on-demand allocation and fine-grained video memory management. It should be noted that "allocation" here refers to the operation of selecting and binding free physical blocks for logical blocks in the free physical block pool, which is different from the meaning of "allocation" in the previous "calling the GPU video memory allocation interface to request a new video memory pool" involving requesting new video memory resources from the operating system or video memory driver.

[0069] For example, when performing an inference task and generating new inter-operator transit tensors and key-value tensors, the inter-operator transit tensors and key-value tensors can be divided into logical blocks, and physical blocks can be requested from the idle physical block pool as needed and allocated to the logical blocks, so that the newly generated inter-operator transit tensor or key-value tensor data can be written into the requested physical blocks.

[0070] For example, a block table can be used to establish a mapping between logical blocks and physical blocks, allowing logically contiguous logical blocks to be mapped to physically non-contiguous physical blocks. For instance, the block table can record the correspondence between logical block indices and physical block numbers. When stored inter-operator tensors are needed during inference, the corresponding physical block can be located based on the mapping in the block table, the inter-operator tensor can be read from the physical block, and relevant calculations can be performed. When historical key-value information needs to be accessed during inference, the corresponding physical block can be located based on the mapping in the block table, the key-value data can be read from the physical block, and attention calculations can be performed. When the data corresponding to a logical block is no longer needed, its corresponding physical block can be released back to the idle physical block pool for reuse in subsequent inference tasks. This approach avoids the reallocation overhead caused by insufficient reserved space in traditional allocation methods, as well as the memory waste caused by excessive reserved space.

[0071] For example, to achieve effective management and reclamation of physical blocks, the block table can record the status information of each physical block being occupied by an operator. Specifically, a counter can be set to count the number of operators currently occupying a physical block. When an operator finishes using the physical block and releases it, the system can decrement the value of the counter by 1; when the value of the counter decrements to 0, it indicates that the physical block is currently not occupied by any operator, and the system can mark the status of the physical block as free so that it can be allocated to other operators for subsequent use.

[0072] In the storage management method provided in at least one embodiment of this disclosure, since the storage space is divided into multiple physical blocks of fixed size before inference begins, and the physical blocks are allocated on demand to the key-value cache and operator-to-operator tensors, only free physical blocks need to be selected and a mapping relationship established through the block table. There is no need to temporarily call the video memory allocation interface to request new video memory resources during inference, thus effectively avoiding inference latency caused by embedding video memory resource request operations into the inference execution process. Furthermore, since the storage space is allocated using fixed-size physical blocks as the basic unit, there is no need to perform splitting and merging operations on the physical blocks, effectively avoiding the generation of video memory fragmentation and improving video memory utilization. Further, the above method can achieve unified storage management and allocation for the key-value cache and operator-to-operator tensors, and their actual storage space supports dynamic sharing and on-demand allocation, thus possessing high resource scheduling flexibility.

[0073] In some examples, chunked prefill can be used during the model's inference process. During chunked prefill, the input sequence is split into multiple consecutive subsequences (chunks) based on a preset chunked size, and the tensors passed between operators are generated during the forward computation of processing individual subsequences.

[0074] Specifically, chunk pre-padding technology breaks down the traditional one-time pre-padding operation into multiple manageable processing units. For example, the input sequence can first be divided into multiple consecutive subsequences, the length of each subsequence determined based on a preset chunk size. The chunk size can be determined based on factors such as hardware resource constraints, memory availability, or performance goals, thereby maintaining high computational utilization while reducing peak memory usage. For example, if the input sequence is 1024 words long and the preset chunk size is 256 words, the input sequence can be divided into 4 pre-padding chunks and processed sequentially in 4 iterations. In some examples, the chunk size can be dynamically adjusted to match the physical block size design, thereby reducing memory fragmentation caused by sequence length. Furthermore, when partitioning the input sequence, if the total number of words in the input sequence is not divisible by the preset chunk size, an incomplete chunk with insufficient words will be generated at the end of the sequence. At this point, a token assembly mechanism can be used to extract tokens from other pending requests (such as requests in the decoding stage or other short requests) and compactly assemble these tokens into an incomplete block until the incomplete block is filled to the preset pre-padded block size. In this way, the pre-padded task of long sequences is smoothly distributed across multiple computation cycles, and the computation batches within each cycle maintain a high data density through assembly operations. This not only avoids excessive instantaneous consumption of computational resources caused by pre-padded single long sequences but also eliminates resource idleness during short sequence decoding.

[0075] During the model's inference process, when processing the forward computation of a single subsequence, an inter-operator transit tensor (hidden state tensor) and a key-value tensor corresponding to the subsequence are generated. The key-value tensor can be stored as a key-value cache for reuse in subsequent subsequence processing or autoregressive decoding stages, thereby improving overall inference efficiency.

[0076] In the above example, the first quantity can be set to be equal to the number of tokens in a single subsequence, that is, the first quantity can be set to the block size of the block pre-filling technique. Since the first logical block segmentation operation in this embodiment is performed on the sequence dimension, this segmentation method and the processing mechanism of the block pre-filling technique are logically highly compatible, without the need to introduce additional data rearrangement or format conversion operations, thereby effectively reducing data processing overhead and memory access latency. Furthermore, combining token assembly technology in the block pre-filling process can also effectively avoid generating incomplete first logical blocks. On this basis, corresponding physical blocks can be allocated to complete logical blocks as needed, thereby eliminating internal fragmentation caused by unused space at the end and effectively reducing the occurrence of memory fragmentation.

[0077] In a block pre-filling scenario, one example of "allocating a physical block for the first logical block" may include steps S1041 to S1042.

[0078] Step S1041: For each inference request, determine the first physical block that has the highest historical usage frequency in the current inference request as the target first physical block.

[0079] Step S1042: Prioritize allocating the target first physical block to the first logical block corresponding to each subsequence of the current inference request.

[0080] For example, in a block pre-filling scenario, for the tensors passed between operators generated from different subsequences of the same inference request, the most frequently used physical block (e.g., Block A) can be maintained for that request through a block table. The tensors passed between operators from different subsequences of the same inference request can all be preferentially allocated to Block A, thereby reusing the same physical block and improving cache hit rate and computational efficiency. For example, for another inference request, another physical block (Block B) can be maintained to ensure memory isolation and memory access locality between requests.

[0081] For example, the above allocation can follow the Least Recently Used (LRU) orchestration principle: for the same request, the frequency of each physical block being used by the request is recorded in the block table, and the physical block with the highest usage frequency (i.e. the highest affinity) is used to supply the tensor writing between operators of the current subsequence, thereby improving cache hit rate and computational efficiency.

[0082] For example, one embodiment of step S101 may include: dividing the storage space into multiple first physical blocks and multiple second physical blocks. The first physical blocks have a first size, and the second physical blocks have a second size. The first size is determined based on the size and a first quantity of the inter-operator transfer tensor, and the second size is determined based on the size and a second quantity of the key-value tensor.

[0083] In some examples, the first size can be determined based on the hidden_size dimension of the inter-operator transit tensor and the first quantity; the second size can be determined based on the num_heads and head_dim dimensions of the key-value tensor and the second quantity.

[0084] For example, assuming the shape of the tensor for inter-operator transfer is [N×S, 2048] and the first quantity is 1024, then the first size can be 1024×2048. For example, assuming the shape of the key-value tensor is [N×S, 4, 128] and the second quantity is 128, then the second size can be 128×4×128.

[0085] It should be noted that the embodiments of this disclosure do not limit the total capacity of the multiple first physical blocks and the total capacity of the multiple second physical blocks, and can be set according to actual needs. Since the size of the first physical blocks and the second physical blocks is fixed, it can also be understood that the embodiments of this disclosure do not limit the number of first physical blocks and the number of second physical blocks.

[0086] In some examples, the total capacity of multiple first physical blocks and the total capacity of multiple second physical blocks can be determined according to a preset ratio. For example, the preset ratio can be configured as 1:1, that is, the multiple first physical blocks and the multiple second physical blocks each occupy 50% of the total storage space capacity. It should be noted that in practical applications, the preset ratio can be flexibly adjusted according to specific business needs or system configuration, for example, it can be set to any other reasonable ratio value, and this disclosure embodiment does not limit this.

[0087] In other examples, the total capacity of multiple first physical blocks can be determined based on the inference service configuration, and then the remaining storage space in the storage space other than the multiple first physical blocks can be determined as the total capacity of the second physical blocks.

[0088] For example, the total GPU memory required for transferring tensors (hidden state tensors) between operators can be pre-estimated based on the server's hardware configuration parameters. Specifically, the maximum context length (MCL) parameter of the input can be obtained first, and the model can be simulated based on this MCL parameter. During the simulation, a reference counting mechanism can be used to track the allocation and release of tensors transferred between operators in GPU memory. Under the premise of satisfying GPU memory reuse constraints, the peak GPU memory requirement during the inference process can be statistically obtained and determined as the total capacity of multiple first physical blocks.

[0089] Furthermore, assuming the peak memory requirement is N×MCL×hidden_size, and a single first physical block is pre-defined to carry data corresponding to M tokens (i.e., the size of a single first physical block is M×hidden_size), the total number of required first physical blocks can be calculated based on the relationship between the peak memory requirement and the size of a single first physical block. The specific calculation formula is, for example: Total number of first physical blocks = (N×MCL) / M.

[0090] An example of "allocating physical blocks for each of the first and second logical blocks" in step S104 may include steps S201 to S202.

[0091] Step S201: Allocate an empty first physical block for the first logical block.

[0092] For example, in step S201, for each first logical block, an idle first physical block can be requested from the idle physical block pool and allocated to that first logical block.

[0093] Step S202: Allocate a free second physical block for the second logical block.

[0094] For example, in step S202, for each second logical block, an idle second physical block can be requested from the idle physical block pool and allocated to that second logical block.

[0095] For example, a free physical block pool table can be maintained to record the identification information of all currently unoccupied physical blocks in the storage space. During the allocation process, a free physical block identifier can be selected from the free physical block pool table, and after the allocation is completed, the free physical block identifier can be removed from the free physical block pool table.

[0096] In some examples, the first dimension and the second dimension are integer multiples of each other. An example of step S104 may include steps S203 to S204.

[0097] Step S203: During the process of allocating physical blocks for the first logical block, in response to the absence of a free first physical block, one or more free second physical blocks are allocated for the first logical block.

[0098] Step S204: During the process of allocating physical blocks for the second logical block, in response to the absence of free second physical blocks, one or more free first physical blocks are allocated for the second logical block.

[0099] For example, when the first size is an integer multiple of the second size, step S203 may include: during the process of allocating physical blocks for the first logical block, in response to the absence of a free first physical block, allocating one or more free second physical blocks for the first logical block. For example, if the first size is the same as the second size, in response to the absence of a free first physical block, one free second physical block can be allocated for the first logical block; if the first size is an integer multiple of the second size greater than 1, in response to the absence of a free first physical block, multiple free second physical blocks can be allocated for the first logical block. In this way, storage allocation can be completed using (multiple) second physical blocks instead of first physical blocks. Accordingly, step S204 may include: during the process of allocating physical blocks for the second logical block, in response to the absence of a free second physical block, allocating one free first physical block for one or more second logical blocks. For example, if the first size is the same as the second size, in response to the absence of a free second physical block, one free first physical block can be allocated for one second logical block; if the first size is an integer multiple of the second size greater than 1, in response to the absence of a free second physical block, one free first physical block can be allocated for multiple second logical blocks. In this way, the remaining space of the larger first physical block can be fully utilized, avoiding storage fragmentation.

[0100] For example, when the second size is an integer multiple of the first size, step S203 may include: during the process of allocating physical blocks for the first logical block, in response to the absence of free first physical blocks, allocating one free second physical block for one or more first logical blocks. For example, if the first size and the second size are the same, in response to the absence of free first physical blocks, one free second physical block can be allocated for one first logical block; if the second size is an integer multiple of the first size greater than 1, in response to the absence of free first physical blocks, one free second physical block can be allocated for multiple first logical blocks. In this way, the remaining space of the larger second physical block can be fully utilized, avoiding storage fragmentation. Correspondingly, step S204 may include: during the process of allocating physical blocks for the second logical block, in response to the absence of free second physical blocks, allocating one or more free first physical blocks for the second logical block. For example, if the first size and the second size are the same, in response to the absence of free second physical blocks, one free first physical block can be allocated for the second logical block; if the second size is an integer multiple of the first size greater than 1, in response to the absence of free second physical blocks, multiple free first physical blocks can be allocated for the second logical block. In this way, storage allocation can be completed using (multiple) first physical blocks instead of second physical blocks.

[0101] The above allocation mechanism enables the mutual appropriation and flexible substitution between the first and second physical blocks, thereby effectively improving the utilization rate of storage resources.

[0102] An example of "managing the mapping relationship between logical blocks and physical blocks based on the block table" in step S104 may include steps S301 to S302.

[0103] Step S301: Manage the mapping relationship between the first logical block and the first physical block based on the first block table.

[0104] Step S302: Manage the mapping relationship between the second logical block and the second physical block based on the second block table.

[0105] In some examples, the mapping relationship between the first logical block and the first physical block and the second logical block and the second physical block can be maintained by maintaining two block tables respectively.

[0106] For example, when the first size and the second size are integer multiples of each other, an example of "managing the mapping relationship between logical blocks and physical blocks based on the block table" in step S104 may include steps S303 to S304.

[0107] Step S303: Manage the mapping relationship between the first logical block, the second logical block, and the first physical block based on the first block table.

[0108] Step S304: Manage the mapping relationship between the first logical block, the second logical block, and the second physical block based on the second block table.

[0109] For example, when the first size and the second size are integer multiples of each other, referring to the aforementioned steps S203 and S204, since the first physical block can be flexibly allocated to the first logical block or the second logical block, the mapping relationship between the first logical block and the first physical block, as well as the mapping relationship between the second logical block and the first physical block, can be recorded in the first block table.

[0110] For example, the mapping relationship recorded in the first table is shown below:

[0111] In this example, the first size is twice the second size. LB_A_01 and LB_A_02 are the identifiers of two first logical blocks, and LB_B_01 and LB_B_02 are the identifiers of two second logical blocks; PB_A_05, PB_A_12, and PB_A_06 are the identifiers of three first physical blocks.

[0112] For example, when the first size and the second size are integer multiples of each other, since the second physical block can be flexibly allocated to the first logical block or the second logical block, the mapping relationship between the first logical block and the second physical block, as well as the mapping relationship between the second logical block and the second physical block, can be recorded in the second block table.

[0113] For example, the mapping relationship recorded in the second table is shown below:

[0114] In this example, the first size is twice the second size. LB_B_03 and LB_B_04 are the identifiers of two second logical blocks, respectively, and LB_A_03 is the identifier of a first logical block; PB_B_01, PB_B_06, PB_B_03, and PB_B_04 are the identifiers of four first physical blocks, respectively.

[0115] It should be noted that the mapping relationship of the block table records mentioned above is only an example, and its specific form can be set according to actual needs.

[0116] For example, if the first size and the second size are the same, one example of "managing the mapping relationship between logical blocks and physical blocks based on the block table" in step S104 may include step S305.

[0117] Step S305: For the first logical block and the second logical block, the mapping relationship between the logical block and the physical block is managed based on the same block table.

[0118] For example, if the first and second dimensions are the same, and the first and second physical blocks are also the same size, all physical blocks can be uniformly numbered (e.g., PB_01~PB_32). The mapping table entries in the block table do not need to be differentiated or specially adapted. Therefore, the mapping relationship between all logical blocks and all physical blocks can be managed uniformly based on a single block table. This method effectively simplifies the design and implementation of the address mapping table, reducing the memory overhead and management complexity during system operation.

[0119] For example, the following is an example of a mapping relationship recorded in the block table:

[0120] Among them, LB_A_01 and LB_A_02 are the identifiers of two first logic blocks, and LB_B_01 and LB_B_02 are the identifiers of two second logic blocks; PB_05 and PB_12 are the identifiers of two first physical blocks, and PB_06 and PB_08 are the identifiers of two second physical blocks. The first and second physical blocks are numbered uniformly.

[0121] It should be noted that the mapping relationship of the block table records mentioned above is only an example, and its specific form can be set according to actual needs.

[0122] The storage management method provided in at least one embodiment of this disclosure may further include step S105.

[0123] Step S105: In response to receiving a data read request, look up the target physical block mapped to the target logical block corresponding to the data read request based on the block table, and read the data corresponding to the target logical block from the target physical block.

[0124] For example, a data read request may include a read request for a stored operator-to-operator tensor (hidden state tensor) or a read request for a key-value cache.

[0125] For example, in step S105, the target physical block where the data corresponding to the data reading request is located can be found based on the logical block-physical block mapping relationship maintained in the block table, and then the data can be read from the target physical block.

[0126] Figure 2 This is a schematic diagram of a storage management method provided in at least one embodiment of the present disclosure.

[0127] For example, Figure 2 An example of an operator flow is shown. The model may include embedding operators, normalization operators, attention operators, addition operators, hybrid expert operators, language model head operators, and sampling operators. The tensors passed between the various operators are called inter-operator transit tensors (hidden state tensors).

[0128] For example, embedding operators can transform discrete symbols (such as words in an input sequence) into continuous, dense vector representations using a learnable embedding matrix, so that subsequent feature extraction layers can process them effectively.

[0129] For example, attention operators are used to capture global dependencies and contextual information between various tokens in an input sequence. They can employ self-attention or cross-attention mechanisms and typically include operations such as linear projection, multi-attention head splitting, attention score calculation, and weighted aggregation.

[0130] For example, addition operators can be used to implement residual connections or feature fusion operations, adding two inputs element-wise to obtain the output. Common operations include element-wise addition, bias addition, and scaling addition.

[0131] For example, the Mixture of Experts (MoE) operator is used to implement a mixed expert mechanism, comprising multiple expert subnetworks, each learning to handle different types of data features or subtasks. The MoE operator introduces a learnable routing mechanism, also known as a gating network. For each input term, the routing mechanism dynamically computes and selects a few of the most relevant expert subnetworks for subsequent computation, rather than activating all expert subnetwork parameters. For example, a MoE operator might contain up to 256 expert subnetworks, but for each term, the routing mechanism only distributes it to at least a few expert subnetworks (e.g., 8) for processing. This sparse activation mechanism allows the model to have a large number of total parameters while maintaining low inference computation, thus significantly improving the model's expressive power. The MoE inference process involves two communication and computation steps: dispatch and combination. Distribution refers to the process of dynamically distributing lexical units to one or more selected expert subnetworks based on the routing table generated by the routing mechanism; combination refers to the process of weighting and fusing the outputs of each expert subnetwork according to the routing weights after the selected one or more expert subnetworks have completed the calculation of the lexical unit, and restoring it into a single lexical unit feature vector.

[0132] For example, the normalization operator can be implemented in the form of layer normalization (LayerNorm), root mean square layer normalization (RMSNorm), etc., and the embodiments disclosed herein are not limited in this regard. Figure 2 The network structures of the normalization operators shown may be the same or different, and this disclosure does not limit this.

[0133] For example, the head operator of a language model can be a linear transformation layer, whose weight matrix dimension is the hidden layer dimension multiplied by the vocabulary size. By mapping the hidden states of the last layer to the vocabulary space and then normalizing them using the normalized exponential function (Softmax), the generation probability of each candidate word can be obtained.

[0134] For example, the sampling operator is configured to determine the corresponding lexical units based on the lexical probability distribution output by the language model head operator. This can be achieved through probabilistic filtering strategies such as Argmax, Top-K, or Top-P. For instance, Argmax selects the candidate lexical unit with the largest value (i.e., the highest probability) in the probability distribution; Top-K sorts the candidate lexical units in descending order of probability and retains the top K elements to form a subset of candidate lexical units; Top-P accumulates the probabilities of candidate lexical units in descending order of probability until the cumulative sum first reaches a preset threshold P, and retains a subset of candidate lexical units within this cumulative range.

[0135] exist Figure 2 In the example, the shape of the operator transfer tensor flowing between the above operators is [N×S, 2048] or [N×S, 151936].

[0136] In the attention operator, the output of the value projection layer (V-Linear) is a value tensor with shape [N,S,4,128], and the output of the rotation position encoding layer (RoPE) corresponding to the key projection layer (K-Linear) is a key tensor with shape [N,4,128,S].

[0137] exist Figure 2 In the example, after the inference service framework is initialized or started, the system can obtain the total amount of available storage resources. After deducting the space occupied by model weights and other basic runtime libraries, the remaining storage resources are determined as dedicated storage space for tensor transfer and key-value cache between operators.

[0138] For example, in Figure 2 In the example, a block pre-filling mechanism can be used. Assuming that the preset block size of the block pre-filling mechanism is 1024, the shape of the tensor between operators corresponding to each subsequence obtained by the block operation can be [1024, 2048].

[0139] For example, the above storage space can be divided into two physical blocks of different sizes ( Figure 2 (Represented by cylinders in the storage space). For example, the size of the first physical block can be set to 1024×2048 based on the block size (1024) and the hidden layer dimension size (2048). For example, assuming the second quantity is 128, the size of the second physical block can be set to 128×4×128 based on the second quantity (128) and the num_heads dimension size (4) and head_dim dimension size (128) of the key-value tensor.

[0140] In this example, an operator-transfer tensor of shape [1024, 2048] can be directly regarded as a first logical block; a value tensor of shape [1024, 4, 128] can be divided into 8 second logical blocks, each with a shape of [128, 4, 128]. The division of the key tensor is similar.

[0141] In some examples, Figure 2 The intermediate activation value tensors within the attention operator can also be stored and managed in the manner described above. For example, for the intermediate activation value tensor output by the rotation position encoding layer corresponding to the query projection layer (Q-Linear), which has a shape of [1024, 32, 128], it can be divided into 2 first logical blocks according to the size of the first physical block; or it can be divided into 64 second logical blocks according to the size of the second physical block. For example, for the intermediate activation value tensor output by the attention layer, which has a shape of [1024, 4, 128], it can be divided into 8 second logical blocks according to the size of the second physical block.

[0142] For example, a first physical block can be allocated to the first logical block, and a second physical block can be allocated to the second logical block. A block table manages the mapping relationship between logical blocks and physical blocks. For instance, the first physical block can be used to store tensors passed between operators. When an operator subsequently needs this tensor as input, the target physical block can be indexed through the mapping relationship in the block table to read the corresponding data. For example, taking the first addition operator in the diagram as an example, its two input tensors are both of shape [1024, 2048], and can be stored in two separate first physical blocks. Similarly, the second physical block can be used to store key-value tensors to form a key-value cache. During subsequent inference, the target physical block can be indexed through the mapping relationship in the block table to read the corresponding data.

[0143] In this example, the size of the first physical block is 32 times the size of the second physical block. In some examples, if there are no free first physical blocks, 32 free second physical blocks can be allocated to one first logical block; if there are no free second physical blocks, free first physical blocks can be allocated to one or more second logical blocks (up to 32 second logical blocks). Through this allocation mechanism, mutual appropriation and flexible substitution between first and second physical blocks can be achieved, thereby effectively improving the utilization rate of storage resources.

[0144] In the above example, the mapping relationship between logical blocks and physical blocks can be managed based on two block tables. For example, the mapping relationship between the first logical block and the second logical block (if any) and the first physical block can be managed based on the first block table; and the mapping relationship between the first logical block (if any) and the second logical block and the second physical block can be managed based on the second block table. In some examples, the mapping relationship between all logical blocks and physical blocks can also be managed through the same block table.

[0145] It should also be noted that the execution order of the various steps of the storage management method in the various embodiments of this disclosure is not limited. Although the execution process of each step has been described in a specific order above, this does not constitute a limitation on the embodiments of this disclosure. The various steps in the storage management method can be executed serially or in parallel, which can be determined according to actual needs.

[0146] For example, compared to the above description, the storage management method provided in at least one embodiment of this disclosure may include more or fewer steps, and the embodiments of this disclosure do not limit this.

[0147] Figure 3 This is a schematic block diagram of a storage management device provided for at least one embodiment of the present disclosure.

[0148] For example, such as Figure 3 As shown, the storage management device 30 provided in at least one embodiment of this disclosure includes a physical partitioning module 31, a first logical partitioning module 32, a second logical partitioning module 33, an allocation module 34, and a management module 35.

[0149] For example, the physical partitioning module is configured to divide the storage space into multiple physical blocks of fixed size.

[0150] For example, the first logical partitioning module is configured to partition the inter-operator transit tensor in the model inference process into one or more first logical blocks, each first logical block including at most a first number of inter-operator transit sub-tensors corresponding to tokens.

[0151] For example, the second logical partitioning module is configured to partition the key-value tensors generated during model inference into one or more second logical blocks, each second logical block including at most a second number of key-value sub-tensors corresponding to lexical units.

[0152] For example, the allocation module is configured to allocate physical blocks for each of the first and second logical blocks.

[0153] For example, the management module is configured to manage the mapping relationship between logical blocks and physical blocks based on a block table.

[0154] In some examples, the physical partitioning module is further configured to divide the storage space into multiple first physical blocks and multiple second physical blocks, wherein the first physical block has a first size and the second physical block has a second size, the first size is determined based on the size of the inter-operator transfer tensor and a first quantity, and the second size is determined based on the size of the key-value tensor and a second quantity.

[0155] In some examples, during model inference, the input sequence is split into multiple consecutive subsequences based on a preset block size, and the operator transfer tensor is generated during the forward computation of processing a single subsequence, wherein the first quantity is equal to the number of tokens in a single subsequence.

[0156] In some examples, the allocation module is further configured to: for each inference request, determine the first physical block that has been used most frequently in the history of the current inference request as the target first physical block; and preferentially allocate the target first physical block to the first logical block corresponding to each subsequence of the current inference request.

[0157] In some examples, where the first size and the second size are integer multiples of each other, the allocation module is further configured to: in the process of allocating physical blocks for the first logical block, in response to the absence of free first physical blocks, allocate one or more free second physical blocks for the first logical block; and in the process of allocating physical blocks for the second logical block, in response to the absence of free second physical blocks, allocate one or more free first physical blocks for the second logical block.

[0158] In some examples, where the first size is an integer multiple of the second size, the allocation module is further configured to: in the process of allocating physical blocks for the first logical block, in response to the absence of a free first physical block, allocate one or more free second physical blocks for the first logical block; and in the process of allocating physical blocks for the second logical block, in response to the absence of a free second physical block, allocate one free first physical block for one or more second logical blocks.

[0159] In some examples, where the second size is an integer multiple of the first size, the allocation module is further configured to: in the process of allocating physical blocks for the first logical block, in response to the absence of a free first physical block, allocate a free second physical block for one or more first logical blocks; and in the process of allocating physical blocks for the second logical block, in response to the absence of a free second physical block, allocate one or more free first physical blocks for the second logical block.

[0160] In some examples, where the first size and the second size are integer multiples of each other, the management module is further configured to: manage the mapping relationship between the first logical block and the second logical block and the first physical block based on the first block table; and manage the mapping relationship between the first logical block and the second logical block and the second physical block based on the second block table.

[0161] In some examples, where the first size and the second size are the same, the management module is further configured to manage the mapping relationship between the logical block and the physical block based on the same block table for the first logical block and the second logical block.

[0162] In some examples, the storage management device also includes an acquisition module configured to: in response to receiving a data read request, look up the target physical block mapped to the target logical block corresponding to the data read request based on the block table, and read the data corresponding to the target logical block from the target physical block.

[0163] In some examples, the block table maintains state information about at least one physical block being occupied by an operator based on a counter.

[0164] It should be noted that the above-mentioned modules can be implemented by software, hardware, firmware or any combination thereof. For example, the equalization module can be implemented as an equalization circuit. The embodiments of this disclosure do not limit its specific implementation.

[0165] It should be understood that the storage management device provided in at least one embodiment of this disclosure can be used to implement the aforementioned storage management method and can also achieve similar technical effects as the aforementioned storage management method, which will not be elaborated here.

[0166] It should be noted that, in the embodiments of this disclosure, the storage management device may include more or fewer modules or units, and the connection relationship between the various modules or units is not limited and can be determined according to actual needs. The specific configuration of each module or unit is not limited; it can be constructed from analog devices according to circuit principles, or from digital chips, or in other suitable ways.

[0167] Figure 4 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0168] For example, such as Figure 4 As shown, the electronic device 400 includes at least one processor 401 and at least one memory 402. The at least one memory 402 includes one or more computer program modules. These computer program modules are stored in the memory 402 and configured to be executed by the at least one processor 401. The one or more computer program modules include instructions for performing the above-described memory management method, which, when executed by the at least one processor 401, can perform one or more steps of the memory management method provided in at least one embodiment of this disclosure. The memory 402 and the processor 401 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0169] For example, processor 401 can be a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), artificial intelligence (AI) accelerator, or other form of processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA); for example, the central processing unit (CPU) can be an x86, ARM, or RISC-V architecture. Processor 401 can be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 400 to perform desired functions.

[0170] For example, memory 402 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0171] Figure 5 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0172] The electronic devices in at least one embodiment of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0173] The electronic device includes at least one processor and a memory. The processor may be referred to as processing device 501 as described below, and the memory may include at least one of ROM 502, RAM 503, and storage device 508 as described below. The memory is used to store programs for performing the methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0174] like Figure 5As shown, the electronic device 500 may include a processing unit 501 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in ROM 502 or a program loaded from storage device 508 into RAM 503. RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interfaces are also connected to bus 504.

[0175] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, displays, speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0176] In particular, according to at least one embodiment of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, at least one embodiment of this disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of at least one embodiment of this disclosure.

[0177] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In at least one embodiment of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In at least one embodiment of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0178] The aforementioned computer-readable medium may be included in the aforementioned electronic device 500; or it may exist independently and not assembled into the electronic device 500.

[0179] Figure 6 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0180] For example, such as Figure 6 As shown, a non-transitory computer-readable storage medium 600 stores computer-readable instructions 601, which, when executed by at least one processor, perform one or more steps of the storage management method described above.

[0181] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard drive for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be... Figure 4 The memory 402 in the memory is described in the foregoing content and will not be repeated here.

[0182] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0183] The following points should be noted regarding this disclosure: (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0184] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0185] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0186] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A storage management method, characterized in that, The storage management method includes: The storage space is divided into multiple physical blocks of fixed size; The operator transit tensor in the model inference process is divided into one or more first logic blocks, and each first logic block includes at most a first number of operator transit sub-tensors corresponding to tokens. The key-value tensors generated during the model inference process are divided into one or more second logical blocks, each second logical block including at most a second number of key-value sub-tensors corresponding to lexical units; A physical block is allocated to each of the first and second logical blocks, and the mapping relationship between logical blocks and physical blocks is managed based on a block table.

2. The method according to claim 1, characterized in that, The process of dividing the storage space into multiple physical blocks of fixed size includes: The storage space is divided into multiple first physical blocks and multiple second physical blocks. Wherein, the first physical block has a first size, the second physical block has a second size, the first size is determined based on the size of the inter-operator transfer tensor and the first quantity, and the second size is determined based on the size of the key-value tensor and the second quantity.

3. The method according to claim 1, characterized in that, During the model inference process, the input sequence is divided into multiple consecutive subsequences based on a preset block size. The inter-operator transfer tensor is generated during the forward computation of processing a single subsequence. Wherein, the first quantity is equal to the number of lexical units in the single subsequence.

4. The method according to claim 3, characterized in that, Allocating physical blocks for the first logical block includes: For each inference request, determine the first physical block that has the highest historical usage frequency of the current inference request as the target first physical block; The target first physical block is preferentially assigned to the first logical block corresponding to each subsequence of the current inference request.

5. The method according to claim 2, characterized in that, When the first size and the second size are integer multiples of each other, the allocation of physical blocks for each logical block in the first logical block and the second logical block includes: During the process of allocating physical blocks for the first logical block, in response to the absence of a free first physical block, one or more free second physical blocks are allocated to the first logical block. During the process of allocating physical blocks for the second logical block, in response to the absence of a free second physical block, one or more free first physical blocks are allocated for the second logical block.

6. The method according to claim 5, characterized in that, When the first size is an integer multiple of the second size, allocating physical blocks for each of the first and second logical blocks includes: During the process of allocating physical blocks for the first logical block, in response to the absence of a free first physical block, one or more free second physical blocks are allocated to the first logical block. During the process of allocating physical blocks for the second logical block, in response to the absence of a free second physical block, a free first physical block is allocated for one or more of the second logical blocks.

7. The method according to claim 5, characterized in that, When the second size is an integer multiple of the first size, allocating physical blocks for each of the first and second logical blocks includes: During the process of allocating a physical block for the first logical block, in response to the absence of a free first physical block, a free second physical block is allocated for one or more of the first logical blocks; During the process of allocating physical blocks for the second logical block, in response to the absence of a free second physical block, one or more free first physical blocks are allocated for the second logical block.

8. The method according to claim 2, characterized in that, When the first size and the second size are integer multiples of each other, the mapping relationship between logical blocks and physical blocks managed based on the block table includes: The first block table manages the mapping relationship between the first logical block and the second logical block and the first physical block; The second block table manages the mapping relationship between the first logical block and the second logical block and the second physical block.

9. The method according to claim 2, characterized in that, When the first size and the second size are the same, the mapping relationship between logical blocks and physical blocks managed based on the block table includes: For the first logical block and the second logical block, the mapping relationship between the logical block and the physical block is managed based on the same block table.

10. The method according to claim 1, characterized in that, The method further includes: In response to receiving a data read request, the system searches for the target physical block mapped to the target logical block corresponding to the data read request based on the block table, and reads the data corresponding to the target logical block from the target physical block.

11. The method according to claim 1, characterized in that, The block table maintains the status information of at least one physical block being occupied by an operator based on a counter.

12. A storage management device, characterized in that, The storage management device includes: The physical partitioning module is configured to divide the storage space into multiple physical blocks of fixed size; The first logical partitioning module is configured to divide the inter-operator transit tensor in the model inference process into one or more first logical blocks, each first logical block including at most a first number of inter-operator transit sub-tensors corresponding to lexical units; The second logical partitioning module is configured to divide the key-value tensor generated during the model inference process into one or more second logical blocks, each second logical block including at most a second number of key-value sub-tensors corresponding to lexical units; The allocation module is configured to allocate physical blocks for each of the first logical block and the second logical block; The management module is configured to manage the mapping relationship between logical blocks and physical blocks based on a block table.

13. An electronic device, characterized in that, The electronic device includes: At least one processor; At least one memory, including one or more computer program modules; The one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the method according to any one of claims 1-11.

14. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions, when executed by at least one processor, perform the method according to any one of claims 1-11.

Images