Data processing method and electronic equipment

Abstract

The invention discloses a data processing method and electronic equipment, and relates to the technical field of computers, and the method comprises the steps: dividing a complete reasoning process of a request into at least one processing stage according to the number of layers of a current reasoning model, and determining a model layer of each stage; and determining a current processing stage, and executing pre-filling reasoning on the request by the corresponding model layer in the first processing node to obtain a key value cache. The second processing node carries out decoding reasoning on the key value cache to obtain the lexical element sequence of the current stage, meanwhile, a new current processing stage is determined, the pre-filling reasoning step and the decoding reasoning step are repeated until the lexical element sequences of all stages are obtained, and the lexical element sequences are spliced to obtain a final result. Therefore, the key value cache is stored and managed by taking the model layer as a unit, pipeline parallelism of three tasks of pre-filling reasoning, network transmission and decoding reasoning is realized, and the problems that computing resources are idle and TTFT is prolonged due to the fact that the key value cache is transmitted by taking the whole request as the unit are solved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more particularly to data processing methods and electronic devices. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) are widely used in scenarios such as intelligent customer service, content generation, and code assistance. Users have placed higher demands on the response speed and concurrency capabilities of inference services, among which Time To First Token (TTFT) has become a key performance indicator for measuring user experience.

[0003] In related technologies, a prefill / decode (P / D) separation architecture is commonly used, which allocates GPU (Graphics Processing Unit) memory on a request-by-request basis and transmits the key-value cache (KV-Cache) of the entire request in memory block order.

[0004] However, this method stores and transmits key-value caches on a per-request basis, which means that when multiple requests are concurrent, the entire layer data transmission for a single request must be completed serially before the Decode node can begin inference. This results in idle computing resources and prolonged TTFT, which urgently needs to be addressed. Summary of the Invention

[0005] This invention provides a data processing method and an electronic device to at least solve the problem in related technologies where key-value caching is transmitted on a per-request basis, resulting in idle computing resources and prolonged TTFT, thereby improving the overall throughput performance of the system.

[0006] This invention provides a data processing method, comprising the following steps: Obtain a data processing request, and based on the current inference model, divide the inference process of the data processing request in the current inference model into at least one processing stage, and determine the model layer of the at least one processing stage; The current processing stage is determined from the at least one processing stage, and the inference operation of the pre-filling stage is performed on the data processing request using the model layer corresponding to the current processing stage in the first processing node to obtain the key-value cache of the current processing stage; The second processing node performs a decoding stage inference operation on the key-value cache of the current processing stage to obtain the word sequence of the current processing stage. At the same time, a new current processing stage is determined from the remaining processing stages, and the step of performing a pre-filling stage inference operation on the data processing request using the model layer corresponding to the current processing stage in the first processing node to obtain the key-value cache of the current processing stage is re-executed to obtain the final processing result.

[0007] The present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the data processing method described in the above embodiments.

[0008] This invention divides the complete inference process of a data processing request into at least one processing stage based on the current inference model layer number, determines the model layer of each stage, and identifies the current processing stage. In the first processing node, the corresponding model layer performs pre-filling inference on the data processing request to obtain a key-value cache. The second processing node performs decoding inference on the key-value cache to obtain the lexical sequence of the current stage, simultaneously determining a new current processing stage. The pre-filling and decoding inference steps are repeated until the lexical sequences of all stages are obtained, and these sequences are concatenated to obtain the final result. Therefore, by storing and managing the key-value cache on a model layer basis, the pipelined parallelism of pre-filling inference, network transmission, and decoding inference is achieved. This solves the problem of idle computing resources and prolonged TTFT caused by transmitting the key-value cache on a per-request basis, thus improving the overall system throughput performance. Attached Figure Description

[0009] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart of a data processing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating memory segmentation in related technologies. Figure 3 This is a schematic diagram illustrating the key-value cache transmission process in related technologies. Figure 4 This is a schematic diagram illustrating the memory block division in an embodiment of the present invention; Figure 5 This is a block diagram of a data processing device provided in an embodiment of the present invention; Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0012] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0013] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] The present invention provides a data processing method, and the execution flow of the data processing method is described in detail below.

[0015] Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention.

[0016] Before introducing the data processing method proposed in the embodiments of the present invention, a brief introduction to the relevant background technology will be given.

[0017] In related technologies, when inventing key-value cache memory space, the system allocates memory based on the specific parameters of the model, treating each inference request as an independent unit. Considering that GPU memory management is based on fixed-size blocks (memory blocks), assuming the system's block_size is set to 128 and the model's total number of layers is 4, then 32 tokens will form a block unit, serving as the basis for memory storage and management. In the inference process of language models, tokens typically represent the words the model infers. Each token is usually composed of a numerical code. The model generates a token for each inference iteration, and each token represents one throughput, matching the corresponding text content in a predefined vocabulary. For example, if the input sequence length of a request is 1024, the number of blocks required for that request is 1024 ÷ 32, totaling 32 blocks. After the Prefill inference phase is completely finished, the system needs to send the complete key-value cache data of these 32 blocks to the Decode node. The Decode node must receive all 32 blocks of data before it can start the inference process of the Decode phase. If multiple requests occur simultaneously in the system, the second request must wait for all key-value cached data from the first request to be transmitted before it can begin transmitting its own data. In this situation, the Decode node can only process the inference task of one request at a time, resulting in relatively low overall efficiency.

[0018] To further understand, consider a simplified model structure containing only three layers (neural network layers). Simultaneously, two users submit inference requests to the model, each request containing an input sequence of 12 tokens. The memory partitioning can be as follows: Figure 2 As shown. Figure 2 In the diagram, q1_token1 to q1_token12 represent the 1st to 12th tokens in request 1, respectively. This demonstrates that the relevant technology employs a token-by-token, layer-by-layer sequential invention and usage mechanism when allocating key-value cache memory. This mechanism is widely used in current system operation. Its basic strategy is to dynamically create and allocate corresponding cache space for each token on demand at different levels, and release or reuse it after use based on actual conditions. The transfer of key-value cache can be as follows... Figure 3 As shown, during the key-value cache transfer process, data is transferred sequentially in memory blocks. Specifically, according to... Figure 2The memory layout of the key-value cache shown indicates that after the first memory block (memory block 1) is transmitted, the Decode node can only receive the complete key-value cache information for the first 10 tokens. However, since the key-value cache data for the 11th token (q1_token11) and the 12th token (q1_token12) has not been fully transmitted, some data is still missing. Therefore, the Decode node cannot perform subsequent inference calculations based on these incomplete caches. It must wait for the second memory block (memory block 2) to be fully transmitted before the Decode node can obtain the complete key-value cache data, including q1_token11 and q1_token12, and only then can it continue to execute the inference task in the Decode phase.

[0019] Therefore, the relevant technology has the following shortcomings: (1) In the large model inference process, calculations are usually performed step by step in a hierarchical structure, with each layer of inference operations being executed independently. When the Prefill node transmits key-value cache data to the Decode node, request 1 contains 12 tokens. It needs to wait for two complete memory blocks to be sent before the key-value cache data of this request can be fully ready. Since the Decode node must rely on these transmitted data when performing the first layer of inference for request 1, it needs to wait for both data blocks to be sent. However, at this time, the key-value cache data corresponding to the second and third layers of request 1 is not yet used or ready, resulting in a significant extension of the waiting time in the Decode stage. This waiting process causes the computing resources of the Decode node to be idle during the corresponding time period, which is not fully utilized, thereby reducing the efficiency and resource utilization of the overall inference process.

[0020] (2) In the key-value caching management mechanism, the invention and maintenance of the cache are based on a single request as the basic unit. The key-value data of each request is transmitted block by block according to the layer order, and there are strict dependencies between different requests. Specifically, the first layer inference operation of request 2 must wait for all layers of request 1 to complete the key-value cache transmission before it can start to execute. This serialized processing method, especially in multi-batch concurrent inference scenarios, will lead to significant latency issues. Due to the lack of parallelization or pre-scheduling mechanisms for key-value caching, the first-term generation time (TTFT) of request 2 is significantly lengthened, thereby affecting the overall inference efficiency and system response performance.

[0021] Based on the aforementioned problems, this invention proposes a data processing method that, within a P / D separation framework, adjusts and optimizes the memory layout structure to store and manage key-value caches on a model-layer basis. Assume a large language model's network structure contains three layers (three model layers), and two different inference requests arrive at the inference computing system simultaneously. When the system's Prefill and Decode nodes receive these two requests, the system can centrally allocate and manage memory based on the actual sequence length of each request when creating and allocating the memory space required for the key-value cache. Specifically, the system will create and centrally store the key-value caches required for the first layer of both requests in the same contiguous memory area; similarly, the key-value caches for the second layer of both requests will also be arranged in the same memory area; the key-value caches for the third layer of both requests will also be centrally stored in the same manner, and so on. Figure 4 As shown, q1_token1~q1_token12 represent the 1st to 12th tokens of request 1, and q2_token1~q2_token12 represent the 1st to 12th tokens of request 2. Memory block 1 contains all the key-value caches for the first layer of requests 1 and 2. Through this hierarchical centralized memory management mechanism, after the Prefill node completes the forward computation of the two requests at the first layer (i.e., the Prefill phase), the generated key-value cache data is uniformly written to the previously allocated first memory area. Subsequently, the system immediately transmits all the first-layer key-value cache data stored in this area and simultaneously begins to execute the Prefill computation of the two requests at the second layer. At the same time, once the Decode node receives the first-layer key-value cache data transmitted from the first area, it immediately uses this data to perform the decoding phase inference of the two requests at the first layer.

[0022] This allows for efficient reuse of the system's computing resources: on one hand, the Prefill node is executing the Level 2 inference of the two requests in parallel; on the other hand, the Decode node is simultaneously executing the Level 1 decoding inference. Once the Level 2 Prefill computation is complete, the resulting key-value cache data is written to the second memory region, and the transmission process begins immediately. Simultaneously, the Prefill node continues executing the Level 3 forward computation of the two requests. Upon receiving the Level 2 key-value cache data, the Decode node immediately begins Level 2 decoding inference, and this process repeats until all levels of computation are complete. This layer-by-layer centralized caching and interleaved execution significantly improves the resource utilization of GPUs or other computing devices, reduces idle waiting time, and achieves efficient parallelism between the Prefill and Decode stages. The parallel advantages and resource reuse benefits of this design become even more pronounced when the number of concurrent requests increases.

[0023] Next, the data processing method proposed in the embodiments of the present invention will be described in detail.

[0024] For example, such as Figure 1 As shown, the data processing method includes the following steps: In step S101, a data processing request is obtained, and based on the current inference model, the inference process of the data processing request in the current inference model is divided into at least one processing stage, and the model layer of at least one processing stage is determined.

[0025] In this context, the data processing request is the user's input. A model layer refers to the structural unit within the model; each computation of a model layer during forward propagation can be considered an independent processing stage.

[0026] In other words, after receiving a data processing request, the system first parses and identifies the request to clarify its specific content and objective. Then, the system comprehensively analyzes the data processing request based on the current inference model, rationally dividing the complete inference process of the data processing request within the current inference model into multiple logically clear and independently executable processing stages. Simultaneously, for each divided processing stage, the system further determines its corresponding model layer, thereby providing accurate and reliable basis for subsequent data processing operations.

[0027] As one possible implementation, in some embodiments, based on the current inference model, the inference process of the data processing request in the current inference model is divided into at least one processing stage, including: determining the sequence length of the data processing request and the number of layers in the current inference model; and dividing the inference process of the data processing request in the current inference model into at least one processing stage based on the sequence length of the data processing request and the number of layers in the current inference model, wherein the number of processing stages is equal to the number of layers in the current inference model.

[0028] Understandably, the sequence length of a data processing request refers to the data dimension of the task, which determines the overhead of a single computation and the required cache size. The longer the sequence, the greater the computational load per layer and the larger the generated key-value cache. The number of model layers in the current inference model refers to the logical depth dimension of the task, which determines the number of standard steps required to complete this request. For example, in a 32-layer model, any request must sequentially go through these 32 steps.

[0029] Specifically, by analyzing and determining the sequence length of the data processing request and the number of layers in the current inference model, the complete inference process of the data processing request can be divided into a number of processing stages equal to the number of layers in the current inference model. For example, for an inference model with N model layers, the entire inference process of the request (whether it is Prefill or Decode) is divided into N consecutive processing stages.

[0030] Therefore, by dividing the system into layers and stages, it supports the generation, storage, and transmission of key-value caches at the model layer level. This meets the optimization requirements of continuous layer-by-layer layout in GPU memory, reduces memory fragmentation, and clears theoretical obstacles for subsequent layer-granular memory management, batch transmission, and pipelined parallel computing, laying a solid foundation.

[0031] In step S102, the current processing stage is determined from at least one processing stage, and the key-value cache of the current processing stage is obtained by performing the inference operation of the pre-filling stage on the data processing request using the model layer corresponding to the current processing stage in the first processing node.

[0032] As is understandable, key-value caching is an optimization technique used to improve the inference performance of language models. Its core principle is to cache and store the key and value vectors corresponding to each calculated token during sequence inference. When generating a new token, the system can directly reuse the previously cached key-value pairs without recalculating them. In this way, the model avoids a large amount of redundant computation, significantly reducing computational complexity and memory access overhead, thereby effectively accelerating inference speed, especially suitable for long text generation and autoregressive decoding tasks.

[0033] The first processing node is the Prefill node in the P / D separation architecture. This node is mainly used to parse and understand the user-provided input information. In this stage, the system first encodes and processes the user input to generate the first token. Simultaneously, it initializes a key-value caching mechanism based on the input content, preparing data for the subsequent autoregressive generation process. This stage heavily relies on the GPU's computing power because it involves a large number of matrix operations and attention mechanisms, making it a typical computationally intensive task. Generally, the longer the user input sequence, the larger the amount of data that needs to be processed and initialized, thus increasing the time required to generate the first token.

[0034] Specifically, the system can execute each processing stage sequentially on the first processing node. That is, the current processing stage refers to the next uncomputed stage with the smallest number. For example, after completing the computation of the i-th model layer, the system immediately identifies the current processing stage as the (i+1)-th model layer. This step ensures that the work of the first processing node can proceed in an orderly and efficient manner along the depth direction of the model, providing a stable and predictable pace for pipeline parallelism. The first processing node processes the user's complete input sequence, that is, it uses the i-th model layer to perform forward propagation computation on the input data, generating a key vector and a value vector (key-value pair) for each token in the sequence, and caching the computation results of each model layer, thus obtaining the key-value cache for each corresponding processing stage.

[0035] In step S103, the key-value cache of the current processing stage is decoded by the second processing node to obtain the word sequence of the current processing stage. At the same time, a new current processing stage is determined from the remaining processing stages, and the step of performing the pre-filling stage inference operation on the data processing request using the model layer corresponding to the current processing stage in the first processing node to obtain the key-value cache of the current processing stage is re-executed to obtain the final processing result.

[0036] Understandably, the second processing node is the Decode node in the P / D separation architecture. This node can iteratively execute the inference task based on the preliminary inference results from the Prefill stage and the pre-generated key-value cache information. In each loop, the system generates only one token as output and immediately updates and expands the corresponding key-value cache after generation to ensure that subsequent inference steps can efficiently utilize the existing intermediate states. Because this stage frequently reads, writes, and updates the key-value cache data in the GPU memory, its performance is highly dependent on the GPU's memory bandwidth resources, making it an overall memory-intensive computational task.

[0037] Specifically, when the second processing node receives the key-value cache of the i-th model layer from the first processing node, it can immediately activate the corresponding model layer to perform inference operations in the decoding stage, obtaining the lexical sequence corresponding to the current processing stage (i.e., the i-th model layer). The output of this stage is not a complete answer, but rather the next one or more lexical sequences generated based on all historical contexts up to the present. This marks the completion of the decoding task for this level and contributes a portion of the final output. While the second processing node is working, the first processing node is not idle. It can immediately select the next (i.e., the (i+1)-th model layer) from the uncomputed levels as the new target, following a predetermined order (usually an incrementing model layer number), and begin performing pre-filling calculations for that layer to generate the key-value cache of the (i+1)-th model layer. This step ensures that production data (i.e., key-value cache) is continuously generated, and that while the first processing node is executing the pre-filling inference of the (i+1)th model layer (i.e., stage i+1), the network is transmitting the output of the ith model layer (i.e., the key-value cache of the ith layer), and the second processing node is executing the decoding inference of the (i-1)th model layer (i.e., stage i-1), all three are performed simultaneously, maximizing resource utilization.

[0038] Thus, the decoding and pre-filling process forms a self-driven loop. Each completion of the loop signifies the end of decoding for one model layer, and the initiation of pre-filling for the next model layer begins. This loop continues until the current inference model outputs a stop symbol or generates a sufficiently long text, thus obtaining the lexical sequences from all processing stages. Since the lexical sequences generated by the second processing node in each loop are fragments of the final answer, the system requires a post-processing step to correctly assemble these lexical fragments, generated in chronological and logical order, into a coherent and complete text, which is then returned to the user as the final result.

[0039] As one possible implementation, in some embodiments, the word sequence of the current processing stage is obtained by performing a decoding-stage inference operation on the key-value cache of the current processing stage through a second processing node, including: obtaining the hidden state of the current processing stage by performing a decoding-stage inference operation on the key-value cache of the current processing stage through a second processing node; and transforming the hidden state of the current processing stage to obtain the word sequence of the current processing stage.

[0040] Specifically, the second processing node can utilize all currently cached historical key-value pairs (key-value cache) and, through the computation of the Transformer decoder, generate a high-dimensional hidden state vector representing the most semantically relevant element. This abstract hidden state vector can then be transformed into a concrete, readable output word through a projection layer and a sampling strategy. Furthermore, during the transformation from hidden state to word, the randomness of the generation can be controlled by adjusting the sampling strategy (such as greedy sampling, temperature sampling, etc.), thereby achieving both accurate and creative text output.

[0041] Thus, the model's understanding of historical dialogues (stored in the key-value cache) and the current semantic computation (i.e., the hidden state) are ultimately translated into the next specific character or word (lexical) in the dialogue, realizing the transformation from context to specific output. Since the complete context information (i.e., the key-value cache) is fully referenced when computing the hidden state, it ensures that the newly generated lexical remains syntactically and semantically consistent with the previous dialogue.

[0042] Furthermore, in some embodiments, after performing the inference operation of the pre-filling stage on the data processing request using the model layer corresponding to the current processing stage in the first processing node to obtain the key-value cache of the current processing stage, the method further includes: determining the target storage location from at least one first memory block of the second processing node based on the sequence length of the data processing request and the model layer corresponding to the current processing stage; and storing the key-value cache of the current processing stage to the target storage location.

[0043] Understandably, the model layer corresponding to the current processing stage determines which large region (i.e., which memory block) the data belongs to; for example, a memory block specifically storing the key-value cache corresponding to the first model layer. The sequence length of the data processing request determines the specific starting offset within this large region. Since the key-value cache for each data processing request is proportional to its sequence length, the system can accurately calculate the start and end addresses of the data block for each data processing request within the memory block.

[0044] Specifically, based on the sequence length characteristics of the data processing request and the model layer corresponding to the current processing stage, the most suitable target storage location is determined from at least one first memory block contained in the second processing node through precise matching. Subsequently, the first processing node can write the key-value cache that was just calculated and temporarily stored in the second memory block to the target storage location through memory copy (or DMA (Direct Memory Access) operation), so that the second processing node has a formal copy of the corresponding data, which the second processing node can access and use at any time.

[0045] Therefore, by completing the calculation of a certain layer on the first processing node, the accurate location is found in the pre-allocated memory of the second processing node according to the length of the request and the current layer number, and the key-value cache of that layer is stored in that location, ensuring that the data is placed in the correct location in the memory of the second processing node in an orderly manner, thereby achieving efficient batch data management and transmission on a layer-by-layer basis.

[0046] Furthermore, in some embodiments, before determining the target storage location from at least one memory block of the second processing node based on the sequence length of the data processing request and the model layer corresponding to the current processing stage, the method further includes: allocating at least one second memory block for the data processing request in the first processing node on a per-model-layer basis for each model layer of the current inference model, and allocating at least one first memory block for the data processing request in the second processing node; wherein the at least one second memory block is configured to store the key-value cache of the current processing stage.

[0047] Specifically, before formally computing using the current inference model, the system can pre-prepare storage space for the data on two processing nodes. That is, as follows: Figure 4 As shown, the key-value caches required by the first model layer in multiple data processing requests are uniformly invented and centrally stored in the same contiguous memory area. The key-value caches required by the second model layer in multiple data processing requests are also uniformly invented and centrally stored in the same contiguous memory area. The third model layer in multiple data processing requests is also centrally stored in the same way, and so on. The second memory block is the memory area in the first processing node. This memory block serves as a temporary working area or sending buffer, used to temporarily store the key-value caches of a certain model layer (a certain processing stage) just calculated during the inference process of the pre-filling stage for each data processing request. The first memory block is the memory area in the second processing node. This memory block serves as the final destination, used to persistently store the key-value caches of each data processing request transmitted from the first processing node for repeated use in the decoding stage.

[0048] Therefore, by completing all time-consuming memory allocation operations in advance, the latency and uncertainty caused by dynamic memory allocation on critical inference computation paths are avoided, ensuring the real-time performance and stability of the inference process.

[0049] Optionally, in some embodiments, both the first memory block and the second memory block use a video memory block of a preset length as the smallest management unit, and the key-value cache of the current processing stage of the data processing request corresponding to the same model layer is continuously stored in the same video memory block or multiple adjacent video memory blocks.

[0050] In essence, a pre-defined memory block refers to a fixed-size memory unit provided by the GPU memory manager (such as the underlying memory pool of CUDA (Compute Unified Device Architecture) or a custom memory allocator). For example, the system can set each minimum managed unit to be a 256KB or 1MB memory block. That is, whether it's the first or second memory block, when the system allocates memory for it, it is essentially creating one or more fixed-size memory blocks. If the amount of key-value cache data for a data processing request is small enough to fit into a single memory block, then it will be stored completely within that single block. If the amount of key-value cache data for a data processing request is large enough to fit into a single memory block, then the system will allocate multiple physically adjacent memory blocks from the memory manager and store the data sequentially in this contiguous, larger memory space.

[0051] Therefore, by storing the key-value caches of multiple data processing requests under the same model layer contiguously, the spatial locality of memory access is significantly enhanced, enabling the GPU to utilize cache and memory bandwidth more efficiently when performing attention calculations, and reducing random access overhead.

[0052] Furthermore, in some embodiments, after the second processing node allocates at least one first memory block for the data processing request, the method further includes: establishing an address mapping table between at least one second memory block and at least one first memory block; wherein the address mapping table is configured to store a one-to-one mapping relationship between the source physical address of the second memory block corresponding to at least one model layer in the first processing node and the target physical address of the corresponding first memory block in the second processing node.

[0053] Specifically, after the first and second processing nodes allocate memory blocks (the second memory block and the first memory block) according to a unified layout, the central management service (such as a distribution service) or the negotiation protocol between nodes can collect the physical memory addresses allocated by both parties and create a mapping entry for each logical data channel (i.e., the data flow at each layer). This ultimately forms a global address mapping table, where each mapping entry uniquely corresponds to a specific source address and a specific destination address. The source physical address is the starting physical address of the second memory block allocated by the first processing node for a specific model layer (such as layer i), and this address is the starting point of data transmission. The destination physical address is the starting physical address of the first memory block allocated by the second processing node for the same model layer (layer i), and this address is the ending point of data transmission.

[0054] Therefore, by establishing precise address mapping in advance, complex cross-node communication is abstracted into a simple and fast table lookup operation, thus providing the most basic and decisive communication guarantee for the pipeline parallelism and high throughput of the entire system.

[0055] Furthermore, in some embodiments, before performing the decoding stage inference operation on the key-value cache of the current processing stage through the second processing node to obtain the word sequence of the current processing stage, the method further includes: determining whether the key-value cache of the current processing stage has completed the storage action at the target storage location; if the key-value cache of the current processing stage has completed the storage action at the target storage location, sending storage completion information to the second processing node through the first processing node.

[0056] Specifically, to achieve a stable and reliable pipelined parallel process, before performing the decoding inference operation on the key-value cache of the current processing stage through the second processing node, it can first determine whether the key-value cache of the current processing stage has completed the storage action at the target storage location. That is, after the first processing node initiates data transfer through a DMA engine or RDMA (Remote Direct Memory Access) operation, it will receive a completion notification from the underlying hardware, informing it that the data has been successfully written to remote memory. At this point, it "determines" that the storage action is complete. After the first processing node confirms that the data has arrived safely, it can send a lightweight storage completion message (Notify) to the second processing node through an independent and reliable communication channel (e.g., a TCP (Transmission Control Protocol) control connection or a message queue based on shared memory).

[0057] Therefore, by introducing an acknowledgment process, the system can detect abnormal situations such as transmission failures earlier and retry or report errors, instead of blindly performing subsequent error calculations, thereby improving the system's fault tolerance.

[0058] Optionally, in some embodiments, the storage completion information is a preset lightweight single-packet notification information, which includes a triplet consisting of request batch identifier information, model layer number information, and storage address pointer of the target storage location.

[0059] Understandably, the request batch identifier is a unique identifier used to distinguish inference requests from different batches or sources being processed simultaneously in the system. In high-concurrency scenarios, the second processing node can simultaneously process notifications from multiple first processing nodes, or notifications from multiple request streams within the same node. Without this identifier, the second processing node will be unable to determine which specific request context the storage completion information corresponds to, leading to task scheduling chaos. The model layer number information explicitly indicates which layer of the current inference model the completed key-value cache belongs to. Since the pipeline parallelism in this embodiment is layer-level, the second processing node needs to know which layer the currently ready data belongs to in order to load the corresponding model layer parameters for calculation and update its internal inference progress status for that request (e.g., marking the i-th layer as ready and ready to start calculation). The target storage location's storage address pointer is a pointer to a specific location in the second processing node's own GPU memory. The second processing node can directly access the data based on the storage address pointer carried in the storage completion information without performing any table lookups or calculations, reducing the address resolution overhead to zero.

[0060] Thus, the lightweight notification information composed of the aforementioned triples, with its small size and accurate information, ensures that tens of thousands of computing tasks can be triggered on time and accurately in complex parallel computing environments, effectively achieving high throughput and low latency.

[0061] It should be noted that, based on the parallelism of the layer-level pipeline, a central scheduler or distributed decision-making mechanism can be introduced. This central scheduler or distributed decision-making mechanism can monitor the load status of the first processing node, network transmission bandwidth, and the second processing node in real time. Based on this, the optimal execution strategy can be dynamically selected for each processing stage (i.e., each model layer), without being bound by a fixed first-processing-second-processing-node path. The execution strategies can include: Strategy 1 (remote decoding, i.e., the scheme in this embodiment), where the i-th layer key-value cache generated by the first processing node is transmitted to the second processing node for decoding. This mode is suitable for scenarios where the second processing node has a light load, sufficient network bandwidth, and the goal is to minimize the latency of the first token; Strategy 2 (local decoding), where, when the system determines that the second processing node is overloaded or the network is congested, the scheduler can instruct the first processing node to perform the decoding stage inference of the i-th layer locally after completing the pre-filling inference, generate the output tokens of that layer, and only send the generated token results (whose data volume is much smaller than the data volume of the key-value cache) to the second processing node.

[0062] Therefore, when the second processing node or the network becomes a bottleneck, the system can automatically "offload" some decoding computation tasks to the relatively idle first processing node, preventing the entire system from being dragged down by slow links, thus maintaining a high overall throughput even under high load. In Strategy 2 mode, what is transmitted is no longer a huge key-value cache, but a lightweight token, which greatly reduces network pressure, especially suitable for deployment environments with limited or fluctuating network bandwidth.

[0063] To facilitate a better understanding of the data processing method of the embodiments of the present invention by those skilled in the art, the following is a detailed description... Figure 4 To elaborate further.

[0064] Step 1: On the server of node 1, start the Prefill inference service; Step 2: On the server at node 2, start the Decode inference service; Step 3: Start the distribution service and begin receiving data processing requests; Step 4: The distribution service receives two data processing requests, sends Request 1 and Request 2 to the Prefill node (first processing node) and the Decode node (second processing node) at the same time, and informs the Prefill node and the Decode node of the connection method so that they can work together to complete the task. Step 5: After receiving the user request, the Decode node, based on the sequence lengths of request 1 and request 2, performs... Figure 4 The memory layout pre-allocates the memory space needed for inference on the GPU card and sends the temporary space memory address invented in step 2 to the Prefill node; Step 6: After receiving user request 1 and request 2, the Prefill node, based on the sequence lengths of request 1 and request 2, performs... Figure 4 The memory layout allocates the memory space required for inference on the GPU card and begins to execute inference computations; Step 7: After the inference calculation of the first layer is completed, the calculation result of the first layer will be cached in the memory space of step 6. This calculation result is the key-value cache of the first model layer. Step 8: Prefill uses the card-to-card communication protocol to sequentially transfer the memory blocks of the first model layer key-value cache to the memory address of Dcode. After the transfer of memory block 1 is completed, it sends a Notify through socket communication to inform the Decode node that the key-value cache of memory block 1 has been transferred. At the same time, the Prefill node continues to perform the second layer inference calculation. Step 9: After receiving the Prefill Notify, the Decode node reads the key-value cache of the first model layer, then starts the first layer Decode stage inference for request 1 and request 2, and prepares to receive the key-value cache of the second layer, and so on until the inference is completed.

[0065] According to the data processing method proposed in this embodiment, the complete inference process of a data processing request is divided into at least one processing stage based on the current inference model layer number, and the model layer of each stage is determined. The current processing stage is determined, and the corresponding model layer in the first processing node performs pre-filling inference on the data processing request to obtain a key-value cache. The second processing node performs decoding inference on the key-value cache to obtain the lexical sequence of the current stage, and simultaneously determines a new current processing stage. The above pre-filling inference and decoding inference steps are repeated until the lexical sequences of all stages are obtained. These lexical sequences are then concatenated to obtain the final result. Thus, by storing and managing the key-value cache on a model layer basis, the pipeline parallelism of the three major tasks—pre-filling inference, network transmission, and decoding inference—is realized. This solves the problem of idle computing resources and prolonged TTFT caused by transmitting the key-value cache on a request-by-request basis, thereby improving the overall throughput performance of the system.

[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that the system according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0067] Next, the data processing apparatus according to an embodiment of the present invention is described with reference to the accompanying drawings.

[0068] Figure 5 This is a block diagram of a data processing apparatus according to an embodiment of the present invention; like Figure 5 As shown, the data processing device 10 includes: a processing module 100, a first inference module 200, and a second inference module 300.

[0069] The processing module 100 is used to acquire data processing requests, and based on the current inference model, divide the inference process of the data processing requests in the current inference model into at least one processing stage, and determine the model layer of at least one processing stage. The first inference module 200 is used to determine the current processing stage from at least one processing stage, and to perform inference operations of the pre-filling stage on the data processing request using the model layer corresponding to the current processing stage in the first processing node to obtain the key-value cache of the current processing stage. The second inference module 300 is used to perform inference operations on the key-value cache of the current processing stage through the second processing node to obtain the word sequence of the current processing stage. At the same time, it determines a new current processing stage from the remaining processing stages and re-executes the step of performing inference operations on the data processing request using the model layer corresponding to the current processing stage in the first processing node to obtain the key-value cache of the current processing stage, so as to obtain the final processing result.

[0070] Optionally, in some embodiments, the processing module 100 is specifically used for: Determine the sequence length of the data processing request and the number of layers in the current inference model; Based on the sequence length of the data processing request and the number of layers in the current inference model, the data processing request is divided into at least one processing stage, wherein the number of processing stages is equal to the number of layers in the current inference model.

[0071] Optionally, in some embodiments, after performing the inference operation of the pre-filling stage on the data processing request using the model layer corresponding to the current processing stage in the first processing node to obtain the key-value cache of the current processing stage, the first inference module 200 includes: The determining unit is used to determine the target storage location from at least one first memory block of the second processing node based on the sequence length of the data processing request and the model layer corresponding to the current processing stage. A storage unit is used to cache and store the key-value pairs of the current processing stage to the target storage location.

[0072] Optionally, in some embodiments, before determining the target storage location from at least one memory block of the second processing node based on the sequence length of the data processing request and the model layer corresponding to the current processing stage, the determining unit is further configured to: For each model layer of the current inference model, at least one second memory block is allocated for the data processing request in the first processing node, and at least one first memory block is allocated for the data processing request in the second processing node. At least one second memory block is configured to store the key-value cache of the current processing stage.

[0073] Optionally, in some embodiments, after the second processing node allocates at least one first memory block for the data processing request, the determining unit is further configured to: Establish an address mapping table between at least one second memory block and at least one first memory block; The address mapping table is configured to store a one-to-one mapping relationship between the source physical address of the second memory block corresponding to at least one model layer in the first processing node and the target physical address of the corresponding first memory block in the second processing node.

[0074] Optionally, in some embodiments, the second inference module 300 is specifically used for: The hidden state of the current processing stage is obtained by performing inference operations on the key-value cache of the current processing stage through the second processing node during the decoding stage. The hidden state of the current processing stage is transformed to obtain the word sequence of the current processing stage.

[0075] Optionally, in some embodiments, before obtaining the lexical sequence of the current processing stage by performing the decoding stage inference operation on the key-value cache of the current processing stage through the second processing node, the second inference module 300 is further configured to: Determine whether the key-value cache in the current processing stage has completed the storage action at the target storage location; When the key-value cache completes the storage operation at the target storage location during the current processing stage, the storage completion information is sent from the first processing node to the second processing node.

[0076] Optionally, in some embodiments, both the first memory block and the second memory block use a video memory block of a preset length as the smallest management unit, and the key-value cache of the current processing stage of the data processing request corresponding to the same model layer is continuously stored in the same video memory block or multiple adjacent video memory blocks.

[0077] Optionally, in some embodiments, the storage completion information is a preset lightweight single-packet notification information, which includes a triplet consisting of request batch identifier information, model layer number information, and storage address pointer of the target storage location.

[0078] For a description of the features in the embodiment corresponding to the data processing device, please refer to the relevant description in the embodiment corresponding to the data processing method, which will not be repeated here.

[0079] According to the data processing apparatus proposed in this embodiment of the invention, the complete reasoning process of a data processing request is divided into at least one processing stage based on the current inference model layer number, and the model layer of each stage is determined. The current processing stage is determined, and the corresponding model layer in the first processing node performs pre-filling inference on the data processing request to obtain a key-value cache. The second processing node performs decoding inference on the key-value cache to obtain the lexical sequence of the current stage, and simultaneously determines a new current processing stage. The above pre-filling inference and decoding inference steps are repeated until the lexical sequences of all stages are obtained. These lexical sequences are then concatenated to obtain the final result. Thus, by storing and managing the key-value cache on a model layer basis, the pipelined parallelism of the three major tasks—pre-filling inference, network transmission, and decoding inference—is realized. This solves the problem of idle computing resources and prolonged TTFT caused by transmitting the key-value cache on a request-by-request basis, thereby improving the overall system throughput performance.

[0080] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. The electronic device may include: The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0081] When the processor 602 executes the program, it implements the data processing method provided in the above embodiments.

[0082] Furthermore, electronic devices also include: Communication interface 603 is used for communication between memory 601 and processor 602.

[0083] The memory 601 is used to store computer programs that can run on the processor 602.

[0084] The memory 601 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0085] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0086] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0087] Processor 602 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention.

[0088] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data processing method embodiments when it is run.

[0089] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0090] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0091] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data processing method embodiments.

[0092] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0093] The data processing method provided by this invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A data processing method, characterized in that, Includes the following steps: Obtain a data processing request, and based on the current inference model, divide the inference process of the data processing request in the current inference model into at least one processing stage, and determine the model layer of the at least one processing stage; The current processing stage is determined from the at least one processing stage, and the inference operation of the pre-filling stage is performed on the data processing request using the model layer corresponding to the current processing stage in the first processing node to obtain the key-value cache of the current processing stage; The second processing node performs a decoding stage inference operation on the key-value cache of the current processing stage to obtain the word sequence of the current processing stage. At the same time, a new current processing stage is determined from the remaining processing stages, and the step of performing a pre-filling stage inference operation on the data processing request using the model layer corresponding to the current processing stage in the first processing node to obtain the key-value cache of the current processing stage is re-executed to obtain the final processing result.

2. The data processing method according to claim 1, characterized in that, The process of dividing the data processing request into at least one processing stage based on the current inference model includes: Determine the sequence length of the data processing request and the number of layers in the current inference model; Based on the sequence length of the data processing request and the number of layers in the current inference model, the inference process of the data processing request in the current inference model is divided into at least one processing stage, wherein the number of processing stages is equal to the number of layers in the current inference model.

3. The data processing method according to claim 2, characterized in that, After obtaining the key-value cache of the current processing stage by performing the inference operation of the pre-filling stage on the data processing request using the model layer corresponding to the current processing stage in the first processing node, the method further includes: Based on the sequence length of the data processing request and the model layer corresponding to the current processing stage, the target storage location is determined from at least one first memory block of the second processing node; Store the key-value cache of the current processing stage to the target storage location.

4. The data processing method according to claim 3, characterized in that, Before determining the target storage location from at least one memory block of the second processing node based on the sequence length of the data processing request and the model layer corresponding to the current processing stage, the method further includes: For each model layer of the current inference model, at least one second memory block is allocated for the data processing request at the first processing node, and at least one first memory block is allocated for the data processing request at the second processing node; The at least one second memory block is configured to store the key-value cache of the current processing stage.

5. The data processing method according to claim 4, characterized in that, After the second processing node allocates the at least one first memory block for the data processing request, the process further includes: Establish an address mapping table between the at least one second memory block and the at least one first memory block; The address mapping table is configured to store a one-to-one mapping relationship between the source physical address of the second memory block corresponding to at least one model layer in the first processing node and the target physical address of the corresponding first memory block in the second processing node.

6. The data processing method according to claim 1, characterized in that, The step of obtaining the lexical sequence of the current processing stage by performing the decoding stage inference operation on the key-value cache of the current processing stage through the second processing node includes: The hidden state of the current processing stage is obtained by performing a decoding inference operation on the key-value cache of the current processing stage through the second processing node. The hidden state of the current processing stage is transformed to obtain the word sequence of the current processing stage.

7. The data processing method according to claim 1, characterized in that, Before performing the decoding stage inference operation on the key-value cache of the current processing stage through the second processing node to obtain the lexical sequence of the current processing stage, the method further includes: Determine whether the key-value cache in the current processing stage has completed the storage action at the target storage location; When the key-value cache in the current processing stage completes the storage operation at the target storage location, the storage completion information is sent from the first processing node to the second processing node.

8. The data processing method according to claim 4, characterized in that, Both the first memory block and the second memory block use a preset length of video memory block as the smallest management unit. The key-value cache of the current processing stage of the data processing request corresponding to the same model layer is continuously stored in the same video memory block or multiple adjacent video memory blocks.

9. The data processing method according to claim 7, characterized in that, The storage completion information is a preset lightweight single-package notification information, which includes a triple consisting of request batch identifier information, model layer number information, and storage address pointer of the target storage location.

10. An electronic device, characterized in that, include: It includes a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the data processing method as described in any one of claims 1-9.

Images