Large model inference method and system based on multi-level cache mechanism, electronic device and storage medium

Abstract

The application provides a large model inference method and system based on a multi-level cache mechanism, an electronic device and a storage medium, and relates to the technical field of computers. The large model inference method based on the multi-level cache mechanism comprises: receiving an original request; determining a pre-populated instance and a decoding instance corresponding to the original request based on the key-value cache distribution and load condition of each instance; determining a plurality of key-value cache blocks of the original request based on a three-level cache mechanism through the pre-populated instance; performing calculation in a pre-population stage based on the key-value cache blocks to obtain a first output token in the pre-population stage; performing decoding stage calculation based on the first output token in the pre-population stage and the key-value cache blocks through the decoding instance to obtain a complete output token corresponding to the original request; and determining an inference result of the original request based on the complete output token. Thus, the hit rate of the prefix cache can be greatly improved, and the problem of uneven load can be effectively alleviated.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to large model inference methods, systems, electronic devices and storage media based on multi-level caching mechanisms. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs), also known as large models, have demonstrated excellent performance in key areas such as natural language processing, image and video generation, leading to a surge in the demand for their inference capabilities. Against this backdrop, efficiently handling increasingly complex large model inference tasks has become a significant challenge for the field of artificial intelligence.

[0003] Current large-scale model inference consists of two stages: prefill and decoding. The prefill stage extracts features from the input sequence in parallel, which is a computationally intensive task. The decoding stage generates the output sequence based on the input features using an autoregressive iterative approach, which is a memory-intensive task. Due to the different resource requirements, processing prefill and decoding in the same inference batch will prevent the first-word latency and throughput from being optimized simultaneously, making it difficult to support high concurrency while meeting Service Level Objectives (SLOs).

[0004] To address the aforementioned challenges, existing technologies can separate the pre-filling and decoding stages and execute them on different AI accelerator cards. This allows for on-demand configuration of parallel methods and service request concurrency for both stages, improving the utilization of computing resources. However, current separation technologies still have significant shortcomings in terms of prefix cache efficiency and load balancing, resulting in high network bandwidth requirements and poor scalability. This is particularly evident in large-scale model inference scenarios with multiple users and high concurrency requests, where low resource utilization persists. Prefix caching is a mechanism for storing intermediate results generated during the pre-filling stage. These intermediate results can be directly used by the subsequent decoding stage, reducing redundant computation. The low efficiency of prefix caching means that intermediate results in the prefix cache are rarely hit during the decoding stage. Summary of the Invention

[0005] This application provides a method, system, electronic device, and storage medium for large model inference based on a multi-level caching mechanism, in order to at least solve the above-mentioned technical problems existing in the prior art.

[0006] According to a first aspect of this application, a large model inference method based on a multi-level caching mechanism is provided, comprising: receiving an original request; determining a pre-filling instance and a decoding instance corresponding to the original request based on the key-value cache distribution and load of each instance; determining multiple key-value cache blocks for the original request based on the pre-filling instance and a three-level caching mechanism; the three-level caching mechanism being a caching mechanism with GPU memory, system memory, and disk as the caching order; performing a pre-filling stage calculation based on the key-value cache blocks to obtain the first output tag of the pre-filling stage; performing a decoding stage calculation based on the first output tag of the pre-filling stage and the key-value cache blocks through the decoding instance to obtain the complete output tag corresponding to the original request; and determining the inference result of the original request based on the complete output tag.

[0007] In one possible implementation, determining the pre-filled instance and decoding instance corresponding to the original request based on the key-value cache distribution and load of each instance includes: splitting the input sequence of the original request into multiple sub-sequence blocks; generating a hash value for each sub-sequence block as an encoding identifier for the sub-sequence block; the hash value of the sub-sequence block is mapped from a sequence formed by concatenating the marker in the sub-sequence block and the encoding identifier of the previous sub-sequence block; comparing the encoding identifiers of the multiple sub-sequence blocks in the original request with the cache block encoding identifiers in each candidate pre-filled instance one by one to obtain the maximum length of prefix matching; determining the first-word latency of the original request in each candidate pre-filled instance based on the length of the original request and the maximum length of prefix matching; determining the candidate pre-filled instance corresponding to the shortest first-word latency as the pre-filled instance corresponding to the original request, and determining a paired decoding instance for the pre-filled instance.

[0008] In one possible implementation, determining the first-word delay of the original request on each of the candidate pre-filled instances based on the length of the original request and the maximum length of the prefix match includes: determining the time for the original request to perform pre-filling calculations on each of the candidate pre-filled instances based on the length of the original request and the maximum length of the prefix match; determining the queuing time as the sum of the pre-filling calculation times of all requests in each of the candidate pre-filled instances; and determining the first-word delay based on the pre-filling calculation time and the queuing time.

[0009] In one possible implementation, determining the first-word delay based on the pre-filling calculation time and the queuing time includes: determining the first-word delay as the sum of the pre-filling calculation time, the queuing time, and the key-value cache block transfer time.

[0010] In one possible implementation, determining multiple key-value cache blocks for the original request based on a three-level caching mechanism through the pre-filling instance includes: searching for key-value cache blocks corresponding to the sub-sequence blocks in the graphics card memory, system memory, and disk sequentially based on the encoding identifier of the sub-sequence blocks through the pre-filling instance; in response to the hit rate of the key-value cache blocks corresponding to the sub-sequence blocks being less than a first threshold, obtaining key-value buffer blocks of the missed sub-sequence blocks from other pre-filling instances, and / or recalculating the key-value buffer blocks of the missed sub-sequence blocks.

[0011] In one possible implementation, obtaining the key-value buffer block of the missed subsequence block from other pre-filled instances, and / or recalculating the key-value buffer block of the missed subsequence block, includes: asynchronously sending a cache request signal to other pre-filled instances holding the key-value buffer block of the missed subsequence block via a message service process in the pre-filled instance; receiving the key-value buffer block sent by other pre-filled instances in response to the cache request signal; and / or pipelined parallelizing the key-value buffer block of the missed subsequence block on multiple computing nodes in the pre-filled instance.

[0012] In one possible implementation, the step of performing the pre-filling phase calculation based on the key-value cache block to obtain the first output marker of the pre-filling phase includes: before the current layer calculation, asynchronously loading the target key-value cache block required for the current layer calculation into the graphics card memory; while performing the current layer calculation based on the target key-value cache block, synchronously loading the key-value cache block required for the next layer calculation; and in response to the end of the current layer calculation, asynchronously writing the target key-value cache block back to the system memory and / or disk.

[0013] In one possible implementation, the step of performing decoding stage calculations based on the first output tag of the pre-filling stage and the key-value cache block through the decoding instance to obtain the complete output tag corresponding to the original request includes: adding the original request to the next processing batch of continuous batch processing through the decoding instance; loading the key-value cache block from local system memory to graphics card memory, and performing decoding stage calculations to obtain the complete output tag.

[0014] In one possible implementation, determining the inference result of the original request based on the complete output tag includes: mapping the complete output tag to the format of the original request to obtain the inference result of the original request.

[0015] According to a second aspect of this application, a large-model inference system based on a multi-level caching mechanism is provided, comprising: a request scheduler, a pre-filling cluster, and a decoding cluster; the pre-filling cluster includes multiple pre-filling instances, and the decoding cluster includes multiple decoding instances; the request scheduler is used to receive an original request; based on the key-value cache distribution and load of each instance, it determines the pre-filling instance and the decoding instance corresponding to the original request; the pre-filling instance is used to determine multiple key-value cache blocks for the original request based on a three-level caching mechanism; the three-level caching mechanism is a caching mechanism with the graphics card memory, system memory, and disk as the caching order; based on the key-value cache blocks, it performs calculations in the pre-filling stage to obtain the first output tag of the pre-filling stage; the decoding instance is used to perform calculations in the decoding stage based on the first output tag of the pre-filling stage and the key-value cache blocks to obtain the complete output tag corresponding to the original request; based on the complete output tag, it determines the inference result of the original request.

[0016] In one embodiment, the request scheduler is further configured to: split the input sequence of the original request into multiple sub-sequence blocks; generate a hash value for each sub-sequence block as an encoding identifier for the sub-sequence block; the hash value of the sub-sequence block is mapped from a sequence formed by concatenating the marker in the sub-sequence block and the encoding identifier of the previous sub-sequence block; compare the encoding identifiers of the multiple sub-sequence blocks in the original request with the cache block encoding identifiers in each candidate pre-fill instance one by one to obtain the maximum length of prefix matching; determine the first-word latency of the original request on each candidate pre-fill instance based on the length of the original request and the maximum length of prefix matching; determine the candidate pre-fill instance corresponding to the shortest first-word latency as the pre-fill instance corresponding to the original request, and determine a paired decoding instance for the pre-fill instance.

[0017] In one possible implementation, the request scheduler is further configured to: determine the time for the original request to perform pre-filling calculations on each of the candidate pre-filling instances based on the length of the original request and the maximum length of the prefix matching; determine the queuing time as the sum of the pre-filling calculation times of all requests in each of the candidate pre-filling instances; and determine the first-word delay based on the pre-filling calculation time and the queuing time.

[0018] In one possible implementation, the request scheduler is further configured to: determine the sum of the pre-filling calculation time, the queuing time, and the key-value cache block transfer time as the first word delay.

[0019] In one possible implementation, the pre-filling instance is further configured to: search for the key-value cache block corresponding to the sub-sequence block in the graphics card memory, system memory, and disk sequentially based on the encoding identifier of the sub-sequence block; in response to the hit rate of the key-value cache block corresponding to the sub-sequence block being less than a first threshold, obtain the key-value buffer block of the missed sub-sequence block from other pre-filling instances, and / or recalculate the key-value buffer block of the missed sub-sequence block.

[0020] In one possible implementation, the pre-fill instance is further configured to: send a cache request signal asynchronously to other pre-fill instances holding key-value buffer blocks of missed subsequence blocks via a message service process in the pre-fill instance; receive key-value buffer blocks sent by other pre-fill instances in response to the cache request signal; and / or compute the key-value buffer blocks of the missed subsequence blocks in a pipelined parallel manner on multiple compute nodes in the pre-fill instance.

[0021] In one possible implementation, the pre-filling instance is further configured to: asynchronously load the target key-value cache block required for the current layer computation into the graphics card memory before the current layer computation; synchronously load the key-value cache block required for the next layer computation while performing the current layer computation based on the target key-value cache block; and asynchronously write the target key-value cache block back to the system memory and / or disk in response to the end of the current layer computation.

[0022] In one possible implementation, the decoding instance is further configured to: add the original request to the next processing batch of the continuous batch processing; load the key-value cache block from the local system memory to the graphics card memory, and perform the calculations of the decoding stage to obtain the complete output tag.

[0023] In one possible implementation, the decoding instance is further configured to: map the complete output token to the format of the original request to obtain the inference result of the original request.

[0024] According to a third aspect of this application, an electronic device is provided, comprising:

[0025] At least one processor; and,

[0026] A memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in this application.

[0028] According to a fourth aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this application.

[0029] This application discloses a large-model inference method, system, electronic device, and storage medium based on a multi-level caching mechanism. The method involves receiving an original request; determining a pre-filling instance and a decoding instance corresponding to the original request based on the key-value cache distribution and load of each instance; determining multiple key-value cache blocks for the original request using the pre-filling instance and a three-level caching mechanism (GPU memory, system memory, and disk as the caching order); performing calculations in the pre-filling stage based on the key-value cache blocks to obtain the first output tag of the pre-filling stage; performing calculations in the decoding stage based on the first output tag of the pre-filling stage and the key-value cache blocks using the decoding instance to obtain the complete output tag corresponding to the original request; and determining the inference result of the original request based on the complete output tag. Therefore, by fully utilizing GPU memory, system memory, and disk storage resources through a multi-level caching mechanism, the hit rate of the prefix cache is significantly improved through hierarchical management and dynamic scheduling of the cache, reducing redundant calculations and unnecessary data transfer. At the same time, the introduced heuristic automatic hotspot shifting framework can monitor and dynamically adjust the cache distribution in real time, effectively alleviating the problem of load imbalance, reducing dependence on network bandwidth, and improving the scalability of the system. In multi-user, high-concurrency large model inference scenarios, these improvements have significantly improved resource utilization, significantly reduced inference latency, and significantly increased throughput, thus providing more competitive performance for large models in complex business scenarios.

[0030] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0031] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, wherein:

[0032] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0033] Figure 1 This paper illustrates a flowchart of a large model inference method based on a multi-level caching mechanism according to an embodiment of this application. Figure 1 ;

[0034] Figure 2 This paper illustrates a flowchart of a large model inference method based on a multi-level caching mechanism according to an embodiment of this application. Figure 2 ;

[0035] Figure 3This paper shows a schematic diagram of the structure of a large model inference system based on a multi-level caching mechanism according to an embodiment of this application;

[0036] Figure 4 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0037] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] Figure 1 This paper illustrates a flowchart of a large model inference method based on a multi-level caching mechanism according to an embodiment of this application. Figure 1 ,like Figure 1 As shown, a large model inference method based on a multi-level caching mechanism includes:

[0039] Step S101: Receive the original request.

[0040] In this embodiment, receiving the initial request is the starting point of the entire large model inference process. The initial request (Prompt) sent by the client is typically in the form of natural language text, images, or other data. These initial requests are first sent to the inference system, which needs to convert them into a format that the large model can process. Specifically, the inference system transforms the initial request into a sequence of tokens, which are the basic units that the large model can understand and process. For example, for text input, the system uses a tokenizer to segment the text into a series of tokens, each token corresponding to a specific word or sub-word unit.

[0041] Step S102: Based on the key-value cache distribution and load of each instance, determine the pre-filled instance and decoding instance corresponding to the original request.

[0042] In this embodiment, the distribution of key-value caches and the current load of each instance are considered when determining the pre-filled instance and the decoding instance. The key-value cache distribution refers to the cached key-value pairs in each instance; these pairs are commonly used intermediate results during large model inference. The load reflects the number of requests currently being processed and the utilization of computing resources in each instance. By analyzing this information, requests can be allocated rationally to ensure the efficiency of the inference process and the full utilization of resources. For example, if a pre-filled instance has a low load and a large number of key-value caches related to the current request, the request may be allocated to this instance to reduce computation time and resource consumption.

[0043] Step S103: By pre-populating instances, multiple key-value cache blocks for the original request are determined based on a three-level caching mechanism.

[0044] In this embodiment, during the pre-filling phase, multiple key-value cache blocks related to the original request are determined according to a three-level caching mechanism. The three-level caching mechanism searches for and uses the cache in the order of GPU memory, system memory, and disk. GPU memory is the fastest storage medium, used to store currently processed data and intermediate results; system memory is slightly slower but has a larger capacity, allowing for the storage of more data; and disk is the slowest storage medium, typically used to store infrequently used data. The required key-value cache block can be searched first in GPU memory; if not found, it will be searched in system memory and then disk in turn. This caching mechanism can fully utilize the characteristics of different storage media, improving data access efficiency.

[0045] Step S104: Based on the key-value cache block, perform the calculation of the pre-filling stage to obtain the first output tag of the pre-filling stage.

[0046] In this embodiment, the computation in the pre-filling stage is based on pre-defined key-value cache blocks. These key-value cache blocks can be used to perform a series of computational operations, typically including feature extraction and vectorization. During the computation, feature vectors related to the input request can be generated, which will serve as input for the subsequent decoding stage. The result of the pre-filling stage is the first output token, which serves as the starting point for the decoding stage. For example, in a natural language processing task, the first output token might be a specific word or sub-word unit, which will guide the decoding stage in generating subsequent text content.

[0047] Step S105: Using the decoding instance, the decoding stage calculation is performed based on the first output tag and key-value cache block of the pre-filling stage to obtain the complete output tag corresponding to the original request.

[0048] In this embodiment, the computation in the decoding phase is based on the first output token and key-value cache block from the pre-filling phase. The decoding instance uses this information to generate the complete output token corresponding to the original request. The decoding process is typically autoregressive, meaning that the output of each step serves as the input for the next. The system progressively generates output tokens based on the feature vectors and key-value cache blocks generated in the pre-filling phase until a complete output sequence is generated. For example, in a text generation task, the decoding phase starts from the first output token and progressively generates subsequent words until a complete sentence or paragraph is generated.

[0049] Step S106: Determine the reasoning result of the original request based on the complete output tags.

[0050] In this embodiment, the inference result of the original request is determined based on the complete output tag generated during the decoding stage. This result is mapped back to the format of the original request so that the client can understand and use it. For example, if the original request is a text generation task, the inference result will be a complete text paragraph; if the original request is an image recognition task, the inference result will be a label or text description describing the image content.

[0051] In this application, by separating the pre-filling and decoding stages and utilizing a three-level caching mechanism, the characteristics of different storage media can be fully utilized to improve data access efficiency. Simultaneously, by rationally allocating pre-filling and decoding instances, the load can be effectively balanced, improving the efficiency of the inference process. This method performs exceptionally well in multi-user, high-concurrency scenarios, significantly improving resource utilization and inference speed, providing strong support for large-scale model inference tasks.

[0052] Figure 2 This paper illustrates a flowchart of a large model inference method based on a multi-level caching mechanism according to an embodiment of this application. Figure 2 ,like Figure 2 As shown, step S102, "Based on the key-value cache distribution and load of each instance, determine the pre-filled instance and decoding instance corresponding to the original request," includes:

[0053] Step S201: Split the input sequence of the original request to obtain multiple sub-sequence blocks.

[0054] In this embodiment, the process of determining the pre-filled instance and the decoding instance first requires splitting the input sequence of the original request. The input sequence is typically a long sequence of tokens, which the system splits into multiple shorter sub-sequence blocks. The purpose of this splitting is to better utilize the caching mechanism and improve the cache hit rate. For example, assuming the input sequence of the original request is a text containing 1024 tokens, the system can split it into multiple sub-sequence blocks of length 256.

[0055] Step S202: Generate a hash value for each subsequence block as the encoding identifier of the subsequence block.

[0056] In this embodiment, to uniquely identify each sub-sequence block, a hash value can be generated as its encoded identifier for each sub-sequence block. This hash value is obtained by concatenating the tags in the sub-sequence block with the encoded identifier of the previous sub-sequence block and then performing a hash mapping. This encoding method ensures that the encoded identifier of each sub-sequence block is unique and can reflect the content and order of the sub-sequence blocks. For example, assuming that sub-sequence block A contains tags 1 to 256 and sub-sequence block B contains tags 257 to 512, then the hash value of sub-sequence block B will be generated based on tags 257 to 512 and the hash value of sub-sequence block A.

[0057] Step S203: Compare the encoding identifiers of multiple sub-sequence blocks in the original request with the encoding identifiers of cache blocks in each candidate pre-filled instance to obtain the maximum length of prefix matching.

[0058] In this embodiment, when determining the pre-filled instance, the system compares the encoding identifiers of multiple sub-sequence blocks in the original request with the cache block encoding identifiers in each candidate pre-filled instance one by one to find the longest prefix matching length. If sub-sequence blocks A, B, C, and D exist in the original request in sequence, and the encoding identifiers of sub-sequence blocks A, B, and C all exist in the cache block encodings of the candidate pre-filled instances, then the maximum prefix matching length is 3.

[0059] Step S204: Based on the length of the original request and the maximum length of the prefix match, determine the first word latency of the original request on each candidate pre-filled instance.

[0060] In this embodiment, the first-word latency refers to the time required from receiving the original request to generating the first output tag. The first-word latency can be estimated based on the length of the original request and the maximum length of the prefix match. Specifically, a pre-filling time prediction model can be used. This model, trained on offline test data, can predict the pre-filling computation time based on the request length and the cache hit length (i.e., the maximum length of the prefix match). For example, assuming the original request length is 1024 tags and the maximum length of the prefix match is 512 tags, the pre-filling computation time can be predicted based on this information, and the queuing time can be added to obtain the first-word latency.

[0061] Step S205: Determine the candidate pre-fill instance corresponding to the shortest first character delay as the pre-fill instance corresponding to the original request, and determine the matching decoding instance for the pre-fill instance.

[0062] In this embodiment, the candidate pre-filled instance with the shortest first-word latency is selected as the pre-filled instance corresponding to the original request. This selection process ensures the efficiency of the inference process, as the shortest first-word latency means the fastest response time. Simultaneously, a paired decoding instance needs to be determined for the selected pre-filled instance. The selection of the decoding instance takes into account the load of the decoding cluster to ensure the efficiency of the entire inference process and the rational utilization of resources.

[0063] In this application, by splitting the input sequence into multiple sub-sequence blocks and generating a unique encoded identifier for each sub-sequence block, matching items in the cache can be quickly identified, thereby improving the cache hit rate. Simultaneously, by predicting the first-word latency based on the prefix matching length and request length, the optimal pre-filled instance can be selected, ensuring the efficiency of the inference process.

[0064] In another embodiment, step S204, "determining the first-word delay of the original request on each candidate pre-padded instance based on the length of the original request and the maximum length of the prefix match," includes:

[0065] Based on the length of the original request and the maximum length of the prefix match, determine the time for the original request to perform pre-padding calculations on each candidate pre-padding instance; sum the pre-padding calculation times of all requests in each candidate pre-padding instance to determine the queuing time; and determine the first-word delay based on the pre-padding calculation time and the queuing time.

[0066] In this embodiment, the process of determining the first-word latency first requires calculating the time required for the original request to perform pre-padding computation on each candidate pre-padding instance. This time is estimated based on the length of the original request (i.e., the number of tags in the input sequence) and the maximum length of the prefix match. Specifically, a pre-padding time prediction model can be used, which is trained based on offline test data and can predict the pre-padding computation time based on the request length and cache hit length. For example, assuming the length of the original request is 1024 tags and the maximum length of the prefix match is 512 tags, the system will predict the pre-padding computation time based on this information.

[0067] In this embodiment, queuing time refers to the sum of the pre-filling calculation times of all requests waiting to be processed in the candidate pre-filling instance. This time reflects how long it would take before processing can begin if the current request is assigned to that instance. Specifically, the queuing time can be obtained by calculating the pre-filling calculation times of all queued requests in each candidate pre-filling instance and adding these times together. For example, if there are 3 requests in a candidate pre-filling instance, and their pre-filling calculation times are 10ms, 20ms, and 30ms respectively, then the queuing time is 60ms.

[0068] In this embodiment, the first character delay is obtained by adding the pre-filling calculation time to the queuing time. For example, assuming the pre-filling calculation time is 50ms and the queuing time is 60ms, then the first character delay is 110ms.

[0069] In this application, by combining pre-filling computation time and queuing time, the processing capacity and current load of each instance can be comprehensively evaluated. This approach ensures that when selecting pre-filled instances, not only computational efficiency but also actual waiting time are considered, thereby achieving efficient resource allocation and rapid response.

[0070] In another embodiment, determining the first-word delay based on the pre-filling calculation time and queuing time includes:

[0071] The sum of the pre-filling calculation time, queuing time, and key-value cache block transfer time is determined as the first word latency.

[0072] In this embodiment, when determining the first-word latency, in addition to considering the pre-filling calculation time and queuing time, the key-value cache block transfer time also needs to be considered. The key-value cache block transfer time refers to the time required to obtain the required key-value cache block from other pre-filled instances. This time includes network transmission latency, cache block loading time, etc. Specifically, the system adds the pre-filling calculation time, queuing time, and key-value cache block transfer time to obtain the final first-word latency.

[0073] In one possible implementation, the key-value cache block transfer time can be calculated as follows:

[0074] Calculating network transmission latency: The network latency between different nodes is measured in advance, typically by sending small data packets and measuring the round-trip time (RTT). Assuming the RTT between two nodes is 20ms, then the one-way transmission latency is 10ms.

[0075] Calculating cache block load time: The system estimates the load time based on the cache block size and network bandwidth. For example, assuming a cache block size of 1MB and a network bandwidth of 100MB / s, the load time would be... .

[0076] Calculate the total transfer time: The key-value cache block transfer time is the sum of network transmission latency and cache block loading time. For example: Key-value cache block transfer time = 10ms + 10ms = 20ms.

[0077] In this application, by comprehensively considering pre-fill computation time, queuing time, and key-value cache block transfer time, the first-word latency can be calculated more accurately. This method considers not only computation and queuing time but also data transmission latency, thus more comprehensively reflecting the actual response time of the system. In multi-user, high-concurrency scenarios, this method can more accurately select the optimal pre-fill instance, ensuring efficient system operation and fast response.

[0078] In another embodiment, step S103, "determining multiple key-value cache blocks for the original request based on a three-level caching mechanism using pre-populated instances," includes:

[0079] By using pre-filled instances, based on the encoding identifier of the subsequence block, the key-value cache block corresponding to the subsequence block is searched sequentially in the graphics card memory, system memory, and disk;

[0080] If the hit rate of the key-value cache block corresponding to the subsequence block is less than a first threshold, the key-value cache block of the missed subsequence block is obtained from other pre-filled instances, and / or the key-value cache block of the missed subsequence block is recalculated.

[0081] In this embodiment, in the pre-filling instance, the corresponding key-value cache block can be searched sequentially in the three-level cache according to the encoding identifier of the sub-sequence block. The specific steps are as follows:

[0082] Graphics card memory search: First, search the graphics card memory for the key-value cache block corresponding to the subsequence block. Graphics card memory is the fastest storage medium and is typically used to store currently processed data and intermediate results. If the required key-value cache block is found in the graphics card memory, it can be used directly, thereby reducing data loading time.

[0083] System memory search: If the required key-value cache block is not found in the graphics card memory, the search will continue in system memory. System memory is slightly slower, but has a larger capacity and can store more data. The corresponding key-value cache block can be found in system memory based on the encoding identifier of the subsequence block.

[0084] Disk search: If the required key-value cache block is not found in system memory, a final search will be conducted on the disk. The disk is the slowest storage medium and is typically used to store infrequently used data. The corresponding key-value cache block can be located on the disk based on the encoding identifier of the subsequence block.

[0085] In this embodiment, if the hit rate of the key-value cache block corresponding to the subsequence block is lower than a set first threshold, the following measures can be taken:

[0086] Retrieving key-value cache blocks from other pre-populated instances: A cache request signal can be asynchronously sent through the message service process within a pre-populated instance to other pre-populated instances that hold key-value cache blocks of missed subsequence blocks. Upon receiving the request, the other pre-populated instances will send the required key-value cache blocks back to the requesting pre-populated instance.

[0087] Recalculate the key-value cache blocks for missed subsequence blocks: If retrieving key-value cache blocks from other pre-populated instances takes too long or is not feasible, you can choose to recalculate the key-value cache blocks for missed subsequence blocks locally. The recalculation process generates the required key-value cache blocks based on the content of the subsequence blocks and the model's computational logic.

[0088] In this application, by sequentially searching for key-value cache blocks in graphics card memory, system memory, and disk, the characteristics of different storage media can be fully utilized to improve data access efficiency. Simultaneously, when the cache hit rate falls below a threshold, cache blocks can be flexibly retrieved from other instances or recalculated, ensuring efficient inference and rational resource utilization.

[0089] In another embodiment, obtaining the key-value buffer block of the missed subsequence block from other pre-filled instances, and / or recalculating the key-value buffer block of the missed subsequence block, includes:

[0090] The message service process in the pre-populated instance asynchronously sends cache request signals to other pre-populated instances holding key-value buffer blocks of missed subsequence blocks; receives key-value buffer blocks sent by other pre-populated instances in response to the cache request signals; and / or,

[0091] The key-value buffer blocks of the missed subsequence blocks are computed in a pipelined parallel manner on multiple compute nodes in the pre-filled instance.

[0092] In this embodiment, in a pre-filled instance, when the local cache fails to hit the key-value buffer block of a subsequence block, a cache request signal can be sent to other pre-filled instances through the message service process. This process is asynchronous, meaning that the pre-filled instance sending the request will not block and wait for a response, but will continue to execute other tasks. The specific steps are as follows:

[0093] Sending a cache request signal: The message service process in the pre-populated instance generates a cache request signal containing the encoded identifier of the missed subsequence block and information about the required key-value buffer block. The process then sends the request signal to other pre-populated instances that hold the key-value buffer block.

[0094] Receiving key-value buffer blocks: After receiving a cache request signal, other pre-populated instances will look up the corresponding key-value buffer block in their local cache and send the block back to the requesting pre-populated instance. The sending process is also asynchronous to ensure efficient system operation.

[0095] In this embodiment, if obtaining the key-value buffer block from other pre-filled instances is not feasible, or the retrieval time is too long, the system will choose to recalculate the key-value buffer block of the missed subsequence block locally. The specific steps are as follows:

[0096] Pipeline parallel computation: Multiple compute nodes in a pre-filled instance compute the key-value buffer blocks of missed subsequence blocks in a pipelined parallel manner. Pipeline parallelism refers to decomposing the computation task into multiple stages, each executed by a different compute node. For example, assuming the computation of a subsequence block is divided into three stages: feature extraction, feature transformation, and feature aggregation, these three stages can be assigned to three compute nodes, with each node completing its corresponding stage task.

[0097] Asynchronous execution: After completing its own task, each computing node passes the result to the next node while continuing to process the task for the next subsequence block. This asynchronous execution method can effectively improve computing efficiency and reduce waiting time.

[0098] In this application, sending cache request signals and receiving key-value buffer blocks asynchronously can effectively utilize network resources and reduce waiting time. Simultaneously, by recompiling the key-value buffer blocks of missed subsequence blocks in a pipelined parallel manner across multiple computing nodes, local computing resources can be fully utilized, improving computational efficiency.

[0099] In another embodiment, step S104, "based on the key-value cache block, performing calculations for the pre-filling stage to obtain the first output tag of the pre-filling stage," includes:

[0100] Before the current layer is computed, the target key-value cache block required for the current layer's computation is asynchronously loaded into the graphics card memory;

[0101] When performing calculations on the current layer based on the target key-value cache block, the key-value cache block required for the next layer calculation is loaded synchronously.

[0102] In response to the completion of the current layer computation, the target key-value cache block is asynchronously written back to system memory and / or disk.

[0103] In this embodiment, during the pre-filling phase, the system asynchronously loads the target key-value cache block required for the current layer's computation into the graphics card memory before the current layer's computation begins. The specific steps are as follows:

[0104] Asynchronous loading: The system initiates an asynchronous loading operation to load the target key-value cache block from system memory or disk into the graphics card memory. This process is asynchronous, meaning the computation task will not block while waiting for the cache block to finish loading. For example, assuming the key-value cache block needed by the current layer is stored in system memory, the system will initiate an asynchronous loading task to load these cache blocks into the graphics card memory.

[0105] Loading status monitoring: During the loading process, the system monitors the loading status to ensure that cache blocks are loaded before computation begins. If the loading time is too long, the system will adjust the loading strategy according to the actual situation, such as increasing bandwidth or optimizing the loading path.

[0106] In this embodiment, during the computation of the current layer, the key-value cache block required for the computation of the next layer can be loaded synchronously. The specific steps are as follows:

[0107] Synchronous Loading: After the current layer's computation begins, a synchronous loading task can be started to load the key-value cache blocks required for the next layer's computation into the graphics card's memory. This process is synchronous; that is, the computation task will wait for the cache blocks to be loaded before continuing execution. For example, assuming the key-value cache blocks required for the current layer's computation have been loaded, the loading task for the next layer's key-value cache blocks will be started immediately.

[0108] Overlapping computation and loading: By decoupling the computation process from the cache block loading process through an asynchronous communication mechanism, overlapping execution of computation and loading can be achieved. This effectively reduces waiting time and improves computational efficiency. For example, the current layer computation may take 10ms, while the loading of the next layer's key-value cache block may take 5ms. Through overlapping execution, the loading of the next layer's key-value cache block can be completed within the last 5ms of the current layer computation.

[0109] In this embodiment, after the current layer's computation is complete, the target key-value cache block is asynchronously written back to system memory and / or disk. The specific steps are as follows:

[0110] Asynchronous write-back: This initiates an asynchronous write-back operation, writing the target key-value cache block from GPU memory back to system memory or disk. This process is asynchronous, meaning the computation task will not block while waiting for the write-back operation to complete. For example, assuming the current layer's computation has finished, an asynchronous write-back task will be initiated to write the target key-value cache block back to system memory.

[0111] Write-back status monitoring: During the write-back process, the write-back status is monitored to ensure that cached blocks can be accessed by other tasks when needed. If the write-back time is too long, the write-back strategy will be adjusted according to the actual situation, such as optimizing the write-back path or increasing bandwidth.

[0112] In this application, by asynchronously loading the target key-value cache block before the current layer's computation begins, waiting time can be reduced and computation efficiency improved. Simultaneously, by synchronously loading the key-value cache block required by the next layer during the current layer's computation, overlapping execution of computation and loading can be achieved, further improving efficiency. Finally, by asynchronously writing back the target key-value cache block after the current layer's computation is completed, the use of graphics card memory can be optimized, reducing video memory usage.

[0113] In another embodiment, step S105, "by using a decoding instance, performing calculations in the decoding phase based on the first output tag and key-value cache block of the pre-filling phase to obtain the complete output tag corresponding to the original request," includes:

[0114] By decoding the instance, the original request is added to the next processing batch in the continuous batch processing;

[0115] The key-value cache block is loaded from the local system memory into the graphics card memory, and the calculations of the decoding stage are performed to obtain the complete output tag.

[0116] In this embodiment, during the decoding phase, the decoding instance adds the original request to the next processing batch in the continuous batch processing. The specific steps are as follows:

[0117] Request queuing: The decoding instance maintains a request queue to manage pending requests. When a new original request arrives, the decoding instance adds it to the end of the queue. For example, assuming there are currently 3 requests in the queue, a new request will be added as the 4th request.

[0118] Batch processing scheduling: The decoding instance periodically retrieves a batch of requests from the queue for processing based on the system configuration and current load. These requests are organized into a batch for batch decoding computation. For example, assuming the system is configured with a batch size of 4, the decoding instance will retrieve the first 4 requests from the queue to form a batch.

[0119] In this embodiment, during the decoding phase, the key-value cache block is loaded from local system memory into the graphics card memory, and the decoding calculation is performed. The specific steps are as follows:

[0120] Cache block loading: The decoding instance initiates a loading operation to load the key-value cache blocks from system memory to GPU memory. This process is synchronous; that is, decoding computations will wait for the cache blocks to be loaded before starting execution. For example, assuming the key-value cache blocks are stored in system memory, the decoding instance will initiate a loading task to load these cache blocks into GPU memory.

[0121] Decoding computation: After loading, the decoding instance uses the key-value cache block and the first output token generated in the pre-filling stage to perform the decoding computation. The decoding process is autoregressive, with the output of each step serving as the input for the next. For example, assuming the first output token is a specific word, the decoding instance uses this word and the key-value cache block to progressively generate subsequent words until a complete output sequence is generated.

[0122] In this application, by adding the original request to the next processing batch in a continuous batch process, the advantages of batch processing can be fully utilized to improve computational efficiency. Simultaneously, by loading the key-value cache block from system memory to graphics card memory and performing decoding calculations, complete output tags can be generated quickly.

[0123] In another embodiment, step S106, "determining the reasoning result of the original request based on the complete output tags," includes:

[0124] Mapping the complete output tokens to the format of the original request yields the inference result of the original request.

[0125] In this embodiment, after completing the decoding stage calculations, the system maps the generated complete output tokens back to the format of the original request to obtain the final inference result. The specific steps are as follows:

[0126] Tag Sequence Reassembly: The complete output tag sequence generated during the decoding phase needs to be reassembled into the format of the original request. For example, if the original request is a text generation task, the tag sequence generated during the decoding phase might be individual words or sub-word units. These tags can be reassembled into a complete text paragraph in the order they were generated.

[0127] Format conversion: The recombined tag sequence can be converted into the format of the original request. For example, if the original request is a text generation task, the tag sequence can be converted into natural language text; if the original request is an image recognition task, the tag sequence can be converted into labels or text descriptions describing the image content.

[0128] Result encapsulation: The transformed result will be encapsulated into a standard response format so that the client can understand and use it. For example, the result might be encapsulated as a JSON object containing information such as the generated text or tags.

[0129] This application enables the accurate mapping of the complete output tokens generated during the decoding phase back to the format of the original request, thereby obtaining the final inference result. This method ensures the accuracy and usability of the inference result, allowing the client to directly use the generated result. Through efficient token recombination and format conversion, inference results conforming to the original request format can be generated quickly, further improving the overall system performance and user experience.

[0130] Figure 3 This paper illustrates a schematic diagram of the structure of a large model inference system based on a multi-level caching mechanism according to an embodiment of this application. Figure 3 As shown, a large model inference system based on a multi-level caching mechanism includes:

[0131] The request scheduler 10, pre-fill cluster 11, and decoding cluster 12 are included; the pre-fill cluster 11 includes multiple pre-fill instances, such as pre-fill instance 110, pre-fill instance 111, pre-fill instance 112, etc., and the decoding cluster 12 includes multiple decoding instances, such as decoding instance 121, decoding instance 122, decoding instance 123, etc.

[0132] The request scheduler 10 is used to receive the original request; based on the key-value cache distribution and load of each instance, it determines the pre-filled instance and decoding instance corresponding to the original request.

[0133] The pre-fill instance is used to determine multiple key-value cache blocks for the original request based on a three-level caching mechanism. The three-level caching mechanism is a caching mechanism with the graphics card memory, system memory, and disk as the caching order. Based on the key-value cache blocks, the calculation of the pre-filling stage is performed to obtain the first output tag of the pre-filling stage.

[0134] The decoding instance is used to perform calculations in the decoding phase based on the first output tag and key-value cache block in the pre-filling phase, to obtain the complete output tag corresponding to the original request; based on the complete output tag, the inference result of the original request is determined.

[0135] In one embodiment, the request scheduler 10 is further configured to: split the input sequence of the original request into multiple sub-sequence blocks; generate a hash value for each sub-sequence block as an encoding identifier for the sub-sequence block; the hash value of the sub-sequence block is mapped from the sequence formed by concatenating the marker in the sub-sequence block and the encoding identifier of the previous sub-sequence block; compare the encoding identifiers of the multiple sub-sequence blocks in the original request with the cache block encoding identifiers in each candidate pre-fill instance one by one to obtain the maximum length of the prefix match; determine the first word latency of the original request on each candidate pre-fill instance based on the length of the original request and the maximum length of the prefix match; determine the candidate pre-fill instance corresponding to the shortest first word latency as the pre-fill instance corresponding to the original request, and determine a paired decoding instance for the pre-fill instance.

[0136] In one embodiment, the request scheduler 10 is further configured to: determine the time for the original request to perform pre-filling calculations on each candidate pre-filling instance based on the length of the original request and the maximum length of the prefix matching; determine the queuing time as the sum of the pre-filling calculation times of all requests in each candidate pre-filling instance; and determine the first-word delay based on the pre-filling calculation time and the queuing time.

[0137] In one embodiment, the request scheduler 10 is further configured to: determine the sum of the pre-filling calculation time, the queuing time, and the key-value cache block transfer time as the first word delay.

[0138] In one possible implementation, the pre-filling instance is further configured to: search for the key-value cache block corresponding to the sub-sequence block in the graphics card memory, system memory, and disk sequentially based on the encoding identifier of the sub-sequence block; in response to the hit rate of the key-value cache block corresponding to the sub-sequence block being less than a first threshold, obtain the key-value buffer block of the missed sub-sequence block from other pre-filling instances, and / or recalculate the key-value buffer block of the missed sub-sequence block.

[0139] In one implementation, the pre-fill instance is further configured to: asynchronously send a cache request signal to other pre-fill instances holding key-value buffer blocks of missed subsequence blocks via a message service process in the pre-fill instance; receive key-value buffer blocks sent by other pre-fill instances in response to the cache request signal; and / or compute the key-value buffer blocks of missed subsequence blocks in a pipelined parallel manner on multiple compute nodes in the pre-fill instance.

[0140] In one possible implementation, the pre-filling instance is further configured to: asynchronously load the target key-value cache block required for the current layer computation into the graphics card memory before the current layer computation; synchronously load the key-value cache block required for the next layer computation while performing the current layer computation based on the target key-value cache block; and asynchronously write the target key-value cache block back to the system memory and / or disk in response to the end of the current layer computation.

[0141] In one possible implementation, the decoding instance is also used to: add the original request to the next processing batch of the continuous batch processing; load the key-value cache block from the local system memory to the graphics card memory, and perform the calculations of the decoding stage to obtain the complete output tag.

[0142] In one possible implementation, the decoding instance is also used to: map the complete output tokens to the format of the original request to obtain the inference result of the original request.

[0143] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0144] Figure 4A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0145] like Figure 4 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0146] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0147] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the large model inference method based on a multi-level caching mechanism. For example, in some embodiments, the large model inference method based on a multi-level caching mechanism can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the large model inference method based on a multi-level caching mechanism described above can be performed. Alternatively, in other embodiments, computing unit 801 may be configured by any other suitable means (e.g., by means of firmware) to perform a large model inference method based on a multi-level caching mechanism.

[0148] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0152] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0153] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0154] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0155] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0156] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A large-model inference method based on a multi-level caching mechanism, characterized in that, The method includes: Receive the original request; Based on the key-value cache distribution and load of each instance, determine the pre-filled instance and decoding instance corresponding to the original request; The pre-filled instance is used to determine multiple key-value cache blocks for the original request based on a three-level caching mechanism; the three-level caching mechanism is a caching mechanism with the graphics card memory, system memory, and disk as the caching order. Based on the key-value cache block, the calculation of the pre-filling stage is performed to obtain the first output tag of the pre-filling stage; Using the decoding instance, the decoding stage is calculated based on the first output tag of the pre-filling stage and the key-value cache block to obtain the complete output tag corresponding to the original request; Based on the complete output tag, determine the reasoning result of the original request; The step of determining the pre-filled instance and decoding instance corresponding to the original request based on the key-value cache distribution and load of each instance includes: The input sequence of the original request is split into multiple sub-sequence blocks; A hash value is generated for each sub-sequence block as the encoding identifier of the sub-sequence block; the hash value of the sub-sequence block is formed by concatenating the tag in the sub-sequence block and the encoding identifier of the previous sub-sequence block. The encoding identifiers of multiple sub-sequence blocks in the original request are compared one by one with the encoding identifiers of cache blocks in each candidate pre-filled instance to obtain the maximum length of prefix matching; Based on the length of the original request and the maximum length of the prefix match, the first word latency of the original request on each of the candidate pre-filled instances is determined; The candidate pre-fill instance corresponding to the shortest first character delay is determined as the pre-fill instance corresponding to the original request, and a matching decoding instance is determined for the pre-fill instance; The step of determining multiple key-value cache blocks for the original request based on the three-level caching mechanism through the pre-filled instance includes: Using the pre-filled instance, based on the encoding identifier of the sub-sequence block, the key-value cache block corresponding to the sub-sequence block is searched sequentially in the graphics card memory, system memory, and disk; In response to the hit rate of the key-value cache block corresponding to the subsequence block being less than a first threshold, the key-value cache block of the missed subsequence block is obtained from other pre-filled instances, and / or the key-value cache block of the missed subsequence block is recalculated.

2. The method according to claim 1, characterized in that, The step of determining the first-word latency of the original request on each of the candidate pre-filled instances based on the length of the original request and the maximum length of the prefix match includes: Based on the length of the original request and the maximum length of the prefix match, the time for the original request to perform pre-filling calculations on each of the candidate pre-filling instances is determined; The queuing time is determined by summing the pre-filling calculation times of all requests in each of the candidate pre-filling instances. The first character delay is determined based on the pre-filling calculation time and the queuing time.

3. The method according to claim 2, characterized in that, Determining the first character delay based on the pre-filled calculation time and the queuing time includes: The sum of the pre-filling calculation time, the queuing time, and the key-value cache block transfer time is determined as the first word latency.

4. The method according to claim 1, characterized in that, The step of obtaining the key-value buffer block of the missed subsequence block from other pre-filled instances, and / or recalculating the key-value buffer block of the missed subsequence block, includes: The message service process in the pre-filled instance asynchronously sends a cache request signal to other pre-filled instances holding key-value buffer blocks of missed subsequence blocks; receives key-value buffer blocks sent by other pre-filled instances in response to the cache request signal; and / or, The key-value buffer block of the missed subsequence block is computed in a pipelined parallel manner on multiple computing nodes in the pre-filled instance.

5. The method according to claim 1, characterized in that, The calculation of the pre-filling phase based on the key-value cache block to obtain the first output tag of the pre-filling phase includes: Before the current layer is computed, the target key-value cache block required for the current layer's computation is asynchronously loaded into the graphics card memory; When performing calculations for the current layer based on the target key-value cache block, the key-value cache block required for the next layer calculation is loaded synchronously. In response to the completion of the current layer calculation, the target key-value cache block is asynchronously written back to system memory and / or disk.

6. The method according to claim 1, characterized in that, The step of calculating the complete output tag corresponding to the original request through the decoding instance, based on the first output tag of the pre-filling stage and the key-value cache block, includes: The original request is added to the next processing batch of the continuous batch processing through the decoding instance; The key-value cache block is loaded from the local system memory into the graphics card memory, and the calculations of the decoding stage are performed to obtain the complete output tag.

7. The method according to claim 1, characterized in that, The step of determining the reasoning result of the original request based on the complete output tag includes: The complete output token is mapped to the format of the original request to obtain the inference result of the original request.

8. A large-model inference system based on a multi-level caching mechanism, characterized in that, The system includes: The system includes a request scheduler, a pre-filled cluster, and a decoding cluster; the pre-filled cluster comprises multiple pre-filled instances, and the decoding cluster comprises multiple decoding instances. The request scheduler is used to receive the original request; and determine the pre-filled instance and decoding instance corresponding to the original request based on the key-value cache distribution and load of each instance. The pre-fill instance is used to determine multiple key-value cache blocks for the original request based on a three-level caching mechanism; the three-level caching mechanism is a caching mechanism with the graphics card memory, system memory, and disk as the caching order; based on the key-value cache blocks, the calculation of the pre-filling stage is performed to obtain the first output tag of the pre-filling stage; The decoding instance is used to perform calculations in the decoding stage based on the first output tag of the pre-filling stage and the key-value cache block to obtain the complete output tag corresponding to the original request; and to determine the inference result of the original request based on the complete output tag. The request scheduler is further configured to: The input sequence of the original request is split into multiple sub-sequence blocks; A hash value is generated for each sub-sequence block as the encoding identifier of the sub-sequence block; the hash value of the sub-sequence block is formed by concatenating the tag in the sub-sequence block and the encoding identifier of the previous sub-sequence block. The encoding identifiers of multiple sub-sequence blocks in the original request are compared one by one with the encoding identifiers of cache blocks in each candidate pre-filled instance to obtain the maximum length of prefix matching; Based on the length of the original request and the maximum length of the prefix match, the first word latency of the original request on each of the candidate pre-filled instances is determined; The candidate pre-fill instance corresponding to the shortest first character delay is determined as the pre-fill instance corresponding to the original request, and a matching decoding instance is determined for the pre-fill instance; The pre-filled instance is also used for: Based on the encoding identifier of the subsequence block, the key-value cache block corresponding to the subsequence block is searched sequentially in the graphics card memory, system memory, and disk; In response to the hit rate of the key-value cache block corresponding to the subsequence block being less than a first threshold, the key-value cache block of the missed subsequence block is obtained from other pre-filled instances, and / or the key-value cache block of the missed subsequence block is recalculated.

9. The system according to claim 8, characterized in that, The request scheduler is also used for: Based on the length of the original request and the maximum length of the prefix match, the time for the original request to perform pre-filling calculations on each of the candidate pre-filling instances is determined; The queuing time is determined by summing the pre-filling calculation times of all requests in each of the candidate pre-filling instances. The first character delay is determined based on the pre-filling calculation time and the queuing time.

10. The system according to claim 9, characterized in that, The request scheduler is also used for: The sum of the pre-filling calculation time, the queuing time, and the key-value cache block transfer time is determined as the first word latency.

11. The system according to claim 8, characterized in that, The pre-filled instance is also used for: The message service process in the pre-filled instance asynchronously sends a cache request signal to other pre-filled instances holding key-value buffer blocks of missed subsequence blocks; receives key-value buffer blocks sent by other pre-filled instances in response to the cache request signal; and / or, The key-value buffer block of the missed subsequence block is computed in a pipelined parallel manner on multiple computing nodes in the pre-filled instance.

12. The system according to claim 8, characterized in that, The pre-filled instance is also used for: Before the current layer is computed, the target key-value cache block required for the current layer's computation is asynchronously loaded into the graphics card memory; When performing calculations for the current layer based on the target key-value cache block, the key-value cache block required for the next layer calculation is loaded synchronously. In response to the completion of the current layer calculation, the target key-value cache block is asynchronously written back to system memory and / or disk.

13. The system according to claim 8, characterized in that, The decoding instance is also used for: Add the original request to the next processing batch in the continuous batch processing; The key-value cache block is loaded from the local system memory into the graphics card memory, and the calculations of the decoding stage are performed to obtain the complete output tag.

14. The system according to claim 8, characterized in that, The decoding instance is also used for: The complete output token is mapped to the format of the original request to obtain the inference result of the original request.

15. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images