Method, apparatus and device for large model inference acceleration

Abstract

The present specification provides a method, device and equipment for large model inference acceleration. The computing service device is configured with a plurality of computing nodes and stores a prefix index structure for indicating a mapping relationship between a token sequence prefix and a computing node storing a cached computing result thereof. The method comprises: performing a global scheduling mechanism, performing prefix matching based on a token sequence of an inference request querying the prefix index structure to determine one or more candidate computing nodes, and selecting a target computing node from the candidate computing nodes according to the real-time load state; performing a local scheduling mechanism, assigning an execution priority to the inference request for scheduling processing according to the degree of prefix matching of the inference request on the target computing node; loading the cached computing result corresponding to the matched prefix, and only calling the large model to perform inference calculation on the non-prefix part of the inference request.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method, apparatus and device for accelerating large model inference. Background Technology

[0002] In recent years, with the rapid increase in the parameter size of large language models, their inference processes in practical applications have posed severe challenges to both computational resources and response time. Existing large model inference systems, especially in distributed computing environments, typically distribute user requests to different computing nodes (such as graphics processing unit servers). To reduce redundant computation, some systems attempt to reuse previously completed computation results by matching the prefix of the user request, i.e., the initial part of the input text, such as through key-value caching.

[0003] However, existing technical solutions still have several shortcomings. First, there is a lack of effective coordination between the scheduling mechanism and the cache state. When selecting computing nodes, the system's scheduling module often relies primarily on the node's immediate load, while the prefix cache related to requests is scattered across various nodes. This separate decision-making mechanism may result in a request containing available cache being dispatched by the scheduling system to a node with low load but no available cache because its corresponding node has a high load. This misses the opportunity for computation reuse, leading to unnecessary duplicate computations and thus reducing the overall system efficiency. Second, the prefix matching granularity in existing technologies is relatively coarse. Most systems only support strict matching based on text or token sequences, and have poor ability to identify requests with similar semantics but different expressions. This results in a low success rate for prefix matching and low overall cache utilization. Finally, within the task processing flow of a single computing node, there is a general lack of priority management based on reuse value. When multiple requests are dispatched to the same computing node, the system usually processes them in a first-come, first-served order, without considering the cache reuse value that different requests can bring. A high-value request that can reuse a large amount of computation results may wait for a long time because it is queued after a low-value request that cannot be reused. This not only worsens the response latency of the high-value request, but also negatively impacts the overall throughput of the system. Summary of the Invention

[0004] In view of this, the purpose of this specification is to solve the technical problems existing in the prior art, such as the disconnect between large model inference scheduling and cache management, low efficiency of prefix matching, and lack of priority scheduling based on reuse value, and to provide an inference acceleration method and system that can coordinate prefix matching, load balancing and task priority sorting, so as to systematically reduce inference latency and improve resource utilization.

[0005] To achieve the above objectives, one or more embodiments of this specification provide a method for accelerating large model inference, applied to a computing service device performing large model inference tasks. The computing service device is configured with multiple computing nodes and is equipped with a hierarchical scheduling mechanism including a global scheduling mechanism and a local scheduling mechanism. The computing service device stores a prefix index structure to indicate the mapping relationship between token sequence prefixes and computing nodes storing their cached computation results. The method includes the following steps:

[0006] The sequence of tokens to receive inference requests;

[0007] The global scheduling mechanism is executed, and based on the token sequence of the inference request, the prefix index structure is queried to perform prefix matching to determine one or more candidate computing nodes, and the real-time load status of the one or more candidate computing nodes is obtained. Then, a target computing node is selected from the candidate computing nodes according to the real-time load status.

[0008] The local scheduling mechanism is executed to assign an execution priority to the inference request based on the prefix matching degree of the inference request on the target computing node, and to schedule the pending requests within the node according to the execution priority.

[0009] The target computing node is invoked so that, according to the scheduling instructions of the local scheduling layer, it loads the cached computing results corresponding to the matching prefix, and calls the large model to perform inference computing on the non-prefix part of the inference request.

[0010] Optionally, the step of querying the prefix index structure based on the token sequence of the inference request to perform prefix matching to determine one or more candidate computing nodes includes:

[0011] Traverse the prefix index structure and calculate the length of the matching token between the token sequence and any prefix stored in the prefix index structure;

[0012] When the proportion of the length of the matching token in the token sequence to the target prefix index is higher than a preset threshold, the computing node corresponding to the target prefix is ​​determined as a candidate computing node;

[0013] When the proportion of the length of the matched token to the token sequence is lower than a preset threshold, the semantic matching process is triggered;

[0014] The similarity between the semantic vector of the inference request and the semantic vector of any prefix stored in the prefix index structure is calculated. If the similarity between the semantic vector of the target prefix index and the semantic vector of the inference request is higher than a preset similarity threshold, the computing node indicated by the target prefix index is determined as a candidate computing node.

[0015] Optionally, the real-time load status includes at least one of the following: GPU utilization of the computing node, video memory occupancy, and request queue length;

[0016] The step of selecting a target computing node from the candidate computing nodes based on the real-time load status includes:

[0017] The suitability score of the candidate computing node is calculated based on the historical cache hit count and the real-time load status, and the node with the highest score is selected as the target computing node.

[0018] Optionally, selecting a target computing node from the candidate computing nodes based on the real-time load status further includes:

[0019] When the fit scores of all candidate computing nodes are lower than a preset low watermark threshold, automatic expansion is triggered, and a new computing node is started as the target computing node.

[0020] Optionally, the large model acceleration method further includes:

[0021] The system periodically acquires the load status of all current computing nodes. When the average load of a certain computing node remains below an idle threshold for a preset period of time, it triggers automatic scaling down, migrates the cached computing results of the certain computing node to other computing nodes, and releases the computing node.

[0022] Optionally, scheduling pending requests within the node according to the execution priority includes:

[0023] Within a preset scheduling task period, according to the preset scheduling weights for different priority queues, a number of user requests corresponding to the scheduling weights are sequentially retrieved from each priority queue for processing.

[0024] Optionally, based on the full-scale scheduling mechanism of the computing service device, waiting time thresholds are preset for different priority queues;

[0025] Assigning an execution priority to the inference request further includes:

[0026] Obtain the waiting time of the first pending request in any inference request queue; when the waiting time exceeds the waiting time threshold corresponding to any request queue, elevate any inference request to a higher priority queue.

[0027] Optionally, the prefix index structure is a Trie tree (prefix tree) or its space-optimized variant, a radix tree.

[0028] Optionally, the cache calculation result is a key vector and a value vector calculated in the self-attention mechanism.

[0029] According to a second aspect of one or more embodiments of this specification, a large model inference acceleration device is proposed, applied to a computing service device performing large model inference tasks. The computing service device is configured with multiple computing nodes and is equipped with a hierarchical scheduling mechanism including a global scheduling mechanism and a local scheduling mechanism. The computing service device stores a prefix index structure to indicate the mapping relationship between token sequence prefixes and computing nodes storing their cached computation results. The device includes:

[0030] The receiving unit is used to receive the token sequence for inference requests;

[0031] A global scheduling mechanism execution unit is used to execute the global scheduling mechanism, query the prefix index structure based on the token sequence of the inference request to perform prefix matching, so as to determine one or more candidate computing nodes, and obtain the real-time load status of the one or more candidate computing nodes, and then select a target computing node from the candidate computing nodes according to the real-time load status.

[0032] The local scheduling mechanism execution unit is used to execute the local scheduling mechanism, assign an execution priority to the inference request based on the prefix matching degree of the inference request on the target computing node, and schedule the pending requests in the node according to the execution priority;

[0033] The inference unit is used to invoke the target computing node so that it loads the cached computing results corresponding to the matching prefix according to the scheduling instructions of the local scheduling layer, and invokes the large model to perform inference computing on the non-prefix part of the inference request.

[0034] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in one or more of the above embodiments by executing the executable instructions.

[0035] As can be seen from the above embodiments, this specification, through one or more embodiments described above, uses a strategy combining prefix matching and load awareness at the global scheduling layer to ensure that requests are directed to the optimal computing nodes, optimizing the use of existing caches. Simultaneously, priority sorting at the local scheduling layer ensures that requests with high reusability are processed first. This combined effect significantly reduces redundant computation and invalid waiting, substantially lowering the generation time of the first token and end-to-end response latency. Secondly, one or more embodiments described above resolve the conflict between cache location and node load, avoiding resource mismatch. By aggregating similar requests to the same node and processing them first, the degree of key-value prefix matching and the actual utilization rate of computing resources are effectively improved, thereby increasing the overall throughput of the system. Furthermore, the layered architecture decouples global resource allocation and local task execution, making the system easier to maintain and expand. Global scheduling can dynamically perceive the state of the entire cluster, while local scheduling focuses on optimizing the efficiency of individual nodes. The two work together to improve the system's adaptability and stability under complex request loads. Attached Figure Description

[0036] Figure 1 This is a system architecture diagram of an exemplary embodiment of a method for accelerating large model inference.

[0037] Figure 2 This is a set of steps in an exemplary embodiment of a method for accelerating large model inference.

[0038] Figure 3 This is a block diagram of a large model inference acceleration device provided in an exemplary embodiment.

[0039] Figure 4 This is a schematic diagram of the structure of a device provided in an exemplary embodiment. Detailed Implementation

[0040] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0041] Figure 1 This is a schematic diagram illustrating the architecture of a large-model inference service device and its interaction with a user, provided as an exemplary embodiment. Figure 1As shown, to implement the large model inference acceleration method described in this embodiment, the large model inference system may include a computing service device 11, a network 12, and several user terminal electronic devices 13, such as PCs (Personal Computers) and mobile phones, to perform large model inference tasks.

[0042] PCs and mobile phones are just some of the types of electronic devices that users can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments in this specification do not limit this. During operation, the electronic device can run a client-side program of an application to achieve the relevant functions of that application. For example, when the electronic device runs a customer service Q&A program, it can act as a client for providing customer service Q&A services. The aforementioned customer service Q&A service client application can be launched and run on the electronic device. This client-side program can be a native application installed on the electronic device, or it can be a mini-program, quick app, or other similar form. Of course, when using web technologies such as HTML5 or similar, the relevant functions can be achieved through a page displayed by a browser. This browser can be a standalone browser application or a browser module embedded in some applications.

[0043] As for the network 12 that enables interaction between electronic devices such as PCs and mobile phones and computing service devices 11, the communication can be implemented using either wired or wireless networks based on the communication methods supported by the respective electronic devices. This specification does not impose any restrictions on this. For example, PC 13 can support both wired and wireless communication, so it can use either wired or wireless networks as needed. Mobile phones 14 typically only support wireless communication, so they can use wireless networks for communication.

[0044] The aforementioned user-end electronic device, as the initiator of the inference request, can carry any application that needs to call large model services, such as a customer service application. Its function is to encapsulate the user's input (such as a piece of text or a question) into an inference request and send it to the aforementioned computing service device 11 via network 12.

[0045] The computing service device 11 serves as a computing service device for performing large model inference tasks. It can be a physical server containing an independent host or a virtual server hosted by a host cluster. During operation, the computing service device 11 can deploy large models or connect to other service devices that deploy large models; this embodiment does not limit this.

[0046] The computing service device 11 is configured with a cluster of computing nodes to perform inference computing tasks in response to large model invocation requests sent by the user. The computing node cluster 110 is a physical or virtual resource pool that executes the actual inference tasks, consisting of multiple computing nodes (1100, 1101, 1102, etc.). In typical application scenarios, each computing node can be one or more high-performance graphics processing units (GPUs, such as NVIDIA A100 or H100) configured in the computing service device 11, or it can be a virtual machine or container instance connected to a local (or cloud-based) computing service master device and allocated on demand in a cloud environment. It is understood that, in one or more embodiments of this specification, the aforementioned computing service device 11 (or the computing service device executing large model inference tasks) is defined as a system device containing the aforementioned computing node cluster 110, and will not be described further hereafter.

[0047] The aforementioned computing service device 11 runs an inference management program to implement the large model inference acceleration method specifically described in this embodiment. In this embodiment, the inference management program of the computing service device 11 incorporates a hierarchical scheduling mechanism, including a global scheduling mechanism and a local scheduling mechanism, to schedule a computing node from the computing node cluster 110 to execute the user-requested inference task. The core function of the global scheduling mechanism is to select one computing node from multiple computing nodes to execute the inference task for a large model inference request sent by the user. The main responsibility of the local scheduling mechanism is to manage and schedule all inference requests assigned to any node based on the global scheduling mechanism. Unlike the global scheduling mechanism, which focuses on the macro-level decision of "which node to select," the local scheduling mechanism focuses on the micro-level scheduling problem of "in what order to execute."

[0048] The aforementioned global scheduling mechanism and local scheduling mechanism are deployed in the form of computer programs within the inference framework of computing service device 11. Alternatively, in another embodiment shown, when the aforementioned multiple computing nodes (1100, 1101, 1102, ...) are physically separated from the computing service device responsible for global scheduling in the form of a computing device cluster, the aforementioned global scheduling mechanism is deployed in the form of computer programs within the inference framework of the global computing service unit responsible for global scheduling, while the local scheduling mechanism is deployed within the inference framework of each computing node.

[0049] In this embodiment, the computing service device 11 is internally configured with a global metadata module 112. Specifically, in this embodiment, the global metadata module 112 is implemented as a prefix index structure to maintain a dynamically updated mapping table. This mapping table records the address of the specific computing node where the token sequence prefix of the processed request is located and its corresponding key-value cache.

[0050] In this embodiment, "token" refers to the smallest semantic / structural unit of data (such as words in text, character segments in code, or sampling points in time-series data); "token sequence" is an ordered set of tokens arranged according to a specific logic (such as chronological order or semantic order) (e.g., the token sequence corresponding to the text "machine learning" is ["machine", "learning"]). For the token sequence, "prefix" refers to a continuous subsequence starting from the beginning of the sequence, which must satisfy the characteristics of "fixed start and variable length" (e.g., the prefix of the sequence [t1,t2,t3,t4] includes [t1], [t1,t2], and [t1,t2,t3]), and the prefix has a hierarchical relationship with the original sequence. "Index" is a mapping structure built to accelerate data retrieval, which associates "search keywords" with "data storage location" through preset rules to avoid traversing the entire data. Therefore, the “token sequence prefix index” described in this embodiment is a retrieval mapping structure built based on the prefix subsequence of the token sequence: using “prefix” as the retrieval keyword, a preset algorithm (such as hash mapping, tree storage) is used to establish the association between the prefix and the storage address (e.g., the address of the computing node) of the corresponding complete token sequence; during retrieval, inputting the target prefix can quickly locate all token sequences containing that prefix without traversing all sequences.

[0051] In another illustrated implementation, the global metadata module 112 is implemented as an efficient prefix index structure, such as a Trie tree or its space-optimized variant, the Radix Tree.

[0052] In this embodiment, key-value caches 3000, 3001, 3002, etc., are constructed within the aforementioned multiple computing nodes. The key-value cache is a high-speed memory area used to store intermediate computation results generated during the inference process of a large model. It is typically allocated in the GPU's video memory (HBM) to achieve minimal access latency. For current mainstream large language models based on the Transformer architecture, these intermediate computation results are mainly key and value vectors calculated by the self-attention mechanism. When the model processes tokens in the input sequence one by one, each token generates a set of key-value pairs, and the computation of subsequent tokens depends on the key-value pairs of all previous tokens. Therefore, caching these key-value pairs allows subsequent requests with the same prefix to directly reuse this part of the computation results, avoiding repeated computation from scratch. This is one of the key technologies for accelerating large model inference.

[0053] based on Figure 1The system architecture shown is for a computing service device that performs large model inference tasks. Figure 2 This specification illustrates a specific implementation of a method for accelerating large model inference, which includes the following steps:

[0054] Step 202: The computing service device receives the token sequence of the inference request.

[0055] In yet another embodiment shown, when Figure 1 When the user client 10 initiates an inference request, for example, by inputting the text "My order shows as shipped, but I can't find the logistics information, what should I do?", the request reaches the service system of the large model via the network. First, the tokenizer, which strictly matches the large model being served (such as the GPT series, LLaMA series, etc.), converts the original text string into a sequence of integer tokens that the model can understand. For example, the above text might be converted into an integer sequence: [101, 2769, 7385, 6738, 2548, 1345, 123, ...]. This token sequence is the underlying data structure operated on by all subsequent scheduling and computation steps.

[0056] Step 204: Execute the global scheduling mechanism, query the prefix index structure based on the token sequence of the inference request to perform prefix matching, so as to determine one or more candidate computing nodes, and obtain the real-time load status of the one or more candidate computing nodes, and then select a target computing node from the candidate computing nodes according to the real-time load status.

[0057] When a new request arrives, the computing service device 11 executes a global scheduling mechanism, utilizing its internally maintained prefix index structure (a radix tree in this embodiment) to find opportunities for cache reuse. Each path in the radix tree from the root node to an internal node represents a token sequence prefix already cached on a certain computing node. This internal node stores identifiers (such as IP addresses or hostnames) of one or more computing nodes (3000, 3001, etc.) containing the key-value cache corresponding to this prefix. During the execution of the global scheduling mechanism, the computing service device 11 uses the token sequence of the new request, starting from the root node of the radix tree, and traverses downwards token by token to find an existing path that shares the longest common prefix with the request token sequence.

[0058] The matching process follows a pre-defined, configurable strategy. The system calculates the longest common prefix length between the new request and existing paths in the tree. A match is considered successful if either of the following conditions is met, and the compute nodes associated with that path are identified as candidate nodes: First, the number of tokens in the longest common prefix is ​​greater than or equal to an absolute threshold (e.g., 50 tokens). This condition aims to capture reuse opportunities in long text requests; even if the prefix accounts for a small percentage of the total length, the computational savings from its absolute length are still considerable. Second, the length of the longest common prefix accounts for a percentage of the total token length of the current request greater than or equal to a relative threshold (e.g., 70%). This condition is more suitable for handling interactive short text requests, ensuring a sufficiently high reuse rate and significant latency improvements. If a prefix cache exists on multiple nodes, all of these nodes are added to the candidate node list. If no matching item is found after traversing the entire index tree, it indicates a completely new request with no cache available. In this case, all compute nodes currently in the "available" state become candidate nodes, and scheduling decisions are purely based on load.

[0059] After identifying one or more candidate computing nodes through prefix matching, the computing server sends status query requests in parallel to all candidate nodes (whether selected due to a successful match or becoming available due to no match) to obtain their latest real-time load data. In this embodiment, the real-time load status is a multi-dimensional data structure, including at least: GPU computing unit utilization (percentage), GPU memory utilization (percentage), and the length of the pending request queue within the node. In more complex implementations, metrics such as network round-trip time (RTT) between nodes may also be included.

[0060] Accordingly, the computing server calculates a comprehensive "fitness score" for each candidate node based on the collected real-time information. This score aims to quantify how suitable a node is to receive new requests at the current moment. As an optional implementation, the formula for calculating this score can be designed as follows:

[0061] S=w1·P match +w2·(1-L gpu )+w3·(1-L mem )+w4·1Q len +1

[0062] Where S is the final fit score; P match The prefix matching degree can be the ratio of the token prefix matching length calculated in step 204 to the total length of the requested token sequence (this value is 0 for nodes with no matching); L gpu This is the normalized GPU utilization (between 0 and 1); Lmem It is the normalized video memory usage rate; Q len This is the request queue length. The +1 in the formula is to avoid a division-by-zero error when the queue length is 0. w1, w2, w3, and w4 are preset weight coefficients, which sum to 1 and can be dynamically or statically adjusted according to the focus of the business scenario. For example, in scenarios that are extremely sensitive to latency, the weight of w1, which represents the value of cache reuse, can be increased; while in scenarios where system stability needs to be prioritized, the weights of w2, w3, and w4, which are related to load, can be increased.

[0063] After calculating the fitness scores of all candidate nodes, the node with the highest score is selected as the final target computing node, and the request, along with the matched prefix information (if any), is dispatched to it. This intelligent decision-making mechanism, which combines "historical information" (prefix matching) and "current state" (real-time load), can effectively avoid sending requests to suboptimal nodes that are "cached but overloaded" or "completely idle but without any cache," thus optimizing global resource allocation.

[0064] Step 206: Execute the local scheduling mechanism, assign an execution priority to the inference request based on the prefix matching degree of the inference request on the target computing node, and schedule the pending requests within the node according to the execution priority.

[0065] As mentioned above, the local scheduling mechanism can be programmed within the scheduling framework invoked by the processor of the computing service device. In the case where the computing node and the computing service device are physically separated, the local scheduling mechanism can also be programmed within the scheduling framework invoked by the processor of the computing node. This embodiment does not limit this.

[0066] In one illustrated implementation, the target computing node maintains multiple priority queues internally; for example, three queues can be set up: P0 (high priority), P1 (medium priority), and P2 (low priority). Based on the local scheduling mechanism, the computing service device (or the target computing node connected to it) places newly arriving requests into the corresponding queue according to the calculated prefix matching degree (i.e., the number of reusable prefix tokens / the total number of requested tokens). These thresholds are configurable, for example:

[0067] Requests with a prefix matching rate of not less than 80%, or those that have been used more than 10 times in the past (i.e., the number of times the historical cache has been hit), are considered high-value requests and are placed in the P0 queue.

[0068] If the prefix matching degree is between 60% (inclusive) and 80%, the request is considered a medium-value request and is placed in the P1 queue.

[0069] If the prefix matching rate is less than 60% (including new requests with no cache hit), the request is considered a low-value request and placed in the P2 queue. In this way, requests that can bring the greatest computational savings and the fastest completion time to this node are given the highest execution priority.

[0070] When a target compute node retrieves a request from the priority queue and submits it for inference, it can employ various scheduling strategies. In one illustrated implementation, the compute node can use a direct priority scheduling mode, processing P0-level tasks first within a preset scheduling cycle. After clearing all P0-level tasks in the cycle, it then processes P1-level tasks, and finally, after clearing all P1-level tasks, it processes P3-level tasks. This leads to a consequence: if there are many P0-level tasks, the compute node may not have time to process P3-level tasks within one or even multiple scheduling cycles, resulting in tasks with low prefix matching scores not being processed in a timely manner and degrading the user experience.

[0071] Therefore, in one of the illustrated embodiments, the computing service device or computing node can set waiting time thresholds for different priority queues; the allocation of an execution priority for the inference request as described in one or more of the above embodiments further includes: obtaining the waiting time of the first pending request in any inference request queue; when the waiting time exceeds the waiting time threshold corresponding to any request queue, promoting the any inference request to a higher priority queue.

[0072] In another implementation shown, a weighted round-robin (WRR) strategy that balances efficiency and fairness is employed. For example, the scheduling weights assigned to queues P0, P1, and P2 can be set to a ratio of 3:2:1. This means that within a scheduling task cycle (e.g., processing 6 consecutive user requests), the target compute node will first attempt to retrieve 3 requests from queue P0, then 2 requests from queue P1, and finally 1 request from queue P2. If a priority queue is empty when it is its turn, its scheduling quota is automatically passed on to the next priority queue. This strategy effectively ensures that high-priority requests are processed first, reducing their waiting time, while preventing low-priority requests from being completely "starved" by the continuous influx of high-priority requests, thus achieving a good balance between maximum efficiency and system fairness.

[0073] However, extreme situations may arise where requests to high-priority queues surge. If, as described in the above implementation, a corresponding number of user requests are retrieved from each priority queue at a time within a scheduling cycle according to pre-set scheduling weights, the surge in requests to high-priority queues could lead to untimely processing. In another implementation proposed in this specification, not only are the scheduling weights of each priority queue considered, but the waiting time of each request is also taken into account during scheduling.

[0074] Step 208: Invoke the target computing node so that it loads the cached computing results corresponding to the matching prefix according to the scheduling instructions of the local scheduling layer, and calls the large model to perform inference computing on the non-prefix part of the inference request.

[0075] Specifically, when the target computing node executes a request, it first loads the key-value pair corresponding to the matching prefix from the key-value cache 3001, and then only performs inference computation on the remaining unmatched non-prefix parts of the request. For each new token computed, a new key-value pair is generated and "concatenated" to the already loaded key-value cache, forming a longer cache sequence. For requests with no matching prefix, the complete computation starts from the first token. This "computation concatenation" process significantly reduces the amount of redundant computation on existing prefix parts and is the core of this solution's inference acceleration. The newly generated key-value pairs are then appended to the key-value cache, finally generating the complete inference result. After inference, the result can be directly returned to client 10, or, if the computing node and computing service device are physically separated, the inference result can be returned to client 10 by the computing node via the computing service device.

[0076] Meanwhile, the aforementioned computing service device will trigger one or more asynchronous state updates. On the one hand, it receives the latest load information (such as GPU utilization, queue length, etc.) reported by the target computing node; on the other hand, and more importantly, if this inference generates a new, longer prefix cache, the relevant metadata (including the new prefix token sequence, its length, and the address of the local node) will be updated by the computing service device in the radix tree of the aforementioned global metadata module 112. In this way, subsequent requests may match this longer, more valuable prefix, forming a positive feedback loop, which makes the system's caching efficiency continuously improve over time.

[0077] In the above implementation, the prefix matching performed by the computing service device in step 204 relies entirely on precise, word-by-word matching of the token sequence. This method is very effective for templated API calls, text directly copied and pasted by users, or frequently asked fixed questions. However, for requests with similar semantics but different wording and word order, such as "How do I bind my bank card?" and "Please tell me the steps to add a bank card", or "What's the weather like in Beijing today?" and "Beijing weather forecast today", the precise matching mechanism cannot recognize their inherent semantic relationship and will treat them as completely unrelated requests, thus missing the opportunity for cache reuse.

[0078] Therefore, in another illustrated embodiment, the prefix index structure stored in the global metadata module 112 is extended. For example, in its radix tree structure, each node representing a valid prefix, in addition to storing the address of the associated computation node, also stores a semantic vector (Semantic Embedding) generated from the prefix token sequence. This semantic vector can be computed offline or near real-time once by a lightweight sentence encoding model (e.g., MiniLM or BGE-small based on the Sentence-BERT architecture) when the prefix cache is first generated and then stored. A lightweight model is chosen to ensure that the vector computation itself does not become a new bottleneck in the scheduling process.

[0079] When a new inference request arrives at the computing service device described in this embodiment, the processing flow of the global scheduling mechanism changes at step 204, forming a hybrid matching strategy: First, the exact token prefix matching described in the above embodiment is still performed, that is, the prefix index structure is traversed, and the matching token length between the token sequence and any prefix stored in the prefix index structure is calculated; when the proportion of the matching token length between the token sequence and the target prefix index to the token sequence is higher than a preset threshold, the computing node corresponding to the target prefix is ​​determined as a candidate computing node; in the global scheduling mechanism, the matching result is given priority because the reuse of exact matching is semantically lossless.

[0080] However, if an exact match fails, or if the found matching prefix is ​​very short (e.g., its length is below a preset semantic trigger threshold, such as 30%), the system will not immediately abandon the search for cached prefixes, but will trigger a semantic matching process. This process includes: using a built-in lightweight sentence encoding model to calculate the semantic vector of the user's reasoning request; subsequently, traversing some or all of the cached prefixes in the aforementioned prefix index structure stored in the global metadata module 112 (which can be pre-filtered, for example, only considering prefixes of similar length), and extracting their pre-stored semantic vectors; calculating the similarity between the semantic vector of the new request and the semantic vector of each cached prefix—for example, in one specific implementation, calculating cosine similarity. Cosine similarity is an indicator that measures the consistency of the directions of two vectors in a high-dimensional space, with a value range between -1 and 1. The closer the value is to 1, the more semantically similar the two texts are; and finding the cached prefix with the highest similarity score that exceeds a preset similarity threshold (e.g., 0.95). If such a prefix is ​​found, even if their token sequences are literally completely different, the system considers them semantically equivalent and can reuse the calculation result of the prefix (i.e., KV cache).

[0081] Once a reusable prefix is ​​found through semantic matching, its associated computing node is identified as a candidate node. Subsequent steps S206 (load awareness and target node selection) and the subsequent process are consistent with the above implementation method, and user requests are directed to the semantically matched computing node.

[0082] In this implementation, by introducing a hybrid prefix matching mechanism that prioritizes exact matching and supplements it with semantic matching, the system's generalization ability and intelligence level are significantly enhanced. For application scenarios such as intelligent customer service, online help, and search engines, users' question formats vary greatly. Semantic matching can penetrate the differences in surface text to uncover the underlying, shared query intent, thereby guiding these semantically equivalent requests to the same computing node for processing. This greatly improves the request reuse rate of the key-value cache and effectively reduces the system's average inference latency.

[0083] In yet another embodiment shown in this specification, in Figure 2 In step 204, after calculating the scores of all candidate nodes according to the global scheduling mechanism, the computing service device checks whether the score of the computing node with the highest score is also lower than a preset low-water mark threshold (for example, the threshold is set to 60 points if the maximum score is 100 points). If even the score of the best candidate node is low, it usually means that all nodes that can provide cache reuse are already under high load or overload, while other idle nodes do not have suitable caches. The entire computing cluster faces a processing capacity bottleneck, and the expected waiting latency for new requests will increase sharply.

[0084] Once this "resource shortage" situation is detected, the pre-set automatic scaling process of the global scheduling mechanism will be triggered. The computing service device will request the creation of one or more new computing node instances by calling the application programming interface (API) provided by the underlying cloud platform or directly calling the software development kit (SDK) of the cloud service provider. During the startup and initialization process, the new node will automatically complete model loading, register within the computing service device, and synchronize a copy of the latest prefix index information from the global metadata module 112 or obtain its access permissions. Once the new node is ready and joined to the computing node cluster 110, the computing service device can distribute the currently pending requests (and subsequent requests that do not find a suitable cache) to this brand-new, completely idle node, thereby quickly relieving cluster pressure.

[0085] In another embodiment illustrated in this specification, to save costs, the computing service device also needs to be able to automatically release redundant computing resources during periods of low business load. To this end, based on a global scheduling mechanism with a pre-configured automatic scaling-down procedure, the computing service device periodically (e.g., every 5 minutes) performs a cluster inspection task. During the inspection, it collects average load metrics for all computing nodes over a past period (e.g., the last 10 minutes), such as average GPU utilization and average video memory utilization.

[0086] If a compute node's average load consistently falls below a preset idle threshold (e.g., GPU utilization consistently below 30%) within a predefined time period, and its local request queue length remains zero or near zero for an extended period, the node is identified as redundant. In this case, the compute service device marks it as "pending scaling down" and immediately stops assigning any new inference requests to that node. This continues until the node has processed all remaining requests in its queue. During the draining process, to avoid losing valuable cache data, the system can selectively migrate the node's key-value cache to other compute nodes. Once the node awaiting scaling down is completely idle, the compute service device will safely terminate the node instance and release its occupied expensive computing resources by calling the cloud platform's API.

[0087] Through this dynamic elastic scaling mechanism tightly coupled with business load, the computing service equipment provided in the above embodiments of this specification can intelligently adapt to the tidal changes in traffic. During peak business periods, computing nodes are automatically added to ensure low latency and high availability for user requests; during off-peak business periods, computing nodes are automatically reduced to avoid resource waste, thereby achieving a balance between cost-effectiveness and service performance.

[0088] Figure 4 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 4As shown, device 400 mainly consists of a communication interface 402, a user interface 404, a processor 406, and a data storage 408. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 410. The communication interface 402 enables device 400 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 402 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 402 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 402 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 402 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0089] User interface 404 includes receiving user input and providing output to the user. Therefore, user interface 404 may include input components such as a keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 404 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 404 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 400 may support remote access from other devices via communication interface 402 or another physical interface (not shown). User interface 404 may be configured to receive user input, the position and movement of which may be indicated by an indicator or cursor described herein. User interface 404 may also be configured as a display device for rendering or displaying text fragments.

[0090] Processor 406 may contain one or more general-purpose processors and / or special-purpose processors.

[0091] Data storage 408 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 406. Data storage 408 may include removable and non-removable components.

[0092] Processor 406 is capable of executing program instructions 418 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 408 to perform the various functions described herein. Data storage 408 may comprise a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 400, enable device 400 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Processor 406 executing program instructions 418 may result in processor 406 using data 412.

[0093] For example, program instructions 418 may include an operating system 422 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 400 and one or more applications 420 (e.g., a browser, social application, or game application). Similarly, data 412 may include operating system data 416 and application data 414. Operating system data 416 is primarily accessible to the operating system 422, while application data 414 is primarily accessible to one or more applications 420. Application data 414 may reside in a file system visible or hidden from the user of device 400.

[0094] Application 420 can communicate with operating system 422 through one or more application programming interfaces (APIs). These APIs help application 420 read and / or write application data 414, transmit or receive information via communication interface 402, receive or display information on user interface 404, etc.

[0095] In some terminology, application 420 may be simply referred to as "app". Furthermore, application 420 can be downloaded to device 400 through one or more online app stores or app markets. However, applications can also be installed on device 400 in other ways, such as through a web browser or a physical interface on device 400 (e.g., a USB port).

[0096] Please refer to Figure 3 A large-scale model inference acceleration device can be applied to, for example... Figure 4 The device shown is used to implement the technical solution described in this specification. Among them, Figure 3 This is a block diagram of a large-model inference acceleration device provided in an exemplary embodiment. Figure 3 As shown, a large model inference acceleration device 30 is applied to a computing service device that performs large model inference tasks. The computing service device is configured with multiple computing nodes and is equipped with a hierarchical scheduling mechanism including a global scheduling mechanism and a local scheduling mechanism. The computing service device stores a prefix index structure to indicate the mapping relationship between the token sequence prefix and the computing node storing its cached computation results. The device 30 includes:

[0097] Receiving unit 302 is used to receive the token sequence of inference request;

[0098] The global scheduling unit 304 is used to execute the global scheduling mechanism, query the prefix index structure based on the token sequence of the inference request to perform prefix matching, so as to determine one or more candidate computing nodes, and obtain the real-time load status of the one or more candidate computing nodes, and then select a target computing node from the candidate computing nodes according to the real-time load status.

[0099] The local scheduling unit 306 is used to execute the local scheduling mechanism, assign an execution priority to the inference request based on the prefix matching degree of the inference request on the target computing node, and schedule the pending requests in the node according to the execution priority.

[0100] The inference unit 308 is used to call the target computing node so that it loads the cached computing results corresponding to the matching prefix according to the scheduling instructions of the local scheduling layer, and calls the large model to perform inference computing on the non-prefix part of the inference request.

[0101] In yet another embodiment shown, the global scheduling unit 304 is further configured to traverse the prefix index structure and calculate the matching token length between the token sequence and any prefix stored in the prefix index structure.

[0102] When the proportion of the length of the matching token in the token sequence to the target prefix index is higher than a preset threshold, the computing node corresponding to the target prefix is ​​determined as a candidate computing node;

[0103] When the proportion of the length of the matched token to the token sequence is lower than a preset threshold, the semantic matching process is triggered;

[0104] The similarity between the semantic vector of the inference request and the semantic vector of any prefix stored in the prefix index structure is calculated. If the similarity between the semantic vector of the target prefix index and the semantic vector of the inference request is higher than a preset similarity threshold, the computing node indicated by the target prefix index is determined as a candidate computing node.

[0105] In yet another embodiment shown, the real-time load status includes at least one of the following: GPU utilization of the compute node, video memory occupancy, and request queue length.

[0106] The global scheduling unit 304 performs the following process: selecting a target computing node from the candidate computing nodes based on the real-time load status. This includes: calculating the suitability score of the candidate computing node based on the historical cache hit count and the real-time load status, and selecting the node with the highest score as the target computing node.

[0107] In another embodiment shown, the device 30 further includes: an expansion unit 310, configured to trigger automatic expansion and start a new computing node as the target computing node when the fit scores of all candidate computing nodes are lower than a preset low watermark threshold.

[0108] In another embodiment shown, the device 30 further includes a scaling-down unit 312, which is used to periodically obtain the load status of all current computing nodes. When the average load of a certain computing node is continuously lower than an idle threshold within a preset period of time, it triggers automatic scaling-down, migrates the cached computing results of the certain computing node to other computing nodes, and releases the computing node.

[0109] In another embodiment shown, the local scheduling unit 306 is further configured to, within a preset scheduling task quantity period, sequentially retrieve and process a number of user requests corresponding to the scheduling weights preset for different priority queues.

[0110] In another embodiment shown, according to the full-scale scheduling mechanism of the computing service device, waiting time thresholds are preset for different priority queues;

[0111] The local scheduling unit 306 is also used to obtain the waiting time of the first pending request in any inference request queue; when the waiting time exceeds the waiting time threshold corresponding to any request queue, the inference request is promoted to a higher priority queue.

[0112] In yet another embodiment shown, the prefix index structure is a Trie tree (prefix tree) or its space-optimized variant, a Radix Tree.

[0113] In yet another embodiment shown, the cache computation result is a key vector and a value vector computed in a self-attention mechanism.

[0114] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0115] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0116] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0117] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0118] What those skilled in the art will understand is:

[0119] In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0120] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0121] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0122] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0123] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0124] This specification uses specific terms to describe embodiments thereof. For example, "one embodiment" and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an alternative embodiment" mentioned twice or more in different places in this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples, without contradiction.

[0125] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A method for accelerating large model inference, applied to a computing service device performing large model inference tasks, wherein the computing service device is configured with multiple computing nodes and has a hierarchical scheduling mechanism including a global scheduling mechanism and a local scheduling mechanism; the computing service device stores a prefix index structure to indicate the mapping relationship between token sequence prefixes and computing nodes storing their cached computation results; the method includes the following steps: The sequence of tokens to receive inference requests; Execute the global scheduling mechanism. Traverse the prefix index structure and calculate the length of the matching token between the token sequence and any prefix stored in the prefix index structure; When the proportion of the length of the matching token in the token sequence to the length of the target prefix index is higher than a preset threshold, one or more computing nodes corresponding to the target prefix are determined as one or more candidate computing nodes; when the proportion of the length of the matching token in the token sequence is lower than a preset threshold, a semantic matching process is triggered; the similarity between the semantic vector of the inference request and the semantic vector of any prefix stored in the prefix index structure is calculated; if the similarity between the semantic vector of the target prefix index and the semantic vector of the inference request is higher than a preset similarity threshold, one or more computing nodes indicated by the target prefix index are determined as one or more candidate computing nodes. And obtain the real-time load status of the one or more candidate computing nodes, and then select a target computing node from the candidate computing nodes based on the real-time load status; The local scheduling mechanism is executed to assign an execution priority to the inference request based on the prefix matching degree of the inference request on the target computing node, and to schedule the pending requests within the node according to the execution priority. The target computing node is invoked so that, according to the scheduling instructions of the local scheduling mechanism, it loads the cached computing results corresponding to the matching prefix, and calls the large model to perform inference computing on the non-prefix part of the inference request.

2. The method according to claim 1, characterized in that, The real-time load status includes at least one of the following: GPU utilization of the computing node, video memory usage, and request queue length. The step of selecting a target computing node from the candidate computing nodes based on the real-time load status includes: The suitability score of the candidate computing node is calculated based on the historical cache hit count and the real-time load status, and the node with the highest score is selected as the target computing node.

3. The method according to claim 1, characterized in that, The step of selecting a target computing node from the candidate computing nodes based on the real-time load status further includes: When the fit scores of all candidate computing nodes are lower than a preset low watermark threshold, automatic expansion is triggered, and a new computing node is started as the target computing node.

4. The method according to claim 1 or 3, further comprising: The system periodically acquires the load status of all current computing nodes. When the average load of a certain computing node remains below an idle threshold for a preset period of time, it triggers automatic scaling down, migrates the cached computing results of the certain computing node to other computing nodes, and releases the computing node.

5. The method according to claim 1, characterized in that, The step of scheduling pending requests within the node according to the execution priority includes: Within a preset scheduling task period, according to the preset scheduling weights for different priority queues, a number of user requests corresponding to the scheduling weights are sequentially retrieved from each priority queue for processing.

6. The method according to claim 1, characterized in that, Based on the local scheduling mechanism of the computing service device, waiting time thresholds are preset for different priority queues; Assigning an execution priority to the inference request further includes: Obtain the waiting time of the first pending request in any inference request queue; when the waiting time exceeds the waiting time threshold corresponding to any inference request queue, elevate any inference request to a higher priority queue.

7. The method according to claim 1, wherein the prefix index structure is a Trie tree (prefix tree) or its space-optimized variant, a radix tree.

8. The method according to claim 1, wherein the cache calculation result is a key vector and a value vector calculated in the self-attention mechanism.

9. A large model inference acceleration device, applied to a computing service device that performs large model inference tasks, wherein the computing service device is configured with multiple computing nodes and is equipped with a hierarchical scheduling mechanism including a global scheduling mechanism and a local scheduling mechanism; The computing service device stores a prefix index structure to indicate the mapping relationship between the token sequence prefix and the computing nodes storing its cached computation results; the device includes: The receiving unit is used to receive the token sequence for inference requests; A global scheduling mechanism execution unit is used to execute the global scheduling mechanism, traverse the prefix index structure, calculate the matching token length between the token sequence and any prefix stored in the prefix index structure; when the proportion of the matching token length between the token sequence and the target prefix index to the token sequence is higher than a preset threshold, determine one or more computing nodes corresponding to the target prefix as one or more candidate computing nodes; when the proportion of the matching token length to the token sequence is lower than a preset threshold, trigger a semantic matching process; traverse and calculate the similarity between the semantic vector of the inference request and the semantic vector of any prefix stored in the prefix index structure; if the similarity between the semantic vector of the target prefix index and the semantic vector of the inference request is higher than a preset similarity threshold, determine one or more computing nodes indicated by the target prefix index as one or more candidate computing nodes; and obtain the real-time load status of the one or more candidate computing nodes, and then select a target computing node from the candidate computing nodes according to the real-time load status. The local scheduling mechanism execution unit is used to execute the local scheduling mechanism, assign an execution priority to the inference request based on the prefix matching degree of the inference request on the target computing node, and schedule the pending requests in the node according to the execution priority; The inference unit is used to invoke the target computing node so that it loads the cached computing results corresponding to the matching prefix according to the scheduling instructions of the local scheduling mechanism, and invokes the large model to perform inference computing on the non-prefix part of the inference request.

10. A computing service device for large-scale model inference tasks, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-8 by executing the executable instructions.

Images