Request scheduling method and electronic equipment

Abstract

The embodiment of the invention provides a request scheduling method and electronic equipment, and the method comprises the steps: obtaining a query sequence which comprises prompt words and a query text; determining first hit information of the cue word in a prefix cache; the prefix cache is located in a shared memory; determining second hit information of the query text in a generation cache; the generation cache is located in a local memory; based on at least one of the first hit information and the second hit information, determining a target reasoning node for performing reasoning calculation on the query sequence in a plurality of local reasoning nodes; and sending the first hit information, the second hit information and the query sequence to the target reasoning node.

Description

Technical Field

[0001] This application relates to the field of computer technology, and includes, but is not limited to, a request scheduling method and an electronic device. Background Technology

[0002] With the continuous growth in demand for large-scale model inference, Transformer-based models face high computational overhead when handling long-sequence text generation tasks. To improve inference efficiency, key-value caching (KV cache) technology is widely used to store the key and value information already computed in the attention mechanism, thereby avoiding redundant calculations, improving inference speed, and enhancing semantic coherence.

[0003] In related technologies, key-value (KV) caches are typically reused only within a single query sequence and are cleared after the current query concludes. Since this type of KV cache can only be reused within a single query sequence, when multiple queries have the same or similar prompts, the corresponding KV pairs must be recalculated each time, resulting in a large amount of redundant computation. Therefore, cross-query sequence KV caches have emerged, aiming to reduce redundant computation and improve overall inference performance by sharing cached data.

[0004] However, related technologies such as Prompt cache and Radix attention, which involve key-value caching across query sequences, determine cache hits through prefix matching and tree traversal, and manage cached content using fixed-format markup languages ​​or Least Recently Used (LRU) strategies. However, these methods lack efficient cache sharing mechanisms in distributed architectures, resulting in low cache utilization between nodes. Furthermore, cache hit determination relies on strict exact matching, making it difficult to adapt to semantically similar but textually inconsistent scenarios, thus impacting cache reuse and overall inference efficiency. Summary of the Invention

[0005] Based on the problems existing in related technologies, this application provides a request scheduling method and an electronic device.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide a request scheduling method, the request scheduling method comprising:

[0008] Obtain the query sequence, which includes prompt words and query text;

[0009] Determine the first hit information of the prompt word in the prefix cache; the prefix cache is located in shared memory;

[0010] Determine the second hit information of the query text in the generation cache; the generation cache is located in local memory;

[0011] Based on at least one of the first hit information and the second hit information, a target inference node for inference calculation of the query sequence is determined among multiple local inference nodes;

[0012] The first hit information, the second hit information, and the query sequence are sent to the target inference node.

[0013] In some embodiments, the prefix cache is used to store a plurality of first key-value pairs; determining the first hit information of the prompt word in the prefix cache includes: retrieving a plurality of first text segments corresponding to the plurality of first key-value pairs from the shared memory; the corresponding first key-value pairs and first text segments have the same identifier; performing text matching on the prompt word and the plurality of first text segments to obtain the first matching information of the prompt word in the prefix cache; in response to the first matching information indicating that there is a first target text segment in the first text segment that matches the prompt word, determining the first identifier of the prompt word hitting the prefix cache and the first target text segment as the first hit information; in response to the first matching information indicating that there is no first target text segment in the first text segment that matches the prompt word, determining that the prompt word does not hit the prefix cache as the first hit information.

[0014] In some embodiments, determining the second hit information of the query text in the generation cache includes: obtaining the second text segment in the local memory and the load status of each local inference node; performing text matching between the query text and the second text segment to obtain the second matching information of the query text in the generation cache; in response to the second matching information indicating that there is a second target text segment in the second text segment that matches the query text, determining the query text hitting the generation cache, the second identifier of the second target text segment, and the load status as the second hit information; in response to the second matching information indicating that there is no second target text segment in the second text segment that matches the query text, determining the query text not hitting the generation cache and the load status as the second hit information.

[0015] In some embodiments, the first hit information includes at least whether the prompt word hits the prefix cache; the second hit information includes at least whether the query text hits the generation cache and the load status of each local inference node; determining the target inference node for inference calculation of the query sequence among multiple local inference nodes based on at least one of the first hit information and the second hit information includes: in response to the first hit information indicating that the prompt word hits the prefix cache and the second hit information indicating that the query text hits the generation cache, determining the first local inference node where the second target KV key-value pair corresponding to the second target text segment matched by the query text is located; wherein, each local inference node corresponds to a local inference memory, the generation cache in each local inference memory includes a portion of the second KV key-value pairs corresponding to the second text segment, and the second KV key-value pairs in different local inference memories are different; in response to the first local inference node having a normal load status, determining the first local inference node as the target inference node; in response to the first local inference node having an abnormal load status, determining the local inference node that satisfies the first load condition as the target inference node; the target inference node includes at least one local inference node.

[0016] In some embodiments, the first hit information includes at least whether the prompt word hits the prefix cache; the second hit information includes at least whether the query text hits the generation cache and the load status of each local inference node; determining the target inference node for inference computation of the query sequence among multiple local inference nodes based on at least one of the first hit information and the second hit information includes: in response to the first hit information indicating that the prompt word hits the prefix cache and the second hit information indicating that the query text does not hit the generation cache, determining the local inference node whose load status satisfies the first load condition as the target inference node.

[0017] In some embodiments, the first hit information includes at least whether the prompt word hits the prefix cache; determining the target inference node for inference calculation of the query sequence among multiple local inference nodes based on at least one of the first hit information and the second hit information includes: in response to the first hit information indicating that the prompt word did not hit the prefix cache, determining the local inference node whose load state meets the first load condition as the target inference node.

[0018] In some embodiments, the prompt word includes at least one prompt subword, the shared memory includes a plurality of first text segments and a weight score for each first text segment, and each first text segment includes at least one text word; determining the first hit information of the prompt word in the prefix cache includes: based on the order of the at least one prompt subword in the prompt word, sequentially matching it with at least one text word of each first text segment to obtain a matching result; in response to the matching result indicating that there is a first target text segment that is the same as the prompt word, obtaining the first hit information of the prompt word hitting the prefix cache; in response to the matching result indicating that some text words in the first target text segment are different from the prompt word, determining the first hit information of the prompt word in the prefix cache based on the weight score of the first target text segment.

[0019] In some embodiments, determining the first hit information of the prompt word in the prefix cache based on the weight score of the first target text segment includes: obtaining the first hit information of the prompt word hitting the prefix cache in response to the weight score being less than the left endpoint value of a preset weight interval; updating the prefix cache based on the partial text words in response to the weight score being within the preset weight interval to obtain the first hit information of the prompt word hitting the prefix cache; and obtaining the first hit information of the prompt word not hitting the prefix cache in response to the weight score being greater than the right endpoint value of the preset weight interval.

[0020] In some embodiments, updating the prefix cache based on the partial text words includes: recalculating the key-value (KV) values ​​of the partial text words to obtain target KV key-value pairs corresponding to the partial text words; updating the KV key-value pairs corresponding to the first target text segment based on the target KV key-value pairs to obtain updated KV key-value pairs corresponding to the first target text segment; and adding the updated KV key-value pairs to the prefix cache.

[0021] Secondly, embodiments of this application provide an electronic device, the electronic device comprising:

[0022] Memory, configured to store computer programs that can run on a processor;

[0023] When a processor is configured to execute the computer program, it performs the following steps: acquiring a query sequence, the query sequence including a prompt word and a query text; determining a first hit of the prompt word in a prefix cache, the prefix cache being located in shared memory; determining a second hit of the query text in a generation cache, the generation cache being located in local memory; determining a target inference node among a plurality of local inference nodes to perform inference computation on the query sequence based on at least one of the first hit information and the second hit information; and sending the first hit information, the second hit information, and the query sequence to the target inference node. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0025] Figure 2 This is an optional flowchart of the request scheduling method provided in the embodiments of this application. Figure 1 ;

[0026] Figure 3 This is an optional flowchart of the request scheduling method provided in the embodiments of this application. Figure 2 ;

[0027] Figure 4 This is a schematic diagram of the hardware architecture for distributed large model inference provided in an embodiment of this application;

[0028] Figure 5 This is a functional diagram of each node in the hardware architecture of the distributed large model inference provided in the embodiments of this application;

[0029] Figure 6 This is a schematic diagram of the distributed large model inference process provided in an embodiment of this application;

[0030] Figure 7 This is a schematic diagram illustrating the algorithm principle for weighting each key-value pair in the prefix cache according to an embodiment of this application;

[0031] Figure 8 This is a schematic diagram of the process of determining the hit rate of prompt words in the prefix cache according to the embodiments of this application. Detailed Implementation

[0032] To more clearly illustrate the purpose, technical solutions, and advantages of the embodiments of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the following description of the embodiments is intended to explain and illustrate the overall concept of the embodiments of this application, and should not be construed as a limitation of the embodiments of this application. In the specification and drawings, the same or similar reference numerals refer to the same or similar parts or components. For clarity, the drawings are not necessarily drawn to scale, and some well-known parts and structures may be omitted from the drawings.

[0033] In some embodiments, unless otherwise defined, the technical or scientific terms used in the embodiments of this application shall have the ordinary meaning understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The word "a" or "an" does not exclude multiple components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," "right," "top," or "bottom" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. When an element such as a layer, film, region, or substrate is referred to as being "above" or "below" another element, the element may be "directly" located "above" or "below" the other element, or there may be intermediate elements present.

[0034] Currently, key-value caching methods across query sequences mainly include Prompt cache and Radix attention. Prompt cache automatically performs a prefix-based cache matching algorithm for long query sequences exceeding 1024 slots. If a cache hit occurs, the cached data is retrieved and the calculation is performed instead; otherwise, the result is recalculated and added back to the cache. To improve cache hit rate, a Prompt Markup Language (PML) method is proposed. This method segments the text in the Prompt into a majority of fixed text and a small portion of variable slots, matching only the fixed text portion. In a distributed architecture, each node performs its own caching, and a routing device routes the query sequence to the node with the cache hit for inference computation.

[0035] Radix attention supports both key-value (KV) caches and generated text caches for both prefixed and generated text. It constructs a Radix Tree, stores the cache in GPU memory, and performs hit detection through depth-first traversal. The traversal follows the prefix matching principle; for partial cache hits, tree nodes can be split. Each node in the tree is marked with its cache hit count, and eviction is performed using the Least Recently Used (LRU) method to improve hit efficiency.

[0036] However, these methods still have drawbacks. In a distributed architecture, the lack of cache sharing between nodes leads to low cache reuse. Prefix matching requires a perfect match, which means that semantically similar prompts cannot be cached, resulting in low cache reuse. PML matching limits the granularity of text segmentation. If the granularity is too fine, it leads to semantic incoherence and adds a lot of extra overhead. If the granularity is too coarse, it has limited effectiveness in long text scenarios.

[0037] To address the problems existing in related technologies, this application provides a request scheduling method that obtains a query sequence, the query sequence including a prompt word and a query text; determines a first hit information of the prompt word in a prefix cache; the prefix cache is located in shared memory; determines a second hit information of the query text in a generation cache; the generation cache is located in local memory; based on at least one of the first hit information and the second hit information, determines a target inference node among multiple local inference nodes to perform inference calculations on the query sequence; and sends the first hit information, the second hit information, and the query sequence to the target inference node.

[0038] In this way, by introducing a prefix cache in shared memory and a generation cache in local memory, cache reuse and efficient query scheduling across query sequences are achieved. Placing the prefix cache in shared memory and the generation cache in local memory, the combined use of the prefix and generation caches effectively reduces redundant computation overhead by leveraging the prefix cache across queries, while improving efficiency in the query phase through the local generation cache. This achieves an efficient cache sharing mechanism in a distributed architecture, improving cache utilization among local inference nodes and enhancing the performance of large model inference services. By selecting the target inference node based on the hit information in the prefix and generation caches of the query sequence, optimal resource allocation can be achieved while ensuring service quality. Therefore, this embodiment not only effectively reduces computational overhead during large model inference but also improves inference speed.

[0039] The request scheduling methods provided in the embodiments of this application can be executed by electronic devices. Figure 1 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 1 The illustrated electronic device 10 includes at least one processor 110, a memory 150, at least one network interface 120, and a user interface 130. The various components in the request scheduling method are coupled together via a bus system 140. It is understood that the bus system 140 is used to implement communication between these components. In addition to a data bus, the bus system 140 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 1 The general labeled all buses as Bus System 140.

[0040] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0041] User interface 130 includes one or more output devices 131 that enable the presentation of media content, and one or more input devices 132.

[0042] Memory 150 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 150 may optionally include one or more storage devices physically located away from processor 110. Memory 150 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 150 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0043] Operating system 151 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0044] The network communication module 152 is used to reach other computing devices via one or more (wired or wireless) network interfaces 120, exemplary network interfaces 120 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0045] The input processing module 153 is used to detect one or more inputs or interactions from one or more input devices 132.

[0046] In some embodiments, the apparatus provided in this application may be implemented in software. Figure 1 A request scheduling device 154 stored in memory 150 is shown. This request scheduling device 154 can be a request scheduling device in a request scheduling method, and can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 1541, a first determination module 1542, a second determination module 1543, a third determination module 1544, and a sending module 1545. These modules are logically connected and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0047] In other embodiments, the apparatus provided in this application can also be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the request scheduling method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0048] In some embodiments, an electronic device can act as a scheduling node in a distributed architecture. The scheduling node can connect to multiple local inference nodes and shared memory via a Compute Express Link (CXL) switch. The scheduling node can read the prefix cache and corresponding text segments in the shared memory via CXL.mem. After matching, the first hit information, the second hit information, and the query sequence are sent to the target inference node via CXL.io.

[0049] The technical solution of this application will now be described in detail with reference to the accompanying drawings.

[0050] Figure 2 This is an optional flowchart of the request scheduling method provided in the embodiments of this application. Figure 1 ,like Figure 2 As shown, the request scheduling method provided in this application embodiment can be implemented through steps S201 to S205:

[0051] Step S201: Obtain the query sequence, which includes prompt words and query text.

[0052] Here, a query sequence can be a user-inputted request that needs to be processed by a large model inference service. A query sequence includes at least a prompt and query text. The prompt guides the model to generate specific content, while the query text is the part the user wants the model to further process or generate. For example, in a natural language processing task, the user-input query sequence might be "Write an article about climate change," where "Write an article about **" is the prompt, and "climate change" is the query text. After receiving the query sequence, the scheduling node uses the matching results of the query sequence as the basis for scheduling.

[0053] In some embodiments, query sequences input by users via terminals such as smartphones, tablets, or server-side applications can be received through a network interface. The scheduling node can temporarily store the query sequences in local memory and allocate tasks based on cache hits.

[0054] Step S202: Determine the first hit information of the prompt word in the prefix cache; the prefix cache is located in shared memory.

[0055] In some embodiments, a prefix cache can refer to a collection of key-value pairs stored in shared memory. This collection caches the calculation results of key-value pairs corresponding to recurring prompts across multiple query sequences. Deployed in shared memory, it can be shared by multiple inference nodes. Because shared memory can be accessed simultaneously by multiple local inference nodes, the prefix cache supports cache reuse across query sequences. For example, when multiple users use the prompt "Please explain machine learning," the relevant key-value pairs only need to be calculated once, and subsequent requests can reuse the calculation results from the prefix cache, thereby reducing redundant calculation overhead.

[0056] Here, the Key in the KV key-value pair is used to calculate the attention weights, and the Value in the KV key-value pair is used for weighted summation to generate the output vector. KV key-value pairs can be used to quickly obtain attention results without recalculating the attention weights, which can significantly improve the inference speed of the model.

[0057] Shared memory can refer to a unified memory area that can be accessed simultaneously by scheduling nodes and multiple inference nodes. It is used to store cached data that can be reused across queries (such as Prefix Cache) to support cache sharing in distributed systems.

[0058] The first hit information indicates whether the prompt word successfully matches the text corresponding to a key-value pair in the prefix cache. When the prompt word successfully matches the text corresponding to a key-value pair, it means the prompt word has hit the prefix cache, and the key-value pair can be directly reused for calculation. When the prompt word does not match the text corresponding to any of the key-value pairs in the prefix cache, it means the prompt word has not hit the prefix cache, and the key-value pair of the current prompt word can be recalculated and the prefix cache updated.

[0059] In some embodiments, the first hit information may further include the identifier of the key-value pair hit when the prompt word hits the prefix cache, and the subsequent target inference node can quickly obtain the corresponding key-value pair based on the identifier.

[0060] In some embodiments, shared memory can correspond to a memory management node. The scheduling node can send a read request to the memory management node to obtain all possible caches in shared memory related to the current prompt word. Subsequently, the scheduling node performs a matching judgment on these caches one by one. The judgment criteria can be the consistency of the text content, or a fuzzy matching mechanism based on attention weights. The fuzzy matching mechanism based on attention weights allows for a certain degree of inconsistency between the prompt word and the text, but it will distinguish them according to their importance. For key-value pairs with high importance, a complete match is required, while for key-value pairs with low importance, inconsistencies can be ignored or lightweight recomputation can be performed.

[0061] Step S203: Determine the second hit information of the query text in the generation cache; the generation cache is located in local memory.

[0062] In some embodiments, a generated cache can refer to a set of key-value pairs stored in local memory, used to cache the key-value pair calculation results corresponding to the generated portion of the content in the current query sequence. Since local memory has lower access latency, a generated cache can be used to reduce the calculation of duplicate tokens within the same query sequence.

[0063] Local memory can refer to the private memory resources of each local inference node and scheduling node, used to store cached data (such as Generated Cache) that is only valid within the current query sequence, with low access latency.

[0064] The second hit information indicates whether the query text matches a cached value in the generated cache. If a match is found, the existing calculation result can be reused directly; otherwise, the generated cache needs to be recalculated and updated. The second hit information can also represent the identifier of the key-value pair that the query text matches in the generated cache, allowing subsequent target inference nodes to quickly retrieve the corresponding key-value pair based on this identifier.

[0065] In some embodiments, the second hit information may also include the load status of each local inference node.

[0066] In some embodiments, after determining a hit in the prefix cache, the scheduling node can further check if the query text exists in the generation cache in local memory. If some tokens in the query text have already been generated previously, the scheduling node can directly read the key-value pairs corresponding to these tokens from the generation cache, thereby avoiding duplicate calculations of the same tokens. The generation cache-based approach can significantly improve the inference efficiency within a single query sequence, especially during the generation of long texts, enabling faster retrieval of required information and reduced computational resource consumption.

[0067] Step S204: Based on at least one of the first hit information and the second hit information, determine the target inference node for inference calculation of the query sequence among multiple local inference nodes.

[0068] Here, a local inference node refers to a model inference execution unit deployed on a local computing unit, used to handle specific query tasks. Inference nodes typically possess certain computing and caching capabilities, enabling them to independently complete the model inference process. For example, in a distributed system, each server node can be configured as a local inference node, with each server node responsible for handling the query tasks assigned to it.

[0069] The target inference node can be one or more nodes selected from multiple inference nodes based on the cache hit rate of the query sequence and the load status of each inference node, making them suitable for executing the current query task. After determining the hit rate of the prefix cache and the generation cache, the scheduling node also needs to comprehensively consider the current load status of each inference node to ensure that the task can be efficiently allocated to the most suitable node for execution. If an inference node has a low load and a high cache hit rate, the scheduling node will preferentially select that inference node as the target inference node.

[0070] In some embodiments, the scheduling node can maintain a load status table that includes the load status of all local inference nodes, and can record information such as CPU utilization, memory utilization, and the number of tasks currently being processed for each node in real time. After the scheduling node determines the cache hit status of the query sequence, it can calculate a score for each inference node based on the load status, and select the inference nodes with the highest or top scores as target inference nodes to optimize overall resource utilization.

[0071] Step S205: Send the first hit information, the second hit information, and the query sequence to the target inference node.

[0072] After completing the hit determination and the target inference node's decision, the scheduling node packages the first hit information, the second hit information, and the query sequence into a task request and sends it to the target inference node. Upon receiving the task request, the target inference node decides, based on the cache hit situation, whether to directly reuse cached data or recalculate to obtain the inference result of the query sequence.

[0073] The scheduling node can send task requests to the target inference node via the CXL.io protocol. CXL.io is a high-performance interconnect protocol that enables low-latency data transmission and is suitable for task scheduling in large-scale distributed systems. Upon receiving the task request, the target inference node begins inference computation, deciding whether to read from the prefix cache in shared memory or the generation cache in local memory based on cache hits. If a cache miss occurs, the target inference node will recalculate the relevant key-value pairs and write the results to the corresponding cache for reuse in subsequent queries.

[0074] This application embodiment achieves cache reuse and efficient query scheduling across query sequences by introducing a prefix cache in shared memory and a generation cache in local memory. Placing the prefix cache in shared memory and the generation cache in local memory, the combined use of the prefix and generation caches effectively reduces redundant computation overhead by leveraging the prefix cache across queries, while improving efficiency during the query phase through the local generation cache. This achieves an efficient cache sharing mechanism in a distributed architecture, improving cache utilization among local inference nodes and enhancing the performance of large model inference services. By selecting the target inference node based on the hit information in the prefix and generation caches of the query sequence, optimal resource allocation can be achieved while ensuring service quality. Therefore, this application embodiment not only effectively reduces computational overhead during large model inference but also improves inference speed.

[0075] In some embodiments, a prefix cache is used to store multiple first key-value pairs, each consisting of a key vector and a value vector, for context modeling during attention computation. The first key-value pairs are pre-computed and cached during the generation of long prompt sequences for reuse across queries, reducing redundant computation overhead. The first key-value pairs can be computed by a large model inference service and written to shared memory via the CXL interface for use in subsequent queries. Step S202 can be implemented through steps S2021 to S2024:

[0076] Step S2021: Obtain multiple first text segments corresponding to the multiple first KV key-value pairs from the shared memory; the corresponding first KV key-value pairs and first text segments have the same identifier.

[0077] In some embodiments, the first text segment may refer to the original text fragment corresponding to the first key-value pair, stored in shared memory. Each first key-value pair corresponds one-to-one with a first text segment; that is, for each first key-value pair, the key-value pair has a unique corresponding first text segment, and the key-value pair and its corresponding text segment use the same identifier. The identifier may be a hash value, a string ID, or other unique encoding form. These identifiers are used to enable fast lookup and association between key-value pairs and text content in the cache, ensuring that the correct key-value cache data can be quickly located and reused in subsequent queries.

[0078] The first text segment is a portion of the input text used by the large model inference service when generating the first key-value pair. For example, when a user inputs "Write an article about climate change", "Write an article about **" is used as the first text segment, and the corresponding key-value pair is saved in the prefix cache, while "Write an article about **" is saved in the shared memory at the location where the text segment is stored.

[0079] Step S2022: Perform text matching between the prompt word and the plurality of first text segments to obtain the first matching information of the prompt word in the prefix cache.

[0080] Text matching refers to the process of comparing the prompt words in the current query sequence with each first text segment in shared memory, either character by character or word by word, to determine whether the current prompt word is consistent with or sufficiently similar to a cached first text segment, thereby deciding whether the first key-value pair corresponding to a cached first text segment can be reused. Text matching can be implemented through exact matching, fuzzy matching, or semantic-based matching, depending on the application scenario and performance requirements.

[0081] Here, by comparing the prompt word with the first text segment in shared memory, it can be determined whether there is a first text segment that is completely identical or has an acceptable difference. If there is a first text segment that is completely identical or has an acceptable difference, it is determined that the prompt word matches the first text segment successfully, and the prompt word hits the prefix cache; if it does not exist, the prompt word does not hit the prefix cache.

[0082] Step S2023: In response to the first matching information indicating that there is a first target text segment in the first text segment that matches the prompt word, the first identifier of the prompt word hitting the prefix cache and the first target text segment is determined as the first hit information.

[0083] Here, the first target text segment can refer to the first text segment that successfully matches the prompt word among multiple first text segments. The first identifier of the first target text segment can be a unique ID or hash value, which is used to quickly locate the KV cache data corresponding to the first identifier in the prefix cache.

[0084] When the first matching information indicates that the prompt word hits the prefix cache, the first hit information can include the hit status (in this case, a hit) and the first identifier corresponding to the hit object (i.e., the first identifier corresponding to the first target text segment). By issuing the first hit information, the target inference node can clearly know in the subsequent inference process whether the current prompt word can reuse the first KV key-value pair in the cache, and which first KV key-value pair the current prompt word can reuse, thereby skipping redundant calculation steps and speeding up the inference process.

[0085] Step S2024: In response to the first matching information indicating that there is no first target text segment in the first text segment that matches the prompt word, the prompt word not hitting the prefix cache is determined as the first hit information.

[0086] When the first matching information indicates that the prompt word did not hit the prefix cache, the first hit information may include the hit status (which is a miss in this case), and the first hit information does not include the first identifier.

[0087] A miss means that the target inference node cannot reuse the existing key-value cache data. It needs to recalculate the first key-value pair corresponding to the prompt word and write the calculation result into shared memory for subsequent queries.

[0088] In this embodiment, the first text segment corresponding to the first KV key-value pair in the cache is obtained from the shared memory. The current prompt word is matched with the existing first text segment by text matching. Based on the matching result, the first hit information (hit or miss) is generated, and the cached data is reused accordingly. This reduces redundant calculations and thus significantly improves the overall efficiency of large model inference.

[0089] In some embodiments, step S203 can be implemented by steps S2031 to S2034:

[0090] Step S2031: Obtain the second text segment in the local memory and the load status of each local inference node.

[0091] The second text segment refers to the text segment stored in local memory that corresponds to the KV pair stored in the generation cache. Here, local memory can be the local memory corresponding to the scheduling node, and the scheduling node can quickly obtain the second text segment through local memory access.

[0092] The second text segment is usually the content of the query part. For example, in a large model inference service on a local inference node, if a query ("The weather is nice today") has been processed and a corresponding key-value pair has been generated, then "The weather is nice today" is cached as the second text segment. The second text segment is stored in the local memory of the scheduling node, and the corresponding key-value pair is stored in the local memory of the local inference node for subsequent similar queries, thereby avoiding duplicate calculations.

[0093] In some embodiments, the load status of a local inference node can refer to the resource consumption of the local inference node when executing inference tasks, including CPU / GPU utilization, memory usage, task queue length, etc. The load status of a local inference node can be used to describe the real-time workload of the node, so that nodes with lower loads can be selected when scheduling tasks, achieving more efficient resource utilization. For example, when the GPU utilization of a node exceeds 90%, it indicates that the node is under high load. In this case, new tasks should be preferentially assigned to nodes with lower loads to balance the overall system performance.

[0094] Step S2032: Perform text matching between the query text and the second text segment to obtain the second matching information of the query text in the generation cache.

[0095] In some embodiments, text matching refers to determining whether there is semantic or structural similarity between the query text and the second text segment by comparing their contents. Text matching can be achieved through methods such as exact string matching, fuzzy matching, or attention-weighted matching. The second matching information is used to characterize whether the query text is generated in the cache.

[0096] By matching the query text with the second text segment, it is possible to quickly determine whether the current query text can reuse the existing second key-value pair to reduce redundant calculations, save computing resources, and speed up inference.

[0097] Step S2033: In response to the second matching information indicating that there is a second target text segment in the second text segment that matches the query text, the query text hits the generation cache, the second identifier of the second target text segment and the load condition as the second hit information.

[0098] In some embodiments, when the second matching information indicates that the query text matches a portion of a plurality of second text segments (i.e., the second target text segment), meaning the query text hits the generation cache, it indicates that the current query text can reuse the content in the generation cache. In this case, the second identifier of the query text hitting the generation cache, the second target text segment, and the load status of each local inference node can be used as the second hit information. The second identifier is used to uniquely identify the second key-value pair in the generation cache, facilitating subsequent access.

[0099] Step S2034: In response to the second matching information indicating that there is no second target text segment in the second text segment that matches the query text, the query text not hitting the generation cache and the load situation is determined as the second hit information.

[0100] In some embodiments, when the second matching information indicates that there is no matching content between the query text and the second text segment, it means that the current query text has not hit the generation cache. In this case, the query text's failure to hit the generation cache and the load status of each local inference node can be used as the second hit information.

[0101] The second hit information can be used for subsequent scheduling strategies. Based on this information, the query text is sent to nodes whose load conditions meet the requirements for processing, so as to avoid resource contention among nodes with high load conditions and improve inference efficiency.

[0102] This application embodiment obtains the second text segment in local memory and the load status of each local inference node, performs text matching between the query text and the second text segment to obtain second matching information, and determines second hit information based on the second matching information. This allows for accurate determination of whether cached data can be reused from the query text, thereby reducing redundant calculations and improving inference efficiency. Simultaneously, by combining load status with scheduling decisions, resource allocation can be optimized, improving overall system performance, and ultimately enabling efficient distributed large-model inference services.

[0103] In some embodiments, the first hit information includes at least whether the prompt word hits the prefix cache; the second hit information includes at least whether the query text hits the generation cache and the load status of each local inference node. Step S204 can be implemented through steps S2041 to S2043:

[0104] Step S2041: In response to the first hit information indicating that the prompt word hits the prefix cache, and the second hit information indicating that the query text hits the generation cache, determine the first local inference node where the second target KV key-value pair corresponding to the second target text segment matched by the query text is located; wherein, each local inference node corresponds to a local inference memory, and the generation cache in each local inference memory includes some second KV key-value pairs corresponding to the second text segment, and the second KV key-value pairs in different local inference memories are different.

[0105] In a distributed inference architecture, each local inference node can be configured with a local inference memory. Local inference memory refers to memory modules deployed on each distributed inference node. Each local inference memory contains a generated cache with several second key-value pairs corresponding to the second text segments, and these second key-value pairs can be different across different local inference memory locations. The local memory corresponding to the scheduling node contains all the second text segments from the local inference memory. Therefore, the query text can be matched against the second text segments in the scheduling node to determine whether the query text hits the generated cache.

[0106] In some embodiments, when the prompt word hits the prefix cache and the query text also hits the generation cache of a local inference node, it means that the query sequence can be cached and reused during inference. In this case, the query sequence can be assigned to the local inference node (i.e., the first local inference node) that stores the generation cache of the matched second KV key-value pair for processing.

[0107] Here, because the second key-value pairs stored in the generation cache of each local inference node are different, only some local inference nodes will have key-value pairs corresponding to the current query text. Therefore, the query must be assigned to the correct node in order to achieve cache reuse.

[0108] Step S2042: In response to the normal load status of the first local inference node, the first local inference node is determined as the target inference node.

[0109] Here, "normal load status" can mean that the first local inference node is in a low load state, that is, the CPU / GPU / CXL and other memory access resources of the first local inference node are not saturated. In other words, the first local inference node has sufficient computing power and memory bandwidth to process the current query sequence, without causing a performance bottleneck or affecting other ongoing inference tasks. At this time, the first local inference node can be identified as the target inference node.

[0110] CPU / GPU / CXL memory access resources not being saturated can refer to CPU / GPU utilization being lower than a preset threshold, CXL memory access latency being low, or memory bandwidth not reaching its peak.

[0111] Step S2043: In response to the abnormal load status of the first local inference node, a local inference node that meets the first load condition is determined as the target inference node; the target inference node includes at least one local inference node.

[0112] In some embodiments, an abnormal load state of the first local inference node may refer to a high load state, such as CPU / GPU utilization exceeding the threshold or increased CXL memory access latency. In this case, the first local inference node cannot be used as the target inference node, and other nodes that meet the first load condition need to be found as alternatives. Meeting the first load condition may refer to a node with a low current load and sufficient resources to process new query sequences.

[0113] In some embodiments, the target inference node refers to the local inference node that is ultimately responsible for performing the query sequence inference computation. The target inference node may include at least one local inference node, that is, the query sequence may be split into multiple parts, and multiple local inference nodes may jointly perform inference.

[0114] In some embodiments, the first hit information includes at least whether the prompt word hits the prefix cache; the second hit information includes at least whether the query text hits the generation cache and the load status of each local inference node. Step S204 can also be implemented by step S2044:

[0115] Step S2044: In response to the first hit information indicating that the prompt word hits the prefix cache, and the second hit information indicating that the query text does not hit the generation cache, the local inference node whose load status meets the first load condition is determined as the target inference node.

[0116] In this embodiment, when a prompt word hits the prefix cache, it indicates that a usable key-value pair exists for the prompt word in the current query sequence, which can be directly reused to reduce computational overhead. When a prompt word hits the prefix cache, it is only necessary to determine whether the query text hits the generation cache and assess the load status of each local inference node.

[0117] If the query text does not generate a cache, it means that there is no available second key-value pair for the query text to be reused. In this case, it is necessary to select a suitable local inference node to perform a complete inference calculation on the query text.

[0118] Here, when the query text is not generated in the cache, the local inference node that meets the first load condition can be selected as the target inference node based on the load status of each local inference node. That is, at least one local inference node with GPU / CPU utilization below a certain threshold and sufficient memory resources can be selected as the target inference node to ensure the efficient execution of the inference task.

[0119] In some embodiments, the first hit information includes at least whether the prompt word hits the prefix cache. Step S204 can also be implemented via step S2045:

[0120] Step S2045: In response to the first hit information indicating that the prefix cache was not hit, the local inference node whose load status meets the first load condition is determined as the target inference node.

[0121] In some embodiments, when the prompt term fails to hit the prefix cache, it indicates that the current query sequence cannot reuse existing cached data, and a completely new computation operation is required during inference. In this case, a suitable local inference node needs to be selected to execute the computation task, which can be determined by whether the load status meets a first load condition.

[0122] Here, a low-load local inference node can be selected as the target inference node, which can avoid the response delay problem caused by resource shortage of high-load local inference nodes and ensure the efficient execution of new computing tasks.

[0123] In this embodiment, the newly generated KV key-value pairs can be stored in the prefix cache and the generation cache respectively, so that they can be reused in subsequent queries.

[0124] This application embodiment achieves efficient cache reuse and reasonable allocation of computing resources by querying the hit information of the sequence in the prefix cache and the generation cache, and dynamically scheduling the target inference node in combination with the load status of each local inference node, thereby improving the overall performance and user experience of the distributed large model inference system.

[0125] Based on the foregoing embodiments, the prompt word includes at least one prompt subword, the shared memory includes multiple first text segments and weight scores for each first text segment, and each first text segment includes at least one text word.

[0126] Here, a prompt word can consist of multiple prompt sub-words, used to represent contextual information in the user's input query sequence. For example, if the prompt word is "Please help me write an article about artificial intelligence," the multiple prompt sub-words could be "please, help me, write, an, about, artificial intelligence, of, article." This word segmentation method allows the system to perform more refined cache matching and improve the hit rate.

[0127] In some embodiments, shared memory may store multiple first text segments and their corresponding weight scores. Each first text segment consists of at least one text word. A text word is the basic unit that constitutes a text segment. For example, the first text segment "The weather is very nice today" consists of the three text words "today," "weather," and "very nice." The weight score is a quantitative assessment of the importance of a first text segment during inference. It can be calculated based on a cross-attention mechanism and is used to determine the prefix cache hit.

[0128] Figure 3 This is an optional flowchart of the request scheduling method provided in the embodiments of this application. Figure 2 ,like Figure 3 As shown, step S202 can also be achieved through steps S301 to S303:

[0129] Step S301: Based on the order of the at least one prompt word in the prompt words, match it sequentially with at least one text word in each first text segment to obtain the matching result.

[0130] In this embodiment, the matching can be performed by matching each prompt word with the text words of each first text segment in the shared memory, one by one, according to the original order of the prompt words in the prompt phrase. For example, if the prompt phrase is "Please help me write an article about artificial intelligence," the system will compare the prompt words "please," "help me," and "write" sequentially with the first text segments stored in the shared memory. The text words of the first text segments stored in the shared memory can be stored in a tree structure. Therefore, it is necessary to check whether each first text segment in the shared memory matches according to the semantic structure and logical order within the prompt phrase, ensuring the accuracy of the matching.

[0131] The matching process can employ string matching algorithms or similarity calculation methods based on attention mechanisms. A match is successful if a text word in the first text segment completely matches multiple current prompt words; otherwise, it is considered a mismatch.

[0132] Step S302: In response to the matching result indicating that there is a first target text segment that is the same as the prompt word, the first hit information of the prompt word hitting the prefix cache is obtained.

[0133] In some embodiments, when the matching result indicates that there is a first target text segment that is completely consistent with the prompt word, it means that the prompt word hits the prefix cache, and the KV cache corresponding to the prompt word can be directly reused without recalculation.

[0134] Step S303: In response to the matching result indicating that some text words in the first target text segment are different from the prompt word, determine the first hit information of the prompt word in the prefix cache based on the weight score of the first target text segment.

[0135] In some embodiments, when the matching result indicates that some text words are inconsistent with the prompt words, it is not immediately determined as a miss. Instead, the weight score of the first target text segment is further referenced for judgment. The weight score can be calculated based on the cross-attention weights applied to each first text segment by the model during the generation phase, reflecting the importance of the first text segment in the inference process. For example, during a token generation process, the model calculates the importance score of each KVpair based on the attention weights between the current token and other tokens related to the current token.

[0136] The higher the weight score, the more critical the first text segment is, and the greater the impact of whether the prefix cache of the first text segment is hit on the final inference result. Therefore, in the prefix cache hit determination, the weight score plays an auxiliary role in decision-making, enabling a more accurate judgment on whether the content of the prefix cache can be reused.

[0137] When some text words do not match the prompt words, a combination of weighted scores and fuzzy matching strategies can be used to further determine whether the prompt words have hit the prefix cache. For example, if a certain first target text segment has a low weighted score, even if some text words do not match the prompt words, the mismatched parts can be ignored, and the prompt word can be determined as a prefix cache hit, thereby reducing unnecessary computational overhead.

[0138] In this embodiment of the application, a preset weight interval [θ1,θ2] can be set, where θ1,θ2∈[0,1), and θ1<θ2. The preset weight interval controls the ambiguity of the prefix cache hit. The smaller θ1,θ2 is, the stricter the cache hit determination, the lower the cache reuse rate, the slower the inference speed, but the higher the inference accuracy.

[0139] The method for determining prefix cache hits based on a combination of sequential matching and weighted judgment in this application can effectively improve cache reuse rate in cross-query scenarios, reduce the computational overhead of key-value pairs, and thus improve the overall efficiency of large model inference.

[0140] In some embodiments, step S303 can be implemented by steps S3031 and S3033:

[0141] Step S3031: In response to the weight score being less than the left endpoint value of the preset weight interval, the first hit information of the prompt word hitting the prefix cache is obtained.

[0142] Here, if the weight score of the first target text segment is lower than the left endpoint value θ1 of the preset weight interval, it indicates that the text segment has a low impact on the subsequently generated content in the entire sequence. In this case, even if the text segment does not completely match the prompt words of the current query, the impact of the text segment can be ignored, and the next text segment can be checked for a match to determine whether the entire cache has been hit.

[0143] In some embodiments, if the weight score of the first target text segment is lower than the left endpoint value θ1 of the preset weight interval, it can be considered that the text segment has hit the prefix cache.

[0144] Step S3032: In response to the weight score being within a preset weight range, the prefix cache is updated based on the partial text words to obtain the first hit information of the prompt word hitting the prefix cache.

[0145] When the weight score of the first target text segment falls within the preset weight range, it is considered that the first target text segment has a certain importance, but is not a critical part. At this time, the non-matching text words (Tokens) in the text segment can be asynchronously recalculated, and the matching status of subsequent text segments can be judged.

[0146] A subset of text terms may refer to a portion of tokens within a text segment. These tokens might influence the inference results. By using a lightweight recalculation method, these effects can be corrected. When processing a long text segment, recalculating only a few key tokens within the segment, rather than recalculating all tokens, reduces computational resource consumption and improves overall processing efficiency.

[0147] Asynchronous recalculation refers to recalculating some mismatched tokens without affecting subsequent cache match determination. Since the calculation result of the tokens that need to be recalculated only depends on the tokens preceding them, the calculation process of these tokens that need to be recalculated can be executed in parallel in the background, and the calculation process of these tokens that need to be recalculated will not block the subsequent cache hit determination process.

[0148] Step S3033: In response to the weight score being greater than the right endpoint value of the preset weight interval, the first hit information of the prompt word not hitting the prefix cache is obtained.

[0149] Here, the right endpoint value θ2 is the upper limit of the preset weight range. When the weight of the first target text segment is greater than θ2, any mismatch between the first target text segment and the prompt word will be considered as a missed prompt word. This is because highly important text segments have a significant impact on the final generated result.

[0150] If the weight score of the first target text segment is higher than the right endpoint of the preset weight interval, and if the text segment fails to completely match the prompt words of the current query, the system determines that the prefix cache has not been hit and stops matching subsequent text segments in the prefix cache. This ensures the accuracy of key text segments, avoids inference bias, and guarantees the quality of the generated content.

[0151] This application embodiment uses different hit determination strategies based on weight scores to dynamically adjust the strictness of prefix cache hits, thereby balancing inference accuracy and cache reuse rate. This effectively controls the consumption of computing resources, can flexibly adapt to the needs of different scenarios, and improves the efficiency of distributed large model inference and user experience.

[0152] In some embodiments, updating the prefix cache based on the partial text words in step S3032 can be achieved through steps S1 to S3:

[0153] Step S1: Recalculate the KV values ​​of the partial text words to obtain the target KV key-value pairs corresponding to the partial text words.

[0154] In some embodiments, the partial text words may be text words in the first target text segment that are different from the prompt words, and may not be consecutive tokens. The key-value values ​​of the partial text words can be recalculated using an attention mechanism to obtain the target key-value pairs corresponding to the partial text words.

[0155] The target key-value pair is used to replace the old value in the prefix cache, thereby updating the cache. The calculation of the target key-value pair takes into account the contextual information of the current query sequence, ensuring that more accurate attention weight calculation results can be provided in subsequent inference.

[0156] Step S2: Based on the target KV key-value pair, update the KV key-value pair corresponding to the first target text segment to obtain the updated KV key-value pair corresponding to the first target text segment.

[0157] Updating key-value pairs can refer to replacing the target key-value pair with the key-value pair in the prefix cache. This process can be performed by overwriting the existing key-value pair or by combining the original key-value pair with the target key-value pair and using a weighted average or similar method to generate a new key-value pair. The updated key-value pairs are better suited to the current query context, thereby improving cache hit rate and reducing redundant calculations.

[0158] Step S3: Add the updated KV key value to the prefix cache.

[0159] By writing the updated key-value pairs to the prefix cache, it can be ensured that subsequent queries can quickly hit the cache when encountering the same or similar prompts, thereby avoiding duplicate calculations.

[0160] This application embodiment can effectively improve the hit accuracy and reuse rate of the prefix cache by recalculating the key-value pairs of some text words and updating the prefix cache of related text segments, thereby reducing unnecessary key-value calculations, significantly reducing inference latency, and improving the real-time response capability of large model inference services.

[0161] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0162] To address the problems existing in related technologies, this application proposes a key-value cache sharing and hit determination method for distributed large-model inference systems. The key-value cache sharing and hit determination method proposed in this invention mainly includes the following three aspects:

[0163] First, a hardware architecture for distributed large model inference is provided, which offloads part of the Prefix cache in the key-value cache to shared memory. Each computing node (i.e., local inference node) in the system directly accesses the shared memory through CXL technology, thereby achieving efficient data sharing.

[0164] Secondly, a method for determining cross-query prefix cache hits is proposed. During the token generation process, the model calculates the cross-attention weights (i.e., weight scores) of each key-value pair in the prefix cache each time a new token is generated, and uses these weights as the basis for weighting the corresponding Prompt text segment. When performing prefix cache hit determination for subsequent queries, low-weight mismatched text segments are ignored, or only a small number of tokens in these mismatched text segments undergo asynchronous, lightweight recalculation.

[0165] Finally, a query request scheduling method is provided, which manages the prefix cache and the generated cache separately. The prefix cache is offloaded to shared memory, while the generated cache is kept in the local memory corresponding to the xPU (e.g., CPU / GPU / TPU). When a prefix cache hit occurs, the query request is scheduled to the designated node for inference computation based on the hit status of the generated cache.

[0166] Figure 4 This is a schematic diagram of the hardware architecture for distributed large model inference provided in an embodiment of this application, such as... Figure 4As shown, the computing resources in the architecture include a scheduling node 401, a memory management node 402, and several distributed local inference nodes 403-1 to 403-n. The local inference nodes can be implemented in the form of xPUs or FPGAs. The memory resources include the local memory 404 of the scheduling node corresponding to the scheduling node and the local memory 405-1 to 405-n of each inference node, as well as shared memory 406 that can be accessed by all nodes. Each node is connected to the CXL switch 407, and each node reads and writes memory through two methods: local direct memory access and CXL memory access.

[0167] Figure 5 This is a functional diagram of each node in the hardware architecture of the distributed large model inference provided in the embodiments of this application, as shown below. Figure 5 As shown, scheduling node 401 and its local memory 404 are responsible for receiving user query sequences and managing their scheduling. Scheduling node 401 stores the received user query sequences in its local memory 404. It can also read the prefix cache and corresponding text segments from shared memory 406 via CXL.mem. For each received query sequence in the form of Prompt+Query, scheduling node 401 first matches the Prompt portion with the text segments corresponding to the key-value pairs (i.e., the first KV key-value pairs) in the prefix cache, and matches the Query portion with the text segments corresponding to the key-value pairs (i.e., the second KV key-value pairs) in the generation cache. Combining the hit results from the generation cache with the load of each inference node, the scheduling node distributes the query sequence to the relevant inference node (i.e., the target inference node 501) for computation via CXL.io.

[0168] The inference node and its local memory are responsible for receiving query tasks from the scheduling node 401 and performing inference computations. The target inference node 501 stores the received query sequence and the generation cache generated during the inference process in its local memory, and reads and writes the prefix cache in shared memory using CXL.mem. Based on the task content issued by the scheduling node, the inference node determines whether the prefix cache in shared memory and the generation cache in local memory can be reused during the inference process, and then performs inference computations.

[0169] When generating each token, the inference node assigns weights to the text segments corresponding to each key-value pair in the prefix cache of shared memory 406 based on the attention calculation results, and synchronizes the reuse records and scoring results to the memory management node 402. Simultaneously, the inference node maintains a locally stored generation cache and sends the results generated for each query and the corresponding text segment changes when the generation cache is modified back to the scheduling node via CXL.io. The model weights and intermediate inference results in local memory are the parameters used by the inference node during inference.

[0170] Memory management node 402 and shared memory 406 are responsible for maintaining the prefix cache in shared memory 406. Memory management node 402 receives prefix cache read and write requests from each inference node and responds according to the shared memory status, including allowing read and write or requiring a wait. Simultaneously, memory management node 402 can analyze the prefix cache read requests from each inference node and, using a specific eviction strategy, evict expired prefix cache entries based on historical reuse patterns. Furthermore, memory management node 402 can collect the weight scores from each inference node for the text segments corresponding to each key-value pair in the prefix cache and provide this data to scheduling node 401 as a reference for the prefix cache hit determination algorithm.

[0171] Figure 6 This is a schematic diagram of the distributed large model inference process provided in the embodiments of this application, such as... Figure 6 As shown, the distributed large model inference process includes steps S601 to S627. The execution subject of steps S601 to S609 is the scheduling node, the execution subject of steps S610 to S620 is the target inference node, and the execution subject of steps S621 to S627 is the memory management node.

[0172] Step S601: Receive user query sequence.

[0173] Here, the query sequence includes Prompt and Query.

[0174] Step S602: Read the Prefix cache text segment and assign weights.

[0175] A read request for shared memory is sent to the memory management node. The memory management node executes step S621. If the memory management node allows reading, it reads the Prefix cache text segment and its corresponding weight score (i.e., weight score) to determine whether the Prefix cache has been hit.

[0176] Step S603: Determine if the Prefix cache is hit.

[0177] Here, Prompt executes step S604 if the prefix cache is hit, and executes step S609 if it is not hit.

[0178] Step S604: Read the Generated cache text segment.

[0179] If Prompt hits the prefix cache in step S603, the generated cache text segment (i.e., the second text segment) is read from the local memory of the scheduling node and used to perform the determination of whether the generated cache has been hit.

[0180] If Prompt is not found in the prefix cache in step S603, the low-load node is selected as the target inference node based on the load of each inference node.

[0181] Step S605: Determine if the Generated cache has been hit.

[0182] Here, if the query hits the cache, step S606 is executed; otherwise, step S609 is executed.

[0183] Step S606: Determine whether the load status of the corresponding node is normal.

[0184] Here, step S607 is executed normally, and step S609 is executed abnormally.

[0185] If the cache generation hit is achieved in step S605, then it is determined whether the load status of the local inference node where the generated cache is located is normal: if the node's load status is normal, then the node where the generated cache is located is selected as the target inference node; if the node's load status is abnormal, then another node with a normal load status or a low-load node is selected as the target inference node.

[0186] Step S607: Select the Generated cache hit node as the target inference node.

[0187] Step S608: The query task is sent to the target inference node.

[0188] The scheduling node sends query tasks to the target inference node, including the query sequence and the hit status of the two-level cache.

[0189] Step S609: Select the low-burden node as the target inference node.

[0190] If the prefix cache misses in step S603, the generation cache misses in step S605, or the node with a cache hit in step S606 has an abnormal load, a low-load node is selected as the target inference node based on the load of each inference node.

[0191] Step S610: Parse the query task.

[0192] The target inference node parses the query task issued by the scheduling node, obtains the query sequence and the hit status of the two-level cache, and begins inference calculation.

[0193] Step S611: Begin reasoning and calculation for the query task.

[0194] Step S612: Confirm whether the Prefix cache has been hit.

[0195] Here, if the Prefix cache is hit, step S613 is executed; if it is not hit, step S614 is executed.

[0196] Step S613: Read the key-value pairs that are hit in the Prefix cache.

[0197] If the target inference node obtains the prefix cache hit information in step S612, the target inference node issues a shared memory read request to the memory management node (i.e., step S624). If reading is allowed, the target inference node reads the key-value pair matched in the prefix cache and reuses the key-value pair for inference computation.

[0198] Step S614: Write the new prefix cache and text segment.

[0199] If the target inference node obtains a prefix cache miss in step S612, the target inference node performs inference calculations, generates new KV key-value pairs and text segments, and sends a shared memory write request to the memory management node. If the write request is agreed upon, the new KV key-value pairs and text segments are written into the shared memory.

[0200] Step S615: Confirm that the Generated cache has been hit.

[0201] Here, if the hit occurs, step S616 is executed; if the hit does not occur, step S617 is executed.

[0202] Step S616: Read the key-value pairs that were hit in the Generated cache.

[0203] If the target inference node obtains the generation cache hit information in step S615, the target inference node reads the matched key-value pair from the generation cache in local memory, and reuses the cache for inference calculation.

[0204] Step S617: Write the new Generated cache and text segment.

[0205] If the target inference node obtains information about a cache miss in step S615, the target inference node performs inference calculations to generate new key-value pairs and text segments. The target inference node writes the newly generated key-value pairs into its local memory and sends the newly generated text segments to the scheduling node.

[0206] Step S618: Weight the Prefix cache.

[0207] During the attention calculation process, the target inference node assigns a weight score to the text segments of each key-value pair in the prefix cache according to the attention weight. The target inference node sends a write request to the memory management node for shared memory, and writes the weight score result to the shared memory if the write is agreed.

[0208] Step S619: The reasoning calculation ends.

[0209] Step S620: Return the generated result.

[0210] Once the inference results are generated, the target inference node will return the results to the scheduling node.

[0211] Step S621: Perform shared memory read / write arbitration.

[0212] The memory management node receives shared memory read / write requests from the scheduling node and all inference nodes, and arbitrates the read / write order of each node based on the memory state.

[0213] Step S622: Determine whether the scheduling node is allowed to read.

[0214] Here, step S602 is allowed to be executed, but step S623 is not allowed to be executed.

[0215] Step S623: Wait.

[0216] Step S624: Perform shared memory read / write arbitration.

[0217] Step S625: Determine whether to allow the target inference node to read or write, or deny the target inference node to write.

[0218] Step S626: Wait.

[0219] Step S627: Execute the Prefix cache eviction policy.

[0220] When the memory management node receives Prefix cache read requests from each inference node, it can count the reuse records of each KV pair in the Prefix cache, run the corresponding eviction policy, and clean up expired Prefix cache.

[0221] This application proposes an adaptive high-hit-rate prefix cache hit determination method, which utilizes the attention weights of intermediate results in inference computation. Based on the cross-attention weights of each key-value pair in the prefix cache during the generation stage, these weights are scored, and the impact of low-weight key-value pairs on prefix cache hits is eliminated or recalculated according to a custom threshold. By employing a fuzzy matching mechanism, the cache reuse rate can be improved, thereby increasing processing speed while maintaining inference accuracy.

[0222] Figure 7 This is a schematic diagram illustrating the algorithm principle for weighting each key-value pair in the prefix cache according to an embodiment of this application, as shown below. Figure 7 As shown, for a model with L layers and H attention heads, assuming that the prefix cache stores M KV pairs, and N steps of computation are performed during the generation of a query, the prefix cache stores several key-value pairs, and the dimensions of both the K vector and the Q vector are d. When performing attention calculation in a certain attention head of a certain layer of the model, the cross-attention weight value corresponding to the i-th K vector when it is generated in the j-th step can be calculated as shown in formula (1):

[0223]

[0224] Where i = 1, 2, ..., M, j = 1, 2, ..., N, and K is a vector. Q vector The sum of the cross-attention weights for all K vectors within a generation step is shown in Equation (2):

[0225]

[0226] During the generation process, each K vector participates in a cross-attention calculation once in each step. Therefore, the expected cross-attention weight of each K vector in this attention head of this layer in all generation steps is as shown in formula (3):

[0227]

[0228] Formula (4) is derived from formula (3):

[0229]

[0230] Similarly, the expected cross-attention weights of each K vector in all attention heads across all layers in all generation steps are shown in Equation (5):

[0231]

[0232] Formula (6) is derived from formula (5):

[0233]

[0234] That is, each key-value pair in the prefix cache has a cross-attention weight expectation during model inference computation, denoted by e. i Characterized by the fact that the expected sum of all KV pairs is 1. Using e i This serves as the weight score for the text segment corresponding to the i-th KVpair in the corresponding Prefix cache. i It is a score for the i-th KV pair already stored in the prefix cache from historical queries, representing the importance of each KV pair currently stored in the prefix cache. It is a statistical result calculated based on the attention weights in historical queries. i It is unrelated to the query sequence currently entered by the user.

[0235] In this embodiment, a threshold interval (i.e., a preset weight interval) [θ1,θ2] can be set, where θ1,θ2∈[0,1), and θ1<θ2. The preset weight interval controls the ambiguity of the prefix cache hit. The smaller θ1,θ2 is, the stricter the cache hit determination, the lower the cache reuse rate, the slower the inference speed, but the higher the inference accuracy.

[0236] Two thresholds θ1 and θ2 control three intervals, e i The three intervals represent the importance of the corresponding i-th KVpair.

[0237] When e i >θ2, corresponding to e i In cases where the pair falls within the maximum range, for highly important key-value pairs, the text must be completely identical for the cache to be considered reusable (i.e., a hit); otherwise, it will lead to inaccurate inference results.

[0238] When e i ≤θ1, corresponding to e i In cases where the key-value pair falls within the minimum interval, even if the text is inconsistent, directly reusing the cache will not significantly affect the inference accuracy. On the contrary, not reusing (but recalculating) will significantly increase the inference time.

[0239] When θ1 <e i ≤θ2, corresponding to e iFor key-value pairs falling within the interval, direct reuse of these pairs with intermediate importance can impact inference accuracy. However, if we don't want to abandon the reuse of subsequent key-value pairs, we can recompile the key-value pair. Recompiling requires using all key-value pairs preceding the current one, consuming significant computation time. Therefore, asynchronous recompiling can be used, where hit checks continue for subsequent content while recompiling, thus masking the time consumption of recompiling.

[0240] Figure 8 This is a flowchart illustrating the process of determining the hit rate of prompt words in the prefix cache, as provided in the embodiments of this application. Figure 8 As shown, the hit determination of the prompt word in the prefix cache can be achieved through steps S801 to S808:

[0241] Step S801: Obtain the prefix cached text segment and its weight score.

[0242] In this embodiment, the scheduling node obtains the prefix cached text segment and its weight score.

[0243] Step S802: Perform text segment matching.

[0244] The scheduling node performs sequential matching and judgment on the Prompt and prefix cached text segments in the query sequence newly entered by the user.

[0245] If a text segment that is inconsistent with the Prompt is found during the text segment matching process, proceed to step S803; if no inconsistent text segment is found, proceed to step S807.

[0246] Step S803: Determine if e i ≤θ1.

[0247] If the text segment corresponds to a weight score of e i If the value is less than or equal to θ1, the inconsistency is ignored, and the matching judgment of subsequent text segments continues until all text segments are traversed, at which point the Prompt is determined to be a cache hit.

[0248] Step S804: Determine if θ1 is correct. <e i ≤θ2.

[0249] If the text segment corresponds to a weight score θ1 <e i If ≤θ2, then some tokens in the text segment are recalculated (attention can be calculated using the KV pair of this token and all previous tokens to obtain a new KV pair), while continuing to match and judge subsequent text segments until all text segments are traversed and the Prompt is determined to be a cache hit.

[0250] Step S805: Partial Token Recalculation.

[0251] Step S806: Prefix cache miss.

[0252] If the weight score corresponding to this text segment is e i If the value is greater than θ2, the matching stops, indicating that the text segment and subsequent text segments have not been cached.

[0253] Step S807: Has the entire text segment been processed?

[0254] Step S808: Prefix cache hit.

[0255] Specifically, the recalculation of tokens in unmatched text segments is performed asynchronously. This is because the recalculation result is only related to all tokens preceding that token and has no relation to subsequent text segments that have not been matched. The asynchronous recalculation process can be processed in parallel with the matching tasks of subsequent text segments. Therefore, the latency caused by asynchronous recalculation can be masked by subsequent processing flows, and the lightweight recalculation operation will not negatively impact the overall inference performance.

[0256] This application implements a prefix cache sharing mechanism with large capacity, low latency, and high bandwidth by using CXL technology. This sharing mechanism has a relatively simple structure, and the memory access initiating node can directly access the shared memory, thus consuming fewer system resources.

[0257] This application provides an adaptive high-hit-rate prefix cache hit determination method. This method utilizes the sparsity characteristics of the attention mechanism to perform fuzzy matching on the Prompt text segments in the cache through cross-attention weight feedback during the generation process. The degree of fuzzy matching can be adjusted by parameters. At the same time, it combines asynchronous lightweight recomputation technology to improve the cache reuse rate while ensuring inference accuracy, thereby improving the overall inference efficiency.

[0258] In this embodiment, a small-capacity but low-latency and high-bandwidth generation cache is deployed locally on the inference node. By using an exact matching mechanism for efficient query scheduling, this generation cache is reused to reduce the computing resources required in the generation stage, thereby further improving the overall inference efficiency.

[0259] In the embodiments of this application, please continue to refer to Figure 1The request scheduling device 154 includes an acquisition module 1541, a first determination module 1542, a second determination module 1543, a third determination module 1544, and a sending module 1545. The acquisition module 1541 is used to acquire a query sequence, which includes a prompt word and query text. The first determination module 1542 is used to determine the first hit information of the prompt word in a prefix cache located in shared memory. The second determination module 1543 is used to determine the second hit information of the query text in a generation cache located in local memory. The generation cache located in local memory. The third determination module 1544 is used to determine a target inference node among multiple local inference nodes to perform inference calculations on the query sequence based on at least one of the first hit information and the second hit information. The sending module 1545 is used to send the first hit information, the second hit information, and the query sequence to the target inference node.

[0260] In some embodiments, the prefix cache is used to store a plurality of first key-value pairs; the first determining module 1542 is further used to obtain a plurality of first text segments corresponding to the plurality of first key-value pairs in the shared memory; the corresponding first key-value pairs and first text segments have the same identifier; perform text matching on the prompt word and the plurality of first text segments to obtain first matching information of the prompt word in the prefix cache; in response to the first matching information indicating that there is a first target text segment in the first text segment that matches the prompt word, determine the first identifier of the prompt word hitting the prefix cache and the first target text segment as the first hit information; in response to the first matching information indicating that there is no first target text segment in the first text segment that matches the prompt word, determine that the prompt word does not hit the prefix cache as the first hit information.

[0261] In some embodiments, the second determining module 1543 is further configured to: obtain the second text segment in the local memory and the load status of each local inference node; perform text matching between the query text and the second text segment to obtain second matching information of the query text in the generation cache; in response to the second matching information indicating that there is a second target text segment in the second text segment that matches the query text, determine that the query text hits the generation cache, the second identifier of the second target text segment, and the load status as the second hit information; in response to the second matching information indicating that there is no second target text segment in the second text segment that matches the query text, determine that the query text does not hit the generation cache and the load status as the second hit information.

[0262] In some embodiments, the first hit information includes at least whether the prompt word hits the prefix cache; the second hit information includes at least whether the query text hits the generation cache and the load status of each local inference node; the third determining module 1544 is further configured to, in response to the first hit information indicating that the prompt word hits the prefix cache and the second hit information indicating that the query text hits the generation cache, determine the first local inference node where the second target KV key-value pair corresponding to the second target text segment matched by the query text is located; wherein, each local inference node corresponds to a local inference memory, and the generation cache in each local inference memory includes a portion of the second KV key-value pairs corresponding to the second text segment, and the second KV key-value pairs in different local inference memories are different; in response to the first local inference node having a normal load status, determine the first local inference node as the target inference node; in response to the first local inference node having an abnormal load status, determine the local inference node that satisfies the first load condition as the target inference node; the target inference node includes at least one local inference node.

[0263] In some embodiments, the first hit information includes at least whether the prompt word hits the prefix cache; the second hit information includes at least whether the query text hits the generation cache and the load status of each local inference node; the third determining module 1544 is further configured to, in response to the first hit information indicating that the prompt word hits the prefix cache and the second hit information indicating that the query text does not hit the generation cache, determine the local inference node whose load status meets the first load condition as the target inference node.

[0264] In some embodiments, the first hit information includes at least whether the prompt word hits the prefix cache; the third determining module 1544 is further configured to, in response to the first hit information indicating that the prompt word does not hit the prefix cache, determine the local inference node whose load state meets the first load condition as the target inference node.

[0265] In some embodiments, the prompt word includes at least one prompt subword, the shared memory includes a plurality of first text segments and a weight score of each first text segment, and each first text segment includes at least one text word; the first determining module 1542 is further configured to, based on the order of the at least one prompt subword in the prompt word, sequentially match it with at least one text word of each first text segment to obtain a matching result; in response to the matching result indicating that there is a first target text segment that is the same as the prompt word, obtain first hit information that the prompt word hits the prefix cache; in response to the matching result indicating that some text words in the first target text segment are different from the prompt word, determine the first hit information of the prompt word in the prefix cache based on the weight score of the first target text segment.

[0266] In some embodiments, the first determining module 1542 is further configured to: obtain first hit information that the prompt word hits the prefix cache in response to the weight score being less than the left endpoint value of a preset weight interval; update the prefix cache based on the partial text words in response to the weight score being within the preset weight interval to obtain first hit information that the prompt word hits the prefix cache; and obtain first hit information that the prompt word does not hit the prefix cache in response to the weight score being greater than the right endpoint value of the preset weight interval.

[0267] In some embodiments, the first determining module 1542 is further configured to recalculate the KV values ​​of the partial text words to obtain target KV key-value pairs corresponding to the partial text words; update the KV key-value pairs corresponding to the first target text segment based on the target KV key-value pairs to obtain updated KV key-value pairs corresponding to the first target text segment; and add the updated KV key-value pairs to the prefix cache.

[0268] It should be noted that the description of the device embodiments in this application is similar to the description of the method embodiments described above, and has similar beneficial effects as the method embodiments; therefore, it will not be repeated. For technical details not disclosed in the device embodiments, please refer to the description of the method embodiments in this application for understanding.

[0269] It should be noted that, in the embodiments of this application, if the above-described request scheduling method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0270] This application provides a storage medium storing executable instructions. When the executable instructions are executed by a processor, the processor will execute the request scheduling method provided in this application.

[0271] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0272] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0273] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0274] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application. It should be understood that "an embodiment" or "one embodiment" mentioned throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence number of the above-described processes does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments of this application are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments.

[0275] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed.

[0276] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A request scheduling method, the request scheduling method comprising: Obtain the query sequence, which includes prompt words and query text; Determine the first hit information of the prompt word in the prefix cache; The prefix cache is located in shared memory; Determine the second hit information of the query text in the generation cache; the generation cache is located in local memory; Based on at least one of the first hit information and the second hit information, a target inference node for inference calculation of the query sequence is determined among multiple local inference nodes; The first hit information, the second hit information, and the query sequence are sent to the target inference node.

2. The request scheduling method according to claim 1, wherein the prefix cache is used to store multiple first KV key-value pairs; Determining the first hit information of the prompt word in the prefix cache includes: Retrieve multiple first text segments corresponding to the multiple first KV key-value pairs from the shared memory; the corresponding first KV key-value pairs and first text segments have the same identifier. The prompt word is matched with the plurality of first text segments to obtain the first matching information of the prompt word in the prefix cache; In response to the first matching information indicating that there is a first target text segment in the first text segment that matches the prompt word, the first identifier of the prompt word hitting the prefix cache and the first target text segment is determined as the first hit information; In response to the first matching information indicating that there is no first target text segment in the first text segment that matches the prompt word, the prompt word not hitting the prefix cache is determined as the first hit information.

3. The request scheduling method according to claim 1, wherein determining the second hit information of the query text in the generation cache includes: Obtain the second text segment in the local memory and the load status of each local inference node; The query text is matched with the second text segment to obtain the second matching information of the query text in the generation cache; In response to the second matching information indicating that there is a second target text segment in the second text segment that matches the query text, the query text hitting the generation cache, the second identifier of the second target text segment, and the load condition are determined as the second hit information; In response to the second matching information indicating that there is no second target text segment in the second text segment that matches the query text, the query text not hitting the generation cache and the load condition are determined as the second hit information.

4. The request scheduling method according to claim 1, wherein the first hit information includes at least whether the prompt word hits the prefix cache; the second hit information includes at least whether the query text hits the generation cache and the load status of each local inference node; The step of determining the target inference node for inference computation on the query sequence among multiple local inference nodes based on at least one of the first hit information and the second hit information includes: In response to the first hit information indicating that the prompt word hits the prefix cache, and the second hit information indicating that the query text hits the generation cache, a first local inference node is determined where the second target KV key-value pair corresponding to the second target text segment matched by the query text is located; wherein, each local inference node corresponds to a local inference memory, and the generation cache in each local inference memory includes some second KV key-value pairs corresponding to the second text segment, and the second KV key-value pairs in different local inference memories are different; In response to the normal load status of the first local inference node, the first local inference node is determined as the target inference node; In response to an abnormal load state of the first local inference node, a local inference node that meets the first load condition is identified as the target inference node; the target inference node includes at least one local inference node.

5. The request scheduling method according to claim 1, wherein the first hit information includes at least whether the prompt word hits the prefix cache; and the second hit information includes at least whether the query text hits the generation cache and the load status of each local inference node; The step of determining the target inference node for inference computation on the query sequence among multiple local inference nodes based on at least one of the first hit information and the second hit information includes: In response to the first hit information indicating that the prompt word hits the prefix cache, and the second hit information indicating that the query text does not hit the generated cache, the local inference node whose load status meets the first load condition is determined as the target inference node.

6. The request scheduling method according to claim 1, wherein the first hit information includes at least whether the prompt word hits the prefix cache; The step of determining the target inference node for inference computation on the query sequence among multiple local inference nodes based on at least one of the first hit information and the second hit information includes: In response to the first hit information indicating that the prefix cache was not hit, the local inference node whose load status meets the first load condition is identified as the target inference node.

7. The request scheduling method according to any one of claims 1 to 6, wherein the prompt word includes at least one prompt subword, the shared memory includes a plurality of first text segments and a weight score of each first text segment, and each first text segment includes at least one text word; Determining the first hit information of the prompt word in the prefix cache includes: Based on the order of the at least one prompt word in the prompt words, it is matched sequentially with at least one text word in each first text segment to obtain the matching result; In response to the matching result indicating that there is a first target text segment that is the same as the prompt word, the first hit information of the prompt word hitting the prefix cache is obtained; In response to the matching result indicating that some text words in the first target text segment are different from the prompt word, the first hit information of the prompt word in the prefix cache is determined based on the weight score of the first target text segment.

8. The request scheduling method according to claim 7, wherein determining the first hit information of the prompt word in the prefix cache based on the weight score of the first target text segment includes: In response to the weight score being less than the left endpoint of a preset weight interval, the first hit information of the prompt word hitting the prefix cache is obtained; In response to the weight score being within a preset weight range, the prefix cache is updated based on the partial text words to obtain the first hit information of the prompt word hitting the prefix cache; In response to the weight score being greater than the right endpoint of a preset weight interval, first hit information is obtained that the prompt word did not hit the prefix cache.

9. The request scheduling method according to claim 8, wherein updating the prefix cache based on the partial text words includes: The key-value pairs of the aforementioned text words are recalculated to obtain the target key-value pairs corresponding to the aforementioned text words; Based on the target key-value pair, the key-value pair corresponding to the first target text segment is updated to obtain the updated key-value pair corresponding to the first target text segment; The updated KV key value is added to the prefix cache.

10. An electronic device, the electronic device comprising: Memory, configured to store computer programs that can run on a processor; When a processor is configured to execute the computer program, it performs the following steps: acquiring a query sequence, the query sequence including prompt words and query text; determining a first hit of the prompt words in a prefix cache; the prefix cache being located in shared memory; Determine the second hit information of the query text in the generation cache; the generation cache is located in local memory; based on at least one of the first hit information and the second hit information, determine the target inference node for inference calculation of the query sequence among multiple local inference nodes; The first hit information, the second hit information, and the query sequence are sent to the target inference node.

Images