Model reasoning method and device

Abstract

The invention provides a model reasoning method and device, and is applied to the technical field of artificial intelligence. The model reasoning method is applied to a first end and comprises the steps that a target key value vector is determined according to a full-quantity key value vector of an input text, the target key value vector comprises a first part, a second part and a third part, the first part comprises key value vectors of head and tail lexical elements, the second part comprises key value vectors of K class clusters, and the third part comprises key value vectors of K class clusters; the K class clusters are class clusters matched with the output first lexical element in the N class clusters, the total key value vector comprises the N class clusters, the third part comprises centroid vectors of the N-K class clusters, and N is greater than K; and sending the target key value vector to the second end to perform decoding calculation.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model reasoning method and apparatus. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in long text processing scenarios (such as document analysis and code generation), the problem of long context reasoning has become increasingly prominent. Summary of the Invention

[0003] In view of this, the present disclosure provides a model reasoning method and apparatus.

[0004] According to a first aspect of this disclosure, a model inference method is provided, applied to a first end, the method comprising: determining a target key value vector based on the full key value vector of the input text, the target key value vector comprising a first part, a second part, and a third part, the first part comprising key value vectors of the first and last words, the second part comprising key value vectors of K clusters, the K clusters being clusters among N clusters that match the first word of the output, the full key value vector comprising N clusters, and the third part comprising centroid vectors of NK clusters, where N is greater than K; and sending the target key value vector to a second end for decoding calculation.

[0005] According to embodiments of this disclosure, determining a target key value vector based on the full key value vector of the input text includes: clustering the full key value vector to obtain N clusters and a key value vector corresponding to each cluster; determining the key value vectors of the first and last words from the N clusters; determining K clusters from the N clusters that match the first word of the output; and determining the centroid vectors of the NK clusters based on the NK clusters.

[0006] According to an embodiment of this disclosure, the method further includes: determining the centroid vectors of the N clusters based on the key value vectors of the N clusters; and saving the key value vectors of the N clusters and the centroid vectors of the N clusters to the main memory of the first end.

[0007] According to embodiments of this disclosure, the method further includes: receiving a target request from a second end and determining at least one incremental centroid vector; sending at least one incremental centroid vector to the second end; wherein the incremental centroid vector represents a centroid vector that is different from the NK centroid vectors in the target key value vector required for decoding the A-th word, and A is an integer greater than 1.

[0008] According to embodiments of this disclosure, clustering the full set of key-value vectors to obtain N clusters and key-value vectors corresponding to each cluster includes: dividing the initial attention matrix corresponding to the input text into multiple attention blocks; determining multiple target attention blocks corresponding to the input text based on the multiple attention blocks; performing attention sparsity calculation based on the multiple target attention blocks to obtain a target attention matrix, wherein the target attention matrix represents the degree of correlation between multiple words in the input text; and dividing the full set of key-value vectors into N clusters based on the target attention matrix, and determining the key-value vectors corresponding to each cluster.

[0009] According to embodiments of this disclosure, determining multiple target attention blocks corresponding to input text based on multiple attention blocks includes: performing anti-diagonal sampling on the attention blocks, calculating the probability value of the anti-diagonal covered elements, where the probability value characterizes the importance of the attention block; if the sum of the target probability values ​​is greater than a first threshold, the attention block corresponding to the target probability value is taken as the target attention block.

[0010] According to embodiments of this disclosure, based on the target attention matrix, the full key-value vector is divided into N clusters, and the key-value vector corresponding to each cluster is determined, including: if the attention score between multiple words in the target attention matrix is ​​greater than a second threshold, the key-value vectors corresponding to the multiple words in the full key-value vector are divided into a cluster, and the attention score represents the degree of association between the multiple words.

[0011] According to a second aspect of this disclosure, a model inference method is provided, applied to a second end. The method includes: receiving a target key-value vector sent by a first end, the target key-value vector being obtained through the following operations: determining the target key-value vector based on the full key-value vector of the input text, the target key-value vector including a first part, a second part, and a third part, the first part including the key-value vectors of the first and last words, the second part including the key-value vectors of K clusters, the K clusters being the clusters among N clusters that match the first word of the output, the full key-value vector including N clusters, and the third part including the centroid vectors of NK clusters; performing decoding calculations based on the target key-value vector, and sequentially outputting multiple words.

[0012] According to an embodiment of this disclosure, decoding calculation is performed based on the target key value vector to sequentially output multiple lexical units, including: decoding calculation of the target key value vector to determine the second lexical unit; using the second lexical unit as the initial lexical unit, decoding calculation is performed sequentially on the centroid vectors of the L clusters corresponding to the Ath lexical unit to determine the (A+1)th lexical unit, where A is an integer greater than 1; and outputting the (A+1)th lexical unit.

[0013] According to an embodiment of this disclosure, taking the second word as the initial word, the centroid vectors of the L clusters corresponding to the Ath word are decoded and calculated sequentially to determine the A+1th word, including: traversing the target key value vector; if the centroid vectors of the L clusters are the same as the centroid vectors of the NK clusters in the target key value vector, the centroid vectors of the L clusters are decoded and calculated to determine the A+1th word.

[0014] According to an embodiment of this disclosure, using the second term as the initial term, the centroid vectors of the L clusters corresponding to the Ath term are decoded and calculated sequentially to determine the (A+1)th term. This includes: traversing the target key-value vector; if the centroid vectors of the L clusters are not the same as the centroid vectors of the NK clusters in the target key-value vector, generating a target request and sending it to the first end; the target request is used to instruct the acquisition of an incremental centroid vector from the memory of the first end; the incremental centroid vector is the centroid vector among the L clusters that is different from the centroid vectors of the NK clusters; receiving the incremental centroid vector sent by the first end; decoding and calculating the centroid vectors of the L clusters, and outputting the (A+1)th term.

[0015] A third aspect of this disclosure provides a model inference apparatus, disposed at a first end, comprising: a first memory; and a first processor, configured to determine a target key value vector based on the full key value vector of the input text, the target key value vector comprising a first part, a second part, and a third part, the first part comprising key value vectors of the first and last words, the second part comprising key value vectors of K clusters, the K clusters being clusters among N clusters that match the first word of the output, the full key value vector comprising N clusters, and the third part comprising centroid vectors of NK clusters, the target key value vector being sent to the second end for decoding calculation, where N is greater than K.

[0016] The fourth aspect of this disclosure provides a model inference apparatus, disposed at a second end, comprising: a second memory; and a second processor for receiving a target key-value vector sent from a first end, the target key-value vector being obtained through the following operations: determining the target key-value vector based on the full key-value vector of the input text, the target key-value vector comprising a first part, a second part, and a third part, the first part comprising key-value vectors of the first and last words, the second part comprising key-value vectors of K clusters, the K clusters being the clusters among N clusters that match the first word of the output, the full key-value vector comprising N clusters, and the third part comprising centroid vectors of NK clusters; performing decoding calculations based on the target key-value vector, and sequentially outputting multiple words.

[0017] A fifth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the model inference method described above.

[0018] A sixth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described model inference method.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 This diagram schematically illustrates an application scenario of the model reasoning method and apparatus according to embodiments of the present disclosure.

[0022] Figure 2 A flowchart illustrating a model inference method according to an embodiment of the present disclosure is shown, wherein the method is applied to a first end;

[0023] Figure 3 A schematic diagram illustrating the centroid vector according to an embodiment of the present disclosure is shown.

[0024] Figure 4 A schematic diagram illustrating block attention dynamic selection according to an embodiment of the present disclosure is shown.

[0025] Figure 5 A flowchart illustrating a model reasoning method according to an embodiment of the present disclosure is shown, wherein the method is applied to a second end;

[0026] Figure 6A The diagram illustrates the interaction between the two ends of the model inference method according to an embodiment of the present disclosure;

[0027] Figure 6B A flowchart illustrating the interaction between the two ends of the model inference method according to an embodiment of the present disclosure is shown schematically.

[0028] Figure 7A One of the effect diagrams of the model reasoning method according to an embodiment of the present disclosure is illustrated schematically;

[0029] Figure 7B A second schematic diagram illustrating the effect of the model reasoning method according to an embodiment of the present disclosure is shown.

[0030] Figure 8A A block diagram of a model inference apparatus according to an embodiment of the present disclosure is shown schematically;

[0031] Figure 8BA second block diagram of a model inference apparatus according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0032] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0033] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0034] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0035] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0037] This disclosure provides a model reasoning method. Before introducing the technical solutions provided by this disclosure, the relevant technologies involved in this disclosure will be explained.

[0038] With the widespread application of Large Language Models (LLMs) in long text processing scenarios (such as document analysis and code generation), the problem of long context reasoning has become increasingly prominent.

[0039] The inference process of large language models is divided into two stages: prefill and decoding. The prefill stage processes the input in parallel, and its computational complexity increases quadratically with the context length, making it computationally intensive. The decoding stage generates output through autoregression, relying on a large-scale key-value cache (KV cache), making it memory-intensive. In the prefill and decoding (P / D) separated architecture, the prefill instance calculates the key-value vector and transmits it over the network to the decoding instance to complete the inference. In long context applications, the computational cost of prefill increases significantly with sequence expansion, leading to excessively high first-word latency (TTFT). Simultaneously, the key-value cache size expands rapidly with model size and context length, causing a surge in GPU memory consumption, limiting batch processing scale and reducing system throughput. Furthermore, cross-instance KV cache transmission is highly sensitive to network bandwidth and latency; insufficient transmission efficiency will block the decoding process and delay inference output.

[0040] To address the aforementioned issues, many related technologies improve inference performance by optimizing attention computation and memory overhead. A typical example is sparse attention, whose basic principle is to reduce the full interaction between the query and the key, retaining only the most important connections. Implementation methods mainly include: fixed-pattern sparsity, such as sliding windows, expanded attention, or block sparsity, to limit the computational scope; content-based dynamic sparsity, such as using Locality Sensitive Hashing (LSH) or Top-k filtering to compute attention only for the most relevant keys; and hierarchical or combinatorial sparsity, such as combining global and local attention to reduce overall complexity while ensuring the transmission of a small amount of global information. Through these mechanisms, the computational complexity of attention can be reduced from quadratic to near-linear, thus supporting efficient inference with very long contexts.

[0041] However, sparse attention methods in related technologies are mainly aimed at long-sequence inference scenarios on a single machine or single card, and have not solved the core bottleneck of P / D separation architecture. In P / D separation, the performance bottleneck mainly comes from the quadratic increase in computation during the pre-filling stage and the overhead of KVcache in memory usage and cross-node transmission during the encoding stage. The optimization goal of sparse attention is limited to reducing computational complexity and single-machine memory overhead, and cannot reduce the overall storage size of KVcache, nor can it alleviate the bandwidth and latency issues of cross-device transmission. Therefore, this type of method has limited effect on shortening TTFT or improving decoding parallelism, and is difficult to directly apply to P / D separation inference systems.

[0042] While sparsification methods in related technologies can effectively reduce the computational pressure of long-context inference on a single machine, their optimization dimensions are fundamentally mismatched with the core bottleneck of the P / D separation architecture. Their shortcomings can be summarized as "unable to solve the fundamental bottleneck of KVCache storage and transmission".

[0043] In the P / D separation architecture, the encoding instance requires a complete KVCache for autoregressive generation. Regardless of the sparse attention computation used by the pre-filled instance, it ultimately needs to generate and transmit a complete, dense KVCache to the decoding instance. Sparsity reduces computational load, not storage load.

[0044] Current sparse attention schemes are completely unable to reduce the overall size of the KVCache. Therefore, the memory bottleneck of decoding instances, the limitation on batch processing size due to the large KVCache, and the ultra-high bandwidth and low latency requirements for transferring the massive KVCache across the network—all these challenges caused by storage scale remain, and may even be further complicated by the metadata introduced by sparse computing models. Furthermore, while pre-filling sparsity accelerates pre-filling computation, potentially allowing the KVCache to be prepared and flooded onto the network faster, it also places higher and more urgent demands on network bandwidth and latency. If the network becomes a bottleneck, it can quickly block the entire system.

[0045] Secondly, such solutions offer limited benefits in reducing first-to-last-word latency: In single-machine scenarios, sparse attention significantly shortens the TTFT by reducing the computational complexity of the pre-filling stage (from O(n²) to ~O(n)). However, in a P / D separation architecture, the TTFT is typically determined by the combined factors of pre-filling computation time, KVCache network transmission time, and the decoding time of the first token. While sparsity optimizes the pre-filling computation time, it is ineffective against the newly introduced critical bottleneck of network transmission time. If network latency is high or bandwidth is insufficient, the computational time saved by pre-filling may be offset by the long transmission time, resulting in insignificant overall TTFT optimization.

[0046] Before providing a further detailed description of the embodiments of this disclosure, the nouns and terms involved in the embodiments of this disclosure will be explained, and the nouns and terms involved in the embodiments of this disclosure shall be interpreted as follows.

[0047] The pre-filling phase occurs after the AI ​​model receives the complete input request, but before it begins generating the first output token. The main tasks of this phase are processing the input request, computing the context representation of all input tokens, and initializing the key-value cache required for the subsequent decoding phase.

[0048] The decoding phase begins after the pre-filling phase is complete. In this phase, the model generates output tokens one by one in an autoregressive manner. Each generated token is added to the already generated sequence and used as input for the next generation.

[0049] Self-attention allows a model to focus on other elements in the input sequence while processing a single element, updating the representation of the current element based on the degree of correlation between these elements. Self-attention is achieved by computing query vectors, key vectors, and value vectors. The attention process includes: first, for each element in the input sequence, a linear transformation is performed to obtain query vectors, key vectors, and value vectors; then, an attention score is calculated by multiplying the current query vector with each target key vector, representing the degree of attention between elements; finally, the value vectors are weighted and summed based on the normalized attention scores (i.e., attention weights) to obtain the self-attention output (i.e., the attention representation).

[0050] An attention scoring function measures the relevance or matching degree between a query vector and a key vector. Its core function is to determine which parts of the input sequence are most relevant to the current output when the model is processing the current task. By calculating the similarity scores between the query vector and each key vector, the attention scoring function provides the basis for subsequent Softmax normalization and weighted summation of values, thereby enabling the focus and selective utilization of key information. Common attention scoring methods include dot product, scaled dot product, and additive attention, which affect how the model pays attention to contextual information and its computational efficiency in different scenarios.

[0051] Figure 1 The diagram illustrates an application scenario of the model reasoning method and apparatus according to embodiments of the present disclosure.

[0052] like Figure 1 As shown, the application scenario according to this embodiment may include a terminal 100, a server 200, and a network 300. The server 200 includes a first server 201 and a second server 202. A large language model may be deployed in the server 200 to execute the model inference method of this embodiment. The first server 201 is used to execute the operation of the large language model in the pre-filling stage; here, the first server 201 corresponds to the first end in this embodiment. The second server 202 is used to operate the method of the large language model in the decoding stage; here, the second server 202 corresponds to the second end in this embodiment. The inference result of the large language model can be achieved through the cooperation of the first server 201 and the second server 202 in the server 200.

[0053] It should be noted that server 200 can be a single server or a server cluster, or it can be a cloud server or cloud server cluster capable of providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. It should be understood that the server mentioned herein is typically a server computer with ample memory and processor resources, but other embodiments are also possible. Furthermore, server 200 is shown only as an example; in practice, other devices or combinations of devices with computing and storage capabilities can be used alternatively or additionally to provide the corresponding services.

[0054] Network 300 serves as a medium for providing a communication link between terminal 100 and server 200. Network 300 also serves as a medium for providing a communication link between first server 201 and second server 202. Network 300 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0055] Users can use terminal 100 to interact with server 200 via network 300 to receive or send messages, etc. Various communication client applications can be installed on terminal 100, such as information retrieval applications, question-and-answer applications, web browser applications, search applications, etc. (for example only).

[0056] Terminal device 100 can be various electronic devices with a display screen and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0057] Server 200 can be a server that provides various services, such as a backend management server that supports the website browsed by the user using the first terminal 100 (for example only). The backend management server can analyze and process data such as received user requests (e.g., input text), and feed back the processing results (e.g., web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0058] For example, server 200 can receive user input data through the front-end interactive interface provided by terminal 100 to obtain user request data. Server 200 can input the user request data into a large language model. First server 201 uses the large language model to understand and analyze the user request data in the pre-filling stage, generating a key-value vector corresponding to the user request data. For example, first server 201 is used to determine the target key-value vector based on the full key-value vector of the input text. The target key-value vector includes a first part, a second part, and a third part. The first part includes the key-value vectors of the first and last words, the second part includes the key-value vectors of K clusters, where K clusters are the clusters that match the first word of the output among N clusters, and the full key-value vector includes N clusters. The third part includes the centroid vectors of NK clusters, where N is greater than K. The target key-value vector is sent to the second end (second server 202) for decoding calculation. Second server 201 uses the large language model to decode and calculate the key-value vector in the decoding stage to generate the request result corresponding to the user request data. For example, the second server 201 is used to receive the target key value vector sent by the first end. The target key value vector is obtained through the following operations: the target key value vector is determined based on the full key value vector of the input text. The target key value vector includes a first part, a second part, and a third part. The first part includes the key value vectors of the first and last words. The second part includes the key value vectors of K clusters. The K clusters are the clusters that match the first word in the output among N clusters. The full key value vector includes N clusters. The third part includes the centroid vectors of NK clusters. The target key value vector is decoded and calculated, and multiple words are output sequentially.

[0059] It should be understood that Figure 1 The number of terminals, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0060] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0061] The following will be based on Figure 1 The described scene, through Figures 2-4 The model reasoning method applied to the first end according to the embodiments of this disclosure will be described in detail.

[0062] The first and second ends are two core components in the P / D separation architecture that are functionally decoupled and have dedicated resources. They work together to realize large language model inference services.

[0063] The first end can be a compute node or server configured and optimized to perform the pre-population phase. The first end can be equipped with a high-performance GPU. For example, the first end can be a server, terminal device, smart network interface card, edge computing device, etc.

[0064] The second end can be a computing node or server configured and optimized for performing the decoding phase. The second end can be a GPU equipped with high-bandwidth memory. For example, the second end can be a server, cloud device, smart network interface card, edge computing device, etc.

[0065] Figure 2 A flowchart illustrating a model inference method according to an embodiment of the present disclosure is shown, wherein the method is applied to a first end. The first end is used to perform operations of a large language model in the pre-filling stage.

[0066] like Figure 2 As shown, the model inference method of this embodiment includes operations S210 to S220.

[0067] In operation S210, the target key value vector is determined based on the full key value vector of the input text. The target key value vector includes a first part, a second part, and a third part. The first part includes the key value vectors of the first and last words. The second part includes the key value vectors of K clusters. The K clusters are the clusters that match the first word of the output among the N clusters. The full key value vector includes N clusters. The third part includes the centroid vectors of NK clusters, where N is greater than K.

[0068] In operation S220, the target key-value vector is sent to the second end for decoding calculation.

[0069] For example, input text can be used by models (such as large language models) to generate output text, answer questions, perform translations, generate summaries, and perform other tasks. Input text is typically natural language text, i.e., language used by humans in daily life, such as English or Chinese. Input text can be in various forms, including but not limited to: sentences, paragraphs, dialogues, keywords, phrases, structured data, and multimodal inputs (e.g., a combination of images, audio, and text input).

[0070] Input text can also include cue messages. Cue messages are a special type of input used to guide or instruct the model to process the input text or generate output text in a specific way. Cue messages can be a simple phrase, sentence, or a series of instructions designed to activate relevant knowledge and capabilities within the model.

[0071] Users can input text through the user interface provided by the terminal device. For example, they can input text information through physical input devices (such as keyboards and / or mice) or virtual buttons (such as touchscreens), or through voice or gestures.

[0072] The input text is converted into a sequence of semantic vectors. Each semantic vector in this sequence is mapped to a key-value vector corresponding to each token in the input text; this is the full key-value vector. Here, a token is a word (lexicon) in the input text. The full key-value vector includes key vectors, value vectors, and the correspondence between them. The key vector (also known as the K vector) represents the token's position within the context; the value vector (also known as the V vector) represents the token's actual content or features.

[0073] The target key-value vector can be a subset of the full key-value vector, and the transmission volume of the target key-value vector is less than that of the full key-value vector. After pre-filling, the first end uses a clustering algorithm (such as K-Means) to divide the full KV cache into N clusters. Each cluster contains KV vectors of tokens with similar semantics. The target key-value vector consists of three parts: First, the key-value vectors corresponding to the first and last tokens of the input text. Second, all key-value vectors of the Top-K clusters corresponding to the first token of the output text, where N is greater than K. Third, the centroid vectors of the NK clusters. Here, the Top-K clusters can be understood as the K clusters among the N clusters that have the highest correlation with the first token of the output text. This embodiment does not impose a specific limitation on the setting of K; the size of K can be set according to the actual application.

[0074] A centroid vector is a condensed representation of all key-value vectors within the same cluster. It contains the core semantics of all tokens in each cluster. For example, a centroid vector could be the average vector of all key vectors within each cluster.

[0075] The target key-value vector is sent to the second end for decoding and calculation to output the result corresponding to the input text.

[0076] Understandably, the first end only needs to transmit a small amount of the target key-value vector to the second end for decoding calculation. Compared to transmitting the full key-value vector to the second end, this reduces the amount of data transmitted from the first end to the second end, thereby reducing network transmission load. Furthermore, due to the smaller amount of data transmitted, the network transmission time from the first end sending the processed data to the second end and receiving the first decoded token is significantly shortened, thus reducing the user-perceived TTFT. Simultaneously, the memory usage required for decoding at the second end is significantly reduced due to the simplification of the KV vector, enabling the second end to simultaneously handle and process more concurrent dialogue context, thereby greatly improving the overall throughput of the entire model inference system.

[0077] As described above, in operation S210, the target key-value vector is determined based on the full key-value vector of the input text. In one possible implementation, this operation may further include the following operations: clustering the full key-value vector to obtain N clusters and a key-value vector corresponding to each cluster; determining the key-value vectors of the first and last words from the N clusters; determining K clusters from the N clusters that match the first word of the output; and determining the centroid vectors of the K clusters based on the K clusters.

[0078] In one example, when clustering the full set of key-value vectors, the K-Means clustering algorithm or other suitable clustering methods can be used to divide them into N clusters based on the semantic similarity between key-value vectors. Each cluster contains key-value vectors corresponding to tokens with high semantic relevance, while maintaining relative semantic independence between clusters.

[0079] When determining the key-value vectors of the first and last tokens from N clusters, the system identifies the affiliation of the first and last tokens in the clustering results of the input text sequence and extracts the key-value vectors corresponding to these first and last tokens separately.

[0080] When determining the K clusters from N clusters that match the first word of the output, the system first generates the first word of the output sequence (output result), and then calculates the similarity score between the query vector of this word and the centroid vectors of each cluster. Based on the similarity ranking, the system selects the K clusters with the highest scores as the clusters that best match the first word of the output. The key-value vectors in these K clusters are fully preserved to ensure the accuracy of the decoding calculation.

[0081] When determining the centroid vectors of NK clusters, for the remaining (NK) clusters that were not selected, the centroid representation of all key-value vectors within each cluster is calculated. The centroid vectors are generated by weighted averaging or other aggregation methods on the key-value vectors within the cluster, which can effectively compress storage space while preserving the core semantic information of the cluster.

[0082] For example, suppose the input text is a technical document containing 1024 tokens. After clustering, it forms 128 clusters (N=128). The system identifies the 16 clusters (K=16) most relevant to the first token of the output. In this case, the target key-value vector will include: the key-value vectors of the first and last tokens (2 tokens each), the complete key-value vectors of the 16 matching clusters (assuming an average of 8 tokens per cluster, for a total of 128 tokens), and the centroid vectors of the remaining 112 clusters. Compared to transmitting the key-value vectors corresponding to all 1024 tokens, the target key-value vector only requires transmitting the equivalent data of approximately 130 tokens, reducing data transmission by approximately 87%.

[0083] Understandably, by clustering and selecting key-value vectors, the storage and transmission overhead of key-value vectors can be significantly reduced while maintaining decoding accuracy. The clustering method ensures centralized management of semantically relevant information, the preservation of the first and last words maintains the integrity of the text boundaries, the selection of Top-K clusters ensures accuracy in the initial stage of decoding, and the use of centroid vectors preserves the semantic essence while compressing storage, thus achieving an effective balance between inference efficiency and accuracy.

[0084] Figure 3 A schematic diagram of the centroid vector according to an embodiment of the present disclosure is shown.

[0085] As described above, the model inference method of this embodiment may further include the following operations: determining the centroid vectors of N clusters based on the key value vectors of N clusters; and saving the key value vectors and centroid vectors of N clusters to the main memory of the first end.

[0086] For example, the centroid vector mathematically represents the center point of all key vectors within the cluster in the vector space.

[0087] The main memory at the first end can be a hardware memory in the first-end computing device used to temporarily store working data that is being directly processed and accessed by the CPU or a dedicated processor (such as a GPU or NPU). For example, the main memory at the first end can be the CPU memory at the first end, or it can be the smart network card at the first end.

[0088] In one example, refer to Figure 3 When determining the centroid vector based on the key-value vectors of N clusters, a mathematical aggregation operation is performed on all key-value vectors within each cluster. The centroid vector can be calculated using an arithmetic mean method, which involves averaging the corresponding dimensions of all key vectors and the corresponding dimensions of all value vectors within the same cluster, thus obtaining the representative key and value vectors for that cluster. Alternatively, the centroid vector can be calculated using a weighted average, where the weights are determined based on the importance score of the tokens in the attention calculation. As a concise representation of clusters, the centroid vector can preserve the core semantic features of the clusters while occupying less storage space.

[0089] The key-value vectors and centroid vectors of N clusters are stored in the CPU memory at the first end, which can be achieved using a hierarchical storage strategy. The complete cluster key-value vector data is stored in main memory to support dynamic retrieval needs during the decoding stage. Simultaneously, the centroid vectors corresponding to all clusters are also stored in a dedicated area of ​​main memory for fast access and similarity calculation. The storage layout can employ a cluster index structure, assigning a unique identifier to each cluster and establishing a mapping relationship between the centroid vector and the complete key-value vector.

[0090] Compared to the aforementioned mechanism that only transmits a portion of the target key-value vector, this storage strategy further optimizes the overall performance of the system. The first end stores the full amount of data in main memory, ensuring the accessibility of key-value vectors for any cluster during decoding, while the pre-computation of centroid vectors significantly improves the efficiency of subsequent similarity matching.

[0091] Understandably, by establishing a complete centroid vector calculation and storage mechanism, the system achieves efficient integration between data processing in the pre-filling stage and dynamic invocation in the decoding stage. The pre-calculation of the centroid vector avoids redundant computational overhead during decoding, while the hierarchical storage strategy optimizes access efficiency while ensuring data integrity, providing a solid data foundation for the high-performance operation of the entire inference system.

[0092] As described above, the model inference method of this embodiment may further include the following operations: receiving a target request from the second end and determining at least one incremental centroid vector; sending at least one incremental centroid vector to the second end; wherein, the incremental centroid vector represents a centroid vector that is different from the NK centroid vectors in the target key value vector required for decoding the Ath word, and A is an integer greater than 1.

[0093] For example, the target request can be a signal generated by the second end during the decoding process if the key value vector required for decoding calculation by the second end during the generation of the Ath word is different from the target key value vector in the second end.

[0094] The incremental centroid vector can be the centroid vector of a cluster whose centroid vector required for decoding calculation when generating the Ath word in the second end is different from the NK centroid vectors of the target key value vector in the second end.

[0095] During the ongoing decoding process, the second end needs to dynamically adjust the required key-value vector combinations based on the semantic changes of the generated lexical units. Based on the complete centroid vector data stored in the main memory of the first end, the system can respond to the real-time needs of the second end and provide targeted incremental centroid vector supplementation.

[0096] In one example, during the generation of the Ath word, if the centroid vector required for decoding calculation during the generation of the Ath word is different from the NK centroid vectors of the target key value vector in the second end, the centroid vector that is different from the NK centroid vectors is determined, i.e., the incremental centroid vector, and the target request is generated and sent to the first end.

[0097] After receiving the target request from the second end, the first end calculates the similarity between the query vector of the A-th word provided by the second end and the centroid vectors of all clusters stored in the main memory of the first end. Through comparative analysis, the incremental centroid vector is determined and then sent to the second end.

[0098] When sending the incremental centroid vector to the second end, the system employs a lightweight transmission protocol, transmitting only the necessary centroid vector data. During transmission, the incremental centroid vector carries cluster identification information, facilitating its integration with existing key-value vector data at the second end.

[0099] For example, based on the decoding example in the aforementioned technical document, suppose that when generating the third term in the output, the decoding content shifts from a technical overview to specific implementation details. The second end finds that the current 16 matching clusters mainly contain overview semantic information, but lack key-value vectors related to implementation details. The second end sends a target request to the first end, requesting the addition of key-value vectors related to "algorithm implementation" and "performance optimization". After receiving the request, the first end calculates the similarity between the query vector of the third term and the centroid vectors of the remaining 112 clusters, identifies 3 clusters highly related to implementation details, and sends the centroid vectors of these 3 clusters as incremental centroid vectors to the second end. When decoding the fifth term in the subsequent decoding, the decoding focus further shifts to "test verification" content, and the first end responds to the request again, adding centroid vectors of 2 clusters related to testing.

[0100] Compared to a static target key-value vector transmission mechanism, this dynamic supplementation strategy achieves precise matching of key-value vector requirements during the decoding process. By transmitting incremental centroid vectors on demand, the system avoids resource waste from excessive data transmission in the initial stage, while ensuring the timeliness and sufficiency of semantic support during decoding.

[0101] Understandably, the incremental centroid vector mechanism achieves a dynamic balance between transmission efficiency and decoding accuracy in the inference system. Based on the complete data storage capacity of the first end and real-time demand feedback from the second end, this mechanism constructs an adaptive key-value vector scheduling system. This enables the entire inference process to intelligently allocate resources according to the semantic evolution of the decoded content, thereby maximizing the overall efficiency of the system while ensuring generation quality.

[0102] Figure 4 A schematic diagram illustrating block attention dynamic selection according to an embodiment of the present disclosure is shown.

[0103] As described above, the operation involves clustering the entire key-value vector to obtain N clusters and a key-value vector corresponding to each cluster. In one possible implementation, this operation may further include the following steps: dividing the initial attention matrix corresponding to the input text into multiple attention blocks; determining multiple target attention blocks corresponding to the input text based on the multiple attention blocks; performing attention sparsity computation on the multiple target attention blocks to obtain a target attention matrix, which represents the degree of association between multiple words in the input text; and dividing the entire key-value vector into N clusters based on the target attention matrix, and determining the key-value vector corresponding to each cluster.

[0104] In one example, the first end can guide the clustering process of key-value vectors through sparse attention computation. The initial attention matrix is ​​the complete attention weight matrix obtained during the pre-filling stage through the dot product of the query vector and the key vector. The dimension of this matrix is ​​the length of the input sequence multiplied by the length of the input sequence. For a long text input containing 30,720 tokens, the initial attention matrix will reach a size of 30,720 × 30,720, containing approximately 940 million attention weight values.

[0105] Dividing the initial attention matrix into multiple attention blocks is the first step in sparsity processing. Specifically, the initial attention matrix can be divided according to a preset block size. For example, refer to... Figure 4 Let the size of the attention block be B×B, where B is the side length parameter of the block. Then, for an input text sequence of length L, the initial attention matrix will be divided into (L / B)×(L / B) attention blocks. For example, when B is set to 64 and the input sequence length is 30720, the attention matrix will be divided into 480×480 64×64 attention blocks, for a total of 230400 attention blocks. Each attention block contains 4096 attention weight values, representing the local attention relationships within the corresponding word / meta region.

[0106] An anti-diagonal sampling strategy can be used to evaluate the importance of each attention block. Specifically, for each attention block, attention weight values ​​are sampled at specific locations along its anti-diagonal direction, and the cumulative sum of these sampled values ​​is calculated as the importance score for that attention block. This process can be represented as:

[0107]

[0108] Where A is the attention mapping matrix, B is the set of attention blocks, |B| represents the number of attention blocks in the set, and b is the element in the attention block covered by the diagonal.

[0109] The most important attention block in the attention mapping matrix can be obtained based on the threshold τ. After selection, partial attention calculation is performed using a mask to reduce the overall computational cost.

[0110] For example, for a 64×64 attention block, the element positions (0,63), (1,62), ..., (63,0) on the anti-diagonal can be sampled, and the attention weight values ​​at these 64 positions are summed to obtain the block's score. Then, the scores of all attention blocks are sorted, and the set of attention blocks with the smallest cumulative score exceeding a preset threshold τ is selected as the target attention block. For example, when τ is set to 0.8, the highest-scoring attention blocks are selected such that the sum of their scores reaches more than 80% of the total score of all attention blocks.

[0111] The selected target attention blocks undergo full attention computation, while unselected attention block regions are filled with zeros or estimated using interpolation based on neighboring blocks. For example, assuming 46,080 target attention blocks (approximately 20%) are selected from 230,400 attention blocks, only these 46,080 blocks need to undergo full attention computation, including softmax normalization, reducing computational cost by about 80% compared to full computation. The target attention matrix obtained in this way retains the main features of the original attention pattern while significantly reducing computational complexity. Each element in the target attention matrix represents the strength of the association between corresponding word pairs; a larger value indicates a stronger semantic association between words.

[0112] The process of dividing the entire set of key-value vectors into N clusters based on the target attention matrix achieves key-value vector organization based on semantic relevance. Specifically, by analyzing the distribution of attention scores among words in the target attention matrix, combinations of words with strong correlations are identified. When the attention scores between multiple words in the target attention matrix are all greater than a second threshold, the key-value vectors corresponding to these words are grouped into the same cluster. For example, setting the second threshold to 0.1, if the pairwise attention scores between words A, B, and C all exceed 0.1, then the key-value vectors corresponding to these three words are grouped into the same cluster. By traversing the entire target attention matrix, multiple such strongly correlated word combinations can be identified, ultimately forming N clusters. The key-value vectors within each cluster have similar semantic features and contextual correlation patterns, providing a foundation for subsequent selective transmission and dynamic loading.

[0113] Understandably, clustering methods based on sparse attention matrices, compared to traditional vector similarity-based clustering algorithms, can more accurately capture the contextual dependencies between words, thus generating clustering results with stronger semantic coherence. This clustering approach not only considers the feature similarity of the key-value vectors themselves but also fully utilizes the association patterns between words revealed by the attention mechanism, enabling key-value vectors within a cluster to work together better during the reasoning process and improving the accuracy of sparse inference.

[0114] As described above, the operation involves determining multiple target attentions corresponding to the input text based on multiple attention blocks. In one possible implementation, this operation may further include: performing anti-diagonal sampling on the attention blocks, calculating the probability value of the anti-diagonal covered elements, where the probability value characterizes the importance of the attention block; and if the sum of the target probability values ​​is greater than a first threshold, then the attention block corresponding to the target probability value is taken as the target attention block.

[0115] In one example, the process of anti-diagonal sampling of attention blocks is as follows: For each B×B attention block, elements are selected along the anti-diagonal direction from the bottom left to the top right. The anti-diagonal refers to the diagonal path within the attention block from coordinates (B-1,0) to coordinates (0,B-1), and the elements on this path represent the attention relationship between word pairs at different distances. For example, in a 64×64 attention block, the anti-diagonal contains 64 elements located at coordinates (63,0), (62,1), (61,2), ..., (1,62), (0,63). The attention values ​​at these positions reflect the complete association spectrum from the maximum word distance to the minimum word distance.

[0116] Calculating the probability value of the anti-diagonal covered elements is a key step in quantifying the importance of an attention block. Specifically, first, all element values ​​along the anti-diagonal of the attention block are extracted, and then these values ​​are summed to obtain the anti-diagonal sum. Subsequently, the anti-diagonal sum is divided by the sum of all elements within the attention block to obtain its probability value. For example, suppose the sum of the anti-diagonal elements of a 64×64 attention block is 25.6, and the sum of all 4096 elements within the block is 128.0. Then, the probability value of this attention block is 25.6 / 128.0 = 0.2. This probability value characterizes the concentration of the main attention weights within the attention block along the anti-diagonal direction. A higher value indicates that the attention pattern within the block more closely matches the typical distance decay characteristic, and therefore, its importance is higher.

[0117] The target probability value refers to the highest probability value selected from all the probability values ​​of the attention blocks, arranged in descending order. The attention blocks corresponding to these target probability values ​​represent the most important attention calculation areas in the input text. For example, for the aforementioned 230,400 attention blocks, after calculating 230,400 corresponding probability values, the top few with the highest probability values ​​are selected in descending order as the target probability values. Assuming the top 46,080 highest probability values ​​are selected as target probability values, these values ​​may include relatively high probability values ​​such as 0.35, 0.32, and 0.28.

[0118] When the sum of the target probability values ​​exceeds a first threshold, the attention block corresponding to each target probability value is identified as the target attention block. The first threshold is a key parameter controlling the degree of sparsity, typically set between 0.8 and 0.9. For example, when the first threshold is set to 0.85, a sufficient number of the highest probability values ​​need to be selected so that the cumulative sum of these probability values ​​reaches more than 85% of the total sum of all probability values. The specific calculation process is as follows: First, calculate the sum of all 230,400 probability values, assuming it to be 100.0; then, accumulate the probabilities sequentially from largest to smallest, stopping the selection when the cumulative sum reaches 85.0; finally, identify the attention blocks corresponding to all probability values ​​involved in the accumulation process as target attention blocks. This method ensures that the selected target attention blocks concentrate 85% of the important attention information in the input text.

[0119] Understandably, the probability calculation method based on anti-diagonal sampling can more accurately identify the intrinsic importance of attention blocks. Anti-diagonal sampling fully considers the distribution pattern of attention weights within the block, and avoids the influence of absolute numerical differences on the selection results through probability quantification, making the selection of target attention blocks more stable and reliable. This refined selection strategy further improves the accuracy of sparse attention calculation, providing higher-quality semantic association information for subsequent key-value vector clustering based on the target attention matrix.

[0120] As described above, the operation involves dividing the entire key-value vector into N clusters based on the target attention matrix and determining the key-value vector corresponding to each cluster. In one possible implementation, this operation may further include the following step: if the attention score among multiple words in the target attention matrix is ​​greater than a second threshold, the key-value vectors corresponding to the multiple words in the entire key-value vector are divided into a single cluster, where the attention score characterizes the degree of association between the multiple words.

[0121] For example, based on the aforementioned determination of the target attention block and generation of the target attention matrix through anti-diagonal sampling, the attention score information in this matrix needs to be used to guide the accurate clustering of the full set of key-value vectors. The target attention matrix retains the most important inter-word association information in the input text, where each element represents the attention weight between corresponding word pairs, providing a quantitative basis for key-value vector grouping based on semantic association.

[0122] Attention score is a numerical metric used in the target attention matrix to measure the strength of the association between lexical units. Specifically, for the element at position (i,j) in the target attention matrix, its value represents the attention weight of the i-th lexical unit to the j-th lexical unit. This weight reflects the degree to which lexical unit i depends on lexical unit j during understanding and processing. Attention scores are usually normalized using softmax and range from 0 to 1. The closer the value is to 1, the stronger the semantic association between lexical units. For example, when processing the sentence "The weather is nice today, let's go for a walk in the park," the attention score between the lexical units "park" and "walk" might reach 0.23, while the attention score between the lexical units "today" and "walk" might only be 0.08, indicating that "park" and "walk" have a stronger semantic association.

[0123] The second threshold is a key parameter for determining whether there is a strong correlation between word pairs, used to distinguish between important and general associations. The setting of the second threshold needs to balance the tightness and coverage of clustering, and is typically set to a value between 0.15 and 0.25. For example, when the second threshold is set to 0.2, only word pairs with an attention score greater than 0.2 are considered to have a strong correlation. In the previous example, the attention score of 0.23 between "park" and "walk" is greater than the second threshold of 0.2, so these two word pairs will be classified into the same cluster; while the attention score of 0.08 between "today" and "walk" is less than the second threshold, so they will not be directly classified into the same cluster.

[0124] Determining the attention score relationships between multiple word units is the core step in the clustering process. Specifically, it is necessary to examine each possible word unit combination in the target attention matrix and determine whether the attention scores between all word unit pairs within the combination are greater than a second threshold. When a group of word units satisfies the condition that the attention scores between every pair of units are greater than the second threshold, these word units are considered to form a semantically coherent and strongly associated group. For example, for the word unit combination {A,B,C}, it is necessary to verify whether the attention scores Attention(A,B), Attention(A,C), Attention(B,A), Attention(B,C), Attention(C,A), and Attention(C,B) are all greater than the second threshold of 0.2. Assuming these six attention scores are 0.28, 0.31, 0.25, 0.33, 0.27, and 0.29 respectively, since all scores are greater than 0.2, word units A, B, and C will be classified into one cluster.

[0125] The process of grouping key-value vectors corresponding to multiple words in the full key-value vector set into clusters achieves key-value vector organization based on semantic relevance. Specifically, once a group of words with strong relevance is identified, the key-value vectors corresponding to these words are extracted from the full key-value vector set and categorized into the same cluster. For example, assuming word A is located at position 15 of the input sequence, word B at position 23, and word C at position 31, then the 15th, 23rd, and 31st key-value vectors will be extracted from the full key-value vector set to form a cluster containing three key-value vectors. The key-value vectors within this cluster will be processed as a whole in subsequent inference processes, maintaining the integrity of the original semantic relevance.

[0126] By traversing all word combinations in the target attention matrix, multiple groups of words that satisfy the strong association condition can be identified, ultimately forming N semantically coherent clusters. The size of each cluster may vary; some clusters may contain 2 key-value vectors, while others may contain 10 or more key-value vectors, depending on the semantic structure of the input text and the association patterns between words.

[0127] Understandably, clustering methods based on attention score thresholds can more accurately reflect the contextual dependencies between words compared to traditional vector distance clustering algorithms, ensuring that key-value vectors within the same cluster are semantically strongly correlated. This clustering approach fully leverages the semantic modeling capabilities of the attention mechanism, resulting in clusters that are not only similar in vector space but, more importantly, semantically interconnected, providing a high-quality key-value vector organization structure for subsequent selective transmission and dynamic reasoning.

[0128] It should be noted that the embodiments disclosed herein do not impose specific limitations on the size of the first threshold and the second threshold, and the size of the first threshold and the second threshold can be adjusted according to actual application requirements.

[0129] The following will be based on Figure 1 The described scene, through Figure 5 The model reasoning method applied to the second end according to the embodiments of this disclosure will be described in detail.

[0130] Figure 5 A flowchart illustrating a model inference method according to an embodiment of the present disclosure is shown, wherein the method is applied to a second end. The second end is used to perform operations of a large language model during the decoding phase.

[0131] like Figure 5 As shown, the model reasoning method of this embodiment includes operations S310 to S320.

[0132] In operation S310, the target key value vector sent by the first end is received. The target key value vector is obtained through the following operation: the target key value vector is determined based on the full key value vector of the input text. The target key value vector includes a first part, a second part, and a third part. The first part includes the key value vectors of the first and last words. The second part includes the key value vectors of K clusters. The K clusters are the clusters that match the first word of the output among the N clusters. The full key value vector includes N clusters. The third part includes the centroid vectors of NK clusters.

[0133] During operation of S320, decoding calculations are performed based on the target key value vector, and multiple tokens are output sequentially.

[0134] For example, the second end acts as a decoding node, specifically responsible for receiving the sparsed target key-value vector and performing the autoregressive generation task.

[0135] The description of the target key vector can be found above. Figure 2 The description in the text will not be repeated here.

[0136] After receiving the target key-value vector, the second end loads this data into the GPU memory for decoding calculations.

[0137] The multiple words output are the components of the output result corresponding to the input text.

[0138] In each decoding step, a fixed number of key-value vector data are maintained in the second-end video memory, specifically including: A first-word key-value vectors, B last-word key-value vectors, K×M key-value vectors of selected clusters, and NK centroid vectors, totaling approximately A+B+K×M+(NK) key-value data corresponding to words.

[0139] Understandably, by adopting the above-mentioned sparse decoding strategy, the second end can significantly reduce memory usage while maintaining the generation quality.

[0140] As described above, in operation S320, decoding calculations are performed based on the target key-value vector, and multiple tokens are output sequentially. In one possible implementation, this operation may further include the following steps: decoding calculations are performed on the target key-value vector to determine the second token; using the second token as the initial token, the centroid vectors of the L clusters corresponding to the A-th token are decoded and calculated sequentially to determine the (A+1)-th token, where A is an integer greater than 1; and the (A+1)-th token is output.

[0141] In one example, the decoding process employs a progressive autoregressive generation approach, where each lexical unit is generated based on the contextual information of the preceding lexical unit and dynamically selected key-value vector data. Upon receiving the target key-value vector, the second end first utilizes the key-value vectors of the first and last lexical units, the key-value vectors of the K initial clusters, and the centroid vectors of the NK clusters to generate the second lexical unit through an attention mechanism.

[0142] After the second word is generated, the decoding process transitions to the iterative generation phase. Using the second word as the initial generated word, for each subsequent word to be generated (where A is greater than 1), the second end needs to calculate the similarity between the query vector of the A-th word and the centroid vectors of all available clusters, thereby determining the centroid vectors of the L clusters most relevant to the current generation state.

[0143] Once the second end determines the L most relevant clusters required for the Ath word, it performs decoding calculations based on the centroid vectors of the L clusters, outputting the (A+1)th word. This process is repeated to output multiple words in sequence, serving as the output result corresponding to the input text.

[0144] As described above, the operation involves: using the second lexical unit as the initial lexical unit, sequentially decoding and calculating the centroid vectors of the L clusters corresponding to the Ath lexical unit to determine the (A+1)th lexical unit. In one possible implementation, this operation can further include: traversing the target key-value vector; if the centroid vectors of the L clusters are the same as the centroid vectors of the NK clusters in the target key-value vector, decoding and calculating the centroid vectors of the L clusters to determine the (A+1)th lexical unit.

[0145] In one example, during the generation of the (A+1)th word, after the Ath word is generated, the system calculates the similarity score between its query vector and the centroid vectors of all clusters, and selects the top L most relevant clusters according to the scores from highest to lowest. These L clusters constitute the core contextual information set required to generate the (A+1)th word.

[0146] For the required L centroid vectors of clusters, check if the L centroid vectors of clusters are among the existing NK centroid vectors of clusters at the second end. If the L centroid vectors of clusters belong to the NK centroid vectors of clusters, then directly call the L centroid vectors of clusters from the NK centroid vectors of clusters for attention calculation.

[0147] For example, suppose that when generating the 5th word, the system determines that the centroid vector of the cluster with cluster number [3,7,15] is needed for calculation. By traversing the target key-value vector, it is found that the cluster [3,7,15] exists in the centroid vectors of the NK clusters initially transmitted. The centroid vector of the cluster with cluster number [3,7,15] can be directly called from the video memory at the second end for decoding calculation, and the 5th word is output.

[0148] As described above, the operation involves: using the second lexical as the initial lexical, sequentially decoding and calculating the centroid vectors of the L clusters corresponding to the Ath lexical to determine the (A+1)th lexical. In another possible implementation, this operation can further include: traversing the target key-value vector; if the centroid vectors of the L clusters are different from the centroid vectors of the NK clusters in the target key-value vector, generating a target request and sending it to the first end. The target request instructs the first end to retrieve the incremental centroid vector from its memory; the incremental centroid vector is the centroid vector among the L clusters that differs from the centroid vectors of the NK clusters; receiving the incremental centroid vector sent by the first end; decoding and calculating the centroid vectors of the L clusters, and outputting the (A+1)th lexical.

[0149] In one example, when the second-end discovers that the required L clusters exceed the coverage of the target key-value vector during the generation of the (A+1)th lexical unit, it initiates an incremental data request mechanism. This typically occurs when the contextual focus shifts significantly during the generation process, such as topic transitions in long text generation or user questions in dialogue systems involving new knowledge domains. The second-end iterates through the cluster indices in the target key-value vector and compares them to find that some clusters among the centroid vectors of the L clusters required for the current generation step are not in the set of NK centroid vectors.

[0150] The generation of target requests employs a batch processing approach to improve network transmission efficiency. The second end first counts all missing cluster IDs and encapsulates these IDs into a structured request message. The target request includes a request identifier, session ID, a list of missing cluster indices, and request priority. Request priority is determined based on the similarity score between the cluster and the current query vector; clusters with higher similarity have higher transmission priority, ensuring that critical context information reaches the second end first.

[0151] After generating the target request and sending it to the first end, the second end enters an asynchronous waiting state. To avoid complete blocking of the generation process, the second end adopts a hybrid computation strategy: attention calculation continues for existing cluster data, while missing clusters are temporarily handled using zero vectors or global average vectors. Although this temporary processing method will affect the computational accuracy to some extent, it can maintain the continuity of the generation process and avoid system response interruptions due to network latency.

[0152] Upon receiving a target request, the first end retrieves the corresponding incremental centroid vector data from its memory. Incremental centroid vectors refer to those cluster centroid vectors that were not included in the initial target key-value vector transmission but are identified as necessary in the current generation step. Based on the cluster index list in the request, the first end extracts the corresponding centroid vectors from the pre-calculated full cluster data and sorts and packages them according to request priority. To reduce network transmission overhead, the first end also performs compression encoding on the incremental centroid vectors.

[0153] Upon receiving the incremental centroid vectors, the second end immediately loads this data into the current computational context, replacing the previous placeholder vectors. The updated set of L cluster centroid vectors contains the complete contextual information needed to generate the (A+1)th lexical unit. The second end then re-executes the attention calculation, using the updated centroid vector data to generate an accurate attention weight distribution, thereby determining the generation probability of the (A+1)th lexical unit.

[0154] For example, when generating the 8th lexical unit, the system determines that 150 clusters with cluster numbers [2,6,13,28,35] are needed. Through traversal, it is found that cluster [2,6,13] can be found in the target key-value vector, but 47 clusters, including [28,35], are not in the NK centroid vectors. The second end generates a target request containing the indices of these 47 missing clusters and sends it to the first end. The first end retrieves the corresponding 47 incremental centroid vectors and transmits them to the second end. Upon receiving this, the second end updates its computation context and generates the 8th lexical unit using the complete information from the 150 clusters.

[0155] Understandably, the dynamic incremental request mechanism ensures that the decoding computation can adapt to the changing context requirements during the generation process. This incremental request mechanism trades slightly increased network communication overhead for higher generation accuracy, thus improving the accuracy of the output results.

[0156] To facilitate understanding of the model reasoning method in the embodiments of this disclosure, the following will be combined with... Figure 6A and 6B This will be further explained through the interaction process between the two ends.

[0157] The pre-filling and decoding phases of the model inference are located on different servers. The pre-filling phase is implemented through the first terminal, and the decoding phase is implemented through the second terminal.

[0158] Figure 6A The diagram illustrates the interaction between the two ends of the model inference method according to an embodiment of the present disclosure; Figure 6B A flowchart illustrating the interaction between the two ends of the model reasoning method according to an embodiment of the present disclosure is shown schematically.

[0159] Reference Figure 6B The pre-filling stage includes operations S410 to S460.

[0160] In operation S410, the initial attention matrix corresponding to the input text is divided into multiple attention blocks.

[0161] In operation S420, multiple target attention blocks are determined based on multiple attention blocks, corresponding to the input text.

[0162] In operation S430, attention sparse calculation is performed based on multiple target attention blocks to obtain the target attention matrix.

[0163] In operation S440, based on the target attention matrix, the full key-value vector is divided into N clusters, and the key-value vector corresponding to each cluster is determined.

[0164] In operation S450, the target key value vector is determined.

[0165] Send the target key-value vector to the second end.

[0166] In operation S460, the key-value vectors of N clusters and the centroid vectors of N clusters are saved to the main memory at the first end.

[0167] The decoding stage includes operations S510 to S570.

[0168] In operation S510, the target key value vector sent by the first end is received. Decoding and calculation are performed based on the target key value vector to obtain the second word.

[0169] In operation S520, the second word is used as the initial word. Based on the output A-th word, the centroid vectors of the L clusters corresponding to the A-th word are determined.

[0170] In operation S530, the target key value vector is traversed to determine whether the centroid vectors of the L clusters are the same as the centroid vectors of the NK clusters in the target key value vector.

[0171] During operation S540, if the centroid vectors of the L clusters are different from the centroid vectors of the NK clusters in the target key vector, a target request is generated and sent to the first end.

[0172] When operating the S550, the incremental centroid vector sent by the first end is received.

[0173] In operation S560, the centroid vectors of L clusters are decoded and calculated to determine the (A+1)th word.

[0174] If the centroid vectors of the L clusters are not the same as the centroid vectors of the NK clusters in the target key value vector, then operation S560 is executed directly.

[0175] In operation S570, if the (A+1)th word carries a sequence end marker, then the (A+1)th words output sequentially will be used as the output result corresponding to the output text.

[0176] If the (A+1)th word does not carry a sequence end marker, then repeat operations S520 to S570.

[0177] For example, the model inference method of this embodiment may include two phases executed sequentially. The first phase is a pre-filling phase, and the second phase is a loop decoding phase.

[0178] In the pre-filling stage, the model receives the input text and divides the initial attention matrix into multiple attention blocks. For each attention block, attention weight values ​​at specific positions along its anti-diagonal direction are sampled, and the cumulative sum of these sampled values ​​is calculated as the importance score of that attention block. All attention block scores are sorted, and the set of attention blocks with the smallest cumulative score exceeding a preset threshold τ is selected as the target attention block, generating a target attention matrix. The attention score information in the target attention matrix guides the clustering of all key-value vectors. By traversing all word combinations in the target attention matrix, multiple groups of words satisfying strong association conditions can be identified, ultimately forming N semantically coherent clusters. The key-value vectors of the first and last words, the top-K clusters matching the first word of the output from the N clusters, and the centroid vectors of the NK clusters are sent as target key-value vectors to the second end. Simultaneously, the key-value vectors of the N clusters and the centroid vectors of the N clusters are stored in the CPU memory of the first end.

[0179] The loop decoding stage model is a process in which multiple tokens of the output result are generated step by step through loop iteration based on the target key value vector sent in the pre-filling stage.

[0180] In the second-end decoding process, the key-value vector in the decoding stage can be divided into three components: a fixed region, which contains the first and last tokens of the request, as these tokens contribute significantly to subsequent attention calculations and are retained; a retrieval region, in which the Q-vector of the current token is dynamically calculated and the K centroids of the clusters generated by clustering are identified, and the KV vectors corresponding to the K centroids are included in the cache of this calculation; and an estimation region, where the KV data corresponding to the remaining unmatched clusters are estimated by weighting the corresponding centroid vectors to ensure that contextual information is preserved as much as possible.

[0181] The second term is obtained by decoding the target key-value vector. The centroid vectors of the L clusters corresponding to the second term are determined. If the centroid vectors of the L clusters exist in the existing centroid vectors of the NK clusters at the second end, they are directly retrieved from the second end's memory for decoding to obtain the third term. If the centroid vectors of the L clusters do not exist in the existing centroid vectors of the NK clusters at the second end, a target request is generated and sent to the first end. The first end retrieves the incremental centroid vectors (i.e., the centroid vectors of the clusters whose centroid vectors differ from the existing centroid vectors of the NK clusters at the second end) that do not exist at the second end. The incremental centroid vectors are integrated with the existing centroid vectors of the clusters at the second end to obtain the centroid vectors of the L clusters. The centroid vectors of the L clusters are then decoded to obtain the third term. The centroid vectors of the L clusters corresponding to the third term are determined… This process continues until the (A+1)th word carries a sequence end marker. Then, the (A+1)th words output sequentially are fed back to the user as the output of the input text.

[0182] The model reasoning method using embodiments of this disclosure has the following characteristics:

[0183] 1. Sparse inference method covering the entire process: For PD-separated large model inference systems, it can realize sparsity acceleration of the entire request lifecycle, including pre-filling, KV transmission and decoding.

[0184] 2. Pre-filled sparsity acceleration: By using cross-attention sampling, only attention blocks with high attention score weights are calculated, which greatly reduces the computational requirements and effectively reduces the arrival time of the first token for long text requests.

[0185] 3. Efficient cross-node KV cache transmission: Through selective, multi-stage, asynchronous KV cache transmission strategy, the bandwidth requirements of the inference system are reduced, and data transmission is prevented from becoming a bottleneck of the inference system.

[0186] 4. Decoding Sparsity Acceleration: From the perspective of GPU memory, a KV cache selective loading strategy with controllable GPU memory cache usage is implemented, avoiding memory explosion of decoding nodes under long text and high concurrency; from the perspective of IO, through the dynamic selective loading strategy of KV cache, the movement of KV cache between CPU memory and GPU memory is greatly reduced, while effectively preserving the model inference accuracy, achieving a balance between efficiency and accuracy.

[0187] The model reasoning method using the embodiments of this disclosure has the following effects:

[0188] 1. Reduce request latency: The TTFT of long text requests can be effectively controlled, and the longer the text is, the more obvious the reduction effect is;

[0189] 2. Improve system throughput: The selective loading strategy of KV cache can effectively control the memory usage of inference requests during the decoding stage, thereby supporting a larger decoding batch size and thus improving throughput;

[0190] 3. Reduce the high requirements of the cluster network for PD separation deployment scheme: The KV buffer transmission volume from the pre-filled node to the decoding node is effectively controlled through sparsification, which reduces the transmission requirements on the network bandwidth between nodes.

[0191] The following is an experimental comparison between the model inference method of this disclosure embodiment and the conventional model inference method.

[0192] The sparse attention acceleration effect during the pre-filling stage is as follows:

[0193] Taking a request with an input length of 30k as an example, the comparison of TTFT and throughput between the conventional attention calculation method and the sparsity force acceleration method is shown in Table 1 below:

[0194]

[0195] As can be seen from the analysis of Table 1, the model inference method of this disclosure has an average TTFT and throughput improvement of more than 10% compared with the conventional attention calculation method of related technologies.

[0196] The acceleration effect of sparse attention during the decoding phase is as follows:

[0197] For a request with an input length of 30k, taking model A as an example, the comparison of KVCache usage per request is shown in Table 2 below:

[0198]

[0199] As shown in Table 1, compared with conventional attention techniques, the model inference method of this disclosure reduces the KV cache usage during the decoding stage by approximately 3.5 times, thereby supporting higher request concurrency and improving the overall system throughput. Similarly, the KVCache transmission volume between the initial pre-filling and decoding nodes is reduced by 3.5 times, which can alleviate the network transmission pressure on the inference cluster.

[0200] Figure 7A One of the effect diagrams of the model reasoning method according to an embodiment of the present disclosure is illustrated schematically.

[0201] Reference Figure 7A The test objective is to evaluate the inference performance gains of enabling the VLLM sparse attention mechanism for the M model in long text scenarios under different input lengths, focusing on quantifying the changes in TTFT, ITL, and total throughput.

[0202] Test configuration: Hardware: L20*1 (TP4); Model: M (sparse attention can be enabled / disabled); Inference framework: vllm-cvx-0.9.3; Input length: 30K–1M; Total requests: 100; Concurrency: 1.

[0203] Test results: On the M-model, after adopting the model inference method of this application, the throughput, TTFT and other inference metrics are significantly improved in scenarios with a maximum length of 1M tokens, and the success rate of long sequences is higher.

[0204] Specifically, in the 300–400K tokens range, the model (sparse version) using the model inference method of this disclosure will improve throughput by about 30%; when it reaches 800K, the baseline model (the conventional model that does not use the model inference method of this disclosure) will basically no longer be generated, while the model using the model inference method of this disclosure can still run.

[0205] TTFT: Across all test lengths, models employing the model inference method of this disclosure can reduce TTFT, with a speed improvement ranging from approximately 16% to 27% (e.g. at 400K); at 800K and above, the baseline model (a conventional model not employing the model inference method of this disclosure) does not produce tokens, while models employing the model inference method of this disclosure continue to operate in the range of 100–1M.

[0206] ITL: Models employing the model inference method of this disclosure can reduce per-token latency by an average of about 50%, and continue to improve above 800K.

[0207] Reference: Improvement rate = ((Sparse Attention Enabled - Community Version Model) ÷ Community Version Model) × 100%. Wherein, Sparse Attention Enabled refers to the model using the model inference method of this disclosure embodiment, the community model is the baseline model, and the conventional model does not use the model inference method of this disclosure embodiment.

[0208] Figure 7B The diagram illustrates the effect of the model reasoning method according to an embodiment of the present disclosure.

[0209] Reference Figure 7B The test objective is to evaluate the performance changes of the inference method of the model in this disclosure under different concurrency levels in long text scenarios, with a focus on quantifying TTFT, ITL and total throughput.

[0210] Test configuration: Hardware: L20*1 (TP4); Model: M (sparse attention can be enabled / disabled); Inference framework: vllm-cvx-0.9.3; Maximum concurrency: 1–100; Total requests: 100.

[0211] Test results: After combining ICI sparse attention, the M-model using the model inference method of this embodiment improves throughput, shortens first-word time, and accelerates token generation for long contexts under multiple concurrency scenarios.

[0212] Among them, throughput: at low concurrency (such as concurrency 1), the model using the model inference method of this disclosure accelerates the processing of 30K tokens by 37.84%; as the concurrency increases to 10–100, the improvement rate is between 22.61% and 18.24% (logarithmic scale).

[0213] TTFT: Improved across all concurrency levels—up to 28.5% at 50 concurrency and at least 13% at 100 concurrency (logarithmic scale).

[0214] ITL: Decreases across all concurrency levels—63.4% at concurrency 1, 51-17% at concurrency 5–100 (and approximately 12% further at concurrency 100, logarithmic scale).

[0215] Reference: Improvement rate = ((Enable sparse attention - Community Edition model) ÷ Community Edition model) × 100%.

[0216] Reference information: TTFT: TimeToFirstToken, the time when the first token is output. ITL: Inter-TokenLatency, the inter-token latency. CommunityModel: The baseline model without sparse attention enabled, i.e., the model that does not use the model inference method of this disclosure embodiment; EnableSparseAttention: The version with sparse attention enabled, i.e., the model that uses the model inference method of this disclosure embodiment. Imp: Represents the percentage improvement relative to the baseline.

[0217] Based on the above-described model reasoning method, this disclosure also provides a model reasoning apparatus. The following will be combined with... Figure 8A and Figure 8B The device is described in detail.

[0218] Figure 8A A block diagram of a model inference apparatus according to an embodiment of the present disclosure is schematically shown, wherein the apparatus is applied to a first end.

[0219] like Figure 8A As shown, the model inference device 600 of this embodiment includes a first memory 610 and a first processor 620.

[0220] The first memory 610 and the first processor 620 are used to determine the target key value vector based on the full key value vector of the input text. The target key value vector includes a first part, a second part, and a third part. The first part includes the key value vectors of the first and last words. The second part includes the key value vectors of K clusters. The K clusters are the clusters that match the first word of the output among N clusters. The full key value vector includes N clusters. The third part includes the centroid vectors of NK clusters. The target key value vector is sent to the second end for decoding calculation, where N is greater than K.

[0221] In some embodiments, the first memory 610 may be CPU memory, and the first processor 620 may be a GPU.

[0222] For the description of other features, please refer to the description of the model reasoning method applied to the first end above, which will not be repeated here.

[0223] Figure 8B A block diagram of a model inference apparatus according to an embodiment of the present disclosure is shown schematically, wherein the apparatus is applied to a second end.

[0224] like Figure 8B As shown, the model inference device 700 of this embodiment includes: a second memory 710 and a second processor 720.

[0225] The second memory 710 and the second processor 720 are used to receive the target key value vector sent by the first end. The target key value vector is obtained through the following operations: the target key value vector is determined based on the full key value vector of the input text. The target key value vector includes a first part, a second part, and a third part. The first part includes the key value vectors of the first and last words. The second part includes the key value vectors of K clusters. The K clusters are the clusters that match the first word in the output among N clusters. The full key value vector includes N clusters. The third part includes the centroid vectors of NK clusters. The target key value vector is decoded and calculated, and multiple words are output sequentially.

[0226] In some embodiments, the second memory 710 may be CPU memory, and the second processor 720 may be a GPU.

[0227] For descriptions of other features, please refer to the description of the model reasoning method applied to the second end above, which will not be repeated here.

[0228] In some embodiments of this disclosure, one or more computer-readable storage media are provided, having stored thereon computer-readable instructions that, when executed, implement a model inference method according to some embodiments of this disclosure. The various steps of the model inference method according to some embodiments of this disclosure can be programmed into computer-readable instructions and stored in the computer-readable storage medium. When such a computer-readable storage medium is read or accessed by a computing device or computer, the computer-readable instructions therein are executed by a processor on the computing device or computer to implement the method described according to some embodiments of this disclosure.

[0229] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods described in the embodiments of this application.

[0230] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0231] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0232] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A model reasoning method applied to a first end, the method comprising: Based on the full key-value vector of the input text, a target key-value vector is determined. The target key-value vector includes a first part, a second part, and a third part. The first part includes the key-value vectors of the first and last words. The second part includes the key-value vectors of K clusters. The K clusters are the clusters that match the first word of the output among N clusters. The full key-value vector includes N clusters. The third part includes the centroid vectors of NK clusters, where N is greater than K. The target key vector is sent to the second end for decoding calculation.

2. The method according to claim 1, wherein determining the target key-value vector based on the full key-value vector of the input text, includes: Cluster the full set of key-value vectors to obtain N clusters and a key-value vector corresponding to each cluster; Determine the key-value vectors of the first and last words from the N clusters; From the N clusters, determine K clusters that match the first word of the output; Based on NK clusters, determine the centroid vectors of the NK clusters.

3. The method according to claim 2, further comprising: Determine the centroid vectors of the N clusters based on their key value vectors; Save the key vectors and centroid vectors of the N clusters to the main memory of the first end.

4. The method according to claim 1, further comprising: Receive a target request from the second end and determine at least one incremental centroid vector; Send at least one incremental centroid vector to the second end; Wherein, the incremental centroid vector represents the centroid vector that is different from the NK centroid vectors in the target key value vector, which is required for decoding the A-th word. A is an integer greater than 1.

5. The method according to claim 2, wherein clustering the full set of key-value vectors to obtain N clusters and a key-value vector corresponding to each cluster includes: The initial attention matrix corresponding to the input text is divided into multiple attention blocks; Based on the multiple attention blocks, determine multiple target attention blocks corresponding to the input text; Attention sparse computation is performed on the multiple target attention blocks to obtain a target attention matrix, which represents the degree of correlation between multiple lexical units of the input text; Based on the target attention matrix, the full key-value vector is divided into N clusters, and the key-value vector corresponding to each cluster is determined.

6. The method according to claim 5, wherein determining a plurality of target attention blocks corresponding to the input text based on the plurality of attention blocks includes: The attention block is sampled along its anti-diagonal line, and the probability value of the anti-diagonal line covering the element is calculated. The probability value represents the importance of the attention block. If the sum of the target probability values ​​is greater than the first threshold, the attention block corresponding to the target probability value is taken as the target attention block.

7. The method according to claim 5, wherein the full key-value vector is divided into N clusters based on the target attention matrix, and the key-value vector corresponding to each cluster is determined, comprising: If the attention score among multiple words in the target attention matrix is ​​greater than the second threshold, the key-value vectors corresponding to the multiple words in the full key-value vector are divided into a cluster, and the attention score represents the degree of association between the multiple words.

8. A model reasoning method applied to a second end, the method comprising: The target key value vector sent by the first end is received. The target key value vector is obtained through the following operations: the target key value vector is determined based on the full key value vector of the input text. The target key value vector includes a first part, a second part, and a third part. The first part includes the key value vectors of the first and last words. The second part includes the key value vectors of K clusters. The K clusters are the clusters that match the first word of the output among N clusters. The full key value vector includes N clusters. The third part includes the centroid vectors of NK clusters. Decoding is performed based on the target key value vector, and multiple tokens are output sequentially.

9. The method according to claim 8, wherein decoding calculation is performed based on the target key value vector to sequentially output multiple tokens, including: The target key value vector is decoded and calculated to determine the second word; Using the second word as the initial word, the centroid vectors of the L clusters corresponding to the Ath word are decoded and calculated sequentially to determine the (A+1)th word, where A is an integer greater than 1; Output the A+1 words.

10. A model reasoning device, disposed at a first end, comprising: First memory; A first processor is configured to determine a target key-value vector based on the full key-value vector of the input text. The target key-value vector includes a first part, a second part, and a third part. The first part includes the key-value vectors of the first and last words. The second part includes the key-value vectors of K clusters, where the K clusters are the clusters that match the first word of the output among N clusters. The full key-value vector includes N clusters. The third part includes the centroid vectors of NK clusters. The target key-value vector is sent to a second end for decoding calculation, where N is greater than K.

Images