Token-level cache matching method and system of large language model and storage medium

Abstract

The invention discloses a Token level cache matching method and system of a large language model and a storage medium. The method comprises the following steps: constructing a local context fragment; generating a context embedding vector of the current Token, and calculating a context entropy value, a semantic consistency index and a semantic change gradient; for the target Token, determining a semantic category of the target Token; inputting the semantic category to which the target Token belongs into a Hash decision maker to obtain a Hash granularity level; if the Hash is the first-level Hash, executing fixed-length Hash and mapping the Hash to a semantic cache bucket based on a semantic theme or a context range of the first-level Hash; if the first-level hash is the second-level hash, dynamically adjusting the hash length according to the semantic similarity with the adjacent Token; if the Hash granularity level of the target Token is a third-level Hash, constructing a high-dimension context representation and executing a fine Hash operation; and matching the hash result with the KV in the pre-stored cache, and executing corresponding operation according to the matching result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a token-level cache matching method, system and storage medium for a large language model. Background Technology

[0002] With the widespread application of Large Language Models (LLMs), inference efficiency and computational resource overhead have become important optimization directions. To improve inference performance, existing systems can adopt caching mechanisms to store intermediate results (such as key-value pairs) when the model processes long text sequences, so that historical calculation results can be reused in subsequent token inference stages, reducing redundant operations.

[0003] Currently, most mainstream key-value caching methods rely on static position indexes or fixed context ranges for cache key matching, which can achieve some computational speedup in certain scenarios. However, when faced with the complex and varied contexts of natural language, especially semantically similar but differently expressed token sequences, traditional matching methods often struggle to accurately identify and hit the cache, leading to redundant calculations and impacting inference efficiency.

[0004] Furthermore, existing technologies have limited consideration of differences in the semantic position, stability, or syntactic features of different tokens within the context, and caching strategies generally lack flexibility and fine-grained processing mechanisms, which to some extent limits further improvement in cache utilization.

[0005] In summary, existing large language model inference caching mechanisms still have room for improvement in dealing with complex semantic structures and improving cache hit rate. Summary of the Invention

[0006] To address the aforementioned technical issues, this application provides a token-level caching matching method, system, and storage medium for a large language model.

[0007] The technical solution provided in this application is described below: The first aspect of this application provides a token-level cache matching method for large language models, the method comprising: For each token in the token sequence to be processed, a local context fragment is constructed based on the adjacent tokens in its context window; The local context fragment is input into a predefined semantic embedding model to generate the context embedding vector of the current token, and the context entropy value, semantic consistency index and semantic change gradient are calculated based on the context embedding vector. For a target token, the semantic category of the target token is determined based on its context embedding vector, context entropy value, semantic consistency, and semantic change gradient. The semantic category to which the target token belongs is input into a pre-configured hash decision maker to obtain the hash granularity level applicable to the target token. The hash granularity level includes at least: first-level hash, second-level hash, and third-level hash. If the target token's hash granularity level is first-level hash, then a fixed-length hash is performed based on its semantic topic or context scope and mapped to the semantic class cache bucket; If the target token's hash granularity level is second-level hash, then the hash length is dynamically adjusted based on the semantic similarity with adjacent tokens, and variable-length hashing is performed in token fragments with high semantic consistency. If the target token's hash granularity level is third-level hash, then the BERT model is used to parse syntactic roles, semantic changes, and nested dependencies, construct a high-dimensional context representation, and perform fine-grained hash operations. The hash result is matched with the key-value pairs in the pre-stored cache, and the corresponding operation is performed based on the matching result.

[0008] Optionally, if the hash granularity level of the target token is a first-level hash, then performing a fixed-length hash and mapping it to a semantic class cache bucket based on its semantic topic or context scope includes: Based on the target token's position in the context window and its semantic category, determine the target token's semantic topic or context scope, and extract the corresponding context feature information; The context features of the target token are hashed using a fixed-length hash algorithm to generate the hash value of the target token. Based on the generated hash value, the target token is mapped to the corresponding semantic class cache bucket, which corresponds to a set of tokens with similar semantic features.

[0009] Optionally, if the target token's hash granularity level is second-level hashing, the hash length is dynamically adjusted based on the semantic similarity with adjacent tokens, and variable-length hashing is performed on token fragments with high semantic consistency, including: Based on the semantic similarity between the target token and adjacent tokens, the semantic similarity score of adjacent tokens is calculated, and the hash length is dynamically adjusted according to the semantic similarity score. Based on the adjusted hash length, a variable-length hash operation is performed within the context fragment consisting of the target token and its adjacent tokens to generate the corresponding hash result.

[0010] Optionally, dynamically adjusting the hash length based on the semantic similarity score includes: When the semantic similarity score indicates high semantic similarity, the hash length is extended; When the semantic similarity score indicates low semantic similarity, the hash length is shortened.

[0011] Optionally, if the hash granularity level of the target token is third-level hashing, then the process of parsing syntactic roles, semantic changes, and nested dependencies using the BERT model to construct a high-dimensional context representation and perform fine-grained hashing operations includes: The BERT model is used to parse the context fragment where the target token is located, and to obtain the syntactic role labeling, context semantic change trend index and dependency structure information between the target token and other tokens. Based on the syntactic roles, semantic change trend indicators, and dependency structure information, a context representation vector containing multi-dimensional syntactic and semantic features is constructed. The context representation vector reflects the structural hierarchy and semantic relationship of the target token in the context. By combining an enhanced hash function, a refined hash operation is performed on the high-dimensional context representation vector to generate a third-level hash result.

[0012] Optionally, matching the hash result with the key-value pairs in the pre-stored cache and performing corresponding operations based on the matching result includes: Based on the hash result generated by the target token in the second-level hash, extract multiple candidate cache keys from the pre-stored cache that are semantically close to the hash result; By combining the Top-K approximate matching method, the matching degree between the candidate cache key and the current hash result is calculated, and the target candidate key with the highest matching degree is selected. For the target candidate key, determine whether the target candidate key is consistent with the context embedding vector of the current target token; If they match, extract and return the cached value corresponding to the target candidate key; If they are inconsistent, a new key-value pair is generated based on the current target token and the corresponding context embedding vector, and the key-value pair is written into the cache.

[0013] Optionally, input the semantic category to which the target Token belongs into a pre-configured hash decision maker to obtain the hash granularity level applicable to the target Token. The hash granularity level at least includes: first-level hash, second-level hash, and third-level hash, including: Based on the context embedding vector of the target Token, calculate its context entropy value (H), semantic consistency index (S), and semantic change gradient (G), where: H is used to measure the distribution balance degree of the context where the target Token is located; S is used to measure the semantic similarity between the target Token and adjacent Tokens within the context window; G is used to reflect the change rate of the context embedding vector of the target Token in the context; Take H, S, and G as input features and input them into a pre-configured hash decision maker to perform rule judgment to determine the hash granularity level of the target Token; The determination criteria for the hash granularity level in the hash decision maker include: When S≥S1 and G≤G1 and H≤H1, it is determined as the first-level hash; When S2≤S<S1 or G1<G≤G2, it is determined as the second-level hash; When S<S2 or G>G2 or H>H2, it is determined as the third-level hash; Among them, S1, S2, G1, G2, H1, and H2 are pre-configured threshold parameters.

[0014] The second aspect of this application provides a Token-level cache matching system for a large language model. The system includes: A construction unit for each Token in the Token sequence to be processed, constructing a local context fragment according to the adjacent Tokens in its context window; A generation and calculation unit for inputting the local context fragment into a pre-defined semantic embedding model to generate the context embedding vector of the current Token, and calculating the context entropy value, semantic consistency index, and semantic change gradient based on the context embedding vector; A determination unit for determining the semantic category of the target Token according to the context embedding vector, context entropy value, semantic consistency, and semantic change gradient of the target Token; A hash level decision unit for inputting the semantic category to which the target Token belongs into a pre-configured hash decision maker to obtain the hash granularity level applicable to the target Token, and the hash granularity level at least includes: first-level hash, second-level hash, and third-level hash; The first hash unit is used to perform a fixed-length hash and map it to the semantic class cache bucket based on its semantic topic or context scope if the hash granularity level of the target token is the first level hash. The second hash unit is used to dynamically adjust the hash length based on the semantic similarity with adjacent tokens if the hash granularity level of the target token is the second level hash, and to perform variable-length hashing in token fragments with high semantic consistency. The third hash unit is used to parse syntactic roles, semantic changes and nested dependencies through the BERT model if the hash granularity level of the target token is the third level hash, to construct a high-dimensional context representation and perform fine-grained hash operations. The matching unit is used to match the hash result with the key-value pairs in the pre-stored cache and perform corresponding operations based on the matching result.

[0015] A third aspect of this application provides a token-level caching matching system for a large language model, the system comprising: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to execute the first aspect and any one of the optional methods in the first aspect.

[0016] A fourth aspect of this application provides a computer-readable storage medium on which a program is stored, which, when executed on a computer, performs the methods of the first aspect and any one of the first aspects.

[0017] As can be seen from the above technical solutions, this application has the following advantages: 1. By constructing local context fragments and calculating context entropy, semantic consistency index and semantic change gradient, the semantic environment of the token can be more accurately characterized, which helps to improve the adaptability of hashing strategies. 2. A multi-level hashing strategy is adopted, which combines elements such as semantic topic, token similarity and syntactic structure to perform fixed-length, variable-length and high-dimensional syntactic-aware hashing operations respectively, making cache key generation more flexible and semantically aware. 3. The cache matching mechanism guided by contextual semantic features can improve the cache hit rate and reduce redundant calculations while ensuring accuracy, thereby significantly improving the overall efficiency of the reasoning process of large language models. 4. The method described can be widely applied to large language model architectures of different sizes and structures, and has good scalability and engineering deployment value. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic flowchart of an embodiment of the token-level cache matching method for a large language model provided in this application; Figure 2 This is a flowchart illustrating an implementation of step S104 in the token-level cache matching method for the large language model provided in this application. Figure 3 This is a flowchart illustrating an implementation of the token-level cache matching method for the large language model provided in this application when the hash is at the first level. Figure 4 This is a flowchart illustrating an implementation of the token-level cache matching method for the large language model provided in this application when the hash is a second-level hash. Figure 5 This is a flowchart illustrating an implementation of the token-level cache matching method for the large language model provided in this application when the hash is at the third level. Figure 6 A schematic diagram of an embodiment of the token-level caching matching system for the large language model provided in this application; Figure 7 This is a schematic diagram of another embodiment of the token-level caching matching system for the large language model provided in this application. Detailed Implementation

[0020] Please see Figure 1 This application first provides an embodiment of a token-level cache matching method for a large language model, which includes: S101. For each Token in the Token sequence to be processed, construct a local context fragment based on the adjacent Tokens in its context window; For each token in the token sequence to be processed, its neighboring tokens in the current context window are extracted (e.g., N tokens to the left and right, where N is a preset window size), forming a local context fragment. This context fragment is used to characterize the semantic neighborhood environment of the current token, thus providing a semantic foundation for subsequent embedding computation. The local context fragment can be constructed by traversing the entire input sequence using a sliding window approach, supporting dynamic window width adjustment to adapt to different token densities and semantic continuity requirements.

[0021] For the sequence of tokens to be processed, a sliding window mechanism can be used to traverse each token in the sequence. Specifically, this includes: Set a context window size parameter W (e.g., W=4) to indicate that W tokens are taken to the left and right of the current token.

[0022] For the t-th TokenT t Extract the token sets on its left and right sides to form a local context fragment C. t ={T t ₋4,...,T t ₋1,T t T t ₊1, ..., T t ₊4}.

[0023] This fragment can be constructed as a token sequence or an embedding vector sequence and used as input for subsequent semantic embedding models. For long texts or paragraphs, sentence boundary constraints can be introduced to avoid extracting context across sentences.

[0024] S102. Input the local context fragment into a predefined semantic embedding model to generate the context embedding vector of the current token, and calculate the context entropy value, semantic consistency index and semantic change gradient based on the context embedding vector; The local context fragment obtained in step S101 is input into a predefined semantic embedding model, which can be Word2Vec, RoBERTa, GPTEncoder or other Transformer architectures, and outputs the context embedding vector corresponding to the current Token.

[0025] Based on this context embedding vector, the following semantic feature metrics are calculated: Context Entropy: Reflects the degree of information clutter in the current token semantic environment; The Semantic Consistency Score measures the semantic coherence between tokens in the current context fragment and can be obtained by calculating the cosine similarity between embedded vectors. Semantic Shift Gradient: Reflects the degree of semantic transformation between tokens, such as the trend of change in the Euclidean distance between the preceding and following embedding vectors.

[0026] These semantic features serve as the basis for determining the semantic category of a token and selecting a hashing strategy.

[0027] The above context fragment C t The steps to input the semantic embedding model are as follows: The embedding model can be a pre-trained BERT-base, RoBERTa, GPTEncoder, or a custom Transformer encoder. Obtain the context-aware vector E for each token. t The dimension of this vector is d (e.g., d=768). Calculate semantic feature metrics: Context entropy value H t : To measure the uniformity of semantic distribution in the context vector of a token, for example, to calculate information entropy based on the activation distribution after Softmax; Semantic consistency S t : Take the average of the embedding vectors cos(Eᵢ, Eⱼ) of any two tokens in the local context; Semantic change gradient G t : Calculate the rate of change between the current token and the previous and subsequent token embeddings, for example, G t =||E t -E t ₋1||+||E t -E t ₊1||.

[0028] These features constitute the input for semantic classification and hash granularity decisions.

[0029] S103. For a target token, determine the semantic category of the target token based on its context embedding vector, context entropy value, semantic consistency, and semantic change gradient. Based on the context embedding vector, context entropy value, semantic consistency, and semantic change gradient of the target token, heuristic rules or lightweight classification models (such as SVM and MLP) are used to semantically classify the token. Semantic categories may include, but are not limited to: Stable semantic classes (such as entity names and structural markers) Medium-variation semantic categories (such as prepositions and conjunctions) Highly dynamic semantic categories (such as sentiment words, verb phrases, etc.) The classification results are used to select an appropriate hash granularity strategy.

[0030] Based on E t H t S t G t The eigenvector F t =[E t H t S t G t Input this into a lightweight classification model, for example: Use a pre-trained MLP model (2-3 fully connected layers, ReLU activation); Alternatively, rule set logic can be used for judgment (such as H). t <threshold 1 and G t <Threshold 2 is classified as "semantically stable"); The classification output is a semantic category label, such as: C1: Stable semantic classes (entity, keyword) C2: Medium variation (prepositions, transition words) C3: Complex dynamic semantic classes (emotion words, predicates) S104. Input the semantic category to which the target Token belongs into the pre-configured hash decision-maker to obtain the hash granularity level applicable to the target Token. The hash granularity level includes at least: first-level hash, second-level hash and third-level hash. The semantic category of the token is input into a pre-configured hash decision generator. The hash decision generator can be a rule-based mapping table or a lightweight neural network driven by feature vectors, and its output includes one of the following three granularity levels: First-level hashing: Suitable for tokens that are semantically stable, have a concentrated context, and exhibit minimal changes; Second-level hashing: suitable for tokens with high context consistency but moderate semantic fluctuation; Third-level hashing: suitable for tokens with complex semantics, multiple dependencies, and key syntactic roles.

[0031] The decision-making process incorporates quantitative standards such as contextual entropy threshold, semantic consistency lower bound, and semantic change gradient curvature to ensure accurate judgment.

[0032] The following is a specific example to illustrate this: Semantic category label C t Input to the hash decision maker and perform the following steps: Hash decision makers can use lookup tables or conditional classification models (such as decision trees); Based on the semantic features H t S t G t For numerical values, select one of the following granularities:

[0033] Preferably, in step S104, the semantic category to which the target token belongs is input into a pre-configured hash decision maker to determine the applicable hash granularity level for the target token. (See [reference needed]). Figure 2 Specifically, it includes the following sub-steps: S1041. Extract the context embedding vector; The context embedding vector of the target token generated in step S102 is used as the input basis for semantic analysis.

[0034] The context embedding vector is a multi-layered encoded representation of the token within the context window, and the dimension can be 768 or 1024, depending on the semantic model used. The context window can be of fixed length (e.g., 8 tokens before and after), or it can be determined by a dynamic window (e.g., several tokens before and after that are semantically related to the current token).

[0035] S1042. Calculate three types of semantic indicators; To achieve a quantitative assessment of the current token semantic environment, this step extracts semantic discriminative features from the following three dimensions: Context entropy value H (ContextEntropy): This is used to measure the semantic distribution balance of the current token's context. Preferably, the distribution complexity can be estimated by performing an information entropy formula or principal component analysis (PCA) on the set of context embedding vectors. When the H value is small, it indicates that the current semantic field is relatively concentrated, with low information redundancy, making it suitable for coarse-grained processing.

[0036] Semantic Consistency Metric S: This is used to measure the semantic similarity between a target token and its adjacent tokens in the context window. Preferably, the cosine similarity between the current token and the embedding vectors of several tokens above and below it can be calculated, and the average value is taken as the S value. The higher the S value, the better the semantic continuity between the token and its adjacent content, indicating that it belongs to a consistent semantic region.

[0037] Gradient of Semantic Shift G: It represents the change amplitude of the target Token embedding vector relative to the embedding vectors of the previous and subsequent Tokens in the context. The change rate can be evaluated using the L2 norm of the vector difference or the Dynamic Time Warping (DTW) method. A lower G indicates that the Token is in a smooth semantic transition region, while a higher G means a semantic jump or logical turn.

[0038] S1043. Determine the hash level based on the combined metrics; Take the above three metrics as input features and input them into a pre-configured hash decision maker to perform rule matching and judgment to determine the hash granularity level of the target Token. In this embodiment, the pre-configured hash decision maker includes a multi-condition matching rule based on thresholds for determining the hash granularity level that the target Token should adopt. The specific rules are as follows: First-level hash (coarse granularity, semantic stability): The condition is: S ≥ S1 and G ≤ G1 and H ≤ H1; The hash Tokens at this level have high semantic continuity and stable changes, and are suitable for overall caching processing.

[0039] Second-level hash (medium granularity, local jump): The condition is: S2 ≤ S < S1 or G1 < G ≤ G2; The hash Tokens at this level are at the semantic conversion edge and require dynamic adjustment of the hash range.

[0040] Third-level hash (fine granularity, semantic complexity): The condition is: S < S2 or G > G2 or H > H2; The hash Tokens at this level have large context fluctuations and complex structures, and require fine-structure modeling.

[0041] Among them, S1, S2, G1, G2, H1, and H2 are the optimal thresholds obtained through offline data statistics or pre-training model parameter tuning. The specific values can be set according to the actual situation of the model. Some preferred specific values are provided below: S1 = 0.85, S2 = 0.65; G1 = 0.3, G2 = 0.6; H1 = 1.2, H2 = 2.0; The above parameters can be obtained through grid search or validation set tuning according to the training data and task scenarios of the large language model.

[0042] S1044. Output the hash granularity level; Based on the matching results above, the hash granularity level that the current target token should adopt is output. This level will serve as a control condition for selecting a fixed-length, variable-length, or structured hash path in subsequent steps.

[0043] S105. If the hash granularity level of the target token is the first level hash, then based on its semantic topic or context scope, perform a fixed-length hash and map it to the semantic class cache bucket. When the hash granularity level is the first level, a semantic class identifier is constructed based on the semantic topic or context scope to which the current token belongs (such as entity category, paragraph title, topic tag, etc.). This identifier is then hashed using a fixed-length hash function (such as SHA-1 truncation, SimHash, etc.) and mapped to the semantic class cache bucket. This cache bucket can be classified according to topic, scope, etc., to achieve high-reusability cache key location.

[0044] Specifically, extract the semantic topic to which the current token belongs (such as tags like "technology" or "medical"), or extract topic words from the local context; Semantic topic text is processed into fixed-length hash values ​​using standard hash algorithms (such as MurmurHash, SHA-1 truncation); Hash values ​​are mapped to semantic class cache buckets (Bucket_ID) for fast cache key location; Example: Hash("climatechange") → Bucket_23.

[0045] S106. If the hash granularity level of the target token is the second level hash, then the hash length is dynamically adjusted according to the semantic similarity with the adjacent tokens, and variable-length hashing is performed in token fragments with high semantic consistency. When the hash granularity level is second, the semantic similarity (such as cosine similarity, Mahalanobis distance, etc.) between the target token and its left and right adjacent tokens is further analyzed, and the range of tokens participating in the hashing is dynamically adjusted accordingly to construct a semantically continuous segment of variable length. Hash calculation is performed within this segment with high semantic consistency to generate a structurally adaptive hash value, thereby enhancing the ability to capture short-term semantic structures. This mechanism is particularly effective for segments with ambiguous semantic boundaries.

[0046] Specifically, the semantic similarity cos(E) between the target token and its neighboring tokens is calculated within the context. t E t-1 ), cos(E t E t+1 ); If the consecutive similarity exceeds a set threshold (e.g., 0.8), these tokens are combined into a semantic fragment S={T}. t-1 T t T t+1}; A dynamic input string is constructed using a sliding window and character concatenation for the semantic fragment, and a variable-length hash algorithm (such as MinHash or LocalitySensitiveHash) is applied to generate a hash key. The hash key is used as the cache key and is compared with the candidate cache key-value set using a Top-K comparison.

[0047] S107. If the hash granularity level of the target token is the third level hash, then the BERT model is used to parse the syntactic roles, semantic changes and nested dependencies, construct a high-dimensional context representation and perform fine-grained hash operations. When the hash granularity level is the third level, the pre-trained BERT model is invoked to perform syntactic structure analysis and dependency parsing on the sentence containing the target token, extracting its syntactic roles (such as subject, object, adverbial, etc.), nested structures in the context (such as relative clauses, verb-object phrases, etc.), and semantic variation paths.

[0048] Based on this structural information, a high-dimensional context vector representation is constructed, and this representation is input into a high-complexity hash function (such as an embedding hash based on an attention mechanism) to perform fine-grained hashing. This strategy is suitable for token scenarios that are semantically flexible but structurally complex.

[0049] Specifically, the BERT model takes a complete sentence containing the target token as input and outputs: The syntactic role label of a token in a sentence; Nested relationship graph in dependency structure tree; Syntactic connectivity features in multi-hop dependency paths; Using this structural information and embedding E with Token t Together, we construct a high-dimensional context representation vector. V t ; V t Input to an Attention-weighted hasher or a deeply embedded hash module to generate a high-precision hash value; Hash values ​​are used for precise retrieval in complex semantic caches.

[0050] S108. Match the hash result with the KV in the pre-stored cache, and perform the corresponding operation based on the matching result.

[0051] The hash result generated in the above steps is used as the key and matched with the key-value pairs (KV) pre-stored in the cache. The matching strategy includes: Fast hash index positioning; Candidate selection based on Top-K semantic approximation; Precise semantic matching verification (e.g., the cosine similarity of the embedded vectors is higher than a preset threshold).

[0052] If a match is found, the corresponding Value is retrieved from the cache and returned to avoid duplicate calculations; if the cache is not hit, a new Key-Value pair is generated and inserted into the cache for subsequent calls, improving model response efficiency.

[0053] Specifically, hash values ​​are used to quickly locate candidate key-value pairs in the cache; If it is a second or third level hash, the following operations also need to be performed: The cosine similarity between the candidate key and the current token is calculated. If the similarity is higher than the set threshold (e.g., 0.95), it is considered a successful match; If a match is found: the corresponding Value is returned for inference calls; If no match is found: Construct a new key-value pair K_new=hash value, V_new=current token, output and store in the cache for later reuse.

[0054] This embodiment proposes a token-level cache matching method for large language models. Based on a hierarchical processing mechanism of context construction, semantic embedding analysis, semantic classification, and multi-granularity hash decision, it has the following beneficial effects: By combining metrics such as context entropy, semantic consistency, and semantic change gradient to perform semantic classification of tokens, and selecting an appropriate hashing strategy accordingly, invalid cache lookups are effectively reduced, and cache hit rate and matching accuracy are improved.

[0055] The hash processing method is dynamically adjusted according to the tokens with different semantic complexity and stability, realizing the transition from fixed length to variable length, and then to fine-grained hashing based on structural information, making the cache matching logic more flexible and adaptable.

[0056] By jointly extracting contextual representations using the embedding model and the BERT model, not only surface similarity at the lexical level is considered, but also deep grammatical and semantic relationships are taken into account, ensuring high-quality comparison between tokens and cache keys.

[0057] This method effectively reduces repetitive token inference operations, lowers overall computational resource consumption, and improves the execution efficiency of large language models during the inference phase by constructing semantic class cache buckets and dynamic token segment reuse strategies.

[0058] This method relies on standard semantic embedding and hashing mechanisms, does not depend on a specific language model architecture, and can be universally deployed in various Transformer-class large models, facilitating engineering implementation and horizontal expansion.

[0059] In summary, the embodiments of this application solve the problems of inconsistent token-level cache granularity, low hit rate, and poor adaptability in existing large language models by using a semantically aware and granular adaptive cache matching method, providing an efficient and refined technical path for the inference optimization of large language models.

[0060] To illustrate the token-level cache matching method for large language models proposed in this application more specifically, the following will further elaborate on the context processing methods, hash strategies, and cache matching mechanisms corresponding to the first-level hash, second-level hash, and third-level hash, based on the processing flow of different hash granularity levels. By combining the semantic complexity of the token with the stability of the context, a hierarchical matching logic is systematically constructed, enabling the caching system to achieve efficient and accurate cache hits in different semantic scenarios. The hash processing paths at each level and their operational details in the specific implementation process are described below: 1. Typical implementation of first-level hashing: In this embodiment, for tokens with a hash granularity level of 1, the system uses a fixed-length hash and maps it to a semantic cache bucket to achieve efficient cache key generation and matching operations. See also... Figure 3 Specifically, it includes the following steps: S1051. Based on the position of the target token in the context window and its semantic category, determine the semantic topic or context scope of the target token, and extract the corresponding context feature information. Based on the target token's position in the context window and its semantic category (such as: person entity, number, place name, abstract concept, etc.), its semantic theme is determined. For example, if the target token is located in "Beijing" in "He is attending a meeting in Beijing", it can be inferred from the surrounding entity structure that it belongs to the "geographical location" theme.

[0061] Specifically, a fixed window is formed by taking several tokens before and after the target token (e.g., taking 2-3 tokens before and after each). Based on part-of-speech tagging, named entity recognition (NER), and rule templates, a context fragment with structural features is extracted. This fragment is then encoded using TF-IDF, LDA, or a lightweight semantic embedding model (such as FastText) to obtain a semantic topic feature vector, which serves as the input for subsequent hashing.

[0062] S1052. The context features of the target token are hashed using a fixed-length hash algorithm to generate a hash value for the target token. Using the context feature vector extracted in step S1051, a fixed-length hash algorithm (such as MurmurHash, CityHash, etc.) is applied to compress the high-dimensional features into a fixed-length hash value.

[0063] For example, first standardize the context feature vector to a fixed-length input (e.g., 64-dimensional); then serialize the standardized vector (e.g., encode floating-point numbers into strings according to precision); finally, input the serialized result into a hash function and output a 128-bit fixed-length hash value.

[0064] S1053. Based on the generated hash value, map the target Token to the corresponding semantic class cache bucket, wherein the semantic class cache bucket corresponds to a set of Tokens with similar semantic features; Using the hash value generated in S1052 as the key, the target token is mapped to the corresponding semantic class cache bucket. Each semantic class cache bucket is used to store tokens with similar topic semantics and their KV pairs for subsequent lookup.

[0065] For example, the first few bits (e.g., the high 8 bits) of the hash value can be used as the bucket number; if a TreeMap or prefix dictionary structure is used to implement the cache bucket, the number directly locates the corresponding bucket; each cache bucket maintains a hash value to a TokenKV mapping list to support fast location.

[0066] S1054, Perform cache matching operation.

[0067] In the mapped cache bucket, based on the complete hash value of the current token, check if there is a completely matching key-value entry. If a match is found, the corresponding cached value is returned directly; if no match is found, a new key-value pair is constructed by combining the current token with its context features and inserted into the cache bucket.

[0068] Specifically, the hash value is used as the primary key for lookup in the target bucket; If the hash value exists, it is further compared with the context feature digest or embedded signature stored in the KV to ensure consistency; If there is a perfect match, the corresponding V value will be used as the inference input or result. If no match is found, the encoding module is called to generate V (such as positional encoding, attention results, etc.) and inserted into the buffer bucket.

[0069] 2. Typical implementation of the second hash level: When the semantic structure of the target token exhibits some variability while maintaining overall consistency, a variable-length hash is employed to adapt to dynamic contextual differences. In this embodiment, the hash length is dynamically adjusted based on semantic similarity with adjacent tokens, and a more flexible token caching matching strategy is implemented in conjunction with semantic consistency. See also... Figure 4 It includes the following steps: S1061. Based on the semantic similarity between the target token and adjacent tokens, calculate the semantic similarity score of adjacent tokens, and dynamically adjust the hash length according to the semantic similarity score; For the target token in the current context window, calculate the similarity of its semantic embeddings with several tokens in the preceding and following directions to evaluate its coherence in the local context.

[0070] Specifically, the current token and its N tokens before and after it (e.g., N=3) are input into the same semantic embedding model (e.g., SimCSE, SBERT); cosine similarity is used to calculate the similarity between the target token and each of its neighboring tokens; an average similarity greater than a preset threshold (e.g., 0.75) indicates high semantic consistency, while the opposite is considered a semantic fluctuation region.

[0071] S1062. Based on the adjusted hash length, perform a variable-length hash operation in the context fragment consisting of the target token and its adjacent tokens to generate the corresponding hash result.

[0072] Based on the semantic similarity results, the start and end positions of the token fragments are dynamically determined to form a variable-length context window for hash processing.

[0073] Specifically, expand outwards from the current token until the similarity is below the threshold or the maximum window length (e.g., 8 tokens) is reached; record the actual token fragments extracted as input for subsequent hash operations.

[0074] S1063. Based on the hash result generated in the second-level hash of the target token, extract multiple candidate cache keys in the pre-stored cache that are semantically close to the hash result; Perform variable-length hash generation on the token fragment determined in S1062. Unlike fixed-length processing, this method allows for variations in the length of the embedded input, improving the flexibility of semantic matching.

[0075] For example, token fragments are encoded into a set of concatenated embedding vectors (e.g., tokenembedding + positionembedding); intermediate digests are generated using hash convolutions (e.g., MinHash or convolutional hashing NetHash); and asymmetric hash compression is performed on the digest values ​​to obtain hash values ​​(e.g., 64~256 bits, automatically adjusted according to fragment length).

[0076] S1064. Using the Top-K approximate matching method, calculate the matching degree between the candidate cache key and the current hash result, and select the target candidate key with the highest matching degree. A Top-K approximate search mechanism is used in the cache to filter out several candidate tokens that are most likely to match from the existing key-value pairs. The Hamming distance or Euclidean distance is calculated between the target token's hash value and the existing hash values ​​in the cache; the Top-K similar entries are quickly extracted from the hash space using an approximate nearest neighbor algorithm (such as FAISS, Annoy); a list of candidate key-value pairs is obtained, ready for precise comparison.

[0077] S1065. For the target candidate key, determine whether the target candidate key is consistent with the context embedding vector of the current target Token; S1066. If they match, extract and return the cached value corresponding to the target candidate key; S1067. If they are inconsistent, a new key-value pair is generated based on the current target token and the corresponding context embedding vector, and the key-value pair is written into the cache.

[0078] In this embodiment, the contextual semantic features of the Top-K candidates are further compared one by one. If there are entries with completely identical or highly similar contextual features, they are considered to be a hit and the cached value is returned; otherwise, a new KV pair is generated and added to the cache.

[0079] Specifically, the similarity of the target token context digest with the feature digests in the candidate key-value pairs is compared; if the similarity is greater than a high threshold (e.g., 0.9), it is considered a hit; if it is not a hit, a new key-value pair is generated and inserted into the corresponding cache bucket according to its variable length range and features.

[0080] 3. Typical implementation of third-level hashing: When the target token is located in a semantically complex region (such as nested long sentences, logical transitions, or syntactic references), fixed- or variable-length hashes are prone to losing deep semantic relationships. Therefore, this embodiment employs a deep language model (such as BERT) for syntactic and semantic dependency parsing, constructs a high-dimensional semantic graph, and performs fine-grained semantic hashing and precise matching operations. (See [link to documentation]). Figure 5 Specifically, it includes: S1071. Use the BERT model to parse the context fragment where the target token is located, and obtain the syntactic role label, context semantic change trend index and dependency structure information between the target token and other tokens. In this embodiment, the sentence boundary or semantic minimum complete unit where the target token is located is first identified, providing a complete information basis for subsequent parsing.

[0081] For example, sentences can be segmented based on punctuation marks in the text (such as commas, periods, questions, or line breaks). If the token is at the end of a sentence or in a complex sentence structure, a sliding window mechanism (such as expanding two sentences on each side) can be introduced to ensure context closure. The current sentence and related clauses are used as the input range to prepare for BERT model analysis.

[0082] Structural modeling is performed on the above sentences to identify semantic paths, subordinate relationships, and referential chains between tokens, capturing non-linear semantic connections.

[0083] The sentence is encoded using Transformer models such as BERT and RoBERTa to obtain the ContextualEmbedding of each Token. Combined with syntactic analysis tools (such as spaCy and Stanza), the dependency graph and role labels (such as subject, object, and adverbial) are extracted. A semantic adjacency graph is built for the Token to record the set of Token nodes that have dependency or nesting relationships with it, forming a structured contextual representation.

[0084] S1072. Based on the syntactic roles, semantic change trend indicators and dependency structure information, construct a context representation vector containing multi-dimensional syntactic and semantic features. The context representation vector reflects the structural hierarchy and semantic relationship of the target token in the context. Based on dependency structure and nesting relationship, features are extracted from local semantic graphs and aggregated into a high-dimensional context vector of the token.

[0085] A weighted graph attention mechanism (GraphAttentionNetwork) can be used to fuse information between the target token and semantically adjacent tokens. The aggregated vector can include: the target token's own embedding + the embeddings of all first-order dependent nodes + the aggregated vector of the path nesting structure, and finally generate a set of structure-enhanced context embeddings to represent the identity and meaning of the token in a complex semantic environment.

[0086] S1073 combines an enhanced hash function to perform a refined hash operation on the high-dimensional context representation vector, generating a third-level hash result.

[0087] Based on the high-dimensional representation vector, a semantically aware hashing algorithm is used to generate fine-grained hash codes, supporting high-discrimination matching. Weighted semantic hashing can be used, which maps the high-dimensional embedding to the hash space through the transformation matrix obtained by training; during the mapping, multiple projection channels (such as topic vectors, dependency path vectors, etc.) are introduced to generate multiple sets of sub-hash codes; S1074. Search for similar entries in the cache using high-dimensional hashing; For the generated fine hash values, perform high-dimensional matching in the cache to find key-value pairs with similar semantic structures. First, establish a semantic hash index library (such as based on HNSW or FAISS) to maintain a structured token cache; set a vector similarity threshold (such as cosine > 0.92) to retrieve Top-K approximate items; if there are key-value pairs with high structural feature matching, mark them as potential hits and prepare for semantic fine matching.

[0088] S1075. Perform semantic feature matching on each matching candidate and update the cache; Each matching candidate is compared against its semantic features to determine if a complete match is found; otherwise, the cache is updated and a new entry is added. The comparison includes: syntactic role consistency (e.g., whether they are the same modifiers), nested structure symmetry, and semantic graph topological consistency. If the consistency score is high (e.g., using structure matching F1 score, reaching 0.9 or above), it is considered a hit, and the corresponding cached value is returned; If a match is not found, the context representation of the target token and the corresponding calculation result are added to the KV cache for subsequent access.

[0089] In this embodiment, third-level hashing is mainly applied to logical transitions and contextual breakpoints in long texts. For example, in the sentence "Although he disagreed, she still insisted on her opinion," the "she" token after "but" can be traced for its referential logic and semantic tension by relying on BERT context modeling, even if there are no obvious semantic clues on the surface of the surrounding tokens. This is a typical third-level processing scenario.

[0090] To implement the aforementioned token-level cache matching method for large language models, this application further provides a corresponding system embodiment. This system can efficiently perform context feature extraction, semantic category determination, hash granularity decision-making, and cache matching operations on the tokens to be processed, thereby improving the cache reuse efficiency and response speed of large language models during the inference phase.

[0091] This system can be integrated into the inference framework of a large language model as part of the token processing module. It can be deployed independently or run in conjunction with the model inference engine. The system design fully considers the dynamic changes in token semantics and adopts a multi-level hash granularity control mechanism, possessing good adaptability and scalability.

[0092] The following will describe in detail the structural composition of this system and the functions of each module with reference to the accompanying drawings, so as to further clarify the technical solution proposed in this application.

[0093] See Figure 6 This application provides an embodiment of a token-level caching matching system for large language models, the system of which includes: Construction unit 601 is used to construct a local context fragment for each Token in the Token sequence to be processed, based on the adjacent Tokens in its context window. The generation calculation unit 602 is used to input the local context fragment into a predefined semantic embedding model, generate the context embedding vector of the current token, and calculate the context entropy value, semantic consistency index and semantic change gradient based on the context embedding vector; The determining unit 603 is used to determine the semantic category of the target token based on the target token's context embedding vector, context entropy value, semantic consistency, and semantic change gradient. The hash level decision unit 604 is used to input the semantic category to which the target token belongs to a pre-configured hash decision unit to obtain the hash granularity level applicable to the target token. The hash granularity level includes at least: first-level hash, second-level hash and third-level hash. The first hash unit 605 is used to perform a fixed-length hash and map it to a semantic class cache bucket based on its semantic topic or context scope if the hash granularity level of the target token is the first level hash. The second hash unit 606 is used to dynamically adjust the hash length based on the semantic similarity with adjacent tokens if the hash granularity level of the target token is the second level hash, and to perform variable-length hashing in token fragments with high semantic consistency. The third hash unit 607 is used to parse syntactic roles, semantic changes and nested dependencies through the BERT model if the hash granularity level of the target token is the third level hash, to construct a high-dimensional context representation and perform fine-grained hash operations. The matching unit 608 is used to match the hash result with the KV in the pre-stored cache and perform corresponding operations based on the matching result.

[0094] Optionally, the first hash unit 605 is specifically used for: Based on the target token's position in the context window and its semantic category, determine the target token's semantic topic or context scope, and extract the corresponding context feature information; The context features of the target token are hashed using a fixed-length hash algorithm to generate the hash value of the target token. Based on the generated hash value, the target token is mapped to the corresponding semantic class cache bucket, which corresponds to a set of tokens with similar semantic features.

[0095] Optionally, the second hash unit 606 is specifically used for: Based on the semantic similarity between the target token and adjacent tokens, the semantic similarity score of adjacent tokens is calculated, and the hash length is dynamically adjusted according to the semantic similarity score. Based on the adjusted hash length, a variable-length hash operation is performed within the context fragment consisting of the target token and its adjacent tokens to generate the corresponding hash result.

[0096] Optionally, the second hash unit 606 is specifically used for: When the semantic similarity score indicates high semantic similarity, the hash length is extended; When the semantic similarity score indicates low semantic similarity, the hash length is shortened.

[0097] Optionally, the third hash unit 607 includes: The BERT model is used to parse the context fragment where the target token is located, and to obtain the syntactic role labeling, context semantic change trend index and dependency structure information between the target token and other tokens. Based on the syntactic roles, semantic change trend indicators, and dependency structure information, a context representation vector containing multi-dimensional syntactic and semantic features is constructed. The context representation vector reflects the structural hierarchy and semantic relationship of the target token in the context. By combining an enhanced hash function, a refined hash operation is performed on the high-dimensional context representation vector to generate a third-level hash result.

[0098] Optionally, the matching unit 608 is specifically used for: Based on the hash result generated by the target token in the second-level hash, extract multiple candidate cache keys from the pre-stored cache that are semantically close to the hash result; Combined with the Top-K approximate matching method, calculate the matching degree between the candidate cache key and the current hash result, and filter out the target candidate key with the highest matching degree; For the target candidate key, determine whether the target candidate key is consistent with the context embedding vector of the current target Token; If they are consistent, extract and return the cache value corresponding to the target candidate key; If they are inconsistent, generate a new key-value pair based on the current target Token and the corresponding context embedding vector, and write the key-value pair into the cache.

[0099] Optionally, the hash level decision unit 604 is specifically used for: Based on the context embedding vector of the target Token, calculate its context entropy value (H), semantic consistency index (S), and semantic change gradient (G), where: H is used to measure the distribution balance degree of the context where the target Token is located; S is used to measure the semantic similarity between the target Token and the adjacent Tokens within the context window; G is used to reflect the change rate of the context embedding vector of the target Token in the context; Take H, S, and G as input features, input them into a pre-configured hash decision maker, and execute rule judgment to determine the hash granularity level of the target Token; The determination criteria for the hash granularity level in the hash decision maker include: When S≥S1 and G≤G1 and H≤H1, it is determined as the first-level hash; When S2≤S<S1 or G1<G≤G2, it is determined as the second-level hash; When S<S2 or G>G2 or H>H2, it is determined as the third-level hash; Among them, S, S2, G1, G2, H1, and H2 are pre-configured threshold parameters.

[0100] Please refer to Figure 7 , this application also provides a Token-level cache matching system for a large language model, including: A processor 701, a memory 702, an input / output unit 703, and a bus 704; The processor 701 is connected to the memory 702, the input / output unit 703, and the bus 704; The memory 702 stores a program, and the processor 701 calls the program to execute any of the above methods.

[0101] This application also relates to a computer-readable storage medium on which a program is stored, characterized in that, when the program is run on a computer, it causes the computer to perform any of the methods described above.

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0106] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A token-level cache matching method for a large language model, characterized in that, The method includes: For each token in the token sequence to be processed, a local context fragment is constructed based on the adjacent tokens in its context window; The local context fragment is input into a predefined semantic embedding model to generate the context embedding vector of the current token, and the context entropy value, semantic consistency index and semantic change gradient are calculated based on the context embedding vector. For a target token, the semantic category of the target token is determined based on its context embedding vector, context entropy value, semantic consistency, and semantic change gradient. The semantic category to which the target token belongs is input into a pre-configured hash decision maker to obtain the hash granularity level applicable to the target token. The hash granularity level includes at least: first-level hash, second-level hash, and third-level hash. If the target token's hash granularity level is first-level hash, then a fixed-length hash is performed based on its semantic topic or context scope and mapped to the semantic class cache bucket; If the target token's hash granularity level is second-level hash, then the hash length is dynamically adjusted based on the semantic similarity with adjacent tokens, and variable-length hashing is performed in token fragments with high semantic consistency. If the target token's hash granularity level is third-level hash, then the BERT model is used to parse syntactic roles, semantic changes, and nested dependencies, construct a high-dimensional context representation, and perform fine-grained hash operations. The hash result is matched with the key-value pairs in the pre-stored cache, and the corresponding operation is performed based on the matching result.

2. The token-level cache matching method for large language models according to claim 1, characterized in that, If the hash granularity level of the target token is first-level hash, then performing a fixed-length hash and mapping it to a semantic class cache bucket based on its semantic topic or context scope includes: Based on the target token's position in the context window and its semantic category, determine the target token's semantic topic or context scope, and extract the corresponding context feature information; The context features of the target token are hashed using a fixed-length hash algorithm to generate the hash value of the target token. Based on the generated hash value, the target token is mapped to the corresponding semantic class cache bucket, which corresponds to a set of tokens with similar semantic features.

3. The token-level cache matching method for large language models according to claim 1, characterized in that, If the target token's hash granularity level is second-level hashing, then the hash length is dynamically adjusted based on the semantic similarity with adjacent tokens, and variable-length hashing is performed on token fragments with high semantic consistency, including: Based on the semantic similarity between the target token and adjacent tokens, the semantic similarity score of adjacent tokens is calculated, and the hash length is dynamically adjusted according to the semantic similarity score. Based on the adjusted hash length, a variable-length hash operation is performed within the context fragment consisting of the target token and its adjacent tokens to generate the corresponding hash result.

4. The token-level cache matching method for large language models according to claim 3, characterized in that, The dynamic adjustment of the hash length based on the semantic similarity score includes: When the semantic similarity score indicates a high semantic similarity, extend the hash length; When the semantic similarity score indicates a low semantic similarity, shorten the hash length.

5. The token-level cache matching method for large language models according to claim 1, characterized in that, If the hash granularity level of the target Token is the third-level hash, the syntactic role, semantic change, and nested dependency relationship are parsed through the BERT model, and a high-dimensional context representation is constructed and a fine-grained hash operation is performed, including: Use the BERT model to parse the context fragment where the target Token is located, and obtain the syntactic role annotation, context semantic change trend index, and dependency structure information between the target Token and other Tokens; Based on the syntactic role, semantic change trend index, and dependency structure information, construct a context representation vector containing multi-dimensional grammar and semantic features, and the context representation vector reflects the structural hierarchy and semantic relationship of the target Token in the context; Combined with an enhanced hash function, perform a refined hash operation on the high-dimensional context representation vector to generate a third-level hash result.

6. The token-level cache matching method for large language models according to claim 3, characterized in that, The matching of the hash result with the KV in the pre-stored cache and the corresponding operations are performed according to the matching result, including: Based on the hash result generated by the target Token in the second-level hash, extract multiple candidate cache keys in the pre-stored cache whose semantic distance is close to the hash result; Combined with the Top-K approximate matching method, calculate the matching degree between the candidate cache key and the current hash result, and screen out the target candidate key with the highest matching degree; For the target candidate key, determine whether the target candidate key is consistent with the context embedding vector of the current target Token; If they are consistent, extract and return the cache value corresponding to the target candidate key; If they are inconsistent, generate a new key-value pair based on the current target Token and the corresponding context embedding vector, and write the key-value pair into the cache.

7. The token-level cache matching method for large language models according to claim 1, characterized in that, Inputting the semantic category to which the target Token belongs into a pre-configured hash decision maker to obtain the hash granularity level applicable to the target Token, including: Based on the context embedding vector of the target Token, calculate its context entropy value (H), semantic consistency index (S), and semantic change gradient (G), where: H is used to measure the distribution balance degree of the context where the target Token is located; S is used to measure the semantic similarity between the target Token and adjacent Tokens within the context window; G is used to reflect the change rate of the context embedding vector of the target Token in the context; Take H, S, and G as input features, input them into a pre-configured hash decision maker, and perform rule judgment to determine the hash granularity level of the target Token; The determination criteria for the hash granularity level in the hash decision maker include: When S≥S1 and G≤G1 and H≤H1, it is determined as the first-level hash; When S2≤S<S1 or G1<G≤G2, it is determined as the second-level hash; When S<S2 or G>G2 or H>H2, it is determined as the third-level hash; Among them, S1, S2, G1, G2, H1, and H2 are pre-configured threshold parameters.

8. A token-level caching matching system for a large language model, characterized in that, The system includes: The building unit is used to construct a local context fragment for each token in the token sequence to be processed, based on the adjacent tokens in its context window. A computational unit is used to input the local context fragment into a predefined semantic embedding model, generate the context embedding vector of the current token, and calculate the context entropy value, semantic consistency index and semantic change gradient based on the context embedding vector; The determining unit is used to determine the semantic category of a target token based on the target token's context embedding vector, context entropy value, semantic consistency, and semantic change gradient. The hash level decision unit is used to input the semantic category to which the target token belongs to a pre-configured hash decision unit to obtain the hash granularity level applicable to the target token. The hash granularity level includes at least: first-level hash, second-level hash and third-level hash. The first hash unit is used to perform a fixed-length hash and map it to the semantic class cache bucket based on its semantic topic or context scope if the hash granularity level of the target token is the first level hash. The second hash unit is used to dynamically adjust the hash length based on the semantic similarity with adjacent tokens if the hash granularity level of the target token is the second level hash, and to perform variable-length hashing in token fragments with high semantic consistency. The third hash unit is used to parse syntactic roles, semantic changes and nested dependencies through the BERT model if the hash granularity level of the target token is the third level hash, to construct a high-dimensional context representation and perform fine-grained hash operations. The matching unit is used to match the hash result with the key-value pairs in the pre-stored cache and perform corresponding operations based on the matching result.

9. A token-level caching matching system for a large language model, characterized in that, The system includes: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, the program performing the method as claimed in any one of claims 1 to 7 when executed on a computer.

Images