Key value cache sharing method, device, equipment and product of large language model

Abstract

The invention discloses a key value cache sharing method and device for a large language model, equipment and a product. The method comprises the steps that request content input to the large language model by a first user is acquired; determining a first segmented word group corresponding to a first part of a shared key value cache of the large language model in the request content, wherein the shared key value cache of the large language model does not have a matched key value; determining an attention score between every two segmented words in the first segmented word group to obtain an attention matrix, determining at least one second segmented word group based on the attention matrix and the first segmented word group, and adding a corresponding key value to a shared key value cache; wherein the attention score between the segmented words in the second segmented word group is greater than the attention score between the segmented words in the second segmented word group and the segmented words outside the second segmented word group. Continuous updating of the shared key value cache is realized, and the hit rate and the utilization rate of the cache key value are remarkably improved.

Description

Technical Field

[0001] This disclosure relates to the field of large language models and data processing, and more specifically, to a method, apparatus, device, and product for key-value cache sharing of large language models. Background Technology

[0002] In the inference process of Large Language Models (LLMs), Key-Value Cache is a core optimization technique. Its basic principle is to store the Key and Value representations of each Token (segmentation) during the attention computation of the Transformer structure. This KV representation allows for direct reuse in subsequent inference stages without repeated computation.

[0003] In related technologies, a fragment-based KV cache sharing mechanism can be adopted. This method divides the word segmentation sequence corresponding to the request content of the large language model into fixed-length word segments and stores the key value corresponding to each word segment. However, the division of word segments is too coarse, lacks flexibility, and has a low reusability. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] Firstly, this disclosure provides a method for sharing key-value caches in a large language model, the method comprising: Get the request content that the first user inputs to the large language model; For the first part of the requested content where there is no matching key value in the shared key value cache of the large language model, the first word segment corresponding to the first part is determined; Determine the attention score between every two words in the first word segmentation group to obtain an attention matrix, and determine at least one second word segmentation group based on the attention matrix and the first word segmentation group; wherein, the attention score between words in the second word segmentation group is greater than the attention score between words in the second word segmentation group and words outside the second word segmentation group; Determine the key value corresponding to the at least one second word segment and add it to the shared key value cache.

[0006] Secondly, this disclosure provides a key-value cache sharing device for a large language model, the key-value cache sharing device comprising: The acquisition module is used to acquire the request content input by the first user to the large language model; The first determining module is used to determine the first word segment corresponding to the first part of the request content where there is no matching key value in the shared key value cache of the large language model; The second determining module is used to determine the attention score between every two words in the first word segmentation group to obtain an attention matrix, and to determine at least one second word segmentation group based on the attention matrix and the first word segmentation group; wherein, the attention score between words in the second word segmentation group is greater than the attention score between words in the second word segmentation group and words outside the second word segmentation group; An addition module is used to determine the key value corresponding to the at least one second word segment and add it to the shared key value cache.

[0007] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.

[0008] Fourthly, this disclosure provides an electronic device, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method described in the first aspect.

[0009] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0010] Through the above technical solution, for the first part of the request content input by the first user to the large language model, where no matching key value exists in the shared key-value cache of the large language model, the first word segment corresponding to the first part is determined. Then, the attention score between every two words in the first word segment is determined, resulting in an attention matrix. Based on the attention matrix and the first word segment, at least one second word segment is determined. Furthermore, the key value corresponding to at least one second word segment is determined and added to the shared key-value cache. The attention score between words within the second word segment is greater than the attention score between words within the second word segment and words outside the second word segment. This method enables key-value caching of the portion of the user's request content where no matching key value exists in the shared key-value cache, achieving continuous updates to the shared key-value cache for subsequent key-value reuse by the large language model, effectively improving model inference efficiency. Furthermore, it can further refine the segmentation of the corresponding word groups based on the attention scores between the word groups, and ensure that the internal dependencies of the segmented word groups are higher than the external dependencies. This extends key-value reuse from fixed-segmentation groups in related technologies to adaptive segmentation groups of arbitrary length, significantly improving the hit rate and utilization of cached key-values.

[0011] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 This is a flowchart illustrating a key-value cache sharing method for a large language model according to an exemplary embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a modular data processing method according to an exemplary embodiment of the present disclosure; Figure 3 This is a schematic diagram illustrating an attention matrix and an attention integral matrix according to an exemplary embodiment of the present disclosure; Figure 4 This is a structural block diagram of a key-value cache sharing system for a large language model, as illustrated in an exemplary embodiment of this disclosure. Figure 5 This is a structural block diagram of a key-value cache sharing device for a large language model according to an exemplary embodiment of the present disclosure; Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0013] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0014] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0015] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0020] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0023] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] In the model's inference process, user input is progressively segmented into a series of tokens. During forward computation, the first step involves attention calculation on these tokens to generate corresponding Keys and Values. The Key represents the token's features, used to match them with the token's query vector to determine their relevance. The Key helps the model understand the importance of each token within the context. The Value represents the token's actual content, used for weighted summation in the attention mechanism to generate the final contextual representation. The Value provides specific information about each token so the model can utilize this information when generating output. Subsequently, when generating the next token, the model does not need to recalculate the Keys and Values ​​of all historical tokens; instead, it directly reads the cached results and only calculates the latest token's portion, significantly reducing computational load. This mechanism is particularly effective in scenarios such as long text inference, interactive dialogue, and multi-turn question answering, significantly shortening the Time-To-First-Token (TTFT) latency and improving overall throughput.

[0025] As large language models are widely deployed in cloud-based inference services and multi-user interactive systems, the need for shared key-value (KV) caches is becoming increasingly apparent. Since different users' inputs often contain the same or similar prefixes (e.g., system prompts, templated content, background knowledge), caching key-value pairs only within a single user session inevitably leads to many redundant calculations, wasting GPU memory and computing power. Therefore, a cross-request KV cache sharing mechanism has emerged. This mechanism allows for the direct reuse of the prefix portion of the key-value pair from the cache when a new user request has the same prefix as an existing request, thereby reducing redundant calculations.

[0026] Key-value cache sharing mechanisms in related technologies include the following two methods: The prefix-matching-based key-value sharing mechanism generates corresponding key-value pairs according to the token sequence of the request when processing a user request for the first time. These key-value pairs are then stored in a shared key-value cache. When a new request arrives, the input prefix is ​​compared with the existing prefix in the cache. If they match, the cached key-value pairs corresponding to the prefixes are reused without recalculation.

[0027] Fragment-based KV sharing mechanism: Divide the token sequence of the request into fixed-size word groups, and then cache the key values ​​corresponding to the word groups. In this way, even if different requests are not strictly related by prefix, as long as they contain the same part, the corresponding key values ​​can be reused directly.

[0028] The aforementioned key-value cache sharing mechanisms have significant limitations in terms of reuse granularity. In particular, the prefix-matching-based key-value sharing mechanism is overly strict, allowing cache reuse only when the prefixes of requests are completely identical. For example, if the cache already contains the request "ABCD," a new request "ABCE" cannot directly share the key-value pairs of the first three tokens; the entire sequence must be recalculated. This mechanism results in a large number of partially overlapping requests being unable to reuse key-value pairs, wasting computational resources.

[0029] The fragment-based key-value sharing mechanism is too crude. For example, the input token sequence is divided into fixed-length word segments, and sharing is possible when two requests contain the same word segments. However, due to the fixed granularity and the frequent overlap of multiple word segments in actual input, fixed-length word segments can lead to redundant calculations or erroneous sharing, resulting in insufficient flexibility. For instance, if "ABCD" is stored as "AB" and "CD", the key value corresponding to "BC" cannot be reused.

[0030] For example, in document analysis scenarios, user requests often contain lengthy background information. If two requests share 300 tokens, the prefix-matching-based key-value sharing mechanism does not support partial reuse and needs to be recalculated from scratch. On the other hand, the fragment-based key-value sharing mechanism may only capture the overlap of multiple tokens at fixed boundaries, leaving the remaining parts unusable. This reduces the reuse rate and lowers the throughput and inference efficiency of large language models.

[0031] It's important to note that the key-value cache sharing mechanism in related technologies typically employs a "full sharing" approach, lacking security guarantees. Attackers can construct inputs that partially overlap with the target user's request. For example, if the target user's input is "My bank card number is 123456," and the attacker submits the prefix "My bank card number is," the attacker's request might hit the cache. The attacker can then observe the hit effect, such as reduced latency or changes in scheduling priority, and gradually deduce subsequent sensitive information. This type of attack constitutes side-channel information leakage; its essence is to reconstruct the target user's true input by exploiting the differences in cache sharing.

[0032] Furthermore, under the fragment-based non-prefix sharing mechanism, attackers can construct inputs containing common templated content, guiding the system to continuously overlap with the target user's requests, thereby achieving indirect inference. Taking the RAG (Retrieval-Augmented Generation) scenario as an example, if an attacker constructs "Based on the following document summary: XXX", where "Based on the following document summary" is a frequently occurring template, it may match a large number of user requests. If this part subsequently contains sensitive data, such as contract content, there is a risk of leakage. Therefore, the relevant technologies lack the identification and isolation of tokens with business risks, leading to the uncontrollable exposure of risky data to other users once sharing occurs.

[0033] In view of this, embodiments of this disclosure provide a method, apparatus, device, and product for sharing key-value caches of large language models to solve the above-mentioned technical problems.

[0034] The embodiments of this disclosure will be further explained below with reference to the accompanying drawings.

[0035] Figure 1 This is a flowchart illustrating a key-value cache sharing method for a large language model according to an exemplary embodiment of this disclosure, with reference to... Figure 1 The key-value cache sharing method may include the following steps: S101: Obtain the request content input by the first user to the large language model.

[0036] The large language model is used to perform model inference on the user's input request content and output the corresponding response. Furthermore, it is also applicable to large speech models, multimodal models, and other models that perform model inference based on text content converted from input content; this disclosure does not impose any limitations on these aspects.

[0037] S102: For the first part of the requested content where there is no matching key in the shared key-value cache of the large language model, determine the first word segment corresponding to the first part.

[0038] Among possible approaches, the key-value cache sharing method also includes: for the request content, if there is a fourth part of a matching key-value in the shared key-value cache of the large language model, retrieve the key-value corresponding to the fourth part from the shared key-value cache; and generate response content for the request content based on the key-value corresponding to the first part and the key-value corresponding to the fourth part.

[0039] It is worth noting that since the shared key-value cache already stores the key-value corresponding to the fourth part, after the large language model is processed, there is no need to repeatedly store the key-value corresponding to the fourth part. Instead, the first part is cached to determine the new reusable fragments that can be stored, thereby enabling efficient and continuous updates to the shared key-value cache.

[0040] For example, such as Figure 2 As shown, the request content is converted into a word segmentation sequence. Before inputting it into the large language model, the retrieval module can first identify the parts of the request content where no matching key value exists. The corresponding word segmentation group for this part is then retrieved from the word segmentation sequence for key-value caching during model inference. Alternatively, the system can identify parts of the request content where no matching key value exists and retrieve the corresponding key value from the shared key-value cache, concatenating it to the word segmentation sequence before inputting it into the large language model. This way, during model inference, the large language model does not need to repeatedly calculate the reusable parts with matching key values; it only needs to calculate the key values ​​for the non-reusable parts without matching key values. The response content is then generated by combining the key values ​​of the reusable parts with the key values ​​of the non-reusable parts. This eliminates the need to calculate the key values ​​of the entire word segmentation sequence corresponding to the request content, effectively improving model response speed and inference efficiency.

[0041] Taking a question-and-answer dialogue system as an example, the system provides the user with the answer output by the model based on the word segmentation sequence corresponding to the user's input question. The specific answer can be determined according to the business scenario, and this disclosure does not impose any restrictions on it.

[0042] S103: Determine the attention score between every two words in the first word segmentation group to obtain the attention matrix, and determine at least one second word segmentation group based on the attention matrix and the first word segmentation group; wherein, the attention score between words in the second word segmentation group is greater than the attention score between words in the second word segmentation group and words outside the second word segmentation group.

[0043] For example, such as Figure 2 As shown, the attention scores of each pair of words in the first segmented group with dynamic length can be determined through the annotation module to obtain the attention matrix. Then, the attention matrix is ​​used to analyze whether the first segmented group satisfies "self-consistency", that is, the self-attention of the words in the segmented group is greater than the dependence on the outside. If it is satisfied, the key value corresponding to the first segmented group can be stored in the shared key value cache. If it is not satisfied, the first segmented group needs to be re-divided, for example, into segmented group 1 and segmented group 2.

[0044] In other words, it is necessary to ensure that the attention score between the inner words of the segmented group stored in the shared key-value cache is greater than the attention score between the inner words and the outer words of the segmented group. This means that the inner words of the segmented group are more dependent and easier to reuse.

[0045] S104: Determine the key value corresponding to at least one second word segment and add it to the shared key value cache.

[0046] The above method enables key-value caching of the portion of a user's request content for which a shared key-value cache does not contain a matching key. This allows for continuous updates to the shared key-value cache, facilitating key-value reuse in large language models and effectively improving model inference efficiency. Furthermore, the method allows for finer-grained segmentation of the corresponding word groups based on attention scores between word segments, ensuring that internal dependencies within the segmented groups are higher than external dependencies. This extends key-value reuse from fixed-segmentation groups in related technologies to adaptive segmentation groups of arbitrary length, significantly improving the hit rate and utilization of cached keys.

[0047] In some possible implementations, the shared key-value cache is also used to store the first hash value of the first content fragment corresponding to each key-value pair and the second hash value of the fragment corresponding to the preset length preceding the first content fragment; the key-value cache sharing method further includes: determining the second content fragment with the preset length preceding the first content fragment in the request content, and retrieving it from the shared key-value cache based on the hash value of the second content fragment; if a third hash value exists among all the second hash values ​​in the shared key-value cache that matches the hash value of the second content fragment, determining the fragment length of the third content fragment corresponding to the third hash value in the first content fragment, and determining a fourth content fragment corresponding to the fragment length in the request content, the fourth content fragment having the same starting position as the second content fragment; if the hash value of the fourth content fragment matches the hash value of the third content fragment... If the first hash value of the segments matches, it is determined that there is a key value in the shared key-value cache that matches the fourth content segment, and the content segment of the request content that is after the fourth content segment of the fourth content segment is taken as the new second content segment; if there is no third hash value among all the second hash values ​​in the shared key-value cache or if the hash value of the fourth content segment does not match the first hash value of the third content segment, the content segment of the request content that is determined from the next position after the start position of the second content segment of the second content segment is taken as the new second content segment; based on the new second content segment, the step of searching in the shared key-value cache based on the hash value of the second content segment is repeated until the new second content segment includes the content segment corresponding to the last preset length in the request content.

[0048] In this embodiment, each historical content fragment in the shared key-value cache stores a corresponding key-value pair, which can be stored in vector form and can be stored separately in the GPU (Graphics Processing Unit). This disclosure does not impose any limitations on this. Each historical content fragment in the shared key-value cache also stores a corresponding word segment, a prefix hash value, and a full-length hash value. The prefix hash value represents the hash value corresponding to the first preset length of the historical content fragment, i.e., the second hash value. The full-length hash value represents the hash value corresponding to the entire historical content fragment, i.e., the first hash value. For example, if a historical content fragment includes 100 characters, the prefix hash value is calculated using the first 8 characters, and the full-length hash value is calculated using the 100 characters.

[0049] For example, since the reused historical content fragments are no longer fixed-length fragments but arbitrary-length fragments, a two-stage matching algorithm can be used to reduce the complexity of the substring matching problem from O(n×m) to O(n+c), where c is the number of candidate positions, achieving millisecond-level retrieval and meeting the performance requirements of online inference systems.

[0050] For example, such as Figure 2As shown, reusable fragment retrieval is performed through the retrieval module. A sliding window quickly filters candidate positions in the request content. The window length is consistent with the fragment length corresponding to the prefix hash. Taking a token sequence of 1-100 characters and a sliding window length of 8 characters as an example, the first character is taken as the candidate position, and the content fragment of 1-8 characters is taken. The shared key-value cache is searched for a prefix hash value that matches the hash value of the content fragment of 1-8 characters. If it exists, the fragment length of the historical content fragment corresponding to the matching prefix hash value is determined. Assuming the fragment length is 30 characters, the content fragment of 1-30 characters is taken in the content request, and the full-length hash value of the historical content fragment is compared with the hash value of the content fragment of 1-30 characters. If they match, it means that the match is successful. Then, the window of the content request is slid to the next window, that is, the 31st character is taken as the candidate position, and the content fragment of 31-38 characters is taken for further matching.

[0051] For example, if no prefix hash value matching the hash value of the content fragment of characters 1-8 is found in the shared key-value cache, or if the full-length hash value of the historical content fragment corresponding to the matching prefix hash value is inconsistent with the hash value of the content fragment of characters 1-30, slide to the next window of the content request, that is, take the second character as the candidate position, take the content fragment of characters 2-9 and continue to match, and so on, until the entire request content has undergone the matching step.

[0052] By storing prefix hashes of historical content fragments, a sliding window can be used to quickly filter candidate positions in new request content. Double hash verification is performed on candidate positions to ensure no collisions and confirm matches, thereby achieving efficient retrieval of dynamically reusable fragments and meeting the needs of online services.

[0053] In other possible implementation methods, historical content fragments stored in the shared key-value cache can be eliminated according to a preset strategy. For example, in the case of insufficient video memory, the least used historical content fragments and their corresponding key-values ​​can be deleted. The specific method can be selected according to the requirements, and this disclosure does not impose any restrictions on it.

[0054] Among other possible implementation methods, non-duplicate storage can also be configured, meaning that the same historical content fragment is stored only once to avoid data redundancy.

[0055] In other possible implementations, storage optimization can be implemented. Specifically, if duplicate sub-fragments exist across multiple historical content segments, these duplicate sub-fragments can be stored as new historical content segments in a shared key-value cache. For example, assuming there are key-value pairs corresponding to "abc", "abd", and "abe", the duplicate "ab" can be cached as a separate segment to improve efficient reuse for subsequent user requests. Specific configurations can be tailored to requirements, and this disclosure does not impose any limitations on this.

[0056] In one possible approach, after obtaining the request content input by the first user to the large language model, the key-value cache sharing method further includes: identifying protected content in the request content and determining unprotected content in the request content other than the protected content. For the first part of the request content where no matching key value exists in the shared key-value cache of the large language model, determining the first word segment corresponding to the first part includes: for the second part of the unprotected content where no matching key value exists in the shared key-value cache, determining the word segment corresponding to the second part, wherein the first word segment includes the word segment corresponding to the second part.

[0057] In this embodiment, the protected content may be user privacy content, content with business risks, or other content that cannot be exposed to other users. The specific settings are determined according to the actual business scenario, and this disclosure does not impose any restrictions on them.

[0058] For example, such as Figure 2 As shown, continuing with the word segmentation sequence "abcdefgijk" as an example, it can be identified through the detection module. Assuming "de" is the protected content, "de" is masked, and the word segmentation sequence corresponding to the unprotected content of "abcdefgijk" excluding "de" is automatically divided into two dynamically lengthed word segments: segmentation group 3 "abc" and segmentation group 4 "fgijk". Then, key-value storage is performed on segmentation group 3 "abc" and segmentation group 4 "fgijk", or key-value storage is performed after a more granular segmentation, depending on the word segmentation dependencies within the segmentation group.

[0059] In one possible approach, for the first part of the requested content where no matching key value exists in the shared key-value cache of the large language model, the first word segment corresponding to the first part is determined. This also includes: for the third part of the protected content where no matching key value exists in the shared key-value cache with a preset marker, the word segment corresponding to the third part is determined. The first word segment includes the word segment corresponding to the third part, and the key value with the preset marker is prohibited from use by users other than the first user.

[0060] In this embodiment, the protected content can also be stored using key-value pairs, or after finer-grained word segmentation, depending on the word segmentation dependencies within the segmentation group. Furthermore, to protect the content security of the protected content, such as... Figure 2As shown, protected content fragments can be marked and stored through the annotation module so that when the same user matches the protected content, the content fragment can be quickly identified as protected based on the marking. For example, the protected content fragment A can be marked based on the user's unique identifier. In this way, while determining that content fragment A is protected, it can also determine whether it can be reused by the first user. The specific settings can be configured according to the requirements, and this disclosure does not impose any restrictions on this.

[0061] Of course, unprotected content fragments can also be marked. For example, a field can be set to indicate whether the corresponding content fragment is protected. For example, 1 indicates protected content and 0 indicates unprotected content. The specific settings can be made according to the needs, and this disclosure does not impose any restrictions on this.

[0062] Thus, through the aforementioned word-segmentation-level sharing mechanism, a content detection and tagging mechanism is introduced into the shared key-value cache to determine whether the input sequence contains protected content on a word-by-word basis. Only keys containing unprotected content are allowed to be shared across users, while keys containing protected content are restricted to use within the same user's session. Through fine-grained control, the security risks of "full sharing" are overcome, achieving a balance between cache security and privacy protection.

[0063] Accordingly, when performing key-value retrieval using the retrieval model, if the matched historical content fragment is unprotected content or if the matched historical content fragment is protected content and the input user of that historical content fragment is the first user, the key-value pair corresponding to the historical content fragment can be obtained (e.g., ...). Figure 2 The KV1 shown is concatenated into the word segmentation sequence to obtain a new word segmentation sequence, which is then input into the large language model. This effectively ensures data security while supporting key-value reuse.

[0064] This method can reuse cached key values ​​of shared key-value cache to improve model inference efficiency, and effectively isolate protected content. It allows key values ​​of unprotected content to be shared across users, while prohibiting key values ​​of protected content from being shared across users. For example, in a multi-tenant environment, it can effectively improve the security and availability of model services.

[0065] In possible ways, identifying protected content in the request content includes: identifying protected content in the request content by at least one of the following methods: determining the portion of the request content that matches a preset rule as protected content; determining the portion of the request content that has business risks as protected content, where the preset model is used to identify the portion of the input content that has business risks; and treating the request content as protected content.

[0066] For example, this embodiment supports multiple methods for detecting protected content. For instance, preset rules can be set, such as using regular expressions to identify content like phone numbers. It can also automatically identify content like names and locations using a pre-trained preset model. Furthermore, a strict policy can be set to block all user-inputted content requests, meaning that all user-inputted content requests are treated as protected data that cannot be shared across users, and only non-user-input content such as system prompts or external knowledge can be shared and reused.

[0067] For example, protected content can be tagged and stored, so that even if an attacker makes inferences through cache hits, they cannot obtain any reuse signals of the protected content, effectively protecting data security.

[0068] In one possible approach, at least one second word segment is determined based on the attention matrix and the first word segment, including: determining all word substrings from the first word to the last word in the first word segment; for each word substring, calculating the first attention score between words within the word substring and the second attention score between words within the word substring and words outside the word substring based on the attention matrix; and determining at least one second word segment based on word substrings where the first attention score is greater than the second attention score.

[0069] For example, such as Figure 3 As shown, for the word segmentation group "abcdef", multiple word substrings can be obtained, which is equivalent to obtaining multiple candidate segments, including a, b, ..., f, ab, ..., ef, abc, ..., abcdef. If it is necessary to calculate the first attention score of the word substring "cde", it is necessary to calculate the internal attention score of the matrix formed by the c, d, and e rows and the c, d, and e columns in the attention matrix. Then, the external attention score of each word in the word substring "cde" with a, b, and f is calculated to obtain the second attention score. The specific value can be determined according to the actual situation, and this disclosure does not impose any restrictions on it.

[0070] It is worth noting that the first attention score can be the sum of all internal attention scores, and the second attention score can be the sum of all external attention scores. Alternatively, the first attention score can be the average of all internal attention scores, and the second attention score can be the average of all external attention scores. The specific values ​​can be set according to the requirements, and this disclosure does not impose any restrictions on them.

[0071] For example, a "abcdef" can eventually yield at least one reusable word segmentation, such as one "abcdef", or two "abc" and "def", or three "ab", "cde", "f", etc., which can be determined based on the actual calculation results.

[0072] Therefore, it is possible to perform a full sequence candidate fragment search for request content fragments that do not match historical content fragments, and finally select the optimal reusable fragment for storage, which significantly improves the hit rate and utilization of cache key values.

[0073] In one possible approach, determining at least one second word segmentation group based on the word segmentation substrings whose first attention score is greater than the second attention score includes: if there are overlapping word segmentation substrings among the word segmentation substrings whose first attention score is greater than the second attention score, taking the word segmentation substring with the largest first attention score among the overlapping word segmentation substrings as the second word segmentation group.

[0074] For example, assuming that both the segmented substring "abc" and the segmented substring "ab" have internal attention scores greater than external attention scores, then the internal attention scores of the two are compared, and the segmented substring with the largest internal attention score is selected as the optimal reusable segment for storage, thereby further improving the hit rate and utilization of cache keys.

[0075] In one possible approach, the first attention score between words within a segmented substring and the second attention score between words within a segmented substring and words outside the segmented substring are calculated based on the attention matrix. This includes: for each first element in the attention matrix, determining the attention submatrix formed by the first element and the element in the first row and first column of the attention matrix, and determining the sum of the elements in the attention submatrix, using the sum as the element value of the element at the same position as the first element in the attention integral matrix; wherein the attention integral matrix and the attention matrix are matrices of the same type; and calculating the first attention score between words within a segmented substring and the second attention score between words within a segmented substring and words outside the segmented substring based on the attention integral matrix.

[0076] It should be noted that when calculating the internal and external attention scores for each segmented substring based on the attention matrix, such as calculating the internal attention score for "bef" as mentioned above, it is necessary to accumulate them sequentially as follows: Figure 3 The attention scores corresponding to each element of the submatrix defined by the coarse circle in the attention matrix are computationally complex and inefficient.

[0077] like Figure 3 As shown, the attention integral matrix can be determined first based on the attention matrix. The attention matrix and the attention integral matrix are matrices of the same type. The element value of each element in the attention integral matrix is ​​equal to the sum of the element values ​​of the submatrix formed by the element at the same position in the attention matrix up to the element in the first row and first column. That is, the attention matrix is ​​preprocessed by the integral image.

[0078] Furthermore, when calculating the internal and external attention scores, the internal and external attention scores of any segmented substring can be calculated in O(1) time. For example, to calculate the internal attention score of "bef", only the following calculation is needed: Figure 3 The four element values ​​selected by the dashed box in the middle, namely the sum of the top-left and bottom-right element values, minus the bottom-left and top-right element values, yield the internal attention score for "bef" without needing to calculate it element-by-element. This allows for O(n) time complexity optimization. 2 The search for candidate segments of the entire sequence can be completed within a short time, effectively improving the storage efficiency of reusable segments.

[0079] To address the shortcomings of existing key-value cache sharing mechanisms in terms of security and reuse granularity, this embodiment provides a selective key-value cache sharing method based on word segmentation granularity. When a new user request arrives, the key-value pairs corresponding to matching historical content fragments are first retrieved from the shared key-value cache and appended to the user request. Missing key-value pairs are filled in using a large language model, generating model inference results and returning them to the user. After inference, reusable fragments and protected content in the user request are analyzed, and sharing permissions for corresponding key-value pairs are disabled. Reusable fragments are further divided and optimized, and the final data fragments are stored in the shared key-value cache for reuse in subsequent requests.

[0080] By introducing content detection before storing reusable fragments, fine-grained segmentation of the user-requested word sequence is performed. Combined with efficient fragment extraction and matching algorithms, secure reuse of key values ​​of unprotected content is achieved. This ensures the secure isolation of protected content while enabling efficient reuse of the context of unprotected content, thereby guaranteeing both data security and efficient reasoning.

[0081] Unlike existing technologies that employ secure but inefficient prohibited-sharing schemes and efficient but insecure indiscriminate-sharing schemes, this embodiment achieves a balance between security and performance through a selective-sharing approach. It effectively improves the reuse rate of storage segments while completely preventing the leakage of protected content and reducing the first token latency (TTFT).

[0082] Based on the same concept, embodiments of this disclosure also provide a key-value cache sharing system for large language models, such as... Figure 4 As shown, the key-value cache sharing system 400 includes a retrieval module 401, a detection module 402, an annotation module 403, and a shared key-value cache 404.

[0083] For example, it can also include an inference engine that deploys a large language model, or a key-value cache sharing system can include an inference engine, or it can interface with different inference engines by modifying the corresponding interfaces. This allows for efficient reuse and secure use of key-value pairs without changing the underlying logic of the large language model.

[0084] For example, by modularizing components such as the retrieval module, detection module, annotation module, and shared key-value cache, a scalable secure key-value management framework is formed. It can also support the replacement and expansion of different content detection tools, such as rule classes and model classes, and has good versatility and portability.

[0085] The retrieval module is responsible for quickly finding and retrieving historical content fragments that match the current user's request context from the shared key-value cache, and then concatenating the corresponding key-value pairs into the inference sequence. The inference engine completes the key-value pairs for missing or recalculated data fragments and outputs the final result based on the complete key-value sequence. After inference, the detection module performs content detection on the input segmented sequence to ensure that the key-value pairs corresponding to protected content are not shared across users. The annotation module analyzes attention distribution to determine which content fragments are reusable and which need to be recalculated in a new context, such as protected content. The shared key-value cache is responsible for storing and managing reusable key-value pairs, supporting efficient indexing, eviction, and scheduling.

[0086] The embodiments of each module of the above system have been described in detail in the corresponding method embodiments, and will not be repeated here.

[0087] Based on the same concept, embodiments of this disclosure also provide a key-value cache sharing device for a large language model, such as... Figure 5 As shown, the key-value cache sharing device 500 includes: The acquisition module 501 is used to acquire the request content input by the first user to the large language model; The first determining module 502 is used to determine the first word segment corresponding to the first part of the request content where there is no matching key value in the shared key value cache of the large language model; The second determining module 503 is used to determine the attention score between every two words in the first word segmentation group, obtain an attention matrix, and determine at least one second word segmentation group based on the attention matrix and the first word segmentation group; wherein, the attention score between words in the second word segmentation group is greater than the attention score between words in the second word segmentation group and words outside the second word segmentation group; Add module 504, used to determine the key value corresponding to the at least one second word segment and add it to the shared key value cache.

[0088] Optionally, the second determining module 503 is used to: Determine all word substrings between the first and last word segments in the first word segmentation group; For each segmented substring, a first attention score between segments within the segmented substring and a second attention score between segments within the segmented substring and segments outside the segmented substring are calculated based on the attention matrix. The at least one second word segment is determined based on the word segment substring whose first attention score is greater than the second attention score.

[0089] Optionally, the second determining module 503 is used to: For each first element in the attention matrix, an attention sub-matrix is ​​determined, consisting of the first element and the element in the first row and first column of the attention matrix. The sum of the elements in the attention sub-matrix is ​​then determined, and the sum is used as the element value of the element in the attention integral matrix that is at the same position as the first element. The attention integral matrix and the attention matrix are of the same type. Based on the attention integral matrix, calculate the first attention score between words within the segmented substring and the second attention score between words within the segmented substring and words outside the segmented substring.

[0090] Optionally, the second determining module 503 is used to: If there are overlapping substrings among the word segments whose first attention score is greater than the second attention score, the word segment with the largest first attention score among the overlapping substrings shall be taken as the second word segment.

[0091] Optionally, after obtaining the request content input by the first user to the large language model, the key-value cache sharing device 500 further includes a recognition module: The identification module is used to identify protected content in the request content and determine unprotected content in the request content other than the protected content. The first determining module 502 is used for: For the second part of the unprotected content where no matching key value exists in the shared key value cache, the word segmentation group corresponding to the second part is determined, and the first word segmentation group includes the word segmentation group corresponding to the second part.

[0092] Optionally, the first determining module 502 is further configured to: For the protected content, if there is no matching key value in the shared key value cache with preset tags for the third part, determine the word segmentation group corresponding to the third part, the first word segmentation group includes the word segmentation group corresponding to the third part, and the key value with preset tags is prohibited from being used by users other than the first user.

[0093] Optionally, the identification module is used for: The protected content in the requested content can be identified by at least one of the following methods: The portion of the request content that matches a preset rule is identified as the protected content; The portion of the request content that contains business risks, as determined by a preset model, is identified as the protected content. The preset model is used to identify the portion of the input content that contains business risks. The requested content is designated as the protected content.

[0094] Optionally, the shared key-value cache is further used to store the first hash value of the first content segment corresponding to each key-value and the second hash value of the segment corresponding to the preset length before the first content segment; The key-value cache sharing device 500 further includes a retrieval module, which is used for: Determine the second content segment of the previously preset length in the request content, and retrieve it in the shared key-value cache based on the hash value of the second content segment; If a third hash value exists among all the second hash values ​​in the shared key-value cache that is consistent with the hash value of the second content fragment, the fragment length of the third content fragment corresponding to the third hash value in the first content fragment is determined, and a fourth content fragment corresponding to the fragment length is determined in the request content, wherein the fourth content fragment has the same starting position as the second content fragment; If the hash value of the fourth content segment is consistent with the first hash value of the third content segment, it is determined that there is a key value in the shared key-value cache that matches the fourth content segment, and the content segment of the preset length in the request content after the fourth content segment is taken as the new second content segment. If the third hash value is not found among all the second hash values ​​in the shared key-value cache, or if the hash value of the fourth content fragment is inconsistent with the first hash value of the third content fragment, the fragment content of the preset length determined from the next position after the start position of the second content fragment in the request content shall be used as the new second content fragment. Based on the new second content fragment, the step of retrieving the hash value based on the second content fragment in the shared key-value cache is repeated until the new second content fragment includes the content fragment corresponding to the last preset length in the request content.

[0095] The key-value cache sharing device 500 further includes a model processing module, which is used for: For the requested content, if there is a fourth part of a matching key value in the shared key-value cache of the large language model, the key value corresponding to the fourth part is obtained from the shared key-value cache; Based on the key values ​​corresponding to the first part and the key values ​​corresponding to the fourth part, a response content is generated for the requested content.

[0096] Based on the same concept, embodiments of this disclosure also provide a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the above-described key-value cache sharing method for a large language model.

[0097] Based on the same concept, this disclosure also provides an electronic device that may include: A storage device on which computer programs are stored; A processing device for executing a computer program stored in a storage device to implement the steps of the key-value cache sharing method for the large language model described above.

[0098] Based on the same concept, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described key-value cache sharing method for a large language model.

[0099] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0100] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0101] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0102] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0103] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0104] In some implementations, communication can be conducted using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can be interconnected with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0105] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0106] The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following: to acquire request content input by a first user to a large language model; for a first part of the request content where no matching key value exists in the shared key-value cache of the large language model, to determine a first word segment corresponding to the first part; to determine the attention score between every two words in the first word segment to obtain an attention matrix, and based on the attention matrix and the first word segment, to determine at least one second word segment; wherein the attention score between words within the second word segment is greater than the attention score between words within the second word segment and words outside the second word segment; and to determine the key value corresponding to the at least one second word segment and add it to the shared key-value cache.

[0107] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0110] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0111] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0112] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0113] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0114] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A key-value cache sharing method of a large language model, characterized in that, The key-value cache sharing method comprises: obtaining request content input by a first user to a large language model; for a first part of the request content, which does not have a matching key-value in the shared key-value cache of the large language model, determining a first token group corresponding to the first part; determining an attention score between each two tokens in the first token group to obtain an attention matrix, and determining at least one second token group based on the attention matrix and the first token group; wherein the attention score between tokens in the second token group is greater than the attention score between a token in the second token group and a token outside the second token group; determining a key-value corresponding to the at least one second token group and adding it to the shared key-value cache.

2. The key-value cache sharing method of a large language model according to claim 1, characterized in that, The method comprises: determining all token substrings between the first token and the last token in the first token group; for each token substring, calculating a first attention score between tokens in the token substring and a second attention score between a token in the token substring and a token outside the token substring based on the attention matrix; based on the token substrings with the first attention score greater than the second attention score, determining the at least one second token group.

3. The key-value cache sharing method of a large language model according to claim 2, characterized in that, The method comprises: for each first element in the attention matrix, determining an attention sub-matrix formed by the first element and the element in the first row and the first column of the attention matrix, and determining the sum of the elements in the attention sub-matrix, taking the sum as the element value of the element in the same position in the attention integral matrix; wherein the attention integral matrix and the attention matrix are the same type of matrix; calculating the first attention score between tokens in the token substring and the second attention score between a token in the token substring and a token outside the token substring based on the attention integral matrix.

4. The key-value cache sharing method of a large language model according to claim 2, characterized in that, The method comprises: if there are overlapping token substrings in the token substrings with the first attention score greater than the second attention score, taking the token substring with the largest first attention score in the overlapping token substrings as the second token group.

5. The key-value cache sharing method of a large language model according to any one of claims 1-4, characterized in that, After obtaining the request content input by the first user to the large language model, the key-value cache sharing method further comprises: identifying protected content in the request content and determining non-protected content in the request content excluding the protected content; The method comprises: For a second part of the unprotected content in which there is no matching key value in the shared key value cache, determine a second token group corresponding to the second part, and the first token group includes the second token group corresponding to the second part.

6. The key-value cache sharing method of a large language model according to claim 5, characterized in that, The determining the first token group corresponding to the first part of the request content in which there is no matching key value in the shared key value cache of the large language model further includes: For a third part of the protected content in which there is no matching key value in the shared key value cache with a preset mark, determine a third token group corresponding to the third part, and the first token group includes the third token group corresponding to the third part. The key value with a preset mark is prohibited from being used by users other than the first user.

7. The key-value cache sharing method of a large language model according to claim 5, characterized in that, The identifying the protected content in the request content includes: The protected content in the request content is identified by at least one of the following ways: Determine the part of the request content that matches the preset rule as the protected content; Determine the part of the request content that has business risks through a preset model as the protected content. The preset model is used to identify the part of the input content that has business risks. The request content is the protected content.

8. The key-value cache sharing method of a large language model according to any one of claims 1-4, characterized in that, The shared key value cache is also used to store the first hash value of each key value corresponding to the first content segment and the second hash value of the segment corresponding to the first content segment with a preset length; The key value cache sharing method further includes: Determine the second content segment with a preset length in the request content, and perform retrieval in the shared key value cache based on the hash value of the second content segment; In the case that there is a third hash value consistent with the hash value of the second content segment in all second hash values in the shared key value cache, determine the segment length of the third content segment corresponding to the third hash value in the first content segment, and determine the fourth content segment corresponding to the segment length in the request content. The fourth content segment has the same starting position as the second content segment; In the case that the hash value of the fourth content segment is consistent with the first hash value of the third content segment, it is determined that there is a key value matching the fourth content segment in the shared key value cache, and the content segment with a preset length after the fourth content segment in the request content is taken as a new second content segment; In the case that there is no third hash value in all second hash values in the shared key value cache or the hash value of the fourth content segment is inconsistent with the first hash value of the third content segment, the segment content with a preset length determined from the next position of the starting position of the second content segment in the request content is taken as a new second content segment; Based on the new second content segment, repeat the step of performing retrieval in the shared key value cache based on the hash value of the second content segment until the new second content segment includes the content segment corresponding to the last preset length in the request content.

9. The key-value cache sharing method of a large language model according to any one of claims 1-4, characterized in that, The key value cache sharing method further includes: For a fourth part of the request content in which there is a matching key value in the shared key value cache of the large language model, a key value corresponding to the fourth part is obtained from the shared key value cache; Based on the key value corresponding to the first part and the key value corresponding to the fourth part, reply content for the request content is generated.

10. A key-value cache sharing apparatus of a large language model, characterized in that, The key value cache sharing device comprises: An obtaining module is configured to obtain request content input by a first user to a large language model; A first determining module is configured to, for a first part of the request content in which there is no matching key value in the shared key value cache of the large language model, determine a first token group corresponding to the first part; A second determining module is configured to determine an attention score between each two tokens in the first token group to obtain an attention matrix, and determine at least one second token group based on the attention matrix and the first token group; wherein the attention score between tokens within the second token group is greater than the attention score between tokens within the second token group and tokens outside the second token group; An adding module is configured to determine a key value corresponding to the at least one second token group and add the key value to the shared key value cache.

11. A computer readable medium having stored thereon a computer program, characterized in that The computer program, when executed by a processing device, implements the steps of the method of any one of claims 1-9.

12. An electronic device, comprising: Comprise: A storage device having a computer program stored thereon; A processing device configured to execute the computer program in the storage device to implement the steps of the method of any one of claims 1-9.

13. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method of any one of claims 1-9. The computer program, when executed by a processor, implements the steps of the method of any one of claims 1-9.

Images