On-demand loading dynamic cache management method and electronic equipment

Abstract

The invention provides an on-demand loading dynamic cache management method and electronic equipment. The method comprises the following steps: acquiring a prefix part in a cue word; performing attention calculation on a plurality of lexical elements included in the prefix part to obtain a key vector, a value vector and an attention score corresponding to each lexical element in the plurality of lexical elements; dividing a plurality of lexical elements included in the prefix part into M groups; selecting m groups with first relevancy from the M groups according to the attention score; m < M; and in response to a received first request sent by the terminal or the edge computing device, issuing first cache data corresponding to the m groups to the terminal or the edge computing device so as to trigger the terminal or the edge computing device to execute a subsequent reasoning process based on the first cache data, and outputting a first lexical element responding to the user question. The method provided by the invention at least can obviously shorten the TTFT.

Description

Technical Field

[0001] This invention relates to the field of key-value caching technology, and more particularly to a dynamic cache management method and electronic device for on-demand loading. Background Technology

[0002] In the autoregressive generation process of large language models, tokens are generated one by one. When calculating the t-th token, the Transformer decoder needs to take the previous t-1 tokens as input and recalculate the key and value vectors, which leads to a large amount of redundant computation and is extremely inefficient. The basic idea of ​​the KV caching strategy is to cache the key and value vectors of all previous tokens after generating the first token. When generating the next token, the cached KV vectors are used directly, and new KV vectors are only calculated for the newly generated token. In this way, a large amount of redundant K and V calculations can be avoided, improving inference efficiency.

[0003] Currently, terminal devices (such as mobile terminals) or edge computing devices (or edge devices) often face the problem of limited computing and storage resources when running large language models. They need to rely on the computing power and storage space of the cloud to request key and value cache data from the cloud to realize the inference of large language models. The high communication latency that may occur between terminal devices or edge computing devices and the cloud will result in an excessively long response time for the generation of the first token, which will reduce the user experience. Summary of the Invention

[0004] This invention provides a dynamic cache management method and electronic device for on-demand loading, which can effectively reduce the amount of data transmitted between terminal devices or edge computing devices and the cloud, and significantly shorten the response time for generating the first token.

[0005] This invention provides an on-demand dynamic cache management method applicable to cloud servers. The method includes: obtaining a prefix portion of a prompt word; performing attention calculation on multiple lexical units included in the prefix portion to obtain a key vector, value vector, and attention score corresponding to each lexical unit; dividing the multiple lexical units included in the prefix portion into M groups; selecting m groups with a first relevance from the M groups based on the attention scores; m < M; the first relevance represents the relevance between the lexical unit in the group and the next lexical unit to be generated; responding to a first request received from a terminal or edge computing device, sending first cache data corresponding to the m groups to the terminal or edge computing device to trigger the terminal or edge computing device to execute subsequent inference processes based on the first cache data and output the first lexical unit responding to the user's question; wherein the first cache data is a portion of all cache data corresponding to the M groups.

[0006] Optionally, based on the attention scores, select m groups with a first relevance from the M groups, including: obtaining a statistical value of the attention score of each group based on the attention score corresponding to at least one word character in each of the M groups; selecting m groups from the M groups whose statistical values ​​are greater than or equal to a preset threshold as the m groups with a first relevance based on the statistical values; or, sorting the M groups based on the statistical values ​​and selecting m groups whose statistical values ​​rank in the top preset percentage as the m groups with a first relevance.

[0007] Optionally, after selecting m groups with a first relevance from the M groups, the method further includes: determining at least one group other than the m groups in the M groups as having a second relevance; storing the key vectors and value vectors corresponding to the words in the m groups with the first relevance according to a first precision and / or compressing them according to a first compression ratio to obtain first cached data; storing the key vectors and value vectors corresponding to the words in the at least one group with the second relevance according to a second precision and / or compressing them according to a second compression ratio; wherein the first precision is higher than the second precision, and the second compression ratio is greater than the first compression ratio.

[0008] This invention also provides an on-demand dynamic cache management method, which can be applied to a terminal or edge computing device. The terminal or edge computing device deploys a large language model, which includes N neural network layers. The method includes: generating a prompt word in response to receiving a user question; the prompt word includes a prefix; sending a first request to the cloud; the first request is used to request the cloud to obtain the first cache data corresponding to the nth layer in the large language model; 1 ≤ n ≤ 4 < N; wherein, the cloud performs attention calculation on multiple lexical units included in the prefix to obtain the key vector, value vector, and attention score corresponding to each lexical unit; receiving the first cache data corresponding to m groups sent by the cloud in response to the first request; wherein, the m groups are groups with a first relevance selected by the cloud from M groups based on the attention score; the first relevance is used to represent the relevance between the lexical unit in the group and the next lexical unit to be generated; the M groups are obtained by dividing the multiple lexical units included in the prefix; m < M; the first cache data is a portion of all cache data corresponding to the M groups; based on the first cache data, performing subsequent inference processes and outputting the first lexical unit responding to the user question.

[0009] Optionally, the subsequent reasoning process is performed, including: during the reasoning process from the (n+1)th layer to the Nth layer, every x layers, the priority of the group corresponding to each word in the prefix part is redefined, and multiple groups with the first relevance are selected from the M groups; x≥2.

[0010] Optionally, every x layers, the priority of the group corresponding to each word in the prefix part is redefined, and multiple groups with first relevance are selected from M groups, including: at the p-th layer of the large language model, attention calculation is performed on all multiple words included in the prefix part to obtain the calculation result corresponding to the p-th layer; the calculation result of the p-th layer includes the key vector, value vector and attention score of each word in all multiple words; there is an x-layer interval between the p-th layer and the n-th layer; based on the calculation result of the p-th layer, l groups with first relevance are selected from the M groups, where l=m or l≠m; l <M; after selecting multiple groups with first relevance from the M groups, the method further includes: sending a second request to the cloud; the second request is used to request the cloud to obtain the second cache data corresponding to multiple words in the l groups; in response to receiving the second cache data sent from the cloud, the second cache data is written to the local cache; the second cache data is input to the p+1-th layer to perform the attention calculation of the p+1-th layer.

[0011] Optionally, the subsequent inference process may further include: determining that at least one group other than l groups in the calculation results of the p-th layer has a second relevance; and writing the key vector and value vector corresponding to at least one group with the second relevance into the standby cache.

[0012] Optionally, every x layers, the priority of the group corresponding to each word in the prefix part is redefined, and multiple groups with first relevance are selected from M groups, including: gradually reducing the number of groups with first relevance according to the arrangement order of different layers in the large language model.

[0013] Optionally, after performing attention calculations on all multiple lexical units included in the prefix part at the p-th layer of the large language model and obtaining the calculation results corresponding to the p-th layer, the method further includes: writing the key vectors and value vectors corresponding to all multiple lexical units into a local cache; after selecting o groups with first relevance from M groups based on the calculation results of the q-th layer, the method further includes: determining at least one other group in the M groups other than the o groups as having second relevance; and releasing the key vectors and value vectors corresponding to at least one group with second relevance from the local cache.

[0014] This invention also proposes a dynamic cache management device based on on-demand loading, applicable to the cloud. The device includes: an acquisition module for acquiring the prefix portion of a prompt word; a calculation module for performing attention calculations on multiple lexical units included in the prefix portion to obtain the key vector, value vector, and attention score corresponding to each lexical unit; a grouping module for dividing the multiple lexical units included in the prefix portion into M groups; a relevance identification module for selecting m groups with a first relevance from the M groups based on the attention scores; m < M; the first relevance represents the relevance between the lexical unit in the group and the next lexical unit to be generated; and a sending module for, in response to receiving a first request from a terminal or edge computing device, sending first cache data corresponding to the m groups to the terminal or edge computing device to trigger the terminal or edge computing device to execute subsequent inference processes based on the first cache data and output the first lexical unit responding to the user's question; wherein the first cache data is a portion of all cache data corresponding to the M groups.

[0015] This invention also proposes a dynamic cache management device based on on-demand loading. This device can be applied to terminal devices or edge computing devices, which deploy a large language model comprising N neural network layers. The device includes: a prompt word module for generating prompt words in response to a received user question; the prompt words include a prefix; a request module for sending a first request to the cloud; the first request requests the cloud to retrieve the first cache data corresponding to the nth layer of the large language model, where 1 ≤ n ≤ 4 < N. The cloud performs attention calculations on multiple lexical units included in the prefix, obtaining the key vector, value vector, and attention score corresponding to each lexical unit; and a receiving module for receiving the first cache data corresponding to m groups sent by the cloud in response to the first request. Among them, m groups are selected by the cloud from M groups based on attention scores and have the first relevance; the first relevance is used to represent the relevance between the word in the group and the next word to be generated; M groups are obtained by dividing the multiple words included in the prefix part; m < M; the first cache data is a part of the data in all cache data corresponding to the M groups; the inference module is used to perform the subsequent inference process based on the first cache data and output the first word in response to the user's question.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the on-demand dynamic cache management methods or multi-agent training methods described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the on-demand dynamic cache management methods or multi-agent training methods described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the on-demand dynamic cache management methods or multi-agent training methods described above.

[0019] The present invention provides an on-demand dynamic cache management method and electronic device. It proposes an on-demand cache management method that performs parallel computation of multiple tokens in the prefix part in the cloud to obtain the key vector, value vector and attention score corresponding to each token. The key vector and value vector themselves, or the data obtained after precision adjustment, compression and other processing, are the K and V cache data. The prefix part is divided into M groups. Based on the attention score, m groups with the highest relevance are selected from the M groups. For example, the m groups with higher relevance are selected. In other words, based on the attention score, the words in the prefix part with higher relevance to the next word in the output are selected, while words with low relevance are ignored. The m groups are the data that the terminal device needs to predict the first token. The remaining data in the other groups of the M groups can be considered as temporarily unnecessary data. Therefore, when the terminal or edge computing device requests the cloud to load cached data, the cloud can only send the cached data corresponding to multiple words in the m groups (the first cached data), without sending the cached data of all M groups to the terminal or edge computing device. This on-demand loading method maintains the integrity of key information and effectively reduces the amount of data transmitted. Compared with the full transmission method in the traditional technical solution, it can shorten the time to generate the first token. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the system architecture of the on-demand dynamic cache management method provided in the embodiments of the present invention.

[0022] Figure 2 This is a schematic diagram illustrating the interaction process between the cloud and the terminal / edge computing device in one embodiment of the on-demand dynamic cache management method provided by the present invention.

[0023] Figure 3 This is a schematic diagram illustrating the process of selecting highly relevant groups and group compression in one embodiment of the on-demand dynamic cache management method provided by the present invention.

[0024] Figure 4 This is a flowchart illustrating the on-demand dynamic cache management method (cloud-based) provided in this embodiment of the invention.

[0025] Figure 5 This is a flowchart illustrating the on-demand dynamic cache management method (on the terminal or edge computing device side) provided in an embodiment of the present invention.

[0026] Figure 6 This is a schematic diagram of the neural network architecture in one embodiment of the on-demand loading dynamic cache management method provided by the present invention.

[0027] Figure 7 This is a schematic diagram of the process of re-determining relevance at layer p in one embodiment of the on-demand dynamic cache management method provided by the present invention.

[0028] Figure 8 This is a flowchart illustrating the auxiliary caching mechanism in one embodiment of the on-demand dynamic cache management method provided by the present invention.

[0029] Figure 9 This is a schematic diagram of the module structure of the dynamic cache management device (cloud) based on on-demand loading provided in an embodiment of the present invention.

[0030] Figure 10 This is a schematic diagram of the module structure of a dynamic cache management device (terminal or edge computing device side) based on on-demand loading provided in an embodiment of the present invention.

[0031] Figure 11 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention.

[0032] Figure label: 901: Acquisition module; 902: Calculation module; 903: Grouping module; 904: Relevance identification module; 905: Sending module; 1001: Prompt word module; 1002: Request module; 1003: Receive module; 1004: Reasoning module; 1110: Processor; 1120: Communication interface; 1130: Memory; 1140: Communication bus. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0034] When running large language models, especially when processing long contextual inputs (long input sequences), mobile terminals and edge computing devices need to build and maintain a massive key-value cache. Limited by the local computing and storage resources of the terminal or edge computing device, relying solely on local full computation would lead to device overload. Therefore, optimization of KV cache data management is necessary. The optimization solutions mainly fall into three categories: The first type is centralized cloud computing. The cloud directly computes the long-context cache and transmits it to the terminal. In other words, the terminal or edge computing device uses the computing power and storage space of the cloud to compute the KV cache data. The cloud and the terminal transmit the KV cache data over the network, enabling the terminal to perform model inference. While this method reduces the local computing pressure on the terminal or edge computing device, traditional solutions rely on the full transmission of cached data between the cloud and the terminal. This means that all KV cache data needs to be transmitted over the network, resulting in a high dependence on the network. Network transmission latency can lead to high communication delays, especially in wireless networks, where the Time To First Token (TTFT) generation time remains too long.

[0035] The second type is overall compressed storage, which reduces the cache size of KV cache data through a uniform compression algorithm. Although it can alleviate memory pressure to some extent, the compression strategy is usually fixed and does not take into account the importance of the differences between different inputs at different inference steps, resulting in a large amount of redundant storage.

[0036] The third type is static pruning, which uniformly removes some tokens from the input sequence before inference begins. This type of method is difficult to guarantee accuracy because the dependency on tokens in the input sequence changes dynamically in different generation steps (inference steps).

[0037] It is evident that some optimization methods in related technologies rely too heavily on the network, while others lack flexibility in caching and fail to fully utilize context redundancy.

[0038] Therefore, a new method is needed to solve at least one of the above problems.

[0039] In view of this, embodiments of the present invention propose a dynamic cache management method and electronic device for on-demand loading, which can realize on-demand loading, hierarchical compression, and dynamic scheduling. It can accelerate the generation of the first token and the overall inference process, significantly reduce the cache size, and reduce the transmission burden without sacrificing inference accuracy.

[0040] Specifically, this embodiment of the invention proposes an on-demand loading and grouping, hierarchical compression mechanism: the cached data corresponding to long contexts is grouped and hierarchically compressed, the highly relevant parts are prioritized for fidelity, the low-relevance parts are loaded with a delay, and the amount of data in the highly relevant parts is gradually reduced according to the order of the layers.

[0041] Among them, the on-demand loading mechanism dynamically loads highly relevant cache blocks during the inference process, rather than loading all cache blocks at once, reducing the latency of the first token and alleviating local memory pressure.

[0042] Optionally, embodiments of the present invention also propose a dynamic scheduling strategy: combining hierarchical reduction and importance prediction, the cached data of tokens stored in the cache is replaced and released, so that the reasoning ability of long context is still maintained under limited resources.

[0043] The following is combined Figures 1-11 This invention describes an on-demand dynamic cache management method and electronic device according to embodiments of the present invention.

[0044] Figure 1 This is a schematic diagram of the hardware system architecture of the on-demand dynamic cache management method provided in this embodiment of the invention. The system architecture includes a cloud, an edge computing device S01, and terminal devices D01, D02, and D03. The edge computing device S01 can provide computing services to the terminal device D03. Terminal devices D01 and D02 can interact directly with the cloud, or they can interact with the cloud through the edge computing device S01. The cloud includes servers or server clusters.

[0045] The terminal devices or edge computing devices deploy large language models. The same large language model can be deployed in the cloud or used only to perform corresponding calculations.

[0046] Figure 2 This is one of the flowcharts illustrating the on-demand dynamic cache management method provided in this embodiment of the invention, combined with... Figure 1 The hardware system architecture shown, from the perspective of interaction between the cloud and terminal / edge computing devices, the on-demand dynamic cache management method provided in this embodiment of the invention may specifically include the following steps: Step 100: In response to receiving a user question, the terminal device or edge computing device generates a prompt word.

[0047] The prompt words include a prefix. For example, the complete context (input sequence) input by the large language model when generating a response generally includes: system prompt words + user prompt words, and may also include dialogue history (which may or may not exist). The prompt words include system prompt words and user prompt words. User prompt words may include multiple parts such as role settings, tasks / instructions, context, input data (the user-inputted question), output instructions, and a few examples. In this embodiment, the prefix refers to the common parts of the prompt words that are unrelated to user privacy data. For example, the prefix includes at least the system prompt words, and may also include common prompt parts of the user prompt words that are unrelated to user privacy data. For example, roles, tasks / instructions, output instructions, examples, etc., may be parts unrelated to user personal information. Therefore, in addition to system prompt words, the prefix may also include these parts unrelated to user personal information. In some embodiments, the prefix may be a system instruction word.

[0048] Optimizing only the prefix is ​​to protect user privacy. User-related data, such as conversation history and user-inputted questions, which may involve privacy, is stored locally on the terminal device. The cloud only calculates and filters the more important data from the common part (prefix), thus reducing the amount of data transmitted over the network. It should be noted that, obviously, with user authorization, all the technical means proposed in this invention (such as on-demand loading and layered compression) can be used to optimize the entire input sequence (complete prompts + possible conversation history).

[0049] Step 101: Obtain the prefix part of the prompt word from the cloud.

[0050] The cloud can obtain the prefix part of the prompt words in two ways: either the terminal generates the corresponding prompt words based on the user's input question and then actively uploads them to the cloud, or the cloud actively obtains them from the terminal.

[0051] Step 102: The cloud performs attention calculations on the multiple tokens included in the prefix part to obtain the calculation results.

[0052] The calculation results include the key vector, value vector, and attention score for each of the multiple tokens. It should be noted that a query vector is also obtained during the actual calculation.

[0053] A large language model is installed in the terminal device or edge computing device. The cloud can deploy a large language model that is exactly the same as the large language model in the terminal device or edge computing device, or it can be used only for performing calculations without installing a large language model.

[0054] Step 103: The cloud divides the multiple lexical units included in the prefix into M groups.

[0055] There are several ways to divide it, such as the following: Method 1: Divide the tokens into M equal groups. The length of each of the M groups is fixed; in other words, the number of tokens in each group is fixed, for example, 2-10 tokens per group.

[0056] In one embodiment, a group may have only one token.

[0057] Method 2: The partitioning is not equal; the length of each group is variable; the number of tokens within a group is not fixed; and the number of tokens in different groups can be the same or different.

[0058] For example, since highly relevant tokens tend to appear in clusters, we can first select several tokens with high attention scores (e.g., attention scores exceeding 0.5) based on the calculated attention scores. Then, we can group each of the selected tokens with a predetermined number of tokens before and after it (e.g., 5 tokens) into a group. Then, we can randomly divide the portion with attention scores below 0.5 into multiple groups in an equal or unequal manner. In this way, we can obtain M groups.

[0059] Step 104: Based on the attention scores, select the m groups with the first relevance from the M groups.

[0060] Where m < M, meaning m groups are a subset of M groups. Relevance represents the correlation between a token in a group and the next token to be generated. In fact, the attention score itself represents the relevance (relevance) between each token and the next token to be inferred, that is, which token in the current input sequence the next token pays more attention to.

[0061] The first relevance can represent the highest relevance, that is, the output that is most relevant to the next token.

[0062] Exemplarily, in some embodiments, such as Figure 3 As shown, based on the attention scores, m groups with the first relevance are selected from the M groups, which can be done using the method defined in steps 1041 to 1042: Step 1041: Based on the attention score corresponding to at least one word in each of the M groups, obtain the statistical value of the attention score for each group.

[0063] For example, the statistical value can be the average. If a group includes 10 tokens, the average of these 10 tokens is calculated as the statistical value of the attention score for this group. In other embodiments, the statistical value can also be a value obtained by linearly transforming the average, such as multiplying the average by 10.

[0064] Next, based on the statistical values, select m groups from the M groups. This can be achieved using either step 10421 or step 10422. Step 10421: Based on the statistical values, select m groups from the M groups whose statistical values ​​are greater than or equal to a preset threshold, and use them as the m groups with the first relevance.

[0065] For example, when the statistical value represents the mean, a preset threshold is set to 0.2-0.4, and groups with a mean attention score greater than or equal to 0.2 are identified as having the first relevance. In this way, m groups can be obtained.

[0066] Alternatively, when a linear transformation is performed on the statistical value based on the mean, such as multiplying it by a constant of 10, the preset threshold can be set to 2-4. Groups exceeding the preset threshold are identified as groups with the first correlation degree. In this way, m groups can be obtained.

[0067] Step 10422: Sort the M groups according to the statistical values, and select the m groups with the highest statistical values ​​by a predetermined percentage as the m groups with the first relevance.

[0068] For example, in the shallow layers (the first n layers), the default percentage is 80%.

[0069] Optionally, in some embodiments, the following continues: Figure 3 As shown, after step 10421 or 10422, the following steps may also be included: Step 1043: Determine at least one group other than m groups out of the M groups as having the second relevance.

[0070] The remaining Mm groups out of the M groups are identified as having a second relevance. The second relevance is lower than the first relevance, indicating that the tokens in the remaining Mm groups are not highly relevant to the next token to be generated through reasoning.

[0071] It should be noted that the embodiments of the present invention only use two relevance levels as examples. In fact, in other embodiments, more than two relevance levels can be set, such as a third relevance level or a fourth relevance level. The level division of relevance is more granular. At least one level with the highest relevance can be selected from multiple relevance levels and added to m groups. The remaining tokens are considered to be tokens that can be ignored in the first round of reasoning (generating the first token).

[0072] Relevance can also be referred to as priority, importance, or correlation.

[0073] Step 1044: Store the key vectors and value vectors corresponding to the words in the m groups with the first relevance according to the first precision and / or compress them according to the first compression ratio to obtain the first cache data.

[0074] Step 1045: Store the key vectors and value vectors corresponding to the tokens in at least one group with second relevance according to the second precision and / or compress them according to the second compression ratio.

[0075] Specifically, for tokens in the m groups with higher relevance (first relevance), higher precision is used for storage and a lower compression ratio is used for compression. For tokens in the other Mm groups with lower relevance, lower precision is used for storage and a higher compression ratio is used for compression. In other words, the first precision is higher than the second precision, and the second compression ratio is higher than the first compression ratio.

[0076] For example, the first precision in step 1044 can be an 8-bit floating-point format (FP8) or a half-precision floating-point format (FP16). The second precision is lower than the first precision; for example, the second precision is an 8-bit integer (INT8).

[0077] For example, the compression method can be similar to that of JPEG images, treating the K and V data as a pixel matrix for compression, preserving low-frequency features and compressing high-frequency features. The compression ratio for m groups should be less than that for Mm groups, resulting in compressed cached data. The cached data corresponding to the m groups is defined as the first cached data. The first cached data includes the data where the calculated key vectors (K) and value vectors (V) corresponding to multiple tokens in each of the m groups are compressed and stored with different precisions; this part of the data is the KV Cache data corresponding to the m groups.

[0078] It should be noted that, in this embodiment of the invention, the cached data can be the K and V vectors themselves, or it can be the cached data after the K and V vectors have been compressed and their precision adjusted.

[0079] Step 105: The terminal device or edge computing device sends the first request to the cloud.

[0080] The first request is used to request the first cached data corresponding to the prefix part of the nth layer in the large language model from the cloud.

[0081] The nth layer is a shallow layer of the large language model, such as one of the first 4 or 5 layers. For example, assuming the large language model has a total of N neural network layers, 1 ≤ n ≤ 4 < N. The cloud layer is used to perform attention calculations on multiple lexical units included in the prefix part, obtaining the key vector, value vector, and attention score corresponding to each lexical unit in the prefix part.

[0082] Step 106: In response to receiving the first request sent by the terminal or edge computing device, the cloud sends the first cached data to the terminal or edge computing device.

[0083] The first cached data contains only the cached data corresponding to the key vectors and value vectors of multiple tokens in the m groups. In other words, the first cached data is a portion of the total cached data corresponding to the M groups, and it is the portion containing key information.

[0084] Step 107: The terminal device or edge computing device receives the first cached data corresponding to the m groups issued by the cloud in response to the first request.

[0085] Step 108: The terminal device or edge computing device performs the subsequent reasoning process based on the first cached data and outputs the first word in response to the user's question.

[0086] For example, if n=1, the cached K and V data sent from the cloud will be used when performing attention calculations in the second layer.

[0087] Thus, according to Figure 2 In the illustrated embodiment, the parallel computation task of multiple tokens in the prefix part of the pre-filling stage (preprocessing stage) of the large language model is handed over to the cloud. After the cloud performs parallel computation, it selects m groups that are more relevant to the first token to be predicted. Only the cached data of the m groups needs to be transmitted to the terminal device or edge computing device. That is, compared with the traditional transmission scheme, only a part of the data needs to be transmitted to the terminal or edge computing device. While retaining key information (highly relevant information, important information), the amount of data transmission is reduced, thereby shortening the response time for generating the first token and improving the user experience.

[0088] according to Figure 2 The interaction flow shown can be seen from the perspective of the cloud alone, such as Figure 4 As shown, the method proposed in this embodiment of the invention may include the following process: Step 401: Obtain the prefix part of the prompt word.

[0089] Step 402: Perform attention calculation on the multiple lexical units included in the prefix part to obtain the key vector, value vector and attention score corresponding to each lexical unit.

[0090] Step 403: Divide the multiple lexical units included in the prefix into M groups.

[0091] Step 404: Based on the attention scores, select the m groups with the first relevance from the M groups.

[0092] Step 405: In response to receiving the first request sent by the terminal or edge computing device, send the first cached data corresponding to m groups to the terminal or edge computing device.

[0093] After the first cached data is sent to the terminal device or edge computing device, it can trigger the terminal or edge computing device to execute the subsequent reasoning process based on the first cached data and output the first token in response to the user's question.

[0094] The first cached data is a portion of the total cached data corresponding to the M groups. Compared to the full cached data transmission, this reduces the amount of data transmitted, thereby reducing the latency of network data transmission and shortening the response time for generating the first token.

[0095] according to Figure 2 The interaction flow shown can be interpreted from the perspective of either the terminal device or the edge computing device, as follows: Figure 5 As shown, the method proposed in this embodiment of the invention may include the following process: Step 501: In response to receiving a user question, generate a prompt word.

[0096] For example, based on the user's question, a user prompt is generated, and then combined with the system prompt to obtain the complete prompt. The prompt, excluding the user's question, may contain other parts that are not visible to the user; that is, the prefix portion may not be displayed to the user.

[0097] Step 502: Send the first request to the cloud.

[0098] The first request is used to request the first cached data corresponding to the nth layer in the large language model from the cloud, for example, 1≤n≤4<N.

[0099] Step 503: Receive the first cached data corresponding to the m groups sent by the cloud in response to the first request.

[0100] Step 504: Based on the first cached data, execute the subsequent reasoning process and output the first word in response to the user's question.

[0101] Compared to related technologies such as full-data caching, static compression, or static pruning, the embodiments of this invention achieve a better balance between efficiency and accuracy. Full-data caching suffers from excessive latency in wireless networks; static compression ignores contextual differences, leading to redundant storage; and static pruning is prone to losing critical information. The embodiments of this invention avoid these problems by combining layered compression with on-demand loading, achieving efficient inference in a cloud-edge collaborative environment.

[0102] For example, such as Figure 6 As shown, the network architecture of a large language model can include N neural network layers. In some embodiments, the relevance of each token in the context (input token sequence) input to the model relative to the next token may dynamically change in different layers. A token that is determined to be highly relevant in a shallow layer (e.g., the first layer) may be considered to be lowly relevant in subsequent layers. This embodiment of the invention, while proposing on-demand loading, also provides a dynamic update mechanism, that is, during the inference process from the (n+1)th layer to the Nth layer, every x layers, the priority of the group corresponding to each word in the prefix part is re-determined, and multiple groups with the first relevance are selected from M groups to ensure that tokens with key information are not ignored, where x≥2.

[0103] For example, such as Figure 6 As shown, there is an interval of x layers between the p-th layer and the first layer, and an interval of x layers between the q-th layer and the p-th layer. For example, x can be any integer from 5 to 15. 1 < p < q < N. If n=1 in the above n-th layer, then the relevance of each token is redefined in the p-th layer, and the relevance of each token is redefined again in the subsequent q-th layer. Thus, if a token in the prefix part is determined to be a necessary part with high relevance, the terminal / edge computing device will immediately request and load the corresponding group (or block) from the cloud, decompress it, and write it to the local cache. This progressive loading avoids the latency caused by full transmission and reduces the local instantaneous storage requirements.

[0104] Specifically, in some embodiments, based on Figure 6 The neural network architecture of the large language model shown is as follows: Figure 7 As shown, every x layers, the priority of the group corresponding to each word in the prefix part is redefined, and multiple groups with the first relevance are selected from M groups. This can be done using methods such as... Figure 7 The process shown is implemented as follows: Step 701: In the p-th layer of the large language model, perform attention calculations on all the multiple lexical units included in the prefix part to obtain the calculation results corresponding to the p-th layer.

[0105] The calculation results of layer p include the key vector, value vector, and attention score of each word in the prefix part. Layer p is separated from layer n (e.g., n=1) by a gap of x layers.

[0106] Step 702: Based on the calculation results of the p-th layer, select the groups with the first relevance from the M groups. l Groups.

[0107] In some embodiments, l =m or l ≠m; l <M.

[0108] Step 703: Send a second request to the cloud.

[0109] The second request is used to request data from the cloud. l Second cache data corresponding to multiple tokens in each group.

[0110] After receiving the second request, the cloud executes it. l Attention calculations are performed on multiple lexical units within each group, resulting in a second cached data. The second cached data can be a set of... l The cached data consists of K and V vectors corresponding to multiple tokens in each group, compressed and stored according to a preset precision (such as the first precision mentioned above).

[0111] Step 704: In response to receiving the second cached data sent from the cloud, write the second cached data into the local cache.

[0112] Step 705: Input the second cached data into the p+1 layer to perform attention calculation in the p+1 layer.

[0113] Optionally, this embodiment of the invention also proposes an auxiliary caching mechanism. When certain tokens in the context are temporarily ignored in early inference steps, their corresponding intermediate representations of K and V vectors are stored in a lightweight backup cache. Once subsequent inference steps require this information again, it can be directly retrieved from the auxiliary cache without recalculation. This mechanism ensures that each context fragment is calculated at most once throughout the entire inference process, improving overall efficiency. Specifically, in subsequent inference processes, such as... Figure 8 As shown, the following process can also be executed: Step 801: Remove the values ​​from the calculation results of the p-th layer. lAt least one other group besides the first group is determined to have a second relevance.

[0114] Step 802: Write the key vector and value vector corresponding to at least one group with the second relevance into the standby cache.

[0115] In other words, the K and V cache data of tokens that are judged to be unimportant or of low relevance in this layer are written to the backup cache. When the cache data of these tokens is needed in the inference of subsequent layers in the same inference step, it can be read from the backup cache. Alternatively, if these tokens are needed in subsequent inference steps, they can be read from the backup cache.

[0116] For example, in the q-th layer of the large language model, attention calculation is performed on all the multiple lexical units included in the prefix part, resulting in the calculation result of the q-th layer; q > p+1; the calculation result of the q-th layer includes the key vector, value vector, and attention score of each of the multiple lexical units. Based on the calculation result of the q-th layer, o groups with the first relevance are selected from M groups, where o ≠ l ; o < M; Next, determine the groups that belong to o but do not belong to M. l At least one word from each group; read the key vector and value vector of at least one word from the backup cache, and then obtain the key vector and value vector of at least one word corresponding to the (q+1)th layer based on the key vector and value vector of at least one word. Specifically, obtaining the key vector and value vector of at least one word corresponding to the (q+1)th layer based on the key vector and value vector of at least one word can be achieved by reading the K and V vectors of at least one word at the p-th layer from the backup cache, and performing attention calculations on multiple intermediate layers from the p-th layer to the q-th layer based on the key vector and value vector of at least one word, to obtain the K and V vectors of at least one word missing during the attention calculation at the (q+1)th layer.

[0117] It should be noted that the embodiments of the present invention also propose a hierarchical compression strategy. Hierarchical compression is a strategy of progressively decreasing the number of highly relevant groups selected from M groups: more context is retained in the front layer to ensure semantic integrity, and redundancy is gradually reduced in the back layer to reduce the amount of computation.

[0118] In some embodiments, the number of groups with the highest or highest relevance is gradually reduced according to the order of layers in the large language model. The first relevance is the part with the highest or highest relevance, and the number can be controlled by setting a preset threshold or a preset percentage for the groups that determine the first relevance, specifically by increasing the preset threshold for attention scores or decreasing the preset percentage.

[0119] For example, refer to Figure 6The example neural network architecture shown controls the preset percentage as follows: In the first layer, the preset percentage is set to 80%, meaning that the groups with the highest attention scores (ranked in the top 80% of all M groups) are selected as having the highest relevance. Then, in the p-th layer, the preset percentage is reduced, for example, to 60%, and the groups with the highest attention scores (ranked in the top 60% of all M groups) are selected. l Each group is considered to have the highest relevance. l <m. Next, at level q, the preset percentage is reduced to 40%, and o groups with the highest statistical values ​​(ranked in the top 40%) are selected from the M groups as the groups with the highest relevance. o < l .

[0120] The preset threshold for attention scores is controlled as follows: In the first layer, the preset threshold is set to 0.2, meaning that m groups with attention scores greater than or equal to 0.2 are selected from all M groups and identified as having the first relevance. Then, in the p-th layer, the preset threshold is increased, for example, to 0.3, and groups with attention scores greater than 0.3 are selected from the M groups. l Each group is considered to have the highest relevance. l <m. Next, at the q-th layer, the preset threshold is increased to 0.5, and o groups with attention scores exceeding 0.5 are selected from the M groups as the groups with the first relevance. o < l .

[0121] Optionally, this embodiment of the invention also proposes a dynamic cache scheduling mechanism. Each time the relevance of each token is re-determined, the key vectors and value vectors corresponding to all the obtained multiple tokens can be written into the local cache. The terminal device or edge computing device will continue to perform relevance prediction in combination with the context, and replace and release the cache corresponding to the tokens that are stored in the local cache but are determined to have low relevance (second relevance, also known as low priority) at the current time, so as to adapt to the limited video memory conditions of the edge computing device or terminal device.

[0122] For example, in the p-th layer of the large language model, attention calculation is performed on all the multiple words included in the prefix part. After obtaining the calculation result corresponding to the p-th layer, the key vector and value vector corresponding to all the multiple words are written to the local cache. After resolving the relevance at level q, based on the o groups with the first relevance, at least one other group among the M groups (excluding the o groups) is determined to have the second relevance (low priority), and the key vector and value vector corresponding to at least one group with the second relevance are released from the local cache.

[0123] The entire inference process of a large language model can include a pre-filling stage (preprocessing stage) and an inference stage (also known as a decoding stage or autoregressive generation stage). In this embodiment of the invention, the parallel computation process of all tokens in the input sequence of the pre-filling stage is handled by the cloud, while the inference stage is executed on the terminal device or edge computing device.

[0124] Based on the above exemplary description, it can be seen that the embodiments of the present invention propose a cache management and inference method based on hierarchical compression and on-demand loading. Specifically, a cloud preprocessing module can be deployed in the cloud, while an edge loading and updating module, an auxiliary cache module, and a dynamic scheduling module can be deployed in the terminal device or edge computing device, forming an efficient cloud-edge collaborative inference closed loop.

[0125] Specifically, during the cloud preprocessing stage, all tokens in the prefix portion of the long context are pre-computed in parallel and transformed into a multi-layered (multiple neural network layers) cache representation (cache data). To reduce size, the cloud can group and compress the cache data: the caches of different network layers are divided into several groups (blocks) according to their relevance. Important parts with high relevance retain higher precision or use a lower compression ratio, while less important parts with low relevance are subjected to stronger compression, lower precision, or even delayed loading. Delayed loading can mean waiting for the network to be idle before sending the data to the terminal or edge computing device. This on-demand loading method maintains the integrity of key information while effectively reducing the amount of data transmitted.

[0126] During the edge inference phase, the terminal or edge computing device does not load all cache at once, but instead adopts an on-demand loading strategy. At the start of inference, only the most relevant portion of the cache is loaded to quickly generate the first token, significantly shortening the Time-To-Through (TTFT). In subsequent generation processes, if certain contexts are deemed necessary (e.g., having primary relevance), the system immediately requests and loads the corresponding group (the compressed group is called a compressed block) from the cloud, decompresses it, and writes it to the local cache. This progressive loading avoids the latency caused by full data transfer while reducing local instantaneous storage requirements.

[0127] To avoid redundant computation, this embodiment of the invention also introduces an auxiliary caching mechanism. When certain contexts are temporarily ignored in early steps, their intermediate representations are stored in a lightweight backup cache. Once subsequent generation steps require this information again, it can be directly retrieved from the auxiliary cache without recomputation. This mechanism ensures that each context fragment is computed at most once throughout the entire inference process, improving overall efficiency.

[0128] Furthermore, in terms of cache updates and scheduling, this embodiment of the invention adopts a layer-by-layer decreasing strategy: retaining more context in the earlier layers to ensure semantic integrity, and gradually reducing redundancy in the later layers to reduce computational load. At the same time, the system combines context importance (relevance) prediction to replace and release low-priority caches to adapt to the limited memory conditions of edge devices.

[0129] Thus, the above-mentioned solution proposed in the embodiments of the present invention forms a dynamic collaboration between the cloud and the edge (terminal / edge computing device): the cloud is responsible for computing and hierarchical compression, while the edge is responsible for lazy loading and cache scheduling, and the two achieve seamless connection through an efficient communication protocol.

[0130] The following describes the dynamic cache management device based on on-demand loading provided in the embodiments of the present invention. The dynamic cache management device based on on-demand loading described below can be referred to in correspondence with the dynamic cache management method based on on-demand loading described above.

[0131] like Figure 9 As shown in the embodiment of the present invention, a dynamic cache management device based on on-demand loading is proposed. This device can be applied in the cloud and includes: Module 901 is used to obtain the prefix part of the prompt word.

[0132] The calculation module 902 is used to perform attention calculation on multiple lexical units included in the prefix part, and obtain the key vector, value vector and attention score corresponding to each lexical unit.

[0133] Grouping module 903 is used to divide the multiple lexical units included in the prefix into M groups.

[0134] The relevance recognition module 904 is used to select m groups with a first relevance from M groups based on the attention score; m < M; the first relevance represents the relevance between the word in the group and the next word to be generated.

[0135] The sending module 905 is used to respond to a first request sent by a terminal or edge computing device, and send first cached data corresponding to m groups to the terminal or edge computing device to trigger the terminal or edge computing device to execute subsequent reasoning processes based on the first cached data and output the first word in response to the user's question; wherein, the first cached data is a portion of all cached data corresponding to the M groups.

[0136] like Figure 10 As shown in the embodiment of the present invention, a dynamic cache management device based on on-demand loading is proposed. This device can be applied to terminal devices or edge computing devices, where a large language model is deployed, and the large language model includes N neural network layers. The device includes: The prompt word module 1001 is used to generate prompt words in response to receiving a user question; the prompt words include a prefix.

[0137] Request module 1002 is used to send a first request to the cloud; the first request is used to request the first cached data corresponding to the nth layer in the large language model from the cloud, where 1≤n≤4<N. The cloud is used to perform attention calculations on multiple lexical units included in the prefix part, obtaining the key vector, value vector, and attention score corresponding to each lexical unit.

[0138] The receiving module 1003 is used to receive the first cached data corresponding to m groups sent by the cloud in response to the first request. Here, the m groups are groups with a first relevance selected by the cloud from M groups based on attention scores; the first relevance represents the relevance between the word element in the group and the next word element to be generated; the M groups are obtained by dividing the multiple words included in the prefix part; m < M; the first cached data is a portion of the total cached data corresponding to the M groups. The reasoning module 1004 is used to perform subsequent reasoning processes based on the first cached data and output the first word in response to the user's question.

[0139] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11 As shown, the electronic device may include a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. In some embodiments, the processor 1110 may include the processor 10 as described in any of the above embodiments. For example, the processor 1110 may be the processor 1110 as described in any of the above embodiments, or the processor 1110 may be a processor array composed of the processors 1110 of any of the above embodiments.

[0140] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the on-demand dynamic cache management method provided by the above methods, the method comprising: Obtain the prefix portion of the prompt word; perform attention calculation on the multiple lexical units included in the prefix portion to obtain the key vector, value vector, and attention score corresponding to each lexical unit; divide the multiple lexical units included in the prefix portion into M groups; select m groups with a first relevance from the M groups according to the attention score; m < M; the first relevance represents the relevance between the lexical unit in the group and the next lexical unit to be generated; in response to receiving a first request sent by the terminal or edge computing device, send the first cached data corresponding to the m groups to the terminal or edge computing device to trigger the terminal or edge computing device to execute the subsequent inference process based on the first cached data and output the first lexical unit responding to the user's question; wherein, the first cached data is a portion of all cached data corresponding to the M groups.

[0142] Or include: In response to receiving a user question, a prompt word is generated; the prompt word includes a prefix; a first request is sent to the cloud; the first request is used to request the first cached data corresponding to the nth layer in the large language model from the cloud; 1≤n≤4<N; wherein, the cloud is used to perform attention calculation on multiple lexical units included in the prefix, to obtain the key vector, value vector and attention score corresponding to each lexical unit; the system receives the first cached data corresponding to m groups sent by the cloud in response to the first request; wherein, the m groups are groups with a first relevance selected by the cloud from M groups based on the attention score; the first relevance is used to represent the relevance between the lexical unit in the group and the next lexical unit to be generated; M groups are obtained by dividing the multiple lexical units included in the prefix; m<M; the first cached data is a portion of the total cached data corresponding to the M groups; based on the first cached data, the subsequent inference process is executed, and the first lexical unit responding to the user question is output.

[0143] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the on-demand dynamic cache management method provided by the methods described above, the method comprising: Obtain the prefix portion of the prompt word; perform attention calculation on the multiple lexical units included in the prefix portion to obtain the key vector, value vector, and attention score corresponding to each lexical unit; divide the multiple lexical units included in the prefix portion into M groups; select m groups with a first relevance from the M groups according to the attention score; m < M; the first relevance represents the relevance between the lexical unit in the group and the next lexical unit to be generated; in response to receiving a first request sent by the terminal or edge computing device, send the first cached data corresponding to the m groups to the terminal or edge computing device to trigger the terminal or edge computing device to execute the subsequent inference process based on the first cached data and output the first lexical unit responding to the user's question; wherein, the first cached data is a portion of all cached data corresponding to the M groups.

[0144] Or include: In response to receiving a user question, a prompt word is generated; the prompt word includes a prefix; a first request is sent to the cloud; the first request is used to request the first cached data corresponding to the nth layer in the large language model from the cloud; 1≤n≤4<N; wherein, the cloud is used to perform attention calculation on multiple lexical units included in the prefix, to obtain the key vector, value vector and attention score corresponding to each lexical unit; the system receives the first cached data corresponding to m groups sent by the cloud in response to the first request; wherein, the m groups are groups with a first relevance selected by the cloud from M groups based on the attention score; the first relevance is used to represent the relevance between the lexical unit in the group and the next lexical unit to be generated; M groups are obtained by dividing the multiple lexical units included in the prefix; m<M; the first cached data is a portion of the total cached data corresponding to the M groups; based on the first cached data, the subsequent inference process is executed, and the first lexical unit responding to the user question is output.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic cache management method for on-demand loading, characterized in that, The method is applied to a cloud server; the method includes: Extract the prefix from the prompt words; Attention is calculated on the multiple lexical units included in the prefix portion to obtain the key vector, value vector and attention score corresponding to each of the multiple lexical units; The prefix portion includes multiple lexical units that are divided into M groups; Based on the attention scores, select m groups with a first relevance from the M groups; m < M; the first relevance represents the relevance between the word in the group and the next word to be generated; In response to receiving a first request from a terminal or edge computing device, the system sends the first cached data corresponding to the m groups to the terminal or edge computing device to trigger the terminal or edge computing device to perform subsequent reasoning based on the first cached data and output the first word in response to the user's question; wherein, the first cached data is a portion of all cached data corresponding to the M groups.

2. The method according to claim 1, characterized in that, Based on the attention scores, select m groups with a first relevance from the M groups, including: Based on the attention score corresponding to at least one word in each of the M groups, the statistical value of the attention score for each group is obtained. Based on the statistical values, select m groups from the M groups whose statistical values ​​are greater than or equal to a preset threshold as the m groups with the first relevance. or, Based on the statistical values, the M groups are sorted, and the m groups with the highest statistical values ​​(ranked by a predetermined percentage) are selected as the m groups with the first relevance.

3. The method according to claim 1 or 2, characterized in that, After selecting m groups with a first relevance from the M groups, the method further includes: At least one group other than the m groups in the M groups is identified as having a second relevance. The key vectors and value vectors corresponding to the words in the m groups with the first relevance are stored according to the first precision and / or compressed according to the first compression ratio to obtain the first cache data; The key vectors and value vectors corresponding to the tokens in at least one group with a second relevance are stored according to a second precision and / or compressed according to a second compression ratio; Wherein, the first precision is higher than the second precision, and the second compression ratio is greater than the first compression ratio.

4. A dynamic cache management method for on-demand loading, characterized in that, The method is applied to a terminal or edge computing device; the terminal or edge computing device deploys a large language model, the large language model comprising N neural network layers; the method includes: In response to receiving a user question, a prompt word is generated; the prompt word includes a prefix. Send a first request to the cloud; the first request is used to request the cloud to obtain the first cached data corresponding to the nth layer in the large language model; 1≤n≤4<N; wherein, the cloud is used to perform attention calculation on the multiple lexical units included in the prefix part to obtain the key vector, value vector and attention score corresponding to each of the multiple lexical units; The system receives first cached data corresponding to m groups issued by the cloud in response to the first request; wherein, the m groups are groups with a first relevance selected by the cloud from M groups based on the attention score; the first relevance is used to represent the relevance between the word in the group and the next word to be generated; the M groups are obtained by dividing the multiple words included in the prefix part; m < M; the first cached data is a portion of the total cached data corresponding to the M groups; Based on the first cached data, the subsequent reasoning process is executed, and the first term in response to the user's question is output.

5. The method according to claim 4, characterized in that, Perform subsequent reasoning, including: During the reasoning process from the (n+1)th to the Nth layer, every x layers, the priority of the group corresponding to each word in the prefix part is re-determined, and multiple groups with the first relevance are selected from the M groups; x≥2.

6. The method according to claim 5, characterized in that, Every x layers, the priority of the group corresponding to each word in the prefix part is re-determined, and multiple groups with the first relevance are selected from the M groups, including: In the p-th layer of the large language model, attention calculation is performed on all the multiple lexical units included in the prefix part to obtain the calculation result corresponding to the p-th layer; the calculation result of the p-th layer includes the key vector, value vector and attention score of each of the multiple lexical units; the p-th layer and the n-th layer are separated by x layers; Based on the calculation results of the p-th layer, select the groups with the first relevance from the M groups. l Groups l =m or l ≠m; l <M; After selecting multiple groups with a first relevance from the M groups, the method further includes: Send a second request to the cloud; the second request is used to request the cloud to obtain the l Second cache data corresponding to multiple tokens in each group; In response to receiving the second cached data from the cloud, the second cached data is written to the local cache; The second cached data is input into the p+1th layer to perform attention calculations in the p+1th layer.

7. The method according to claim 6, characterized in that, The subsequent reasoning process also includes: The calculation results of the p-th layer are divided by the above. l At least one other group besides the first group is determined to have a second relevance; Write the key vector and value vector corresponding to at least one group with the second relevance to the standby cache.

8. The method according to claim 5, characterized in that, Every x layers, the priority of the group corresponding to each word in the prefix part is re-determined, and multiple groups with the first relevance are selected from the M groups, including: The number of groups with the first degree of relevance is gradually reduced according to the arrangement order of the different layers in the large language model.

9. The method according to claim 6, characterized in that, After performing attention calculations on all multiple lexical units included in the prefix portion at the p-th layer of the large language model, and obtaining the calculation result corresponding to the p-th layer, the method further includes: Write the key vectors and value vectors corresponding to all the multiple lexical units into the local cache; Based on the calculation results of the q-th layer, after selecting o groups with the first relevance from the M groups, the method further includes: At least one group other than the o groups in the M groups is identified as having a second relevance; Release the key vector and value vector corresponding to at least one group with the second relevance from the local cache.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the on-demand dynamic cache management method as described in any one of claims 1 to 3 or 4-9.

Images