Data processing method and apparatus, terminal device, and computer-readable storage medium

Abstract

The application discloses a data processing method and device, a terminal equipment and a computer readable storage medium. The method comprises the following steps: obtaining current key-value cache information and current conversation information; determining whether the current key-value cache information meets a predetermined condition; if the current key-value cache information meets the predetermined condition, determining a target compression parameter according to the current conversation information; and compressing the current key-value cache data according to the target compression parameter to obtain processed key-value cache data. The method can correspondingly adopt different compression parameters to implement key-value cache compression under different conversation conditions, so that the compression processing can be adjusted to adapt to the current conversation condition, the flexibility of the key-value cache compression processing can be improved, the key-value cache compression process and the current conversation information have better adaptability, and the rationality and processing efficiency of the key-value cache compression processing are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a data processing method, apparatus, terminal device, and computer-readable storage medium. Background Technology

[0002] With the widespread application of large language models, key-value cache-based inference has become an important technique for improving model inference efficiency. During inference, the key-value cache stores intermediate calculation results corresponding to historical contexts to avoid recalculating historical content in subsequent inference processes, thereby reducing computational overhead and improving response efficiency. Since the size of the key-value cache typically grows continuously with the length of the context, it can consume significant memory resources in long-dialogue scenarios. Therefore, it is usually necessary to compress the key-value cache while still meeting inference requirements to reduce memory consumption.

[0003] In related technologies, key-value caches are usually compressed using preset compression parameters. This compression method is not very flexible and it is difficult to balance compression effect and processing efficiency, which in turn affects user experience. Summary of the Invention

[0004] This application provides a data processing method, apparatus, terminal device, and computer-readable storage medium, which can implement key-value cache compression with different compression parameters corresponding to different dialogue situations. This allows the compression process to be adjusted to adapt to the current dialogue situation, improving the flexibility of key-value cache compression processing, making the key-value cache compression process more adaptable to the current dialogue information, and improving the rationality and processing efficiency of key-value cache compression processing.

[0005] The technical solution adopted by this invention to solve the problem is as follows: On the one hand, this application provides a data processing method, including: Retrieve the current key-value cache information and the current conversation information; Determine whether the current key-value cache information meets the predetermined conditions; if the current key-value cache information meets the predetermined conditions, determine the target compression parameters based on the current dialogue information; The current key-value cache data is compressed according to the target compression parameters to obtain the processed key-value cache data.

[0006] In some embodiments of this application, the target compression parameters are determined based on the current dialogue information, including: Categorize the current dialogue information to determine the dialogue type; Determine the basic compression parameters based on the current key-value cache information; Determine the target compression parameters based on the dialogue type and the basic compression parameters.

[0007] In some embodiments of this application, the basic compression parameters include at least one of a first parameter and a second parameter, wherein the first parameter characterizes the amount of data to be compressed, the second parameter characterizes the amount of data to be retained, and the target compression parameters include at least one of a third parameter and a fourth parameter. Based on the dialogue type and basic compression parameters, determine the target compression parameters, including: Determine the importance of the current dialogue information based on the dialogue type; If the importance level indicates that the current dialogue is a key dialogue, decrease the first parameter to obtain the third parameter, and / or increase the second parameter to obtain the fourth parameter; If the importance level indicates that the current dialogue is not a critical dialogue, increase the first parameter to obtain the third parameter, and / or decrease the first parameter to obtain the fourth parameter.

[0008] In some implementations of this application, after obtaining the current key-value cache information, the process includes: Retrieve memory cache information; Determine the key-value cache threshold based on the current key-value cache information and memory cache information; If the current key-value cache information is greater than or equal to the key-value cache threshold, the key-value cache information is determined to meet the predetermined conditions.

[0009] In some embodiments of this application, the target compression parameters include a third parameter and a fourth parameter; The current key-value cache data is compressed according to the target compression parameters to obtain the processed key-value cache data, including: Based on the third and fourth parameters, retrieve the key-value cache data to be compressed and the key-value cache data to be retained from the current key-value cache data; Based on the key-value cache data to be compressed, determine the candidate key-value cache data; The candidate key-value cache data and the key-value cache data to be retained are reorganized to obtain the processed key-value cache data.

[0010] In some embodiments of this application, the key-value cache data to be compressed and the key-value cache data to be retained are obtained from the current key-value cache data according to the third and fourth parameters, including: The first location information is determined based on the third parameter and the location index information; The second location information is determined based on the third parameter, the location index information, and the fourth parameter; Select key-value cache data to be compressed from the current key-value cache data based on the first position information, and obtain key-value cache data to be retained from the key-value cache data based on the second position information.

[0011] In some embodiments of this application, candidate key-value cache data is determined based on the key-value cache data to be compressed, including: Retrieve the text data corresponding to the key-value cache data to be compressed; Semantic compression is performed on text data to generate summary data; Based on the summary data and system prompts, candidate key-value cache data is determined.

[0012] Secondly, embodiments of the present invention also provide a data processing apparatus, comprising: The acquisition module is used to acquire the current key-value cache information and the current dialogue information; The judgment module is used to determine whether the key-value cache information meets the predetermined conditions; if the current key-value cache information meets the predetermined conditions, the target compression parameters are determined based on the current dialogue information. The compression module is used to compress the current key-value cache data according to the target compression parameters to obtain the processed key-value cache data.

[0013] Thirdly, this application also provides a terminal device, which includes: One or more processors; Memory; and One or more applications, wherein the applications are stored in memory and configured to be executed by a processor to implement the data processing method of any of the first aspects.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps of the data processing method of any of the first aspects.

[0015] The beneficial effects of this invention are as follows: By acquiring the current key-value cache information and the current dialogue information, and determining that the current key-value cache information meets predetermined conditions, a target compression parameter is determined based on the current dialogue information, and the current key-value cache data is compressed according to the target compression parameter. Since the target compression parameter is determined in conjunction with the current dialogue information, different compression parameters can be used to implement key-value cache compression under different dialogue conditions. This allows the compression process to adapt to the current dialogue situation, improving the flexibility of key-value cache compression processing, enhancing the adaptability between the key-value cache compression process and the current dialogue information, and improving the rationality and efficiency of key-value cache compression processing. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating one embodiment of the data processing method provided in this invention. Figure 3 This is a flowchart illustrating a specific embodiment of the method for compressing KV Cache provided in this invention. Figure 4 This is a flowchart illustrating a specific embodiment of initializing system prompts provided by this invention. Figure 5 This is a flowchart illustrating a specific embodiment of the calculation of compression parameters n and m provided in this invention. Figure 6 This is a flowchart illustrating a specific embodiment of the selective deletion of tokens provided in this invention. Figure 7 This is a flowchart illustrating a specific embodiment of data compression provided in this invention. Figure 8 This is a flowchart illustrating a specific embodiment of cache reorganization provided in this invention. Figure 9 This is a flowchart illustrating a specific embodiment of adjusting compression parameters n and m provided in this invention. Figure 10 This is a schematic block diagram of the data processing device provided in the embodiments of the present invention; Figure 11 This is a schematic diagram of an embodiment of the terminal device provided in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of the stated features.

[0020] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0021] It should be noted that since the method in this application embodiment is executed in a terminal device, the processing objects of each terminal device exist in the form of data or information, such as time, which is essentially time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data that exist so that the terminal device can process them. Specific details will not be elaborated here.

[0022] This application provides a data processing method, apparatus, terminal device, and computer-readable storage medium, which will be described in detail below.

[0023] Please see Figure 1 , Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of this application. The data processing system may include a terminal device 100, which integrates a data processing unit, such as... Figure 1 Terminal devices in the process.

[0024] In this embodiment, the terminal device 100 is mainly used to obtain current key-value cache information and current dialogue information; determine whether the current key-value cache information meets predetermined conditions; if the current key-value cache information meets predetermined conditions, determine the target compression parameter according to the current dialogue information; compress the current key-value cache data according to the target compression parameter to obtain the processed key-value cache data. Different compression parameters can be used to implement key-value cache compression in different dialogue situations, so that the compression process can be adjusted to adapt to the current dialogue situation, which can improve the flexibility of key-value cache compression processing, make the key-value cache compression process better adaptable to the current dialogue information, and improve the rationality and processing efficiency of key-value cache compression processing.

[0025] In this embodiment, the terminal device 100 can be an independent server, a server network, or a server cluster. For example, the terminal device 100 described in this embodiment includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud server composed of multiple servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0026] It is understood that the terminal device 100 used in the embodiments of this application can be a device that includes both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. Such a device may include: cellular or other communication devices having a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display. Specifically, the terminal device 100 may be a desktop terminal or a mobile terminal, and the terminal device 100 may also be one of a mobile phone, tablet computer, laptop computer, etc.

[0027] Those skilled in the art will understand that Figure 1 The application environment shown is merely one application scenario of the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include those that are more specific to this application. Figure 1 The number of more or fewer terminal devices shown, for example Figure 1 Only one terminal device is shown in the diagram. It is understood that the data processing system may also include one or more other services, which are not specified here.

[0028] In addition, such as Figure 1 As shown, the data processing system may also include a memory 200 for storing data, such as key-value cache information, such as current key-value cache information, such as current dialogue information, such as compression parameters, such as target compression parameters, etc.

[0029] It should be noted that, Figure 1The schematic diagram of the data processing system shown is merely an example. The data processing system and scenario described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of data processing systems and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0030] First, this application provides a data processing method. The execution subject of the data processing method is a data processing device, which is applied to a terminal device. The data processing method includes: obtaining current key-value cache information and current dialogue information; determining whether the current key-value cache information meets predetermined conditions; if the current key-value cache information meets predetermined conditions, determining target compression parameters based on the current dialogue information; and compressing the current key-value cache data according to the target compression parameters to obtain processed key-value cache data.

[0031] like Figure 2 The diagram shown is a flowchart of an embodiment of the data processing method in this application. The data processing method may include the following steps S201 to S203, as detailed below: Step S201: Obtain the current key-value cache information and the current dialogue information.

[0032] Key-value cache (KV cache) is a type of intermediate computation result generated and stored by a large language model during inference. It records the key and value vectors corresponding to historical contexts, allowing for direct reuse of historical computation results in subsequent inference processes. This avoids recalculating historical contexts and improves the model's inference efficiency. Key-value caches are typically generated and stored progressively based on previous interactions between the large language model and the user. As the user continues to interact with the model in multiple rounds, the information in the key-value cache usually increases continuously.

[0033] In this step, the current key-value cache information can be related to the current key-value cache state, such as the number of tokens corresponding to the current key-value cache, the storage space occupied, the cache length, and the cache size. The large language model can be of various types, such as large language models deployed in the cloud, edge devices, or terminal devices, especially in-vehicle large language models applied in vehicles. Since in-vehicle platforms are typically limited by the computing power and storage resources of in-vehicle chips, the key-value cache grows continuously as users interact multiple times, easily consuming a significant amount of storage resources. Therefore, it is even more necessary to improve resource utilization efficiency through reasonable key-value cache management.

[0034] Simultaneously, this step also acquires current dialogue information. This current dialogue information is not the dialogue text itself, but rather information that characterizes the current dialogue content, such as the content type, semantic features, topic features, or other information reflecting the current dialogue situation. The purpose of acquiring this information is to provide a basis for determining the compression strategy for the key-value cache, enabling subsequent processing to be adjusted according to the current interaction, rather than relying solely on the key-value cache itself. By simultaneously acquiring both the current key-value cache information and the current dialogue information, fundamental data support is provided for determining whether compression is necessary and for defining compression parameters, making subsequent compression processing more aligned with the current model's operating state.

[0035] Step S202: Determine whether the current key-value cache information meets the predetermined conditions; if the current key-value cache information meets the predetermined conditions, determine the target compression parameters based on the current dialogue information.

[0036] In this step, the predetermined conditions are pre-set conditions used to trigger key-value cache compression processing. Examples include the number of tokens corresponding to the current key-value cache reaching a preset threshold, the cache length reaching a preset range, and the cache occupying storage resources reaching a preset proportion. First, the key-value cache information is used to determine whether it meets the predetermined conditions. If the determination result indicates that the current key-value cache has not yet met the predetermined conditions, it means that the current key-value cache can still meet the model inference requirements, and compression processing is not required. However, if the determination result indicates that the current key-value cache meets the predetermined conditions, it means that the current key-value cache has reached the state where compression processing is needed, and then the compression parameter determination process begins.

[0037] Different types of dialogues have varying degrees of impact on subsequent reasoning. Therefore, using fixed compression parameters may not yield good results. In contrast, this step flexibly determines the target compression parameters based on the current dialogue information. The target compression parameters can represent the compression degree, compression ratio, compression range, or other parameters related to compression processing. For example, when the current dialogue information indicates that the conversation is general casual conversation, a relatively large compression degree can be determined to free up more cache resources; when the current dialogue information indicates that the conversation involves navigation, vehicle control, or other relatively important interactions, a relatively small compression degree can be determined to retain more contextual information related to the current interaction. By determining the target compression parameters based on the current dialogue information, the compression parameters can be adjusted according to the current interaction, providing a basis for subsequent key-value cache compression and thus improving the flexibility of the compression process.

[0038] Step S203: Compress the current key-value cache data according to the target compression parameters to obtain the processed key-value cache data.

[0039] After determining the target compression parameters, this step compresses the current key-value cache data according to these parameters to obtain processed key-value cache data. The current key-value cache data refers to the key-value cache data currently stored in the large language model. Since this application uses target compression parameters determined based on the current dialogue information, this step can perform data compression according to the determined target compression parameters, rather than always using a uniform compression method for all key-value cache data. After compression, processed key-value cache data with lower resource consumption is obtained, reducing key-value cache resource usage while still meeting the needs of key-value cache usage during subsequent large language model inference.

[0040] Through the above steps, this application realizes the process of dynamically determining compression parameters and completing key-value cache compression based on the current running status. This is beneficial to improving the flexibility of key-value cache compression, improving the efficiency of storage resource utilization, and providing support for large language models to continuously conduct multi-turn dialogues. It is especially suitable for application scenarios with relatively limited computing and storage resources, such as in-vehicle large language models.

[0041] In one specific implementation, after obtaining the current key-value cache information, the process includes: obtaining memory cache information; determining the key-value cache threshold based on the current key-value cache information and the memory cache information; and determining that the key-value cache information meets predetermined conditions if the current key-value cache information is greater than or equal to the key-value cache threshold.

[0042] This implementation provides specific criteria for determining predetermined conditions, and determines whether the current key-value cache has reached a state where compression is required.

[0043] Specifically, the first step is to obtain memory cache information. This information represents the current state of storage resources used to store key-value cache data, such as the available memory capacity, current available memory size, reserved cache space, or other information reflecting cache storage resource status. Especially for large in-vehicle language models typically deployed on automotive computing platforms, the memory resources of the in-vehicle hardware limit the speed at which the key-value cache grows, making it easier to approach the device's maximum available cache capacity. Therefore, the timing of subsequent compression processing can be determined based on the current memory situation.

[0044] Secondly, the key-value cache threshold is determined by combining the current key-value cache information and the memory cache information. The key-value cache threshold can be understood as the target size that the current device can allow the key-value cache to grow to. When the current key-value cache exceeds this size, it is considered that the key-value cache has reached a state that requires compression.

[0045] For example, in one embodiment, the number of tokens corresponding to the current key-value cache, current_tokens, can be monitored in real time, and the key-value cache threshold, T_threshold, can be dynamically calculated based on the total memory capacity, total_memory, and the average storage space occupied by the key-value cache corresponding to a single token, avg_kv_cache_size_per_token. The calculation formula is as follows: T_threshold=(total_memory×0.7) / (avg_kv_cache_size_per_token).

[0046] After calculating the key-value cache threshold T_threshold, the current_tokens are compared with T_threshold. When the current_tokens is greater than or equal to T_threshold, it can be determined that the current key-value cache information meets the predetermined conditions, and the subsequent key-value cache compression process can be started.

[0047] By employing the above methods, key-value cache compression operations can be initiated promptly when the current key-value cache is close to the device's capacity, while the current key-value cache continues to participate in model inference when cache resources are still relatively abundant. This makes the timing of key-value cache compression more reasonable, improves memory resource utilization efficiency, and helps ensure the continuity and stability of the large language model inference process.

[0048] In one specific implementation, the target compression parameters are determined based on the current dialogue information, including: classifying the current dialogue information to determine the dialogue type; determining the basic compression parameters based on the current key-value cache information; and determining the target compression parameters based on the dialogue type and the basic compression parameters.

[0049] In this embodiment, the basic compression parameters are first determined, and then the target compression parameters are determined by combining the dialogue type and the basic compression parameters, so that the target compression parameters can reflect the current key-value cache state and adapt to the current dialogue situation.

[0050] Specifically, the current dialogue information can first be categorized to determine its corresponding dialogue type. The dialogue type characterizes the content category of the current dialogue, and can include, for example, instruction-based dialogues, casual conversations, security-related dialogues, or other types of dialogues. The categorization object here is not the dialogue text itself, but rather a type of information that characterizes the content of the current dialogue. By analyzing the current dialogue information, the category to which the current dialogue belongs can be obtained, providing a basis for subsequently determining compression parameters.

[0051] Subsequently, basic compression parameters are determined based on the current key-value cache information. These basic compression parameters can be understood as a set of compression parameters initially determined based on the current key-value cache state before adjustments are made in conjunction with the current dialogue content. They reflect the compression requirements of the current key-value cache under normal circumstances. For example, the basic compression parameters may include a first parameter n and a second parameter m, where the first parameter represents the amount of data to be compressed, and the second parameter represents the amount of data to be retained. In one embodiment, the basic compression parameters can be calculated based on the number of tokens corresponding to the current key-value cache, current_tokens, using the following formula: n = min(300, current_tokens × 0.25); m = min(100, current_tokens × 0.1); where the function min(a, b) represents taking the smaller value between parameters a and b.

[0052] The above calculation method can dynamically obtain the basic compression parameters as the current key-value cache size changes, so that the basic compression parameters can reflect the basic compression requirements corresponding to the current key-value cache.

[0053] After obtaining the basic compression parameters, these parameters are adjusted accordingly based on the previously determined dialogue type to obtain the target compression parameters. Since the basic compression parameters already reflect the current key-value cache size, and the dialogue type further reflects the category of the current interaction content, adjusting the basic compression parameters in conjunction with the dialogue type allows the final target compression parameters to simultaneously consider both the current key-value cache state and the current interaction situation, providing a more reasonable parameter basis for subsequent key-value cache compression processing.

[0054] In this way, the target compression parameters are no longer determined solely by the key-value cache size, but are further adjusted based on the current dialogue situation on the basis of the basic compression parameters, which helps to improve the adaptability and flexibility of key-value cache compression processing.

[0055] In one specific implementation, the basic compression parameters include at least one of a first parameter and a second parameter. The first parameter is used to characterize the amount of data to be compressed, and the second parameter is used to characterize the amount of data to be retained. The target compression parameters include at least one of a third parameter and a fourth parameter. The target compression parameters are determined based on the dialogue type and the basic compression parameters, including: determining the importance of the current dialogue information based on the dialogue type; if the importance indicates that the current dialogue is a critical dialogue, decreasing the first parameter to obtain the third parameter, and / or increasing the second parameter to obtain the fourth parameter; if the importance indicates that the current dialogue is a non-critical dialogue, increasing the first parameter to obtain the third parameter, and / or decreasing the first parameter to obtain the fourth parameter.

[0056] In this embodiment, the importance of the current dialogue information is first determined based on the dialogue type. Then, the basic compression parameters are adjusted according to the importance to obtain the final target compression parameters used for key-value caching compression processing. The importance level characterizes the significance of the current dialogue for subsequent model inference and can be further divided into key dialogues and non-key dialogues. For example, after dialogue type analysis, safety-related dialogues, vehicle control commands, or navigation commands can be identified as key dialogues, while ordinary chatter can be identified as non-key dialogues. Of course, the above classification method is only an example; those skilled in the art can use other methods to determine the importance of the current dialogue according to actual application needs.

[0057] When the current dialogue is determined to be critical, to retain more information relevant to the current interaction during subsequent reasoning, the amount of data to be compressed can be appropriately reduced (i.e., the first parameter can be decreased), while the amount of data to be retained can be appropriately increased (i.e., the second parameter can be increased). The adjusted first and second parameters then serve as the third and fourth parameters, respectively, to form the final target compression parameters. Conversely, when the current dialogue is determined to be non-critical, the amount of data to be compressed can be appropriately increased, while the amount of data to be retained can be appropriately decreased, resulting in new third and fourth parameters, forming the corresponding target compression parameters.

[0058] By adopting the above approach, we can further consider the importance of the current dialogue on the basis of basic compression requirements, so that the subsequent key-value cache compression processing can adopt a more reasonable compression strategy. This is conducive to improving the flexibility and rationality of key-value cache compression processing, while taking into account the efficiency of memory resource utilization and the need for context information in subsequent model inference.

[0059] In one specific implementation, the target compression parameters include a third parameter and a fourth parameter. Compressing the current key-value cache data according to the target compression parameters to obtain processed key-value cache data includes: obtaining key-value cache data to be compressed and key-value cache data to be retained from the current key-value cache data according to the third and fourth parameters; determining candidate key-value cache data based on the key-value cache data to be compressed; and recombining the candidate key-value cache data and the key-value cache data to be retained to obtain the processed key-value cache data.

[0060] In this embodiment, different data regions are divided from the current key-value cache data according to the target compression parameters, and each region is compressed and retained separately, ultimately completing the reconstruction of the key-value cache data. The third parameter represents the amount of data to be compressed, and the fourth parameter represents the amount of data to be retained.

[0061] First, based on the third and fourth parameters, the key-value cache data to be compressed and the key-value cache data to be retained can be determined from the current key-value cache data. The key-value cache data to be compressed refers to the key-value cache data that needs to be compressed in this instance, while the key-value cache data to be retained refers to the data that does not need to be compressed and will be directly retained in the reconstructed key-value cache.

[0062] Subsequently, candidate key-value cache data is determined based on the key-value cache data to be compressed. Candidate key-value cache data refers to the new key-value cache data obtained after compressing the key-value cache data to be compressed. This new data can replace the original key-value cache data, thereby reducing the storage resources occupied by the key-value cache. It should be noted that this implementation does not limit the specific compression method used to generate the candidate key-value cache data; any data processing method capable of compressing the key-value cache can be used to obtain the corresponding candidate key-value cache data.

[0063] After obtaining the candidate key-value cache data, the candidate key-value cache data is recombined with the key-value cache data to be retained to form a new key-value cache structure, which is then used as the processed key-value cache data in subsequent model inference. The underlying logic of the above process is that not all key-value cache data needs to be regenerated, but only the data that needs to be compressed is processed, while the data that does not need to be compressed is retained, thereby reducing unnecessary data processing.

[0064] By adopting the above method, while reducing the resource consumption of key-value cache, we can retain some key-value cache data that is still needed for subsequent inference, improve the efficiency of key-value cache compression processing, and ensure that the reconstructed key-value cache can continue to support the subsequent large language model inference process.

[0065] In one specific implementation, obtaining the key-value cache data to be compressed and the key-value cache data to be retained from the current key-value cache data according to the third parameter and the fourth parameter includes: determining the first position information according to the third parameter and the position index information; determining the second position information according to the third parameter, the position index information and the fourth parameter; selecting the key-value cache data to be compressed from the current key-value cache data according to the first position information; and obtaining the key-value cache data to be retained from the key-value cache data according to the second position information.

[0066] This implementation introduces location index information, enabling the regions to be compressed and retained to be determined through location positioning, thereby improving the accuracy of the key-value cache reorganization process. The location index information is used to characterize the positional relationships between different parts of the current key-value cache data.

[0067] In one embodiment, the location index information may include the location index corresponding to the end position of the system prompt, i.e., system_end_idx. Since the system prompt typically remains unchanged throughout the dialogue, its corresponding key-value cache usually serves as the basis for all subsequent inference processes. Therefore, system_end_idx can represent the end position of the key-value cache corresponding to the system prompt, which is also the start position of the key-value cache corresponding to the first historical dialogue segment after the system prompt. Subsequent areas to be compressed and areas to be retained can be determined using system_end_idx as the positioning benchmark, thereby avoiding impact on the key-value cache corresponding to the system prompt. For example, in one embodiment, the deletion start position delete_start can be first determined as system_end_idx, and the deletion end position delete_end can be determined according to the third parameter, specifically expressed as: delete_end = system_end_idx + n - 1, where n represents the third parameter, i.e., the amount of data to be compressed this time. Thus, the location range corresponding to the key-value cache data to be compressed can be determined, i.e., [delete_start, delete_end]. Subsequently, the starting position of the retention area, `keep_start`, is further determined based on the third parameter, specifically expressed as: `keep_start = system_end_idx + n`, where `m` represents the fourth parameter, i.e., the amount of data to be retained. Then, the position range corresponding to the key-value cache data to be retained is determined, i.e., `[keep_start, keep_start + m - 1]`. Thus, the key-value cache data to be compressed can be determined based on the first position information, and the key-value cache data to be retained can be determined based on the second position information.

[0068] After determining the two parts of data mentioned above, the original text content of the token corresponding to the area to be compressed can be saved, and the key-value cache data to be compressed can be compressed to generate corresponding new key-value cache data. Then, the newly generated key-value cache data is inserted after `system_end_idx`, and the aforementioned key-value cache data to be retained is rearranged after the newly generated key-value cache data, thus completing the reorganization of the entire key-value cache structure. The rearrangement of the key-value cache data to be retained after the newly generated key-value cache data does not involve recalculating the key-value cache corresponding to the part to be retained. Rather, because the length of the compressed key-value cache is usually smaller than the length of the original key-value cache, it is necessary to adjust the storage location of the key-value cache to be retained in the cache space so that the reconstructed key-value cache re-forms a contiguous storage structure.

[0069] By employing the above methods, the key-value cache corresponding to the system prompt words can always remain stable. At the same time, it can quickly complete the replacement of the region to be compressed and the reorganization of the region to be retained, thereby improving the efficiency of key-value cache compression and cache reconstruction, and providing a stable key-value cache structure for subsequent large language models to continuously perform multi-round inference.

[0070] In one specific implementation, candidate key-value cache data is determined based on the key-value cache data to be compressed, including: obtaining the text data corresponding to the key-value cache data to be compressed; performing semantic compression on the text data to generate summary data; and determining candidate key-value cache data based on the summary data and system prompt word information.

[0071] This implementation first extracts the text data corresponding to the key-value cache data to be compressed, uses the text content as the subsequent compression object, and then regenerates the corresponding key-value cache based on the compressed text, thereby completing the construction of candidate key-value cache data.

[0072] Since the key-value cache is generated during the forward inference process of the large language model on the user's historical interaction content, each part of the key-value cache corresponds to a certain historical text content. Obtaining the text data corresponding to the key-value cache data to be compressed means obtaining the original text content corresponding to the generation of that part of the key-value cache. Subsequently, semantic compression is performed on the text data to generate summary data. Semantic compression aims to maintain the main semantic information of the original text by summarizing, generalizing, or simplifying the text content, so that the generated summary data can express the main content of the original text with a shorter text length, rather than simply truncating or deleting the text.

[0073] In one embodiment, the length of the summary data can be controlled to maintain a certain proportional relationship with the length of the text to be compressed. For example, the number of tokens corresponding to the summary, n_compressed, can be set to satisfy the following relationship: n_compressed = n × 0.4, where n represents the third parameter, i.e., the number of tokens corresponding to the key-value cache data to be compressed. The above ratio is only an example, and those skilled in the art can adopt other ratios or other summary length control methods according to actual application needs. By controlling the summary length, the generated summary can retain the main semantic information while further reducing the storage resources required for subsequent key-value cache generation.

[0074] After generating the summary data, candidate key-value cache data is determined based on the summary data and system prompts. System prompts are pre-set messages that define the model's role, behavioral rules, or response style before the dialogue begins, and typically remain unchanged throughout the multi-turn dialogue. Since the subsequently generated key-value caches need to be consistent with the current model context, the summary data can be appended to the system prompts and used as new input to the large language model for forward inference. Forward inference refers to the process by which the large language model performs feature calculations on the input text layer by layer according to the computational flow of each layer of the neural network, ultimately generating the corresponding key-value caches. Because the input content has been replaced by summary data instead of the original text, re-performing forward inference generates a new key-value cache corresponding to the summary data, which is then used as candidate key-value cache data to replace the original key-value cache data to be compressed.

[0075] Since the summary can preserve the main semantic information of the original text, and the candidate key-value cache generated by re-performing forward inference can be consistent with the summary data, a smaller candidate key-value cache can be used to replace the original larger key-value cache to be compressed. This reduces the storage resources occupied by the key-value cache while still providing contextual information corresponding to the compressed text for subsequent large language model inference, thereby improving the rationality of key-value cache compression processing and the continuity of subsequent inference processes.

[0076] As a specific example, with Figure 3 Based on the process shown, combined with Figures 4 to 9 This paper describes the KV Cache compression method proposed in this application.

[0077] In this embodiment, the in-vehicle large language model is used as the application object. After the vehicle starts, the KV Cache initialization process is executed first, such as... Figure 4 As shown, the system first loads predefined system prompts and inputs them into the large language model for forward inference, generating corresponding key and value vectors to form a KV Cache for each system prompt. Subsequently, the generated KV Cache is stored at the beginning of the cache area, and a memory protection flag is set to ensure that the KV Cache corresponding to the system prompt is a non-deletable region participating in subsequent inference. Simultaneously, the end index of the KV Cache corresponding to the system prompt, `system_end_idx`, is recorded as the position reference during subsequent KV Cache compression and reorganization. The initial value of `current_tokens` is set according to the number of tokens corresponding to the system prompt, completing the KV Cache initialization.

[0078] After initialization is complete, as follows Figure 5 As shown, the system continuously monitors the current size of the KV Cache, continuously counts the current number of tokens (current_tokens), and dynamically calculates the KV Cache threshold T_threshold by combining the total memory capacity (total_memory) available for storing the KV Cache and the average KV Cache space occupied per token (avg_kv_cache_size_per_token). Then, current_tokens is compared with T_threshold. When current_tokens is less than T_threshold, it indicates that the current KV Cache still has sufficient storage space, and the large language model continues to perform normal inference. When current_tokens is greater than or equal to T_threshold, the KV Cache compression process is triggered, and the basic compression parameters n (the first parameter mentioned above) and m (the second parameter mentioned above) are calculated based on the current number of tokens. n represents the amount of data to be compressed in this operation, and m represents the amount of data to be retained in this operation. Furthermore, before entering the compression process, it can also combine... Figure 9 The process described analyzes the current dialogue content, identifying whether it's a command-based, casual, or security-related conversation, and determines its importance accordingly. If the dialogue is deemed critical, parameter n is appropriately decreased and parameter m is increased to preserve more contextual information; if it's deemed non-critical, parameter n is increased and parameter m is decreased to improve the compression ratio. Simultaneously, historical compression records are checked to determine if the content to be compressed has already been compressed. If duplicate compression exists, the corresponding content is skipped; otherwise, normal compression is performed. After compression, the parameters used and the compression effect are recorded, and optimization suggestions are generated based on the results for further adjustment of compression parameters in subsequent compression processes.

[0079] Subsequently, as Figure 6As shown, using `system_end_idx` as the positioning reference, the KV Cache regions to be compressed and those to be retained are determined. Specifically, the deletion start position is determined as `delete_start = system_end_idx`, and the deletion end position is determined as `delete_end = system_end_idx + n - 1`, thus deleting the KV Cache corresponding to the interval [delete_start, delete_end]. Then, the retention start position `keep_start = system_end_idx + n` is calculated, and the interval [keep_start, keep_start + m] is retained. 1] Corresponding KV Cache. At the same time, the original text content corresponding to the deleted token is saved for subsequent semantic compression.

[0080] like Figure 7 As shown, a lightweight compression model is called on the saved original text to perform semantic compression, generating a summary text that expresses the original semantics, and controlling the summary length to satisfy: n_compressed = n × 0.4. Then, the summary text is concatenated to the system prompt words and input again into the large language model for forward inference, regenerating a new KV Cache corresponding to the summary. After generation, the compression quality can be verified using semantic similarity evaluation. When the semantic similarity meets the preset requirements, the newly generated compressed KV Cache is accepted; otherwise, the compression parameters are adjusted and the compression process is re-executed.

[0081] Finally, as Figure 8 As shown, the newly generated compressed KV Cache is inserted into the position corresponding to system_end_idx, and the original m Tokens corresponding to the KV Cache are moved to the new compressed KV Cache, so that the reconstructed KV Cache forms a contiguous storage structure again. Then, all relevant pointers and position indices are updated, the attention mask matrix is ​​regenerated, and the current number of tokens is updated: current_tokens = system_tokens + n_compressed + m.

[0082] After cache reorganization, the attention score distribution after reorganization can be further checked to verify the integrity of the KV Cache. If the verification passes, the current round of KV Cache compression is completed, and the KV Cache monitoring phase is restarted. If the integrity verification fails, the process is rolled back to the pre-compression state, and the compression parameters are adjusted before re-executing the compression process. This cycle is repeated to achieve dynamic compression and continuous optimization of the KV Cache, reducing memory consumption while ensuring the continuity and stability of the large language model inference process.

[0083] To better implement the data processing method in the embodiments of this application, based on the data processing method, the embodiments of this application also provide a data processing apparatus, such as... Figure 10 As shown, the data processing device 300 includes: The acquisition module 310 is used to acquire the current key-value cache information and the current dialogue information; The judgment module 320 is used to determine whether the key-value cache information meets the predetermined conditions; if the current key-value cache information meets the predetermined conditions, the target compression parameters are determined based on the current dialogue information. Compression module 330 is used to compress the current key-value cache data according to the target compression parameters to obtain the processed key-value cache data.

[0084] In this embodiment, by acquiring the current key-value cache information and the current dialogue information, and determining that the current key-value cache information meets predetermined conditions, a target compression parameter is determined based on the current dialogue information, and the current key-value cache data is compressed according to the target compression parameter. Since the target compression parameter is determined in conjunction with the current dialogue information, different compression parameters can be used to implement key-value cache compression under different dialogue conditions. This allows the compression process to adapt to the current dialogue situation, improving the flexibility of key-value cache compression processing, enhancing the adaptability between the key-value cache compression process and the current dialogue information, and improving the rationality and efficiency of key-value cache compression processing.

[0085] In some embodiments of this application, the determination module 320 determines the target compression parameters based on the current dialogue information, including: Categorize the current dialogue information to determine the dialogue type; Determine the basic compression parameters based on the current key-value cache information; Determine the target compression parameters based on the dialogue type and the basic compression parameters.

[0086] In some embodiments of this application, the basic compression parameters include at least one of a first parameter and a second parameter. The first parameter characterizes the amount of data to be compressed, and the second parameter characterizes the amount of data to be retained. The target compression parameters include at least one of a third parameter and a fourth parameter. The judgment module 320 determines the target compression parameters based on the dialogue type and basic compression parameters, including: Determine the importance of the current dialogue information based on the dialogue type; If the importance level indicates that the current dialogue is a key dialogue, decrease the first parameter to obtain the third parameter, and / or increase the second parameter to obtain the fourth parameter; If the importance level indicates that the current dialogue is not a critical dialogue, increase the first parameter to obtain the third parameter, and / or decrease the first parameter to obtain the fourth parameter.

[0087] In some embodiments of this application, after obtaining the current key-value cache information, the determination module 320 is further configured to: Retrieve memory cache information; Determine the key-value cache threshold based on the current key-value cache information and memory cache information; If the current key-value cache information is greater than or equal to the key-value cache threshold, the key-value cache information is determined to meet the predetermined conditions.

[0088] In some embodiments of this application, the target compression parameters include a third parameter and a fourth parameter; Compression module 330 compresses the current key-value cache data according to the target compression parameters to obtain processed key-value cache data, including: Based on the third and fourth parameters, retrieve the key-value cache data to be compressed and the key-value cache data to be retained from the current key-value cache data; Based on the key-value cache data to be compressed, determine the candidate key-value cache data; The candidate key-value cache data and the key-value cache data to be retained are reorganized to obtain the processed key-value cache data.

[0089] In some embodiments of this application, the compression module 330 obtains key-value cache data to be compressed and key-value cache data to be retained from the current key-value cache data according to the third parameter and the fourth parameter, including: The first location information is determined based on the third parameter and the location index information; The second location information is determined based on the third parameter, the location index information, and the fourth parameter; Select key-value cache data to be compressed from the current key-value cache data based on the first position information, and obtain key-value cache data to be retained from the key-value cache data based on the second position information.

[0090] In some embodiments of this application, the compression module 330 determines candidate key-value cache data based on the key-value cache data to be compressed, including: Retrieve the text data corresponding to the key-value cache data to be compressed; Semantic compression is performed on text data to generate summary data; Based on the summary data and system prompts, candidate key-value cache data is determined.

[0091] This application also provides a terminal device that integrates any of the data processing devices provided in this application. The terminal device includes: One or more processors; Memory; and One or more applications, wherein the applications are stored in memory and configured to be executed by a processor from the steps of the data processing method in any of the embodiments described above.

[0092] This application also provides a terminal device that integrates any of the data processing devices provided in this application. For example... Figure 11 As shown, it illustrates a structural schematic diagram of the terminal device involved in the embodiments of this application. Specifically: The terminal device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 11 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 401 is the control center of the terminal device. It connects various parts of the terminal device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, it performs various functions and processes data of the terminal device, thereby providing overall monitoring of the terminal device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0093] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0094] The terminal device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0095] The terminal device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0096] Although not shown, the terminal device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the terminal device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows: Retrieve the current key-value cache information and the current conversation information; Determine whether the current key-value cache information meets the predetermined conditions; if the current key-value cache information meets the predetermined conditions, determine the target compression parameters based on the current dialogue information; The current key-value cache data is compressed according to the target compression parameters to obtain the processed key-value cache data.

[0097] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0098] Therefore, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the data processing methods provided in embodiments of this application. For example, the computer program loaded by the processor can execute the following steps: Retrieve the current key-value cache information and the current conversation information; Determine whether the current key-value cache information meets the predetermined conditions; if the current key-value cache information meets the predetermined conditions, determine the target compression parameters based on the current dialogue information; The current key-value cache data is compressed according to the target compression parameters to obtain the processed key-value cache data.

[0099] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0100] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0101] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0102] The data processing method, apparatus, terminal device, and computer-readable storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, include: Retrieve the current key-value cache information and the current conversation information; Determine whether the current key-value cache information meets the predetermined conditions; If the current key-value cache information meets the predetermined conditions, the target compression parameters are determined based on the current dialogue information; The current key-value cache data is compressed according to the target compression parameters to obtain the processed key-value cache data.

2. The data processing method according to claim 1, characterized in that, Determining the target compression parameters based on the current dialogue information includes: The current dialogue information is categorized to determine the dialogue type; Based on the current key-value cache information, determine the basic compression parameters; The target compression parameters are determined based on the dialogue type and the basic compression parameters.

3. The data processing method according to claim 2, characterized in that, The basic compression parameters include at least one of a first parameter and a second parameter, wherein the first parameter characterizes the amount of data to be compressed, and the second parameter characterizes the amount of data to be retained; the target compression parameters include at least one of a third parameter and a fourth parameter. The step of determining the target compression parameters based on the dialogue type and the basic compression parameters includes: Based on the dialogue type, determine the importance of the current dialogue information; If the importance level indicates that the current dialogue is a key dialogue, decrease the first parameter to obtain the third parameter, and / or increase the second parameter to obtain the fourth parameter; If the importance level indicates that the current dialogue is not a critical dialogue, the first parameter is increased to obtain the third parameter, and / or the first parameter is decreased to obtain the fourth parameter.

4. The data processing method according to claim 1, characterized in that, After obtaining the current key-value cache information, the following steps are included: Retrieve memory cache information; Determine the key-value cache threshold based on the current key-value cache information and the memory cache information; If the current key-value cache information is greater than or equal to the key-value cache threshold, it is determined that the key-value cache information meets the predetermined condition.

5. The data processing method according to claim 1, characterized in that, The target compression parameters include a third parameter and a fourth parameter; The step of compressing the current key-value cache data according to the target compression parameters to obtain processed key-value cache data includes: Based on the third parameter and the fourth parameter, obtain the key-value cache data to be compressed and the key-value cache data to be retained from the current key-value cache data; Based on the key-value cache data to be compressed, candidate key-value cache data are determined; The candidate key-value cache data and the key-value cache data to be retained are recombined to obtain the processed key-value cache data.

6. The data processing method according to claim 5, characterized in that, The step of obtaining the key-value cache data to be compressed and the key-value cache data to be retained from the current key-value cache data according to the third parameter and the fourth parameter includes: The first location information is determined based on the third parameter and the location index information; The second location information is determined based on the third parameter, the location index information, and the fourth parameter; The key-value cache data to be compressed is selected from the current key-value cache data according to the first location information, and the key-value cache data to be retained is obtained from the key-value cache data according to the second location information.

7. The data processing method according to claim 5, characterized in that, The step of determining candidate key-value cache data based on the key-value cache data to be compressed includes: Obtain the text data corresponding to the key-value cache data to be compressed; The text data is semantically compressed to generate summary data; Based on the summary data and system prompt information, candidate key-value cache data is determined.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the current key-value cache information and the current dialogue information; The judgment module is used to determine whether the key-value cache information meets predetermined conditions; If the current key-value cache information meets the predetermined conditions, the target compression parameters are determined based on the current dialogue information; The compression module is used to compress the current key-value cache data according to the target compression parameters to obtain the processed key-value cache data.

9. A terminal device, characterized in that, The terminal device includes: one or more processors, a memory, and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the data processing method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps of the data processing method according to any one of claims 1 to 7.

Images