Large language model reasoning optimization method, system and equipment and storage medium

Abstract

The invention discloses a big language model reasoning optimization method, system and device and a storage medium, which are corresponding schemes, in the scheme, a data clustering and quantification mechanism is introduced in a big language model reasoning process, when the size of a KV cache exceeds a preset threshold value, firstly, KV cache data is subjected to clustering division, and when the size of the KV cache exceeds the preset threshold value, the KV cache data is subjected to clustering division; and performing low-precision quantization processing on part of data, and replacing the other part of data with a clustering center, thereby achieving the purpose of reducing the KV cache size. Through the scheme of the invention, the video memory occupation in the reasoning process can be obviously reduced, the reasoning efficiency is improved, and the utilization rate of system resources is optimized.

Description

Technical Field

[0001] The present invention relates to the field of computer GPU (graphics processing unit) video memory optimization technology, and in particular to a large language model reasoning optimization method, system, device and storage medium. Background Art

[0002] Large language model (LLM) reasoning is an important development in the field of natural language processing in recent years. As the model size gradually increases, its computing requirements in the reasoning stage also increase significantly. During the reasoning process, the model generates output through a series of complex matrix operations and nonlinear activation functions, which places high demands on computing resources, especially video memory. As the model reasoning proceeds, the video memory consumption of the process will continue to increase. This is mainly reflected in the need to cache certain intermediate results during the reasoning process, that is, the data called keys and values. These cached data are usually called KV caches.

[0003] Caching historically generated KVs is a key mechanism for preserving previous states during the inference process of large language models, especially when using autoregressive generation. Its function is to retain the previously calculated key and value pairs at each step of inference, thereby avoiding repeated calculations and improving inference efficiency. However, as the sequence length increases, the storage requirements of the KV cache increase linearly. Especially when processing long sequence inputs, the video memory occupied by the KV cache will increase significantly, becoming one of the main bottlenecks of the system. For some large models (such as GPT-3, LlaMa3.1, etc.), the video memory required for the KV cache during inference may even exceed the video memory required for the model's own parameters, thereby limiting the model's ability to perform inference on devices with limited resources. Among them, GPT-3 is an open source generative pre-training model released by OpenAI, and -3 is the version number of the model; LlaMa3.1 is an open source large language model released by Meta, and 3.1 is the version number of the model.

[0004] Therefore, how to effectively reduce the KV cache's occupation of video memory, thereby improving reasoning efficiency and system resource utilization, has become a key technical challenge in large language model reasoning, and is also one of the problems that need to be solved urgently at this stage.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a large language model reasoning optimization method, system, device and storage medium, which can significantly reduce the storage requirements of the KV cache, thereby effectively reducing the memory occupancy, improving reasoning efficiency and system resource utilization.

[0007] The objective of the present invention is achieved through the following technical solutions: A large language model reasoning optimization method, comprising: Determine whether the KV cache of the large language model exceeds a preset threshold; wherein the KV cache represents a key-value cache; If it exceeds the preset threshold, the data in the KV cache is clustered to obtain several cluster categories; the data in the KV cache is rearranged according to the cluster categories, that is, the data belonging to the same cluster category is reorganized so that the data in the same cluster category are arranged in adjacent positions, and for each cluster category, the attention score of its cluster center is calculated respectively, and the cluster categories are arranged in descending order according to the attention score; Quantify the data in a part of cluster categories according to the data category; replace the data in another part of cluster categories with the cluster center of the cluster category to which it belongs; After quantization and cluster center replacement, the data becomes the optimized KV cache data, which is used for subsequent reasoning of the large language model.

[0008] A large language model reasoning optimization system, comprising: A KV cache amount determination module is used to determine whether the KV cache of the large language model exceeds a preset threshold; wherein the KV cache represents a key value cache; The clustering module is used to cluster the data in the KV cache to obtain several cluster categories when the KV cache exceeds a preset threshold; rearrange the data in the KV cache according to the cluster categories, that is, reorganize the data belonging to the same cluster category so that the data in the same cluster category are arranged in adjacent positions, and for each cluster category, calculate the attention score of its cluster center respectively, and arrange the cluster categories in descending order according to the attention score; A quantification module is used to quantify the data in a part of the cluster categories according to the data categories; The cluster center replacement module is used to replace the data in another part of the cluster category with the cluster center of the cluster category to which it belongs; The inference module is used to use the optimized KV cache data for subsequent inference of the large language model; the data after quantization and cluster center replacement processing is the optimized KV cache data.

[0009] A processing device, comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0010] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0011] It can be seen from the technical solution provided by the present invention that a data clustering and quantization mechanism is introduced in the large language model reasoning process. When the size of the KV cache exceeds the preset threshold, the KV cache data is first clustered and divided, and then part of the data is subjected to low-precision quantization processing, and the other part is replaced by the cluster center, so as to achieve the purpose of reducing the KV cache size. Through the solution of the present invention, the video memory occupancy in the reasoning process can be significantly reduced, the reasoning efficiency can be improved, and the utilization rate of system resources can be optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A flowchart of a large language model reasoning optimization method provided by an embodiment of the present invention; Figure 2 A schematic diagram of cluster rearrangement provided by an embodiment of the present invention; Figure 3 A schematic diagram of quantizing a KV cache according to different quantization bit widths provided by an embodiment of the present invention; Figure 4 A schematic diagram of a large language model reasoning optimization system provided by an embodiment of the present invention; Figure 5 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0015] First, the terms that may be used in this article are explained as follows: The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0016] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0017] The following is a detailed description of a large language model inference optimization method, system, device and storage medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, they shall be carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer.

[0018] Embodiment 1 The embodiment of the present invention provides a large language model reasoning optimization method, such as Figure 1 As shown, it mainly includes the following steps: Step S1: Determine whether the KV cache of the large language model exceeds a preset threshold.

[0019] In the embodiment of the present invention, KV cache represents key-value cache, and the relevant principles have been introduced in the aforementioned background technology and will not be repeated here.

[0020] Step S2: If the preset threshold is exceeded, the data in the KV cache is clustered to obtain several cluster categories.

[0021] In the embodiment of the present invention, the threshold value may be set according to the specification of the hardware device, and the present invention does not limit the specific value.

[0022] In an embodiment of the present invention, the data in the KV cache is rearranged according to the clustering category, that is, the data belonging to the same clustering category is reorganized so that the data in the same clustering category is arranged in adjacent positions; and, for each clustering category, the attention score of its cluster center is calculated respectively, and the clustering categories are arranged in descending order according to the attention score; thereafter, step S3 is entered to perform quantization and cluster center replacement processing.

[0023] In an embodiment of the present invention, the attention score of the cluster center can be calculated in the following manner: given a query (Q), for each cluster category, the similarity between the cluster center and the given query is calculated, and then the scale is adjusted (to alleviate the problem of excessive or small gradients and ensure numerical stability), and then the softmax function (normalized exponential function) is applied to convert the scaled similarity into a probability distribution (i.e., the importance weight of the cluster center relative to the given query), and finally the cluster centers are weighted and summed in combination with the probability distribution to obtain the attention score.

[0024] Step S3: quantify the data in a part of the cluster categories according to the data categories; and replace the data in another part of the cluster categories with the cluster centers of the corresponding cluster categories.

[0025] In the embodiment of the present invention, low-precision quantization processing is performed on part of the data, and the other part is replaced by cluster centers, so as to achieve the purpose of reducing the KV cache size.

[0026] Preferably, the cluster categories are sorted in descending order in the solution provided above. In this part, the data from the front part of the cluster categories (that is, the part of the cluster categories with higher attention scores of the cluster centers) are taken out for low-precision quantization processing, and the remaining part is replaced by the cluster centers.

[0027] Step S4: After quantization and cluster center replacement processing, the data is the optimized KV cache data, and the optimized KV cache data is used for subsequent reasoning of the large language model.

[0028] The main principle of the above scheme provided by the embodiment of the present invention is: in the process of large language model inference, when the KV cache exceeds the preset threshold, the KV cache data is clustered by introducing a clustering algorithm to divide the data into several cluster categories; then, low-precision quantization processing is performed on part of the cluster data, and the cluster center point is used to replace the unquantized cluster data, thereby significantly reducing the storage requirements of the KV cache. This method effectively reduces the video memory occupied by the KV cache, and improves the utilization of system resources by reducing memory usage.

[0029] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0030] 1. Initialize the cluster center.

[0031] In the embodiment of the present invention, when the size of the KV cache reaches or exceeds a preset threshold, it is necessary to perform clustering processing on the data in the KV cache. The preset threshold can be flexibly set according to the current hardware conditions, for example, not exceeding 80% of the total capacity of the device. In order to ensure the effectiveness of the clustering process and the reasonable distribution of the cluster centers, the cluster centers must first be initialized, and then clustering processing is performed.

[0032] Initialize cluster centers. The relevant steps include: (1) select clustering algorithm; (2) determine the number of cluster centers; (3) initialize cluster centers.

[0033] (1) Select a clustering algorithm.

[0034] In the embodiment of the present invention, a distance-based clustering algorithm may be selected for clustering.

[0035] Exemplary: K-means algorithm or other distance-based clustering algorithms can be selected; if K-means algorithm is used, K-means++ optimization strategy (i.e., algorithm for selecting initial values ​​by K-means algorithm) is used for subsequent initialization of different cluster centers, and the positions of cluster centers are dynamically adjusted during the clustering process, while clustering results are iteratively updated to ensure convergence and accuracy.

[0036] (2) Determine the number of cluster centers.

[0037] In an embodiment of the present invention, there are multiple cluster centers, and the number of cluster centers can be dynamically adjusted according to the scale and distribution of the KV cache data to ensure a balance between accuracy and storage efficiency. Specifically, the optimal number of cluster centers can be determined by a heuristic algorithm or a cross-validation method, and iteratively updated according to data changes during the reasoning process to adapt to the storage and computing requirements in different scenarios.

[0038] (3) Initialization of cluster centers.

[0039] In the embodiment of the present invention, the initialized cluster center can be obtained by selecting the initialization strategy of the cluster center.

[0040] Exemplary: Initialization strategies may include: random selection, K-means++ optimization strategy or method based on prior information; if random selection is adopted, several data points are randomly selected from the KV cache data as the initial cluster centers to simplify the initialization process. If the K-means++ optimization strategy is adopted, the uniform distribution of cluster centers is ensured by increasing the distance weight to improve the stability and accuracy of the clustering effect. If a method based on prior information is adopted, the existing data distribution knowledge in the prior information is used to initialize the cluster centers, thereby accelerating the clustering process and improving the quality of the final clustering results. The choice of different strategies depends on the specific data characteristics and reasoning requirements to achieve the best clustering effect.

[0041] Preferably, since the attention score is involved in the generation process of the KV cache, the cluster center initialization process can be optimized thereby; including but not limited to the following methods: randomly selecting the attention score as a probability distribution function, selecting the cluster center according to the attention score, or using a mixed strategy of partial selection based on the attention score and partial random selection to optimize the cluster center initialization process.

[0042] 2. Clustering calculation.

[0043] In an embodiment of the present invention, after completing the initialization of the cluster center, the next step will be entered - the calculation process of clustering the key (Key) and value (Value) data in the KV cache. The core of this step is to divide the data in the KV cache into several cluster categories through a clustering algorithm, and associate each type of data with its corresponding cluster center. The main process is as follows: calculate the distance between each data in the KV cache and each cluster center respectively, classify each data into the cluster category to which the cluster center closest to it belongs, and continuously update each cluster center; finally, all data in the KV cache are clustered to obtain several cluster categories, each cluster category corresponds to a cluster center, and contains the data in the corresponding KV cache.

[0044] Specifically, the distance between each data point and all cluster centers is calculated first. When calculating for the first time, the cluster centers involved here are the initialized cluster centers obtained by the above scheme. This part is based on different distance measurement methods, such as Euclidean distance, cosine similarity or other measurement methods suitable for data characteristics. Through these distance calculations, the similarity between each data point and all cluster centers can be evaluated.

[0045] After that, each data point is classified into the category of the cluster center that is closest to it. This process is called the assignment phase, in which each data point is marked as belonging to a specific cluster. This assignment applies not only to the key data, but also to the corresponding value data to ensure that the entire KV cache data can be accurately classified into the corresponding cluster.

[0046] As each data point is classified, the cluster centers are continuously updated. Through multiple iterations, the positions of these cluster centers are gradually optimized so that they can better represent the data distribution of the corresponding categories. Eventually, the clustering process will gradually stabilize and form several cluster categories, each of which corresponds to a cluster center and contains the key-value pair data closest to the center.

[0047] 3. Rearrange according to the clustering results.

[0048] After completing the clustering calculation, the embodiment of the present invention will rearrange the data in the KV cache according to the clustering results. Specifically, the data belonging to the same cluster is reorganized so that the data items in the same cluster are arranged in adjacent positions. In addition, it is necessary to calculate the attention score of the center point of each cluster to determine the order of ranking between clusters. Generally, the center point of the cluster with a higher attention score will be arranged in a more forward position.

[0049] like Figure 2 As shown in Figure 1, an example of cluster rearrangement is provided. After clustering is completed, 10 data are divided into 3 cluster categories. During the data rearrangement process, the data items belonging to the same cluster are arranged in order: first, the data belonging to cluster 1 (data 1, 3, 4) are placed in adjacent positions, then the data belonging to cluster 2 (data 2, 5, 10) are placed, and finally the data belonging to cluster 3 (data 6, 7, 8, 9) are placed.

[0050] 4. Low-precision quantization.

[0051] In the embodiment of the present invention, data in a set proportion of cluster categories are selected, and the key data and value data therein are quantized according to a set number of bits.

[0052] For example, low-precision quantization can be applied to data in the first 70% to 80% of cluster categories. This selection range helps to reduce video memory usage while ensuring the accuracy of the inference results. If the proportion of quantized data is too small, it may have a significant impact on the accuracy of the inference results.

[0053] In the low-precision quantization process, the embodiment of the present invention adopts different quantization strategies for the data in the KV cache. Figure 3As shown in the figure, an example of quantization of key data and value data is provided, that is, the key data is quantized by 2 bits (2-bit), while the value data is quantized by 1 bit (1-bit). Specifically, quantizing the key data by 2 bits means compressing the representation of each key data into 4 possible discrete values. This can significantly reduce storage requirements while still maintaining the basic information of the key data for efficient retrieval and matching. For value data, a lower bit width can be used, and 1-bit quantization can be used to compress it into 2 possible states (usually 0 or 1). This extreme quantization method helps to further reduce storage usage and is suitable for scenarios with relatively low precision requirements. Although 1-bit quantization may lose some precision, for most reasoning tasks, its impact is within an acceptable range and can significantly improve the storage efficiency of the system.

[0054] 5. Cluster center replacement.

[0055] In the embodiment of the present invention, for the data in the remaining cluster categories (i.e., the cluster data not quantized with low precision), the cluster center point is used to replace the data of the entire cluster. Specifically, since the cluster center point is generated by calculating the average value of all data in the cluster category to which it belongs, using the cluster center point to replace the original data can not only significantly reduce the storage requirements, but also improve the computing efficiency while maintaining the representativeness of the data.

[0056] However, since the cluster center is only an approximate representation of the entire cluster data, in order to avoid losing important information about the distribution of the original data, the number of data in the cluster category needs to be recorded, which can be used to adjust the weight information (i.e., the weight of each cluster category) in the subsequent reasoning process of the large language model, thereby optimizing the efficiency and accuracy of the subsequent reasoning process. The number of recorded data will play a key role in the subsequent reasoning and calculation process. For example, some calculations in the reasoning process may need to combine the number of data points to restore a more accurate result. At this time, this can be achieved by adjusting the relevant weight information in the subsequent reasoning process of the large language model.

[0057] 6. Update and iteration of KV cache.

[0058] After completing the processing of the first to fifth parts above, the data after quantization and cluster center replacement processing, that is, the optimized KV cache data, is obtained. After the optimized KV cache data is stored, it can be used for the subsequent reasoning process of the large language model. The related reasoning process involved can refer to conventional technology, and the present invention will not elaborate on it.

[0059] At the same time, iteration can continue in the subsequent reasoning process. That is, if the size of the KV cache exceeds the preset threshold again, the processing of the first to fifth parts above will be performed again.

[0060] The above-mentioned solution provided by the embodiment of the present invention, when the KV cache size exceeds a preset threshold, divides the KV cache into several clustering groups through data clustering technology, and then combines cluster center replacement processing and quantization strategy to significantly reduce the storage requirements of the KV cache, thereby effectively reducing video memory occupancy and improving reasoning efficiency and system resource utilization.

[0061] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0062] Embodiment 2 The present invention also provides a large language model reasoning optimization system, which is mainly used to implement the method provided in the above embodiment, such as Figure 4 As shown, the system mainly includes: A KV cache amount determination module is used to determine whether the KV cache of the large language model exceeds a preset threshold; wherein the KV cache represents a key value cache; The clustering module is used to cluster the data in the KV cache to obtain several cluster categories when the KV cache exceeds a preset threshold; rearrange the data in the KV cache according to the cluster categories, that is, reorganize the data belonging to the same cluster category so that the data in the same cluster category are arranged in adjacent positions, and for each cluster category, calculate the attention score of its cluster center respectively, and arrange the cluster categories in descending order according to the attention score; A quantification module is used to quantify the data in a part of the cluster categories according to the data categories; The cluster center replacement module is used to replace the data in another part of the cluster category with the cluster center of the cluster category to which it belongs; The inference module is used to use the optimized KV cache data for subsequent inference of the large language model; the data after quantization and cluster center replacement processing is the optimized KV cache data.

[0063] Considering that the main technical details involved in the system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0064] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0065] Embodiment 3 The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0066] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0067] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example: The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.; The output device may be a display terminal; The memory may be a random access memory (RAM) or a non-volatile memory such as a disk storage.

[0068] Embodiment 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0069] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0070] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A large language model reasoning optimization method, characterized in that: include: Determine whether the KV cache of the large language model exceeds a preset threshold; wherein the KV cache represents a key-value cache; If it exceeds the preset threshold, the data in the KV cache is clustered to obtain several cluster categories; the data in the KV cache is rearranged according to the cluster categories, that is, the data belonging to the same cluster category is reorganized so that the data in the same cluster category are arranged in adjacent positions, and for each cluster category, the attention score of its cluster center is calculated respectively, and the cluster categories are arranged in descending order according to the attention score; Quantify the data in a part of cluster categories according to the data category; replace the data in another part of cluster categories with the cluster center of the cluster category to which it belongs; After quantization and cluster center replacement, the data becomes the optimized KV cache data, which is used for subsequent reasoning of the large language model.

2. A large language model reasoning optimization method according to claim 1, characterized in that: Before clustering the data in the KV cache, the step of initializing the cluster center is also included. The relevant steps include: Select a clustering algorithm, that is, select a distance-based clustering algorithm for clustering; Determine the number of cluster centers: There are multiple cluster centers. According to the data scale and distribution of the KV cache, the number of cluster centers is determined by a heuristic algorithm or a cross-validation method, and iteratively updated during the reasoning process; Cluster center initialization: The initialized cluster center is obtained by the selected cluster center initialization strategy.

3. A large language model reasoning optimization method according to claim 2, characterized in that: Also includes: Optimize the cluster center initialization process; That is, the generation process of the KV cache involves attention scores, and the attention scores are randomly selected as probability distribution functions, cluster centers are selected according to the attention scores, or a mixed strategy of partial selection based on attention scores and partial random selection is adopted to optimize the cluster center initialization process.

4. A large language model reasoning optimization method according to claim 2, characterized in that: The initialization strategies include: random selection, K-means++ optimization strategy or a method based on prior information; If random selection is used, several data points are randomly selected from the KV cache data as the initial clustering centers; If the K-means++ optimization strategy is adopted, the cluster centers are evenly distributed by increasing the distance weight; the K-means++ optimization strategy is an algorithm for selecting initial values ​​for the K-means algorithm, and the K-means algorithm is a K-means algorithm; If there is prior information, a method based on prior information is used to initialize the cluster center using the existing data distribution knowledge in the prior information.

5. A large language model reasoning optimization method according to claim 1 or 2, characterized in that: The data in the KV cache is clustered to obtain several clustering categories including: Calculate the distance between each data in the KV cache and each cluster center respectively, classify each data into the cluster category to which the cluster center closest to it belongs, and continuously update each cluster center; Finally, all the data in the KV cache is clustered to obtain several cluster categories. Each cluster category corresponds to a cluster center and contains the data in the corresponding KV cache.

6. A large language model reasoning optimization method according to claim 5, characterized in that: The quantizing of the data in a part of the cluster categories according to the data categories includes: selecting the data in a part of the cluster categories that are at the front in the rearranged data, and quantizing the key data and the value data therein according to the set number of bits.

7. A large language model reasoning optimization method according to claim 1, characterized in that: When the cluster center of the corresponding cluster category is used as a substitute, the amount of data in the cluster category is recorded at the same time, and the recorded amount information is used to adjust the weight information in the subsequent reasoning process of the large language model.

8. A large language model reasoning optimization system, characterized in that: include: A KV cache amount determination module is used to determine whether the KV cache of the large language model exceeds a preset threshold; wherein the KV cache represents a key value cache; The clustering module is used to cluster the data in the KV cache to obtain several cluster categories when the KV cache exceeds a preset threshold; rearrange the data in the KV cache according to the cluster categories, that is, reorganize the data belonging to the same cluster category so that the data in the same cluster category are arranged in adjacent positions, and for each cluster category, calculate the attention score of its cluster center respectively, and arrange the cluster categories in descending order according to the attention score; A quantification module is used to quantify the data in a part of the cluster categories according to the data categories; The cluster center replacement module is used to replace the data in another part of the cluster category with the cluster center of the cluster category to which it belongs; The inference module is used to use the optimized KV cache data for subsequent inference of the large language model; the data after quantization and cluster center replacement processing is the optimized KV cache data.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images