Large language model reasoning optimization method, system, device and storage medium

By clustering and quantizing the data cached in KV during the inference process of large language models, the problem of excessive demand for KV cache storage is solved, which significantly reduces the memory usage and improves the inference efficiency and resource utilization.

CN119323264BActive Publication Date: 2025-03-25UNIV OF SCI & TECH OF CHINA
2 Cites 0 Cited by

Patent Information

Application Number
CN202411881594.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-12-19
Publication Date
2025-03-25
Estimated Expiration
2044-12-19

AI Technical Summary

Technical Problem

During the inference process of large language model, the storage demand for KV caches has increased linearly, resulting in a significant increase in video memory usage, becoming the main bottleneck of the system, limiting the model's ability to reason on devices with limited resources.

Method used

By determining whether the KV cache exceeds the preset threshold, if it exceeds, the data in the KV cache is clustered, several cluster categories are obtained, and the data is rearranged according to the cluster category, the attention score of the cluster center is calculated, and the cluster center is replaced is quantified and cluster center is replaced, reducing the storage demand of the KV cache.

Benefits of technology

It significantly reduces the video memory usage during the inference process, improves the inference efficiency and system resource utilization.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The present invention discloses a large language model reasoning optimization method, system, device and storage medium, which are corresponding schemes. In the scheme, a data clustering and quantization mechanism is introduced in the large language model reasoning process. When the size of the KV cache exceeds the preset threshold, the KV cache data is first clustered and divided, and then part of the data is subjected to low-precision quantization processing, and the other part is replaced by the cluster center, so as to achieve the purpose of reducing the KV cache size. Through the scheme of the present invention, the video memory occupancy in the reasoning process can be significantly reduced, the reasoning efficiency can be improved, and the utilization rate of system resources can be optimized.
Need to check novelty before this filing date? Find Prior Art