Compression method, device and equipment of key value cache, storage medium and program product

Abstract

This application relates to a method, apparatus, device, storage medium, and program product for key-value caching compression. The method includes: obtaining the lexical sequence and the number of lexical caches corresponding to each network layer in a large language model; for any network layer, splitting the lexical sequence into key sink lexicals, task instruction lexicals, and context semantic lexicals according to the position of each lexical in the network layer's lexical sequence; selecting target semantic lexicals that match the task instruction lexicals of the network layer from the context semantic lexicals based on the number of lexical caches; compressing the key-value information corresponding to the key sink lexicals and target semantic lexicals in the network layer to obtain compressed key-value inference information for the network layer; and caching the compressed key-value inference information for all network layers. This method can improve the accuracy of key-value information, thereby improving the inference accuracy of the large language model.

Images