Model reasoning method and device, electronic equipment and storage medium

Abstract

The invention provides a model reasoning method and device, electronic equipment and a storage medium, and the method comprises the steps: responding to a lexical element sparse instruction, obtaining lexical elements to be discarded from the key value cache information of each layer of a large model, the lexical elements to be discarded of each layer being determined after lexical element sparse processing; and carrying out interlayer combination on the lexical elements to be discarded of each layer, and executing model reasoning operation based on an interlayer combination result. According to the method, the purposes of saving the memory, accelerating the attention calculation and improving the model reasoning effect can be achieved in the whole model reasoning process at the same time, in addition, the influence on hardware is mainly reflected in the aspect of memory saving, the memory or on-chip storage of an artificial intelligence chip can be reduced, and therefore most existing large models can be directly supported, and the model reasoning efficiency is improved. And extra training cost is not needed.

Images