A step-mapping based language model speculation reasoning acceleration algorithm

Abstract

The present application relates to the field of artificial intelligence and computer, more particularly, the present application relates to a language model speculation reasoning acceleration method and system based on step mapping, which is applied to a computing device deployed with a target language model and a draft language model, the method sets the maximum number of steps of draft generation and limits the running step number K, constructs a draft language model containing M identical processing modules, pre-configures and solidifies a static step mapping function g(k)=k, calibrates the single-step cost offline under the reference input shape (,,D) and sets the threshold value, forms a reproducible configuration record, during reasoning, the target hidden state vector h is reused and fused with the token embedding vector e in the fusion layer, and only the corresponding processing module is activated according to the step to generate the candidate sequence; the target language model verifies the candidate sequence vectorization to obtain the accepted prefix length A, and according to the acceptance statistics, the speculation step number K is adaptively adjusted up or down according to the period T, and the update result is written into the configuration record to support subsequent reproducible calling.

Images