Virtual batches in large language model inferences

Abstract

This document describes systems and techniques directed at virtual batches in large language model (LLM) inferences. An LLM, at least partially deployed on an electronic device, generates a dependency map for a plurality of inference tokens. The inference tokens can be based on a same input, different inputs, or a mixture of both. The dependency map indicates sequential or otherwise logical dependence of each inference token. The LLM can further generate a plurality of virtual batches based on the dependency map. The plurality of virtual batches includes masked portions indicating positions in one or more of the plurality of virtual batches that do not have an active token reference.

Images