Device for ensemble optimization of small language models through latency-capable local switching units

Abstract

Device for latency-optimized routing of subtasks to an ensemble of locally operated small language models, comprising: a) a routing unit (10) with a hardware-level routing algorithm for selecting the locally available language model that is optimally suited for a given subtask with respect to latency and capability profile, b) a latency measurement module (20) for continuously recording the inference latency of each available language model, wherein the latency decision is made in the range of 1-2 milliseconds, c) a task decomposition unit (30) for decomposing an input query into a plurality of specialized subtasks, each of which can be assigned to a single language model, and d) a result aggregation unit (40) for combining the partial results of the activated language models into a coherent overall result.