Hybrid language model cascade with hardware topology-based dynamic model selection

Abstract

Device for dynamically cascading speech processing models of different parameter sizes on a local data processing system, comprising: a) a topology detection unit (10) that identifies the available computing units of the data processing system at runtime, continuously records their respective utilization, available memory and current energy consumption and provides this as a topology data set (11); b) a model selection unit (20) that determines the optimal combination of at least two speech processing models of different parameter sizes based on the topology data set (11) and a query complexity assessment, wherein smaller models are provided for edge computing units and larger models for more powerful computing units;c) a cascading unit (30) that transparently distributes incoming requests to the language processing models determined by the model selection unit (20) and switches between models without interrupting the processing process if the hardware topology or resource availability changes during processing; d) a result aggregation unit (40) that combines partial results from different models into a single response.