Device for service-level agreement-based orchestration of speech processing agents with deterministic latency guarantee

Abstract

Orchestration device (10) for service agreement-based control of speech processing agents with deterministic latency guarantee, comprising: a) an agreement management unit (12) that sets and stores a latency threshold and an associated percentile criterion for each incoming request; b) a latency monitoring unit (20) that measures the response times of all available speech processing models in real time and records them as a latency histogram; c) a context-dependent model selection unit (30) that, based on the request context, the current latency values, and the set agreement thresholds, selects the speech processing model that is most likely to deliver the best result quality while adhering to the agreement; d) a preemptive buffer unit (40) that preprocesses frequently requested contexts and stores their results for immediate retrieval;e) an automatic failover unit (50) which, in the event of an imminent exceedance of the latency threshold, redirects the request to a local speech processing model.;