Edge device inference chain cache for local speech processing agents

Abstract

Device for local caching and reuse of inference steps for language processing agents on edge devices, comprising: a) a persistent cache (10) that stores completed inference chains together with their associated input contexts, context vectors and results as indexed entries on a local storage medium, wherein the storage is performed without transferring data to a network-based service; b) a semantic analysis unit (20) that computes a context vector for each incoming request and compares it with the context vectors of the entries stored in the cache (10) in order to identify semantically similar inference chains;c) a reuse logic (30) which, when a configurable similarity threshold between the incoming request and a stored entry is exceeded, reuses the stored inference steps in whole or in part, recalculating only differing substeps; d) an adaptive decay value calculation unit (40) which determines an individual decay value for each entry in the cache (10), based on the semantic stability of the input context and the frequency of reuse.