A heterogeneous device large language model adaptive collaborative reasoning method for dynamic edge environment

Abstract

This invention discloses an adaptive collaborative inference method for large language models on heterogeneous devices in dynamic edge environments. Applied to physical network environments composed of multiple heterogeneous physical devices with physical computing chips, physical memory, and network communication interfaces, this method adaptively segments and distributes a complete large language model across multiple physical devices during collaborative inference. The method first monitors and predicts the operating status of the physical devices, then solves for the large language model reallocation scheme that maximizes net time gain, and finally minimizes the pause time of collaborative inference by switching allocation schemes. This invention effectively addresses the dynamic fluctuations in network and computing resources in edge environments, enabling smooth switching of collaborative inference among heterogeneous physical devices, thereby reducing overall collaborative inference time and significantly improving system resource utilization efficiency.

Images