Edge-end collaborative reasoning method and system based on large model characteristics and dynamic network

Abstract

The application discloses an edge-end collaborative reasoning method and system based on large model characteristics and dynamic networks, and belongs to the technical field of large model deployment reasoning. The collaborative reasoning method comprises the following steps: constructing a relationship model between edge-end network resources and model interaction data and an edge-end collaborative triggering mechanism based on network resources, and jointly determining whether to trigger edge-end collaboration through model confidence and network delay; designing a collaborative reasoning enhancement strategy based on speculative sampling under different network environments to optimize the large and small model collaborative reasoning process; by introducing a collaborative triggering function based on confidence and network delay, the system can realize on-demand collaboration by real-time sensing of model output quality and communication environment changes, meet the dual requirements of real-time performance and performance on the premise of ensuring accuracy, fully consider the computing resource occupation, reasoning delay and network transmission bottleneck of large models in edge-end heterogeneous environments, and improve the reasoning efficiency through an efficient collaborative reasoning strategy.

Images