Large language model reasoning optimization system, method and equipment and storage medium

Abstract

The embodiment of the invention provides a big language model reasoning optimization system, method and equipment and a storage medium. The system comprises an asynchronous KV cache pre-fetching module, a heterogeneous Specific decoding module, a dynamic model re-fragmentation scheduling module, a Memory-Aware dynamic batch module and a scheduling control engine, and can organically integrate the asynchronous KV cache pre-fetching module, the heterogeneous Specific decoding module, the dynamic model re-fragmentation scheduling module and the Memory-Aware dynamic batch module, so that the dynamic model re-fragmentation scheduling module and the Memory-Aware dynamic batch module are integrated, and the dynamic model re-fragmentation scheduling module and the Memory-Aware dynamic batch module are integrated. And a set of closed-loop adaptive optimization system is formed under the unified coordination of a scheduling control engine, so that module-level linkage, adaptive coordination and performance feedback closed loop in the reasoning process are realized, and finally, the throughput, the response speed and the resource utilization rate in the reasoning stage are remarkably improved on the premise of not influencing the accuracy of a large language model.

Images