Large model inference scheduling method and system, storage medium and computer program product
By having computing nodes actively report status data and performance metrics, and adjusting to a computing node-led scheduling mode, the problems of load imbalance and high latency in traditional solutions are solved, achieving more efficient load balancing and stability.
CN121809702BActive Publication Date: 2026-06-26IFLYTEK CO LTD
2 Cites 0 Cited by
Patent Information
- Application Number
- CN202610285806.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-10
- Publication Date
- 2026-06-26
- Estimated Expiration
- 2046-03-10
AI Technical Summary
Technical Problem
Traditional distributed scheduling schemes cannot adapt to the characteristics of large model inference loads, resulting in uneven distribution of inference tasks, decreased throughput, and high inference latency.
Method used
By having computing nodes proactively perceive their own status data, calculate performance metrics, and report them to the gateway, the gateway uses these metrics to allocate inference tasks and adjusts to a computing node-led scheduling mode to achieve load balancing.
Benefits of technology
It improves the overall system throughput, enhances inference efficiency and service stability, and solves the problems of unbalanced load and high latency in traditional solutions.
✦ Generated by Eureka AI based on patent content.
Abstract
The application discloses a large model inference scheduling method and system, a storage medium and a computer program product, and relates to the technical field of large model inference scheduling. The application changes the "blind pushing" mode of the gateway in the traditional scheme, and adjusts to be dominated by the computing node. The computing node can dynamically and comprehensively perceive state data of the node, such as processor state and inference task state borne by the node. Based on the state data, the performance index of the computing node can be determined. The computing node actively reports the performance index to the gateway. The gateway can allocate target computing nodes for target inference tasks to be allocated based on the latest performance indexes of the computing nodes, better realize load balancing, and then improve the overall throughput of the system, improve inference efficiency, and improve the stability of inference service.
Need to check novelty before this filing date? Find Prior Art