Request allocation method, system and equipment based on heterogeneous computing system and medium

By obtaining inference task information and heterogeneous computing power equipment performance parameters in heterogeneous computing system, determining the key-value cache reading time and selecting appropriate equipment, the problem of poor balance in inference request allocation is solved, and the saving of computing power resources and cost reduction is achieved.

CN119690687AActive Publication Date: 2025-03-25LANGCHAO ELECTRONIC INFORMATION IND CO LTD
8 Cites 9 Cited by

Patent Information

Application Number
CN202510221607.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-02-27
Publication Date
2025-03-25
Estimated Expiration
2045-02-27

AI Technical Summary

Technical Problem

In heterogeneous computing systems, the allocation balance of inference requests is poor, resulting in a lot of wasted computing resources, especially when calling separate memory or local memory.

Method used

By obtaining inference task information and performance parameters of heterogeneous computing power equipment, the reading time required for key-value cache is determined, and the appropriate heterogeneous computing power equipment is selected based on the reading time, the preset allocation number is determined based on the target reading time and performance parameters, and the request allocation result is determined based on the preset allocation number and the concurrent request number.

Benefits of technology

It improves the allocation balance of inference requests, saves computing resources, reduces costs, and speeds up the execution of inference tasks.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a request allocation method and system based on a heterogeneous computing system, equipment and a medium, relates to the technical field of computers, and determines reading time required for using key value cache when a plurality of heterogeneous computing power equipment accesses a memory under the condition of preferentially considering the use condition of a key value cache mechanism. Requests are reasonably distributed by considering the performance difference of the computing power information of the heterogeneous computing power equipment, the memory information corresponding to the memory extension and the characteristics of the reasoning task information. Furthermore, the reasonability of request allocation is improved according to a comparison relationship between the preset allocation request quantity and the concurrent request quantity and different strategies for judging whether allocation conditions are met or not. Therefore, the technical problem that more computing power resources are wasted due to the fact that the allocation balance of the reasoning request is poor when a separated memory or a local memory is called can be solved, and the technical effects that the reasoning task information is reasonably allocated to the heterogeneous computing power equipment, so that the allocation balance is improved, the computing power resources are saved, and the cost is reduced are achieved.
Need to check novelty before this filing date? Find Prior Art