Model reasoning computing power scheduling method and device for data center

Abstract

The invention belongs to the technical field of resource scheduling, and discloses a model reasoning computing power scheduling method and device for a data center, and the method comprises the steps that an intelligent service module obtains a capacity expansion and contraction triggering condition based on a reasoning framework; the monitoring component module calculates GPU operation data to obtain a GPU calculation core load rate and sends the GPU calculation core load rate to the intelligent service module; the intelligent service module determines a current load stage according to a GPU calculation core load rate and a capacity expansion and contraction trigger condition; and the intelligent service module determines the computing power scheduling range of the large language model in the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool according to the current load stage, so that the large language model processes the user request based on the computing power scheduling range. Through collaborative analysis of the intelligent service module and the capacity expansion and contraction triggering condition, collaborative management of the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool can be realized, and continuity and stability of computing power resource supply of an enterprise in an internet data center are guaranteed.

Description

Technical Field

[0001] This application relates to the field of resource scheduling technology, and in particular to a method and apparatus for scheduling model inference computing power in a data center. Background Technology

[0002] In existing technologies, for large Internet data center enterprises, due to the significant real-time and bursty nature of Internet services, the traffic distribution of their internal large language model inference tasks generally exhibits pulse-like characteristics. The pressure of inference tasks varies significantly at different times, posing a great challenge to the resource scheduling capabilities of GPUs.

[0003] Typically, the pure on-premises deployment solution for Internet data centers relies entirely on the enterprise's own GPU clusters for inference tasks, using statically partitioned resource pools to handle these tasks. This solution can lead to service interruptions during sudden surges in inference tasks because the resource pools cannot be elastically scaled up, and the resource pools are severely idle during off-peak periods, creating a heavy asset burden for cost-effective Internet data center enterprises. The pure cloud resource deployment solution relies on elastic GPU instances in the public cloud, dynamically adjusting resources through cloud-native tools. While it can cope with traffic fluctuations, for Internet data center enterprises that have already deployed a large amount of business and data locally, complete reliance on cloud resources can lead to uncontrolled costs and increased risks of data leakage, making it difficult to meet the real-time response and data localization requirements of various Internet data center businesses.

[0004] In summary, the purely offline deployment solution for Internet data centers faces the problem of insufficient peak computing power and idle resources during off-peak periods, while the purely cloud-based deployment solution faces an imbalance between flexibility and cost. Together, these two issues have become the core bottlenecks restricting the efficient and economical implementation of large language model inference services in Internet data centers. Summary of the Invention

[0005] This application provides a method and apparatus for scheduling model inference computing power in a data center. Through collaborative analysis of intelligent service modules and scaling trigger conditions, it enables collaborative management of local intelligent computing base resource pools and cloud-based dynamic elastic computing power pools. Based on GPU computing core load rate analysis, it prioritizes the use of low-cost local intelligent computing base resource pools during off-peak periods and expands to the cloud-based dynamic elastic computing power pool during peak periods, avoiding resource idleness and shortages. Dynamic resource adjustments reduce enterprise expenses and ensure the continuity and stability of computing power resource supply for internet data center enterprises.

[0006] In a first aspect, embodiments of this application provide a method for scheduling model inference computing power in a data center, the method comprising: The intelligent service module obtains the scaling-up / scaling-down trigger conditions based on the inference framework; The monitoring component module calculates GPU operating data, obtains the GPU computing core load rate, and sends it to the intelligent service module. The intelligent service module determines the current load stage based on the GPU computing core load rate and the scaling up / down trigger conditions; The intelligent service module determines the computing power scheduling range of the large language model from the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool based on the current load stage, and enables the large language model to process user requests based on the computing power scheduling range.

[0007] Furthermore, the scaling up / down trigger conditions include: GPU computing core load rate less than 60%, GPU computing core load rate greater than or equal to 60%, and GPU computing core load rate less than or equal to 30%.

[0008] Furthermore, the intelligent service module determines the current load stage based on the GPU computing core load rate and scaling trigger conditions, including: When the GPU computing core load rate is less than 60%, the current load phase is a low load phase. When the GPU computing core load rate is greater than or equal to 60%, the current load stage is a high load stage; When the GPU computing core load rate is less than or equal to 30%, the current load phase is the load decline phase.

[0009] Furthermore, the intelligent service module determines the computing power scheduling range of the large language model from the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool based on the current load stage, enabling the large language model to process user requests based on the computing power scheduling range, including: When the current load phase is a low load phase, the computing power scheduling scope is the first inference task module in the local intelligent computing base resource pool, and the user request is input into the first inference task module in the local intelligent computing base resource pool for processing.

[0010] Furthermore, the intelligent service module determines the computing power scheduling range of the large language model from the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool based on the current load stage, enabling the large language model to process user requests based on the computing power scheduling range, including: When the current load phase is a high load phase, determine whether the local intelligent computing base resource pool meets the deployment requirements of the second inference task module; if so, deploy the second inference task module in the local intelligent computing base resource pool, and use the second inference task module as the computing power scheduling range, and input user requests into the second inference task module for processing; If not, request a deployment node from the cloud dynamic elastic computing power pool; in response to the deployment permission instruction of the cloud dynamic elastic computing power pool, deploy the second inference task module on the node permitted by the cloud dynamic elastic computing power pool; After deployment, the second inference task module will be used as the computing power scheduling scope, and user requests will be processed by the second inference task module of the cloud dynamic elastic computing power pool.

[0011] Furthermore, the method also includes: When the current load phase is the load rollback phase, the intelligent service module releases the nodes in the cloud dynamic elastic computing power pool and reclaims the second inference task modules deployed on the nodes.

[0012] Secondly, embodiments of this application provide a model inference computing power scheduling device for a data center, the device comprising: The intelligent service module is used to obtain the scaling-up and scaling-down trigger conditions based on the inference framework; determine the current load stage based on the GPU computing core load rate and the scaling-up and scaling-down trigger conditions; determine the computing power scheduling range of the large language model in the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool based on the current load stage, and enable the large language model to process user requests based on the computing power scheduling range. The monitoring component module is used to calculate GPU operating data, obtain the GPU computing core load rate, and send it to the intelligent service module.

[0013] Furthermore, the scaling up / down trigger conditions include: GPU computing core load rate less than 60%, GPU computing core load rate greater than or equal to 60%, and GPU computing core load rate less than or equal to 30%.

[0014] Furthermore, the intelligent service module is specifically used to determine the current load stage as a low load stage when the GPU computing core load rate is less than 60%; to determine the current load stage as a high load stage when the GPU computing core load rate is greater than or equal to 60%; and to determine the current load stage as a load decline stage when the GPU computing core load rate is less than or equal to 30%.

[0015] Furthermore, the intelligent service module is also used to release nodes in the cloud dynamic elastic computing power pool and reclaim the second inference task module deployed on the nodes when the current load phase is the load fallback phase.

[0016] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following: This application provides a model inference computing power scheduling method for data centers. This method enables collaborative management of local intelligent computing base resource pools and cloud-based dynamic elastic computing power pools through the collaborative analysis of intelligent service modules and scaling trigger conditions. Based on the analysis of GPU computing core load rates, the method prioritizes the use of low-cost local intelligent computing base resource pools during off-peak periods and expands to the cloud-based dynamic elastic computing power pool during peak periods, avoiding resource idleness and insufficient resources. Dynamic resource adjustments reduce enterprise expenses and ensure the continuity and stability of computing power resource supply for internet data center enterprises. Attached Figure Description

[0017] Figure 1 A flowchart of a data center model inference computing power scheduling method provided as an exemplary embodiment of this application.

[0018] Figure 2 This is a structural diagram of a data center model inference computing power scheduling device provided as an exemplary embodiment of this application. Detailed Implementation

[0019] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0020] Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0021] Please see Figure 1 This application provides a method for scheduling model inference computing power in a data center, which specifically includes the following steps: Step S1: The intelligent service module obtains the scaling-up / scaling trigger conditions based on the inference framework.

[0022] In some embodiments, the scaling up / down triggering conditions include: GPU computing core load rate less than 60%, GPU computing core load rate greater than or equal to 60%, and GPU computing core load rate less than or equal to 30%.

[0023] Setting the percentages to 60% and 30% can meet the resource optimization needs of the system, further avoid ineffective scaling up and down, reduce system costs, and improve the utilization rate of resources in the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool.

[0024] In some embodiments, a tiered computing power scheduling rule is defined through CRO, and the inference task module is labeled "service: llm-inference" and bound with scheduling priorities. The first priority is the local intelligent computing base resource pool (node ​​pool ID: local-pool-01), and the second priority is the cloud dynamic elastic computing power pool (node ​​pool ID: cloud-pool-01). Then, the inference service parameters are configured according to the intelligent service module. First, the service name is defined (e.g., "llm-chat-service"). Then, the TensorRT-LLM inference framework is adopted. Finally, scaling trigger conditions are set, limiting the minimum number of instances to 1 and the maximum number of instances to 5. The single instance resource configuration is 1 GPU card, 8-core CPU, and 16Gi memory. The maximum number of instances and the single instance configuration can be customized as needed; otherwise, the default configuration of 1 minimum instance and 5 maximum instances will be used.

[0025] Specifically, an "instance" can be a GPU server in the cloud.

[0026] The local intelligent computing base resource pool includes hardware such as GPU servers and network switches, which can serve as a local resource pool to provide multiple local nodes for intelligent service modules, enabling local resource scheduling and saving costs. The cloud dynamic elastic computing power pool connects to the computing power interface of cloud vendors through APIs, which can provide more flexible cloud nodes for system resource scheduling and avoid insufficient node resources.

[0027] Preferably, the local intelligent computing base resource pool includes, but is not limited to, computing hardware, storage hardware, network hardware, infrastructure hardware, dedicated acceleration hardware, and management and security hardware.

[0028] Specifically, the computing hardware includes, but is not limited to, GPU servers, CPU servers, and resource scheduling nodes. GPU servers can be configured with models such as the NVIDIA DGX H100 and Huawei Atlas 800T A2, integrating 8-16 H100 / Ascend 910B accelerator cards per machine, paired with 2TB of DDR5 memory and a 400G network interface, thus supporting distributed training of trillion-parameter models. GPUs are interconnected via NVLink to achieve ultra-high bandwidth of 3.6TB / s. CPU servers can be configured with Inspur NF5488A5 (dual-socket Intel Xeon 64-core + 1TB memory) and Huawei TaiShan 2480, supporting data preprocessing, distributed task scheduling, and Kubernetes cluster management based on the large memory capacity. Resource scheduling nodes can be run by Dell PowerEdge R760 (64 cores + 512GB memory) to achieve heterogeneous computing power pooling and task orchestration.

[0029] Specifically, the storage hardware can provide high-speed data access, disaster recovery backup, and hot / cold data tiering, including but not limited to all-flash storage nodes, distributed object storage, and backup storage nodes. Among them, the all-flash storage nodes can adopt Huawei OceanStor Pacific 9920 (24×30TB NVMe SSD per node), support the RoCEv2 RDMA protocol, with a throughput of ≥100GB / s and a latency of <100μs; the distributed object storage can adopt Ceph clusters or Alibaba Cloud OSS gateways, be compatible with the S3 protocol, and have EB-level scalability, reducing storage costs by more than half; the backup storage nodes can adopt IBM TS4500 tape library, enabling it to support offline disaster recovery and PB-level cold data storage.

[0030] Specifically, the network hardware enables low-latency communication, traffic isolation, and security protection. This includes, but is not limited to, parameter plane network devices, storage plane network devices, service plane network devices, and management network devices. Parameter plane network devices can utilize Huawei CloudEngine XH9210 or NVIDIA Quantum-2 InfiniBand switches, supporting GPUDirect RDMA with latency less than 1μs, enabling lossless scaling to tens of thousands of cards. Storage plane network devices can use Huawei CE8850-64CQ, supporting the NVMe-oF protocol, achieving end-to-end RDMA optimization and bandwidth isolation to ensure efficient data loading. Service plane network devices can use Huawei CE6885, supporting SRv6 / VXLAN multi-tenant isolation and integrating DDoS protection and load balancing. Management network devices can use Huawei CE6857 out-of-band management switches, enabling independent plane monitoring of IPMI / BMC hardware status.

[0031] Specifically, the infrastructure hardware is adapted to environmental needs to maintain high-density deployment, heat dissipation, power supply, and environmental stability. This includes, but is not limited to, high-density cabinets, liquid cooling systems, power distribution systems, and intelligent monitoring modules. The high-density cabinets can utilize Huawei FusionModule 8000, which supports 80kW power density per cabinet and integrates liquid / air cooling hybrid heat dissipation and intelligent PDUs. The liquid cooling system can use GRC immersion liquid-cooled cabinets, with a single cabinet heat dissipation capacity ≥200kW, PUE ≤1.1, and power consumption reduced by 40%. The power distribution system can use Emerson Liebert UPS and Huawei FusionPower 2000, with dual-path AC power and diesel generator redundancy supporting modular expansion. The intelligent monitoring module can use Raritan PX3 intelligent PDUs for port-level power monitoring and Supermicro Superserver 2029U for full-stack monitoring of Prometheus+ELK.

[0032] Specifically, dedicated acceleration hardware can optimize computing power for specific scenarios, further improving efficiency and energy efficiency. This includes, but is not limited to, AI accelerator cards, DPU smart network cards, quantum computing interfaces, and optical interconnect devices. Among these, the AI ​​accelerator card can utilize Huawei Ascend 910B and Graphcore IPU-M2000, which support sparse computing and graph neural network acceleration, resulting in a 30% improvement in energy efficiency. The DPU smart network card can utilize NVIDIA BlueField-3, which supports network protocol offloading and storage virtualization, further releasing CPU computing power. The quantum computing interface can utilize Origin Quantum's QPanda 2.0 gateway, enabling hybrid scheduling of classical and quantum computing. The optical interconnect device can utilize Huawei OptiXtrans DC908, achieving ultra-low latency interconnection across data centers, reducing latency by 70%.

[0033] Specifically, the management of security hardware includes, but is not limited to, security protection devices and edge access devices; among them, security protection devices can adopt Huawei USG6600E firewalls and hardware encryption machines; edge access devices can adopt industrial-grade edge servers Huawei Atlas 500 to further support edge-cloud collaborative inference.

[0034] In some embodiments, the construction of a computing cluster is also included.

[0035] Specifically, firstly, the CB Edge Ultra full-domain computing power management platform, which supports distributed container orchestration, is adopted, and the CB Edge Ultra master node is deployed to complete the access configuration of the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool; secondly, the intelligent service module and monitoring component module are deployed, and the collection rules for monitoring indicators such as GPU computing core load rate and task response latency are configured; finally, a model asset repository supporting distributed storage protocols is built to store large language model files, such as files for Tongyi Qianwen and the Llama series, and the mounting mapping relationship with the inference task module is configured.

[0036] The collection rules for the monitoring metrics include: the collection object is the computing core of each GPU node, i.e., the inference task module; the collection frequency is once per second for GPU computing core load rate and once per 500 milliseconds for task response latency; GPU computing core load rate includes single-card real-time utilization, node average utilization, and total cluster utilization, and task response latency includes P50 / P90 / P99 quantile latency; the collected data is pushed in real time to the metric storage unit in the intelligent service module via HTTP / GRPC protocol, such as storing it in a time-series database; if three consecutive collection failures occur, a local caching mechanism will be triggered to store the most recent valid data and report it.

[0037] The mounting and mapping relationship between the model asset repository and the inference task module is configured based on a distributed storage protocol. Specifically: Model asset repository path: In distributed storage, directories are divided according to model type and version, for example, / llm-models / {model_name} / {version}; Inference task module path: The preset model loading directory within the inference task module, for example, / mnt / models / {model_name}; Mapping rule: The path / llm-models / qwen / 7b-v1 of the model asset repository is mounted to the path / mnt / models / qwen of the inference task container through the storage configuration file, achieving a one-to-one mapping and supporting read-only permissions.

[0038] In some embodiments, when the inference task module starts, it needs to load a pre-trained large language model file to run the inference logic. Through pre-configured mount mappings, the inference task module can directly access model files (such as weight files and configuration files) in the model asset repository from its internal mount path (e.g., / mnt / models / qwen), eliminating the need to store the model internally. This saves storage space for the inference task module and ensures that all inference task modules use the same model version. For example, this mapping can be referenced in the deployment command using the `--data="llm-model: / mnt / models / Qwen"` parameter, enabling the inference framework to load the model from the ` / mnt / models / Qwen` path, ensuring the normal startup and operation of the inference service.

[0039] Step S2: The monitoring component module calculates the GPU running data, obtains the GPU computing core load rate, and sends it to the intelligent service module.

[0040] The monitoring component module continuously calculates and monitors GPU operating data, obtaining the GPU computing core resource load rate and sending it to the intelligent service module. The intelligent service module then determines the stage of the load and takes appropriate action. Continuous monitoring through the monitoring component module enables the monitoring of GPU core load rate and other data, providing a data foundation for the flexible scaling of each resource node.

[0041] Step S3: The intelligent service module determines the current load stage based on the GPU computing core load rate and the scaling up / down trigger conditions.

[0042] In some embodiments, the intelligent service module determines the current load stage based on the GPU computing core load rate and scaling trigger conditions, including: When the GPU computing core load rate is less than 60%, the current load phase is a low load phase.

[0043] When the GPU computing core load rate is greater than or equal to 60%, the current load stage is a high load stage.

[0044] When the GPU computing core load rate is less than or equal to 30%, the current load phase is the load decline phase.

[0045] Among them, the intelligent service module realizes the collaborative management of the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool by analyzing and elastically scheduling based on the GPU computing core load rate and the scaling trigger conditions. With the GPU computing core load rate as the core indicator, combined with the node scheduling, it realizes the automation and precision of the large language model inference computing power scheduling, avoiding the problems of over-scaling and response latency.

[0046] In step S4, the intelligent service module determines the computing power scheduling range of the large language model in the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool according to the current load stage, and enables the large language model to process user requests based on the computing power scheduling range.

[0047] In some embodiments, a data center is a complex collection of facilities, including hardware such as servers, storage devices, and switches, as well as software and physical infrastructure supporting the hardware operation. It serves as the foundation for technologies such as data analysis and model training, providing the necessary data and computing power for these technologies. This application adjusts the computing power scheduling range for large language models within the data center, which can indirectly maintain the data center and improve its operational stability and continuity.

[0048] In some embodiments, the intelligent service module determines the computing power scheduling range of the large language model in the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool according to the current load stage, and enables the large language model to process user requests based on the computing power scheduling range, including: when the current load stage is a low load stage, the computing power scheduling range is the first inference task module in the local intelligent computing base resource pool, and the user request is input into the first inference task module in the local intelligent computing base resource pool for processing.

[0049] Specifically, during the low-load phase, user inference requests are processed by the first inference task module. The intelligent service module sets the computing power scheduling range of the first inference task module to the local intelligent computing base resource pool, limiting its scheduling of nodes in the cloud dynamic elastic computing power pool, thereby saving system resources and maintaining stable traffic.

[0050] In some embodiments, the intelligent service module determines the computing power scheduling range of the large language model in the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool according to the current load stage, and enables the large language model to process user requests based on the computing power scheduling range, including: when the current load stage is a high load stage, determining whether the local intelligent computing base resource pool meets the deployment of the second inference task module; if so, deploying the second inference task module in the local intelligent computing base resource pool, using the second inference task module as the computing power scheduling range, and inputting the user request into the second inference task module for processing.

[0051] If not, request a deployment node from the cloud dynamic elastic computing power pool; in response to the deployment permission instruction from the cloud dynamic elastic computing power pool, deploy the second inference task module on the node permitted by the cloud dynamic elastic computing power pool.

[0052] After deployment, the second inference task module will be used as the computing power scheduling scope, and user requests will be processed by the second inference task module of the cloud dynamic elastic computing power pool.

[0053] During high-load phases, the intelligent service module prioritizes adding a second inference task module and sets the scheduling scope of the second inference task module to the local intelligent computing base resource pool. If the local intelligent computing base resource pool is insufficient (at which point the resources in the local intelligent computing base resource pool are in a pending state), it requests the deployment of a preset number of nodes from the cloud dynamic elastic computing power pool and deploys the second inference task module according to the number of nodes permitted by the cloud dynamic elastic computing power pool.

[0054] The default number of nodes is 1, and the system can be expanded according to user needs by allowing users to preset different numbers of nodes. This application can dynamically acquire nodes based on the availability of local and cloud resources even under high load conditions, improving system flexibility.

[0055] The first and second inference task modules deployed can be used directly when the large language model obtains the user's inference task. This increases the available resource modules for parsing and inference tasks of the large language model. Flexible scheduling is achieved based on the first inference task module in the local intelligent computing base resource pool and the second inference task module in the cloud dynamic elastic computing power pool, enabling the large language model to run continuously and stably.

[0056] In some embodiments, the method further includes: when the current load phase is a load fallback phase, the intelligent service module releases the nodes in the cloud dynamic elastic computing power pool and reclaims the second inference task module deployed on the nodes.

[0057] During the load rollback phase, the intelligent service module first releases nodes from the cloud dynamic elastic computing power pool and retains nodes from the local intelligent computing base resource pool, minimizing the consumption of resource nodes in the system, quickly saving system resource costs, and avoiding excessive expansion; it also reclaims the second inference task module deployed on the nodes, making the nodes in the cloud dynamic elastic computing power pool reusable, further saving the node consumption of the cloud dynamic elastic computing power pool.

[0058] The data center model inference computing power scheduling method provided in the above embodiments can achieve collaborative management of the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool through the collaborative analysis of intelligent service modules and scaling trigger conditions. Based on the analysis of GPU computing core load rate, the low-cost local intelligent computing base resource pool is prioritized during low load periods, and the system is expanded to the cloud dynamic elastic computing power pool during high load periods to avoid resource idleness and insufficient resources. Dynamic resource adjustment reduces enterprise expenses and ensures the continuity and stability of computing power resource supply for Internet data center enterprises.

[0059] Please see Figure 2 Another embodiment of this application provides a model inference computing power scheduling device for a data center, the device comprising: The intelligent service module 101 is used to obtain the scaling-up trigger conditions based on the inference framework; determine the current load stage based on the GPU computing core load rate and the scaling-up trigger conditions; determine the computing power scheduling range of the large language model in the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool based on the current load stage, and enable the large language model to process user requests based on the computing power scheduling range.

[0060] The monitoring component module 102 is used to calculate GPU running data, obtain the GPU computing core load rate, and send it to the intelligent service module.

[0061] In some embodiments, the scaling up / down triggering conditions include: GPU computing core load rate less than 60%, GPU computing core load rate greater than or equal to 60%, and GPU computing core load rate less than or equal to 30%.

[0062] In some embodiments, the intelligent service module is specifically used to determine that the current load stage is a low load stage when the GPU computing core load rate is less than 60%; to determine that the current load stage is a high load stage when the GPU computing core load rate is greater than or equal to 60%; and to determine that the current load stage is a load decline stage when the GPU computing core load rate is less than or equal to 30%.

[0063] In some embodiments, the intelligent service module is also used to release nodes in the cloud dynamic elastic computing power pool and reclaim the second inference task module deployed on the nodes when the current load phase is a load fallback phase.

[0064] The specific limitations of the model inference computing power scheduling device for a data center provided in this embodiment can be found in the embodiment of the model inference computing power scheduling method for a data center described above, and will not be repeated here. Each module in the above-described model inference computing power scheduling device for a data center can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0065] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0066] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for scheduling computing power for model inference in a data center, characterized in that, The method includes: The intelligent service module obtains the scaling-up / scaling-down trigger conditions based on the inference framework; The monitoring component module calculates GPU operating data, obtains the GPU computing core load rate, and sends it to the intelligent service module. The intelligent service module determines the current load stage based on the GPU computing core load rate and the scaling up / down trigger conditions; The intelligent service module determines the computing power scheduling range of the large language model in the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool based on the current load stage, and then enables the large language model to process user requests based on the computing power scheduling range.

2. The data center model inference computing power scheduling method according to claim 1, characterized in that, The scaling up / down trigger conditions include: GPU computing core load rate less than 60%, GPU computing core load rate greater than or equal to 60%, and GPU computing core load rate less than or equal to 30%.

3. The data center model inference computing power scheduling method according to claim 2, characterized in that, The intelligent service module determines the current load stage based on the GPU computing core load rate and the scaling up / down trigger conditions, including: When the GPU computing core load rate is less than 60%, the current load phase is a low load phase; When the GPU computing core load rate is greater than or equal to 60%, the current load stage is a high load stage; When the GPU computing core load rate is less than or equal to 30%, the current load phase is the load decline phase.

4. The data center model inference computing power scheduling method according to claim 3, characterized in that, The intelligent service module determines the computing power scheduling range of the large language model from the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool based on the current load stage, and instructs the large language model to process user requests based on the computing power scheduling range, including: When the current load phase is a low load phase, the computing power scheduling range is the first inference task module in the local intelligent computing base resource pool, and the user request is input into the first inference task module in the local intelligent computing base resource pool for processing.

5. The data center model inference computing power scheduling method according to claim 4, characterized in that, The intelligent service module determines the computing power scheduling range of the large language model from the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool based on the current load stage, and instructs the large language model to process user requests based on the computing power scheduling range, including: When the current load phase is a high load phase, determine whether the local intelligent computing base resource pool meets the deployment requirements of the second inference task module; if so, deploy the second inference task module in the local intelligent computing base resource pool, and use the second inference task module as the computing power scheduling range, and input user requests into the second inference task module for processing; If not, request a deployment node from the cloud dynamic elastic computing power pool; in response to the deployment permission instruction of the cloud dynamic elastic computing power pool, deploy the second inference task module on the node permitted by the cloud dynamic elastic computing power pool; After deployment, the second inference task module is used as the computing power scheduling range, and user requests input into the cloud dynamic elastic computing power pool are processed by the second inference task module.

6. The data center model inference computing power scheduling method according to claim 5, characterized in that, Also includes: When the current load phase is the load rollback phase, the intelligent service module releases the nodes in the cloud dynamic elastic computing power pool and reclaims the second inference task modules deployed on the nodes.

7. A model inference computing power scheduling device for a data center, characterized in that, The device includes: The intelligent service module is used to obtain the scaling-up trigger conditions based on the inference framework; determine the current load stage based on the GPU computing core load rate and the scaling-up trigger conditions; determine the computing power scheduling range of the large language model in the local intelligent computing base resource pool and the cloud dynamic elastic computing power pool based on the current load stage, and enable the large language model to process user requests based on the computing power scheduling range. The monitoring component module is used to calculate GPU operating data, obtain the GPU computing core load rate, and send it to the intelligent service module.

8. The data center model inference computing power scheduling device according to claim 7, characterized in that, The scaling up / down trigger conditions include: GPU computing core load rate less than 60%, GPU computing core load rate greater than or equal to 60%, and GPU computing core load rate less than or equal to 30%.

9. The data center model inference computing power scheduling device according to claim 8, characterized in that, The intelligent service module is specifically used to determine that the current load stage is a low load stage when the GPU computing core load rate is less than 60%; to determine that the current load stage is a high load stage when the GPU computing core load rate is greater than or equal to 60%; and to determine that the current load stage is a load decline stage when the GPU computing core load rate is less than or equal to 30%.

10. The data center model inference computing power scheduling device according to claim 9, characterized in that, The intelligent service module is also used to release nodes in the cloud dynamic elastic computing power pool and reclaim the second inference task module deployed on the nodes when the current load phase is a load fallback phase.

Images