Inference task optimal computing power node allocation method and system for heterogeneous environment
By constructing a node feature library and task descriptors, and combining reinforcement learning algorithms to optimize task-node allocation, the adaptability and global performance issues of inference task allocation in heterogeneous environments are solved, achieving efficient resource utilization and stable operation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGZHOU TIANCHENG NETWORK TECHNOLOGY CO LTD
- Filing Date
- 2026-04-22
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies for inference task allocation in heterogeneous environments suffer from poor adaptability, single optimization objectives, insufficient global performance, and poor stability, making it difficult to meet the needs of complex scenarios.
By collecting heterogeneous cluster node features in real time, building a node feature library, receiving inference task features, generating standardized descriptors, constructing an evaluation model, using reinforcement learning algorithms to optimize task-node allocation, and introducing a dynamic rescheduling mechanism, we can achieve accurate matching and globally optimal allocation of tasks and nodes.
It achieves precise matching of tasks and nodes, avoids resource waste, improves the overall performance and resource utilization of the cluster, reduces inference latency and operating costs, and ensures stable cluster operation.
Smart Images
Figure CN122431877A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of distributed computing power scheduling technology, and in particular to a method and system for optimal allocation of computing power nodes for inference tasks in heterogeneous environments. Background Technology
[0002] With the rapid development of artificial intelligence technology and the large-scale deployment of AI models such as Large Language Models (LLM), Computer Vision (CV), and multimodal models, inference tasks are characterized by high concurrency, low latency, multi-model hybrid deployment, and significant differences in computing power requirements. To meet the computing power requirements of different inference tasks and reduce deployment costs, existing AI computing power clusters generally adopt heterogeneous architectures, consisting of various types of computing power nodes such as CPUs, GPUs, NPUs, and FPGAs. Different types of nodes have significant differences in computing power performance, GPU / RAM capacity, bandwidth, power consumption, and inference latency. For example, GPU nodes are suitable for high-computation-intensity parallel inference tasks (such as the large model pre-filling stage), NPU nodes are suitable for low-power, high-throughput inference tasks (such as edge CV inference), CPU nodes are suitable for lightweight inference tasks, and FPGA nodes are suitable for customized, low-latency inference tasks.
[0003] The allocation of computing nodes for inference tasks is a core issue in heterogeneous cluster management, as the allocation strategy directly determines the overall performance, resource utilization, inference latency, and operating costs of the cluster. However, existing methods for allocating inference tasks in heterogeneous environments have many shortcomings and are difficult to meet the needs of complex scenarios, as follows: 1. Existing allocation methods mostly rely on manual configuration or simple load balancing algorithms (such as round-robin or random allocation), which are static allocation modes and cannot perceive the characteristics of inference tasks and the state changes of computing nodes in real time, resulting in poor task and node adaptability. For example, allocating large model inference tasks with high computational intensity to CPU nodes will lead to a significant increase in inference latency; allocating lightweight inference tasks to high-performance GPU nodes will result in a waste of computing resources.
[0004] 2. Existing methods often employ single-dimensional optimization objectives, focusing solely on inference latency or node load balancing, while neglecting multi-dimensional factors such as computing power matching, communication overhead, SLA (Service Level Agreement) constraints, and energy efficiency costs. For example, some methods, in an effort to reduce latency, allocate all high-priority tasks to high-performance GPU nodes, leading to GPU node overload and consequently reducing the overall cluster throughput. Other methods, in an effort to balance the load, distribute tasks evenly across nodes, ignoring the compatibility between tasks and node computing power, resulting in low inference efficiency.
[0005] 3. Existing methods fail to differentiate between the characteristics of different types of inference tasks, especially the differences in computational requirements between the pre-filling and decoding stages of large model inference, CV and NLP tasks, and multimodal tasks. The use of a uniform allocation strategy leads to low allocation accuracy. For example, the pre-filling stage of large model inference is computationally intensive and involves large amounts of data, requiring high-performance GPU nodes, while the decoding stage is computationally intensive and has high concurrency, allowing allocation to NPU or CPU nodes. Existing methods cannot achieve this phased allocation, resulting in low resource utilization.
[0006] 4. Existing methods often employ local optimization strategies, considering only the performance optimization of a single task or node, without allocating resources from a global cluster perspective. This leads to low overall cluster throughput and resource utilization, and easily results in resource bottlenecks and waste. For example, when a node has a low load, allocating a large number of tasks to that node can cause it to become overloaded later, while other nodes still have idle resources, creating an unbalanced resource allocation situation.
[0007] Therefore, there is an urgent need for an optimal computing power node allocation method for inference tasks in heterogeneous environments that can achieve precise task and node matching, global optimal allocation, and dynamic self-adaptation. This method would solve the problems of poor adaptability, single optimization objective, insufficient global performance, and poor stability of existing allocation methods, improve the overall performance and resource utilization of heterogeneous clusters, and reduce inference latency and operating costs. Summary of the Invention
[0008] To address the shortcomings of existing technologies, this invention provides a method and system for optimal allocation of computing power nodes for inference tasks in heterogeneous environments, thereby solving the technical problems of inference task allocation in heterogeneous environments.
[0009] In a first aspect, the present invention provides a method for optimal allocation of computing power nodes for inference tasks in heterogeneous environments, comprising the following steps: S1. Real-time collection of multi-dimensional features of each computing node in the heterogeneous cluster to build a node feature library; S2. Receive the inference task input from the outside, perform multi-dimensional analysis on the inference task, extract task features, and generate a standardized task descriptor based on the task features. S3. Based on the node feature library and task descriptor, construct an evaluation model; The evaluation model is used to calculate the comprehensive fit score of each inference task on each candidate computing power node, and the target computing power node is selected based on the comprehensive fit score. S4. Construct a multi-objective optimization function with the objectives of minimizing the average inference latency of the cluster, maximizing the overall throughput of the cluster, satisfying the SLA latency constraints of all inference tasks, and balancing the load of each target computing node. S5. Solve the multi-objective optimization function using a reinforcement learning algorithm to obtain the globally optimal task-node allocation matrix; S6. The globally optimal task-node allocation matrix is distributed to the node agents of each target computing power node, and the node agents start the inference task execution. S7. Monitor the running status of each inference task and the load status of each target computing node in real time. When the preset rescheduling trigger conditions are met, trigger the dynamic rescheduling process.
[0010] Secondly, the present invention provides an optimal computing power node allocation system for inference tasks in heterogeneous environments, comprising: The node awareness module is used to collect multi-dimensional features of each computing node in a heterogeneous cluster in real time and build a node feature library. The task parsing module is used to receive inference tasks from external input, perform multi-dimensional parsing of the inference tasks, extract task features, and generate standardized task descriptors based on the task features. The adaptation evaluation module calls the evaluation model built based on the node feature library and standardized task descriptors to obtain the comprehensive adaptation score of each inference task on each candidate computing power node, and selects the target computing power node based on the comprehensive adaptation score. The multi-objective optimization function construction module is used to construct multi-objective optimization functions with the objectives of minimizing the average inference latency of the cluster, maximizing the overall throughput of the cluster, satisfying the SLA latency constraints of all inference tasks, and balancing the load of each target computing node. The optimal allocation module uses a reinforcement learning algorithm to solve the multi-objective optimization function to obtain the globally optimal task-node allocation matrix; The scheduling and execution module is used to distribute the globally optimal task-node allocation matrix to the node agents of each target computing power node, and the node agents start the inference task. The rescheduling module is used to monitor the running status of each inference task and the load status of each target computing node in real time. When the preset rescheduling trigger conditions are met, the dynamic rescheduling process is triggered.
[0011] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements all the steps of the method for optimal computing power node allocation for inference tasks in heterogeneous environments.
[0012] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements all the steps of the method for optimal computing power node allocation for inference tasks in heterogeneous environments.
[0013] The beneficial effects of this invention are as follows: 1. This invention comprehensively captures differences in node performance and task requirements through multi-dimensional feature acquisition and analysis, constructs a multi-dimensional adaptability evaluation model, achieves accurate matching between tasks and nodes, fully leverages the hardware advantages of different types of heterogeneous nodes, avoids waste of computing resources, and adapts to different types of inference task scenarios. 2. This invention uses multi-objective optimization as the optimization function, taking into account multiple objectives such as inference latency, cluster throughput, resource utilization, SLA constraints, and energy efficiency cost. It employs an improved reinforcement learning algorithm to solve for the globally optimal allocation matrix, avoiding local optima and improving the overall performance of the cluster. 3. This invention introduces a dynamic rescheduling mechanism to monitor the cluster status and task running status in real time, respond promptly to abnormal situations such as node failures, task fluctuations, and resource changes, trigger task migration and strategy optimization, ensure stable cluster operation, and improve system reliability. Attached Figure Description
[0014] Figure 1 This is a flowchart illustrating the method of an embodiment of the present invention; Figure 2 This is a flowchart illustrating the process of solving a multi-objective optimization function using a reinforcement learning algorithm in an embodiment of the present invention. Figure 3 This is a schematic diagram of the system framework of Embodiment 2 of the present invention; In the diagram, 100 is the node perception module; 200 is the task parsing module; 300 is the adaptation evaluation module; 400 is the multi-objective optimization function construction module; 500 is the optimal allocation module; 600 is the scheduling execution module; and 700 is the rescheduling module. Detailed Implementation
[0015] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings: Example 1 like Figure 1 As shown, this embodiment provides a method for optimal computing power node allocation for inference tasks in heterogeneous environments, including the following steps: S1. Real-time collection of multi-dimensional features of each computing node in the heterogeneous cluster to build a node feature library; In this embodiment, the heterogeneous cluster consists of at least two types of computing power nodes selected from CPU, GPU, NPU, and FPGA. For example, the heterogeneous cluster set up in this embodiment consists of 8 nodes, including 2 GPU nodes, 3 NPU nodes, 2 CPU nodes, and 1 FPGA node, used to process LLaMA-7B large model inference tasks and YOLOv8 CV detection tasks. The multi-dimensional features include hardware features, runtime status features, and performance benchmark features. Specifically, it includes the following steps: S11. Hardware features are obtained by combining offline collection and online verification. The hardware features are the inherent attributes of the nodes. The offline collection is performed when the computing node joins the cluster to initialize the computing node feature library. The online verification is performed once every hour to correct hardware feature deviations. The hardware features include node type identifier, number of cores, peak computing power, video memory / memory capacity and bandwidth, power consumption limit, network interface bandwidth, hardware architecture version and supported inference acceleration framework. S12. The aforementioned running status features are dynamic attributes of the node, reflecting the current load and running status of the node. The sampling frequency is 1 time / second. After sampling, outliers are removed by sliding window filtering to avoid abnormal data affecting the allocation strategy. The running status features include current CPU utilization, GPU utilization, available video memory capacity, available memory capacity (GB), task queue status, running status, and network status. S13. The performance benchmark features are used to quantify the processing capabilities of nodes for different types of inference tasks. They are obtained through offline pre-training and online updating. The offline pre-training is performed when a node joins the cluster, and the online update cycle is 24 hours. Each update uses random sampling to run the standard model and update the benchmark data to ensure the accuracy of the benchmark features.
[0016] The preset standard models cover mainstream inference task types, including ResNet50 (CV classification), YOLOv8 (CV detection), BERT-base (NLP classification), LLaMA-7B (large model generation), and Stable Diffusion (multimodal generation). The performance benchmark features include: Inference latency includes single-frame inference latency (CV task, single-token inference latency); Inference throughput: the number of tasks that can be processed per unit of time; Resource usage: maximum GPU memory usage and maximum system memory usage during inference. S14. The collected hardware features, operating status features, and performance benchmark features are standardized and processed to construct a node feature library. The node feature library is updated in real time, and the update frequency is consistent with the operating status feature collection frequency.
[0017] S2. Receive the reasoning task from external input, perform multi-dimensional analysis on the reasoning task, extract task features, and generate a standardized task descriptor based on the task features. The task descriptor is used to uniquely identify the feature attributes of a reasoning task; specifically, it includes the following steps: S21. Receive inference tasks submitted by external systems; The aforementioned inference task submission methods support multiple methods such as API interface submission and batch import, and each inference task carries a unique task ID.
[0018] S22. Perform multi-dimensional analysis on each reasoning task and extract task features; In this embodiment, the task features include: Task types include CV, NLP, and multimodal tasks (image and text generation, speech to text, and large model inference). The computational and resource requirements of different task types vary significantly. Model size is characterized by the number of model parameters, such as LLaMA-7B and BERT-base. The larger the model size, the higher the demand for computing power and GPU memory. Computational intensity is represented by the number of floating-point operations per frame / token (FLOPs), reflecting the computational complexity of the task. The higher the computational intensity, the higher the requirement for node computing power. Memory requirements include the sum of the memory or video memory used by the model during inference and the memory or video memory used by the input and output data. Batch size, the number of samples processed in each inference iteration; SLA delay constraint: the maximum allowed time from task submission to completion; Task priorities are divided into three levels: high, medium, and low. In this embodiment, the priority weights are set to 1.2, 1.0, and 0.8, respectively. High-priority tasks are given priority to meet SLA constraints and are assigned a higher fit weight during allocation. Input and output data volume: the total capacity of task input and output data.
[0019] S23. Standardize the parsed task features to generate standardized task descriptors. Each task descriptor corresponds one-to-one with the inference task ID and contains all task feature information.
[0020] S3. Construct an evaluation model based on the node feature library and standardized task descriptors. Obtain the comprehensive fit score of each inference task on the candidate computing power nodes through the evaluation model, and select target computing power nodes based on the comprehensive fit score; specifically including the following steps: S31. Based on the memory requirements and computational intensity of each inference task, select candidate computing power nodes that meet the basic constraints. S32. Construct an evaluation model that includes computing power matching degree, memory matching degree, bandwidth matching degree, historical performance fit degree, load saturation degree, communication cost, and energy efficiency cost; obtain the comprehensive fit score of each inference task on each candidate computing power node through the evaluation model. ,Right now: In the formula, Representational reasoning task In candidate computing power nodes The overall fit score; , , , Reasoning tasks In candidate computing power nodes The matching degree of computing power, memory, bandwidth, historical performance fit, and communication cost; , Candidate computing power nodes Load saturation and energy efficiency costs; These are the corresponding weighting coefficients; In this embodiment, the inference task In candidate computing power nodes Computing power matching degree Represented as: In the formula, For the first The computational intensity of each inference task; For the first Available computing power of each candidate computing node; When the task computation intensity is less than or equal to the node's available computing power This indicates a perfect match in computing power; when the task's computational intensity exceeds the node's available computing power... This indicates insufficient computing power and a reduced matching degree.
[0021] In this embodiment, the memory matching degree is... Represented as: In the formula, For the first Memory requirements for each inference task; For the first Available memory or video memory for each candidate computing node; For the first The total memory or video memory of each candidate computing node; When the task's memory requirements perfectly match the node's available memory. .
[0022] In this embodiment, the bandwidth matching degree The degree of matching between the task's data transmission requirements and the node's network bandwidth is represented by the following formula: In the formula, Indicates the first The total amount of input and output data for each inference task; Indicates the first Network bandwidth of each candidate computing node; Indicates the first Estimated reasoning time for each reasoning task.
[0023] In this embodiment, the historical performance fit degree The formula for representing the performance of similar tasks at this node in the past is: In the formula, For the first time in history Class reasoning task in the first Success rate on candidate computing nodes; For the first time in history Class reasoning task in the first Average inference latency on candidate computing nodes.
[0024] In this embodiment, the candidate computing nodes load saturation This represents the current load status of a node; its calculation formula is: In the formula, Candidate computing power nodes Number of tasks currently running; Candidate computing power nodes The maximum number of concurrent tasks.
[0025] In this embodiment, the communication cost is... The time cost required for data transmission in a task is represented by the following formula: In the formula, Indicates the first Network bandwidth of each candidate computing node; Indicates the first The total amount of input and output data for each inference task.
[0026] In this embodiment, the candidate computing nodes Energy efficiency cost The energy efficiency of a node is characterized by the following formula: In the formula, For the first Real-time power consumption of each candidate computing node; For the first Available computing power of each candidate computing node.
[0027] S33. Sort the comprehensive adaptation scores of each inference task on each candidate computing power node in descending order, and take the top K candidate computing power nodes as the target computing power nodes. This embodiment is based on the comprehensive adaptation score. Judgment and reasoning task With candidate computing power nodes The higher the score, the better the matching degree of the reasoning task. With candidate computing power nodes The better the match, the more suitable it is for reasoning tasks. Assigned to candidate computing nodes .
[0028] S4. Construct a multi-objective optimization function with the objectives of minimizing the average inference latency of the cluster, maximizing the overall throughput of the cluster, satisfying the SLA latency constraints of all inference tasks, and balancing the load of each target computing node. The multi-objective optimization function Represented as: ; In the formula, Indicates average inference delay; For the overall throughput of the cluster; The target computing power node load imbalance; SLA violation rate; , , , These are the target weight coefficients.
[0029] The average inference delay The average inference latency for all inference tasks in the cluster is: ; In the formula, The total number of reasoning tasks; For reasoning tasks At the target computing power node Inference delay.
[0030] The overall throughput of the cluster This represents the number of inference tasks that the cluster can process per unit of time, i.e.: The aforementioned node load imbalance Characterizes the degree of load balance among nodes. In the formula, For target computing power nodes Load saturation; Average load saturation; The SLA violation rate The proportion of tasks exceeding the SLA latency constraint, i.e.: In the formula, The number of tasks exceeding the SLA latency constraint; This represents the total number of tasks.
[0031] In this embodiment, the constraints of the multi-objective optimization function are: Memory constraints: For each target computing power node All allocated to each target computing node The total memory requirements of the inference tasks shall not exceed the available memory or video memory capacity of the node; that is: , To assign elements to the matrix, When, it indicates a reasoning task. Assigned to target computing nodes ; When, it indicates a reasoning task. Not allocated to the target computing power node ; Concurrency constraints: For each target computing power node All allocated to each target computing node The number of tasks cannot exceed the node's maximum concurrent task count, that is: ; SLA constraint: For each inference task Distribute to target computing nodes The subsequent inference delay does not exceed the task's SLA delay constraint. ,Right now: ; Power consumption constraint: for each target computing power node Real-time power consumption does not exceed the node power consumption limit. ,Right now: ; Assignment constraints: for each inference task Only one target computing power node is allocated. ,Right now: N is the total number of computing nodes in the cluster.
[0032] S5. Solve the multi-objective optimization function using a reinforcement learning algorithm to obtain the globally optimal task-node allocation matrix; In this embodiment, a sparsity constraint of the allocation matrix is added to the loss function of the reinforcement learning algorithm Actor network to avoid overload caused by assigning multiple tasks to the same node. The sparsity constraint term is: , The constraint coefficients are used to guide the allocation matrix to become sparse through sparsity constraint terms, achieving a balanced task allocation; task priority weights are introduced into the loss function of the Critic network. The delay error of high-priority tasks is given a relatively high weight.
[0033] Solving multi-objective optimization functions, such as Figure 2 As shown, the specific steps are as follows: S51. Initialize the parameters of the reinforcement learning algorithm and define the environment state, action space and reward function of reinforcement learning; Initialize learning rate Convergence threshold Number of Worker processes K, Experience replay pool capacity C, Sparsity constraint coefficient λ, Discount factor Initial parameters θ of Actor network and Critic network ; The environmental state described is the current comprehensive state of the cluster, including node characteristics, task characteristics, and the current allocation state, namely: , The node state matrix; This is the task state matrix; Assign the current matrix, elements ; The actions described Assigning decisions to nodes of a single task, i.e., for the first node... One unassigned task, select a target computing node. The action space can be represented as follows: The reward function is constructed based on a multi-objective optimization function and is used to evaluate the quality of actions, guiding the algorithm to converge toward the global optimum. .
[0034] S52, Set the cluster environment status The system is divided into multiple sub-states, each corresponding to an independent Worker process. Each Worker process collects node state data in parallel within its corresponding sub-state. Based on the current environment state s and the action probabilities output by the Actor network, an action is selected. Perform the task and receive a reward. and the next state and empirical samples Stored in the experience replay pool; details are as follows: Each worker process is based on its corresponding sub-state Calculate the action probability distribution using an Actor network. : In the formula, These are the parameters of the Actor network; Indicates the transpose operation; This is a state-action feature mapping function used to map substates... With action Mapped to high-dimensional feature vectors; For the first k Sub-states corresponding to each Worker process; This refers to the action space corresponding to the current Worker process; Based on the probability distribution of actions, a roulette wheel method is used to select actions. Execute, and update the current allocation matrix after execution. To obtain the next substate ; The reward for the current action is calculated based on the real-time calculation results of the multi-objective optimization function. ; .
[0035] empirical samples Stored in the experience replay pool; and assigned priority weights to each sample. ;Right now: ; In the formula, For the Critic network in state The following is a value estimate; It is a very small number; S53. The main process extracts B samples from the experience replay pool using a priority sampling strategy. Each worker process calculates the policy gradient of the Actor network and the value gradient of the Critic network based on the sampled samples, as follows: The Actor network is used to output action probabilities, and its loss function is... This includes policy loss and sparsity constraints, namely: The policy gradient of the Actor network is expressed as: ; In the formula, Represents the parameters of the Actor network Find the gradient; This is represented as the policy gradient of the Actor network, used to update the Actor network parameters and guide the optimization direction of the task-node allocation policy; These are the parameters of the Actor network; The gradient of the logarithm of the policy probability reflects the parameters of the Actor network. The degree of influence on the probability of choosing an action in the allocation process; In the state Select the assigned action The probability of; The dominant function; The number of samples collected; Importance weight; For the first The state corresponding to each sample; For the first The action corresponding to each sample.
[0036] The Critic network described above is used to evaluate state value, and its loss function is... Based on temporal difference error construction, task priority weights are introduced to optimize the accuracy of value assessment, i.e. ; In the formula, The objective function is... For the target Critic network in state The following is a value estimate; For discount reasons; For the first The priority weight of each sample corresponds to the task. For the Critic network in state The following is a value estimate; The value gradient of the Critic network is expressed as: ; In the formula, The value gradient of the Critic network is used to optimize the accuracy of value assessment. These are all the trainable parameters of the Critic network; Represents the trainable parameters of the Critic network Find the gradient.
[0037] The calculation process employs gradient clipping to restrict the gradient norm to the [0,1] interval, thus avoiding gradient explosion or vanishing problems; that is: ; in, Indicates the gradient after pruning; This is the clipping threshold; Represents the gradient norm; This represents the gradient of pruning, which is the policy gradient of the Actor network or the value gradient of the Critic network.
[0038] S54. After receiving the policy gradient and value gradient data from each Worker process, the main process merges the policy gradient and value gradient data using a weighted average method to obtain the merged global gradient, i.e.: ; In the formula, For the first Gradient weights of each Worker process; For the first The policy gradient or value gradient data computed by each Worker process; The fused global gradient; This represents the total number of Worker processes. Based on the fused global gradient, the parameters of the global Actor and Critic networks are updated uniformly using stochastic gradient descent, achieving asynchronous parameter updates and global synchronization. ; ; In the formula, The learning rate; , These are the global gradients of the fused Actor and Critic networks, respectively. , These are the updated Actor network parameters and all trainable parameters of the Critic network, respectively. , These represent the parameters of the Actor network before the update and all trainable parameters of the Critic network, respectively.
[0039] S55. When the gradient change is consistently less than the convergence threshold, convergence is determined, parameter updates are stopped, and the globally optimal task-node assignment matrix is output; if convergence is not achieved, return to S52.
[0040] S6. Distribute the globally optimal task-node allocation matrix to the node agents of each target computing power node, and the node agents start the inference task. In this embodiment, the allocation matrix data is encrypted using the AES-256 encryption algorithm during the distribution process. The node agents deployed on each target computing power node monitor the distribution end in real time. After receiving the allocation instruction, they first decrypt and verify the instruction. The verification includes the integrity and legality of the allocation matrix. After the verification is passed, the node agent parses the allocation instruction, extracts the task list corresponding to the node, and starts the inference task execution in descending order of task priority. During the execution, the node's local inference acceleration framework is called to load the model file and input data corresponding to the task and complete the inference calculation.
[0041] S7. Monitor the running status of each inference task and the load status of each target computing node in real time. When the preset rescheduling trigger conditions are met, trigger the dynamic rescheduling process.
[0042] In this embodiment, the dynamic rescheduling triggering condition includes at least one of the following: Node load trigger: The load saturation of a certain node. And the duration exceeds 30 seconds. In this embodiment, the load saturation threshold is... Set to 0.85; Delay triggering: The real-time inference latency of a task exceeds 1.5 times its SLA latency constraint and lasts for more than 10 seconds; Node failure trigger: A node experiences a hardware failure, network interruption, or software crash, causing the task to fail to execute normally; Resource change trigger: New computing nodes join the cluster or existing nodes leave the cluster, causing a change in the total cluster resources; Task fluctuation trigger: A batch of high-priority tasks exceeding 50% of the current cluster's maximum throughput arrives within a short period of time; Energy efficiency trigger: The energy efficiency cost of a certain node. And the duration exceeded 1 minute. This represents the energy efficiency cost threshold.
[0043] In this embodiment, the dynamic rescheduling process is as follows: S71. Mark the abnormal tasks and abnormal nodes that trigger rescheduling, and record the abnormal information; S72. Temporarily migrate tasks on abnormal nodes to the top 3 candidate computing power nodes with the best adaptability scores. Use a breakpoint resume mechanism to save the current execution progress and intermediate data of the tasks. During the temporary migration, prioritize the migration of high-priority tasks and start execution immediately after the migration is completed. Low-priority tasks are migrated gradually. S73. Re-execute steps S3~S5, and re-select candidate nodes, calculate fitness scores, and solve for a new optimal allocation matrix based on the current cluster status and task status. S74. The new allocation matrix is gradually distributed to each node, and a batch switching strategy is adopted. Each switch does not exceed 20% of the total number of tasks. The task execution status is monitored in real time during the migration process.
[0044] Example 2 like Figure 3 As shown, this embodiment provides an optimal computing power node allocation system for inference tasks in heterogeneous environments, including: The node awareness module 100 is used to collect multi-dimensional features of each computing node in the heterogeneous cluster in real time and build a node feature library. The task parsing module 200 is used to receive external input reasoning tasks, perform multi-dimensional parsing of the reasoning tasks, extract task features, and generate standardized task descriptors based on the task features. The adaptation evaluation module 300 calls the evaluation model built based on the node feature library and standardized task descriptors to calculate the comprehensive adaptation score of each inference task on each candidate computing power node, and selects the target computing power node based on the comprehensive adaptation score. The multi-objective optimization function construction module 400 is used to construct a multi-objective optimization function with the objectives of minimizing the average inference latency of the cluster, maximizing the overall throughput of the cluster, satisfying the SLA latency constraints of all inference tasks, and balancing the load of each target computing node. The optimal allocation module 500 uses a reinforcement learning algorithm to solve the multi-objective optimization function to obtain the globally optimal task-node allocation matrix; The scheduling and execution module 600 is used to distribute the globally optimal task-node allocation matrix to the node agents of each target computing power node, and the node agents start the inference task. The rescheduling module 700 is used to monitor the running status of each inference task and the load status of each target computing node in real time. When the preset rescheduling trigger conditions are met, the dynamic rescheduling process is triggered, the allocation matrix is updated and distributed.
[0045] Example 3 This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements all the steps of the optimal computing power node allocation method for inference tasks in heterogeneous environments as described in Embodiment 1.
[0046] In this embodiment, the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. A processor, coupled to the memory, is used to execute computer programs stored in the memory.
[0047] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form.
[0048] Example 4 This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements all the steps of the optimal computing power node allocation method for inference tasks in heterogeneous environments as described in Embodiment 1.
[0049] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.
Claims
1. A method for optimal computing power node allocation for inference tasks in heterogeneous environments, characterized in that, Includes the following steps: S1. Real-time collection of multi-dimensional features of each computing node in the heterogeneous cluster to build a node feature library; S2. Receive the inference task input from the outside, perform multi-dimensional analysis on the inference task, extract task features, and generate a standardized task descriptor based on the task features. S3. Construct an evaluation model based on the node feature library and standardized task descriptors; The evaluation model is used to calculate the comprehensive fit score of each inference task on each candidate computing power node, and the target computing power node is selected based on the comprehensive fit score. S4. Construct a multi-objective optimization function with the objectives of minimizing the average inference latency of the cluster, maximizing the overall throughput of the cluster, satisfying the SLA latency constraints of all inference tasks, and balancing the load of each target computing node. S5. Solve the multi-objective optimization function using a reinforcement learning algorithm to obtain the globally optimal task-node allocation matrix; S6. The globally optimal task-node allocation matrix is sent to the node agents of each target computing power node, and the node agents start the inference task. S7. Monitor the running status of each inference task and the load status of each target computing node in real time. When the preset rescheduling trigger conditions are met, trigger the dynamic rescheduling process.
2. The optimal computing power node allocation method for inference tasks in heterogeneous environments according to claim 1, characterized in that: In S2, the task features include: Task types include CV, NLP, multimodal, and large model inference. Model size is represented by the number of model parameters; The strength is calculated by the number of floating-point operations per frame or per token; Memory requirements include the sum of the memory or video memory used by the model during inference and the memory or video memory used by the input and output data. Batch size, the number of samples processed in each inference iteration; SLA delay constraint: the maximum allowed time from task submission to completion; Task priorities are divided into three levels: high, medium, and low. Input and output data volume: the total capacity of task input and output data.
3. The optimal computing power node allocation method for inference tasks in heterogeneous environments according to claim 1, characterized in that: S3 includes: S31. Based on the memory requirements and computational intensity of each inference task, select candidate computing power nodes that meet the basic constraints. S32. Construct an evaluation model that includes computing power matching degree, memory matching degree, bandwidth matching degree, historical performance fitting degree, load saturation, communication cost, and energy efficiency cost; The comprehensive fit score of each inference task on each candidate computing power node is obtained based on the evaluation model. S33. Sort the comprehensive adaptation scores of each inference task on each candidate computing power node in descending order, and take the top K candidate computing power nodes as the target computing power nodes.
4. The optimal computing power node allocation method for inference tasks in heterogeneous environments according to claim 1, characterized in that: In step S4, the constraints of the multi-objective optimization function are: Memory constraint: For each candidate computing power node, the sum of the memory requirements of all inference tasks allocated to each target computing power node shall not exceed the available memory or video memory capacity of the node. Concurrency constraint: For each target computing power node, the total number of inference tasks allocated to each target computing power node shall not exceed the maximum number of concurrent tasks of the node; SLA constraint: For each inference task, the inference delay after being allocated to the target computing node shall not exceed the SLA delay constraint of the task. Power consumption constraint: For each target computing power node, the real-time power consumption shall not exceed the node's power consumption limit; Allocation constraint: Each inference task is assigned to only one target computing power node.
5. The optimal computing power node allocation method for inference tasks in heterogeneous environments according to claim 4, characterized in that: In step S5, a sparsity constraint on the allocation matrix is added to the loss function of the Actor network in the reinforcement learning algorithm. The sparsity constraint term guides the allocation matrix to become sparse, thereby achieving a balanced distribution of tasks. By introducing task priority weights into the loss function of the Critic network in the reinforcement learning algorithm, relatively high weights are assigned to the delay errors of high-priority tasks.
6. The optimal computing power node allocation method for inference tasks in heterogeneous environments according to claim 5, characterized in that: In step S5, the multi-objective optimization function is solved, including: S51. Initialize the parameters of the reinforcement learning algorithm and define the environment state, action space and reward function of reinforcement learning; S52, Set the cluster environment status The system is divided into multiple sub-states, each corresponding to an independent Worker process. Each Worker process collects node state data in parallel within its corresponding sub-state. Based on the current environment state s and the action probabilities output by the Actor network, an action is selected. Perform the task and receive a reward. and the next state and empirical samples Stored in the experience replay pool; S53. The main process extracts B samples from the experience replay pool using a priority sampling strategy. Each worker process calculates the policy gradient of the Actor network and the value gradient of the Critic network based on the sampled samples and the loss function. S54. After receiving the policy gradient and value gradient data from each Worker process, the main process uses a weighted average to fuse the policy gradient and value gradient data to obtain the fused global gradient. Then, based on the fused global gradient, the parameters of the global Actor and Critic networks are updated uniformly using the stochastic gradient descent method. S55. When the gradient change is continuously less than the convergence threshold, convergence is determined, parameter updates are stopped, and the globally optimal task-node allocation matrix is output; if convergence is not achieved, return to step S52.
7. The optimal computing power node allocation method for inference tasks in heterogeneous environments according to claim 6, characterized in that: In step S7, the dynamic rescheduling triggering condition includes at least one of the following: Node load trigger: The load saturation of a node exceeds the load saturation threshold and lasts for more than 30 seconds; Delay triggering: The real-time inference latency of a task exceeds 1.5 times its SLA latency constraint and lasts for more than 10 seconds; Node failure trigger: A node experiences a hardware failure, network interruption, or software crash, causing the task to fail to execute normally; Resource change trigger: New computing nodes join the cluster or existing nodes leave the cluster, causing a change in the total cluster resources; Task fluctuation trigger: A batch of high-priority tasks exceeding 50% of the current cluster's maximum throughput arrives within a short period of time; Energy efficiency trigger: The energy efficiency cost of a certain node exceeds the energy efficiency cost threshold and the duration exceeds 1 minute; The dynamic rescheduling process is as follows: S71. Mark the abnormal tasks and abnormal nodes that trigger rescheduling, and record the abnormal information; S72. Temporarily migrate tasks on abnormal nodes to the top 3 candidate computing power nodes with the best adaptability scores. Use a breakpoint resume mechanism to save the current execution progress and intermediate data of the tasks. During the temporary migration, prioritize the migration of high-priority tasks and start execution immediately after the migration is completed. Low-priority tasks are migrated gradually. S73. Re-execute S3~S5 to solve for the new optimal task-node allocation matrix; S74. The new optimal task-node allocation matrix is gradually distributed to each target computing power node.
8. An optimal computing power node allocation system for inference tasks in heterogeneous environments, characterized in that, include: The node awareness module is used to collect multi-dimensional features of each computing node in a heterogeneous cluster in real time and build a node feature library. The task parsing module is used to receive inference tasks from external input, perform multi-dimensional parsing of the inference tasks, extract task features, and generate standardized task descriptors based on the task features. The adaptation evaluation module calls the evaluation model built based on the node feature library and standardized task descriptors to obtain the comprehensive adaptation score of each inference task on each candidate computing power node, and selects the target computing power node based on the comprehensive adaptation score. The multi-objective optimization function construction module is used to construct multi-objective optimization functions with the objectives of minimizing the average inference latency of the cluster, maximizing the overall throughput of the cluster, satisfying the SLA latency constraints of all inference tasks, and balancing the load of each target computing node. The optimal allocation module uses a reinforcement learning algorithm to solve the multi-objective optimization function to obtain the globally optimal task-node allocation matrix; The scheduling and execution module is used to distribute the globally optimal task-node allocation matrix to the node agents of each target computing power node, and the node agents start the inference task. The rescheduling module is used to monitor the running status of each inference task and the load status of each target computing node in real time. When the preset rescheduling trigger conditions are met, the dynamic rescheduling process is triggered.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements all the steps of the optimal computing power node allocation method for inference tasks in heterogeneous environments as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements all the steps of the optimal computing power node allocation method for inference tasks in heterogeneous environments as described in any one of claims 1-7.