An algorithm scheduling method and system of an ARM cluster server

Abstract

The application relates to the technical field of computers and discloses a computing power scheduling method and system for an ARM cluster server, which comprises the following steps: S1, obtaining task information of a to-be-scheduled inference task and node state information of each computing node; S2, determining model residence states on each computing node and model loading costs; S3, predicting effective computing power of each computing node within a preset time window; S4, executing results are predicted based on the model loading costs and the effective computing power, and uncertainty information is obtained; S5, a confidence degree meeting a performance constraint is determined according to the execution results and the uncertainty information; S6, a target computing node is determined based on the confidence degree and a scheduling score; and S7, the task is scheduled to the target computing node for execution. Through the uncertainty information of the predicted value, the confidence degree of the to-be-scheduled inference task meeting the preset performance constraint is constructed, and the target computing node is determined based on the confidence degree and the scheduling score, so that the computing power scheduling process considers the predicted results and fluctuation conditions simultaneously.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a computing power scheduling method and system for an ARM cluster server. Background Technology

[0002] With the continuous development of artificial intelligence inference tasks, ARM cluster servers are widely used in edge computing and distributed inference scenarios due to their low power consumption and high energy efficiency. ARM cluster servers are usually composed of multiple computing nodes based on the ARM architecture, which are interconnected through a network to form a unified computing resource pool to support the parallel processing of large-scale inference tasks. On this basis, computing power scheduling methods are used to allocate the inference tasks to be scheduled among multiple computing nodes to improve the overall resource utilization efficiency and task processing performance. At present, the computing power scheduling of ARM cluster servers is usually based on the current resource occupancy or historical load information of each computing node to allocate the inference tasks to be scheduled, for example, selecting computing nodes with lower processor utilization or more available resources to execute tasks.

[0003] However, in current technology, the allocation of inference tasks to be scheduled is based solely on the resource status of each computing node, without quantitative analysis of the uncertainty of task execution results. This makes it difficult to accurately determine the actual possibility of the inference tasks to be scheduled meeting performance constraints when node load fluctuates or operating status changes. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a computing power scheduling method and system for ARM cluster servers, which solves the problem that existing scheduling methods do not quantitatively analyze the uncertainty of task execution results, making it difficult to accurately determine the actual possibility of tasks meeting performance constraints.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a computing power scheduling method for an ARM cluster server, comprising: S1. Obtain the task information of the inference task to be scheduled, as well as the node status information of each computing node in the ARM cluster; S2. Based on the task information and the node status information of each computing node, determine the model dwell status of the inference task to be scheduled on each computing node, and determine the corresponding model loading cost. S3. Based on the node status information of each computing node, predict the effective computing power of each computing node within a preset future time window; S4. Based on the model loading cost and the effective computing power, predict the execution result of the scheduled inference task on each computing node, and obtain the uncertainty information of the corresponding prediction result; S5. Based on the execution results and their uncertainty information, determine the confidence level of the inference task to be scheduled on each computing node to satisfy the preset performance constraints; S6. Based on the confidence level and scheduling score corresponding to each computing node, determine the target computing node, wherein the scheduling score is at least related to the execution result, model loading cost and uncertainty information; S7. Schedule the inference task to be scheduled to the target computing node for execution.

[0006] Preferably, step S1 specifically includes: Obtain task information for the inference task to be scheduled, including task identifier, model identifier, task priority, and preset performance constraints; Obtain node status information of each computing node in the ARM cluster, including processor utilization, frequency status, temperature status, memory resource status, and model residency information; Based on the task information and the node status information of each computing node, a set of candidate computing nodes corresponding to the inference task to be scheduled is established.

[0007] Preferably, step S2 specifically includes: Based on the model identifier in the task information, query the model residency information corresponding to each computing node to determine whether each computing node has resided the target model corresponding to the inference task to be scheduled. If it is determined that the target model has been resident in each computing node, the target model is further determined to be in a reusable state in the corresponding computing node based on the version information and resource usage status of the target model, so as to obtain the model resident status corresponding to each computing node. For computing nodes whose model residency status indicates that the target model is not resided or is not in a reusable state, the model loading cost corresponding to each computing node is determined based on the amount of data loaded by the target model and the data input / output bandwidth of the corresponding computing node.

[0008] Preferably, step S3 specifically includes: Based on the node status information of each computing node, node features that characterize the running status of the computing node are extracted. The node features include processor utilization, frequency status, temperature status, memory bandwidth usage status, and NUMA access status. The node features are input into a preset effective computing power prediction model to predict the available computing power of each computing node within a preset future time window, so as to obtain the effective computing power prediction value corresponding to each computing node. The predicted effective computing power value is used as the schedulable computing power index for each computing node within the preset future time window.

[0009] Preferably, step S4 specifically includes: Based on the model loading cost and effective computing power prediction value corresponding to each computing node, the execution result prediction value of the inference task to be scheduled on each computing node is calculated respectively, and the execution result prediction value includes the predicted execution time. The predicted execution result is input into the uncertainty assessment model to estimate the uncertainty of the predicted execution result, so as to obtain the uncertainty information corresponding to each computing node. The uncertainty assessment model is any one or more of heteroscedasticity regression model, quantile regression model, or MCDropout-based neural network model.

[0010] Preferably, step S5 specifically includes: Based on the predicted execution results and uncertainty information of each computing node, the distribution information of the execution results of the inference task to be scheduled on each computing node is constructed. The distribution information of execution results corresponding to each computing node is compared with the preset performance constraints to determine the probability value of the scheduled inference task on each computing node satisfying the preset performance constraints. The probability value corresponding to each computing node is determined as the confidence level corresponding to each computing node.

[0011] Preferably, step S6 specifically includes: The confidence level of each computing node is compared with a preset confidence threshold, and computing nodes with a confidence level greater than or equal to the preset confidence threshold are determined as candidate scheduling nodes; For each candidate scheduling node, a corresponding scheduling score is calculated based on the predicted execution result, model loading cost, and uncertainty information. The candidate scheduling node with the best scheduling score is selected as the target computing node.

[0012] Preferably, the preset confidence threshold is dynamically adjusted based on exogenous environmental variables, specifically including: Obtain the exogenous environment variables corresponding to the inference task to be scheduled, wherein the exogenous environment variables include at least one of system load status, task priority, and network status; Based on the exogenous environmental variables, determine the corresponding confidence threshold adjustment amount, and update the preset confidence threshold based on the confidence threshold adjustment amount to obtain the adjusted confidence threshold; The confidence level of each computing node is re-compared with the adjusted confidence level threshold to determine the candidate scheduling nodes.

[0013] Preferably, step S7 specifically includes: The inference task to be scheduled is sent to the target computing node; The target computing node is controlled to call the resident target model or perform the loading operation of the target model according to the corresponding model resident status. Control the target computing node to execute the scheduled inference task, and obtain the actual execution result information corresponding to the scheduled inference task.

[0014] A computing power scheduling system for an ARM cluster server, the system comprising: The task information acquisition module acquires the task information of the inference task to be scheduled and the node status information of each computing node in the ARM cluster. The model residency analysis module determines the model residency status of the inference task to be scheduled on each computing node based on the task information and the node status information of each computing node, and determines the corresponding model loading cost. The effective computing power prediction module predicts the effective computing power of each computing node within a preset future time window based on the node status information of each computing node. The execution result prediction and uncertainty assessment module, based on the model loading cost and the effective computing power, predicts the execution result of the scheduled inference task on each of the computing nodes, and obtains the uncertainty information of the corresponding prediction result; The confidence determination module determines the confidence level of the inference task to be scheduled on each computing node based on the execution result and its uncertainty information. The scheduling decision module determines the target computing node based on the confidence level and scheduling score of each computing node, and schedules the inference task to be scheduled to the target computing node for execution.

[0015] This invention provides a computing power scheduling method and system for ARM cluster servers. It has the following beneficial effects: 1. This invention introduces uncertainty information of the predicted execution result, constructs the confidence level of the inference task to be scheduled to meet the preset performance constraints, and determines the target computing node based on the confidence level and scheduling score, so that the computing power scheduling process takes into account the prediction result and its fluctuation.

[0016] 2. This invention determines the model residency status on each computing node and calculates the model loading cost when the target model is not resided or cannot be reused, so that the scheduling process can explicitly consider the model loading cost and avoid scheduling based solely on the current resource status of the node.

[0017] 3. This invention predicts the effective computing power within a preset future time window based on the processor utilization, frequency status, temperature status, memory bandwidth usage status, and NUMA access status of each computing node, enabling the scheduling process to reflect the actual available computing power of each computing node.

[0018] 4. This invention dynamically adjusts the preset confidence threshold based on at least one of system load status, task priority, and network status, enabling the candidate scheduling node selection process to adapt to different task scenarios and cluster operating states. Attached Figure Description

[0019] Figure 1 This is a flowchart of a computing power scheduling method for an ARM cluster server according to the present invention; Figure 2 This is an architecture diagram of a computing power scheduling system for an ARM cluster server according to the present invention. Detailed Implementation

[0020] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see the appendix Figure 1 This invention provides a computing power scheduling method for an ARM cluster server, comprising: S1. Obtain the task information of the inference task to be scheduled, as well as the node status information of each computing node in the ARM cluster; Furthermore, step S1 specifically includes: Obtain task information for the inference task to be scheduled, including task identifier, model identifier, task priority, and preset performance constraints; Obtain the node status information of each computing node in the ARM cluster. The node status information includes processor utilization, frequency status, temperature status, memory resource status, and model residency information. Based on the task information and the node status information of each computing node, a set of candidate computing nodes corresponding to the inference task to be scheduled is established.

[0022] Specifically, the task information of the inference task to be scheduled and the node status information of each computing node in the ARM cluster are first obtained as the data basis for subsequent model dwell state determination, effective computing power prediction and scheduling decision. Specifically, by obtaining the task identifier, model identifier, task priority, and preset performance constraints corresponding to the inference task to be scheduled, the model requirements and performance requirements of the current inference task to be scheduled are determined. The model identifier is used to represent the target model corresponding to the inference task to be scheduled, and the preset performance constraints are used to represent the performance conditions that the inference task to be scheduled needs to meet during execution. Based on the task information, further obtain the node status information of each computing node in the ARM cluster. The node status information includes processor utilization, frequency status, temperature status, memory resource status, and model residency information. Through the node status information, on the one hand, we can characterize the current running status of each computing node, and on the other hand, we can characterize the basic residency status of the target model in each computing node, thereby providing input data for subsequent model residency status analysis and effective computing power prediction. After obtaining task information and node status information, a set of candidate computing nodes is established based on the model requirements, performance requirements and node status information of the inference task to be scheduled. The set of candidate computing nodes is used to limit the processing scope of subsequent model dwell status determination, effective computing power prediction, execution result prediction and confidence calculation. For example, when the model identifier corresponding to a certain inference task to be scheduled is Model_A, and the preset performance constraint is that the response latency does not exceed 120ms, a set of candidate computing nodes corresponding to the inference task to be scheduled can be established based on the node status information of each computing node, and subsequent scheduling processing can be carried out on the basis of the set of candidate computing nodes.

[0023] S2. Based on the task information and the node status information of each computing node, determine the model dwell status of the inference task to be scheduled on each computing node, and determine the corresponding model loading cost. Furthermore, step S2 specifically includes: Based on the model identifier in the task information, query the model residency information corresponding to each computing node to determine whether each computing node has resided the target model corresponding to the inference task to be scheduled. Given that the target model has been established in each computing node, the version information and resource usage status of the target model are used to determine whether the target model is in a reusable state in the corresponding computing node, so as to obtain the model residency status of each computing node. For computing nodes where the target model is not resident or is not in a reusable state, the model loading cost for each computing node is determined based on the amount of data loaded from the target model and the data input / output bandwidth of the corresponding computing node.

[0024] Specifically, after obtaining the task information of the inference task to be scheduled and the node status information of each computing node, the model residency status of the inference task to be scheduled on each computing node is further determined, and the corresponding model loading cost is determined, so as to provide a basis for subsequent execution result prediction and scheduling score calculation. Among them, the model residency status is used to characterize the direct callability of the target model on the corresponding computing node, and the model loading cost is used to characterize the cost required to load the target model to the corresponding computing node when the target model is not resided or is not reusable. Based on the model identifier in the task information and combined with the model residency information of each computing node, it is determined whether each computing node has a target model corresponding to the inference task to be scheduled. In other words, by matching the model identifier corresponding to the inference task to be scheduled with the currently recorded loaded model information of each computing node, it can be determined whether the target model already exists in the corresponding computing node. This allows subsequent computing nodes to be distinguished into nodes that have the target model and nodes that do not, thus providing a premise for subsequent reuse status judgment and loading cost calculation. Given that a target model is already resident in a computing node, the version information and resource occupancy status of the target model are further combined to determine whether the target model is in a reusable state in the corresponding computing node. This yields the model resident status for each computing node. The version information indicates whether the target model resident in the current node is consistent with the model version required by the inference task to be scheduled. The resource occupancy status indicates whether the target model currently has the resource conditions to be invoked again. If the version information is consistent and the resource occupancy status meets the invocation conditions, the target model on the computing node can be determined to be in a reusable state. If the version information is inconsistent, or although it is resident but the current resource occupancy status does not meet the invocation conditions, the target model on the computing node is determined not to be in a reusable state. Through the above processing, we can avoid making a simple judgment based solely on "whether the model has been loaded", thus enabling the model resident status to more accurately reflect the actual carrying capacity of the corresponding computing node for the inference task to be scheduled. For computing nodes whose model residency status indicates that the target model is not resided or is not in a reusable state, the model loading cost for each computing node is further determined based on the amount of data loaded from the target model and the data input / output bandwidth of the corresponding computing node. In this embodiment, the model loading cost can be expressed by the following formula: ; in, Indicates the first The model loading cost corresponding to each computing node This indicates the amount of data loaded for the target model. Indicates the first The data input and output bandwidth of each computing node is used to determine the model loading cost. When the target model is already residing on a computing node and can be reused, the corresponding model loading cost can be determined as 0. When the target model is not residing or is residing but cannot be reused, the corresponding model loading cost is determined according to the relationship between the amount of data loaded by the model and the data input and output bandwidth of the node. By introducing the model loading cost, the loading cost of the target model can be incorporated into the subsequent execution result prediction and scheduling score calculation process, instead of making scheduling judgments based solely on the current resource status of the node. For example, for a scheduled inference task, its corresponding model is identified as Model_A. If Model_A is already resident in compute node N1, and its version information is consistent with the current task requirements, and its resource occupancy status meets the conditions for re-calling, then the model resident status of compute node N1 can be determined to be reusable, and its model loading cost is 0. If Model_A is not resident in compute node N2, or although it is resident but the model version is inconsistent, then the target model needs to be reloaded. Assuming that the loading data volume of Model_A is 2GB and the data input / output bandwidth of compute node N2 is 1GB / s, then the model loading cost of compute node N2 can be determined to be 2s. Based on the above processing, in the subsequent scheduling process, it is possible to distinguish between "compute nodes that can directly call the target model" and "compute nodes that need to bear the additional cost of model loading", thereby providing a basis for further determining the target compute node.

[0025] S3. Based on the node status information of each computing node, predict the effective computing power of each computing node within a preset future time window. Furthermore, step S3 specifically includes: Based on the node status information of each computing node, node features that characterize the running status of the computing node are extracted. These node features include processor utilization, frequency status, temperature status, memory bandwidth usage status, and NUMA access status. Input the node features into the preset effective computing power prediction model to predict the available computing power of each computing node within the preset future time window, so as to obtain the effective computing power prediction value corresponding to each computing node. The effective computing power prediction value is used as the schedulable computing power indicator for each computing node within a preset future time window.

[0026] Specifically, in this embodiment, after determining the model residency status and model loading cost corresponding to each computing node, the effective computing power of each computing node within a preset future time window is predicted based on the node status information of each computing node, so as to provide a computing power representation for subsequent execution result prediction. The effective computing power is used to represent the available computing power of the corresponding computing node that can actually be used for the scheduled inference task within the preset future time window, which is different from the nominal computing power represented only by the static hardware parameters of the node. Based on the node status information of each computing node, node features representing the operating status of the computing nodes are extracted. These node features include processor utilization, frequency status, temperature status, memory bandwidth usage status, and NUMA access status. Among them, processor utilization is used to represent the current processor resource usage of the corresponding computing node, frequency status is used to represent the current operating frequency level of the processor, temperature status is used to represent the current thermal state of the node, memory bandwidth usage status is used to represent the degree of competition for memory bandwidth resources, and NUMA access status is used to represent whether the node accesses memory across NUMA regions. Thus, by extracting node features, the current multidimensional operating status of the computing nodes can be transformed into the input information required for subsequent effective computing power prediction. After obtaining the node features, the node features are input into a preset effective computing power prediction model to predict the available computing power of each computing node within a preset future time window, so as to obtain the effective computing power prediction value corresponding to each computing node. In this embodiment, the effective computing power prediction model is used to establish the correspondence between node features and available computing power within a preset future time window, and its effective computing power prediction value can be expressed as: ; in, Indicates the first Predicted effective computing power of each computing node within a preset future time window. Indicates the first Processor utilization of each computing node Indicates the first The frequency state of each computing node Indicates the first Temperature status of each computing node Indicates the first Memory bandwidth usage status of each computing node Indicates the first NUMA access status of each compute node The prediction function corresponding to the effective computing power prediction model is represented by the above processing, which can map the current running state of the computing node to the available computing power in the corresponding future time window, thereby avoiding scheduling judgment based solely on the current instantaneous resource status. After obtaining the effective computing power prediction value corresponding to each computing node, the effective computing power prediction value is used as the schedulable computing power indicator of each computing node within the preset future time window. In other words, in the subsequent execution result prediction process, the static computing power parameters or instantaneous resource reserves of the computing node are no longer directly used. Instead, the effective computing power prediction value is used to characterize the actual carrying capacity of the computing node for the scheduled inference task within the preset future time window, so that the subsequent execution result prediction and target computing node determination process are more in line with the actual operating status of each computing node. For example, for candidate computing nodes N1 and N2, if computing node N1 currently has low processor utilization, normal temperature, low memory bandwidth usage, and no significant cross-NUMA region access, then a higher effective computing power prediction value can be obtained through the effective computing power prediction model. If computing node N2 has a similar processor utilization to computing node N1, but its temperature is higher, its memory bandwidth usage is larger, and it has more cross-NUMA region access, then a relatively lower effective computing power prediction value can be obtained. Based on the above effective computing power prediction values, the actual available computing power of different computing nodes for scheduled inference tasks within a preset future time window can be more accurately reflected in subsequent steps.

[0027] S4. Based on model loading cost and effective computing power, predict the execution results of the inference task to be scheduled on each computing node, and obtain the uncertainty information of the corresponding prediction results. Furthermore, step S4 specifically includes: Based on the model loading cost and effective computing power prediction value corresponding to each computing node, the execution result prediction value of the inference task to be scheduled on each computing node is calculated respectively. The execution result prediction value includes the predicted execution time. The predicted execution results are input into the uncertainty assessment model to estimate the uncertainty of the predicted execution results, so as to obtain the uncertainty information corresponding to each computing node. The uncertainty assessment model is any one or more of the following: heteroscedasticity regression model, quantile regression model, or MC Dropout-based neural network model.

[0028] Specifically, in this embodiment, after obtaining the model loading cost and effective computing power prediction value corresponding to each computing node, the execution result of the inference task to be scheduled on each computing node is further predicted, and the uncertainty information of the corresponding prediction result is obtained to provide a basis for subsequent confidence calculation. The execution result prediction value is used to characterize the expected execution of the inference task to be scheduled on the corresponding computing node, and the uncertainty information is used to characterize the fluctuation or credibility of the execution result prediction value, so that the subsequent scheduling process not only considers the prediction result itself, but also the stability of the prediction result. Based on the model loading cost and effective computing power prediction value corresponding to each computing node, the execution result prediction value of the inference task to be scheduled on each computing node is calculated respectively. In this embodiment, the execution result prediction value includes at least the predicted execution time. For ease of explanation, the predicted execution time of the inference task to be scheduled on the i-th computing node can be expressed as: ; in, This represents the predicted execution time of the inference task to be scheduled on the i-th computing node. This represents the computational workload corresponding to the inference task to be scheduled. This represents the predicted effective computing power of the i-th computing node. This represents the model loading cost corresponding to the i-th computing node. Thus, the predicted execution time takes into account both the computation processing time of the scheduled inference task on the corresponding computing node and the model loading time introduced when the target model is not resident or cannot be reused, so that the predicted execution result can more completely reflect the actual processing cost of the corresponding computing node for the current task. After obtaining the predicted execution results for each computing node, the predicted execution results are input into the uncertainty assessment model to estimate the uncertainty of the predicted execution results, thereby obtaining the uncertainty information for each computing node. In this embodiment, the uncertainty information can be used to characterize the dispersion, range, or fluctuation of the predicted execution results. Accordingly, the uncertainty assessment model can be any one or more of heteroscedastic regression, quantile regression, or MCDropout-based neural network models. Heteroscedastic regression can be used to output the mean and variance of the predicted execution results, quantile regression can be used to output the upper and lower quantile intervals of the predicted execution results, and MCDropout-based neural network models can characterize the fluctuation of the prediction results through multiple random inactivation inference results. By further estimating the uncertainty of the predicted execution results, the subsequent processing can no longer treat the predicted execution results as a definite value, but rather as a prediction result with a fluctuation range. For example, for the same inference task to be scheduled, if the model loading cost of computing node N1 is low and the effective computing power prediction value is high, then the predicted execution time calculated according to the above formula is short. If the predicted execution time of computing node N2 is close to that of computing node N1, but the fluctuation range corresponding to its uncertainty information is large, it indicates that the prediction result on this computing node is less stable. In this case, in subsequent steps, the uncertainty information can be further combined to distinguish different computing nodes, thereby providing a basis for the confidence calculation to meet the preset performance constraints. Thus, based on the model loading cost and the effective computing power prediction value, the execution result prediction value and uncertainty information corresponding to each computing node can be obtained, so that the subsequent scheduling process can simultaneously have the ability to characterize execution efficiency and prediction stability.

[0029] S5. Based on the execution results and their uncertainty information, determine the confidence level of the inference task to be scheduled on each computing node to satisfy the preset performance constraints; Furthermore, the S5 steps specifically include: Based on the predicted execution results and uncertainty information of each computing node, construct the distribution information of the execution results of the inference task to be scheduled on each computing node; The distribution information of execution results corresponding to each computing node is compared with the preset performance constraints to determine the probability value of the scheduled inference task on each computing node satisfying the preset performance constraints. The probability value corresponding to each computing node is determined as the confidence level of each computing node.

[0030] Specifically, after obtaining the predicted execution results and uncertainty information for each computing node, the confidence level of the inference task to be scheduled on each computing node is further determined to provide a basis for the subsequent selection of candidate scheduling nodes. The confidence level is used to characterize the possibility that the inference task to be scheduled on the corresponding computing node satisfies the preset performance constraints, so that the subsequent scheduling process can not only judge based on the predicted execution results themselves, but also distinguish each computing node based on the distribution of the predicted results. Specifically, based on the predicted execution results and uncertainty information of each computing node, the distribution information of the execution results of the inference task to be scheduled on each computing node is constructed. In this embodiment, the predicted execution result can characterize the predicted execution time on the corresponding computing node, and the uncertainty information can characterize the fluctuation, distribution range or dispersion of the predicted execution time. Based on the predicted execution result and uncertainty information, the distribution information of the execution results of the inference task to be scheduled on each computing node can be further formed, thereby expanding the original single prediction result into a distribution result that can characterize its range of change, so as to be used for subsequent comparison with preset performance constraints. After obtaining the execution result distribution information corresponding to each computing node, the execution result distribution information is compared with the preset performance constraints to determine the probability value of the inference task to be scheduled on each computing node satisfying the preset performance constraints. In this embodiment, the preset performance constraint is that the execution time is not greater than the latency threshold. At that time, the first The probability value corresponding to each computing node can be expressed as: ; in, Indicates the inference task to be scheduled is in the 1st... The probability value of satisfying the preset performance constraints on each computing node. Indicates the inference task to be scheduled is in the 1st... The predicted execution time random variable on each computing node This represents the latency threshold corresponding to the preset performance constraints. Therefore, by comparing the execution result distribution information with the preset performance constraints, the execution result distribution corresponding to each computing node can be further transformed into a probability value that satisfies the preset performance constraints. After obtaining the probability values ​​corresponding to each computing node, the probability values ​​corresponding to each computing node are determined as the confidence level of each computing node. In other words, the confidence level essentially represents the degree of possibility that the inference task to be scheduled meets the preset performance constraints on the corresponding computing node. The predicted value of the execution result and the uncertainty information can be uniformly transformed into a quantitative indicator that can directly participate in the subsequent scheduling and screening, so that the subsequent computing node screening process has a clear probability judgment basis. For example, for the same inference task to be scheduled, if the predicted execution time of computing node N1 is concentrated around 100ms and the fluctuation range is small, a higher probability value can be obtained under the preset performance constraint that the execution time does not exceed 120ms. If computing node N2 has a similar average predicted execution time to computing node N1, but its fluctuation range is larger, then its probability value of meeting the preset performance constraint is relatively low. Thus, the confidence level of computing node N1 can be determined to be higher, thereby providing a basis for the selection of subsequent candidate scheduling nodes. This allows us to obtain the confidence level of each computing node based on the predicted execution result and uncertainty information, enabling the subsequent scheduling decision-making process to simultaneously reflect both the prediction of the execution result and the stability of the prediction result.

[0031] S6. Based on the confidence level and scheduling score of each computing node, determine the target computing node, wherein the scheduling score is at least related to the execution result, model loading cost and uncertainty information; Furthermore, step S6 specifically includes: The confidence level of each computing node is compared with a preset confidence threshold, and computing nodes with a confidence level greater than or equal to the preset confidence threshold are determined as candidate scheduling nodes. For each candidate scheduling node, the corresponding scheduling score is calculated based on the predicted execution result, model loading cost, and uncertainty information. The candidate scheduling node with the best scheduling score is selected as the target computing node.

[0032] Specifically, after obtaining the confidence level of each computing node, the target computing node is determined based on the confidence level and scheduling score of each computing node. The confidence level is used to reflect the degree of probability that the inference task to be scheduled will meet the preset performance constraints on the corresponding computing node, and the scheduling score is used to further distinguish the computing nodes that meet the confidence level requirements. Thus, on the basis of ensuring that the task meets the preset performance constraints, the target computing node is determined by comprehensively considering the predicted value of the execution result, the model loading cost, and the uncertainty information. First, the confidence level of each computing node is compared with a preset confidence threshold. Computing nodes with a confidence level greater than or equal to the preset confidence threshold are identified as candidate scheduling nodes. In other words, the preset confidence threshold is used to screen each computing node, excluding computing nodes with a low probability of meeting the preset performance constraints. Only computing nodes that meet the confidence requirements are retained to participate in the subsequent scheduling score calculation. This ensures that the determination of subsequent target computing nodes is based on meeting the preset performance constraints, avoiding the use of computing nodes with good resource status but insufficient probability of meeting performance constraints as the final scheduling objects. After determining the candidate scheduling nodes, a scheduling score is calculated for each candidate node based on the predicted execution result, model loading cost, and uncertainty information. In this embodiment, the scheduling score is used to comprehensively characterize the scheduling adaptability of each candidate scheduling node to the current inference task to be scheduled. The scheduling score corresponding to each candidate scheduling node can be represented as: ; in, Indicates the first Scheduling scores of candidate scheduling nodes Indicates the inference task to be scheduled is in the 1st... The predicted execution time in the predicted execution result values ​​of each candidate scheduling node Indicates the first The model loading cost corresponding to each candidate scheduling node This represents the amount of risk determined by uncertain information. , and These represent the weighting coefficients corresponding to prediction execution time, model loading cost, and risk, respectively. They can be integrated into the same scheduling score to comprehensively evaluate candidate scheduling nodes by considering execution efficiency, model loading cost, and prediction result fluctuations. After obtaining the scheduling scores corresponding to each candidate scheduling node, the candidate scheduling node with the best scheduling score is determined as the target computing node. When the scheduling score is used to characterize the comprehensive scheduling cost, the candidate scheduling node with the smallest scheduling score can be determined as the target computing node. Correspondingly, according to the definition of the scheduling score, the candidate scheduling node with the best scheduling score can be taken as the final scheduling result. Within the range of candidate scheduling nodes that meet the preset performance constraints, the computing node with the better comprehensive cost can be further selected as the target computing node. For example, for the same inference task to be scheduled, if both computing node N1 and computing node N2 meet the pre-set confidence threshold requirements, then both can be used as candidate scheduling nodes to participate in the scheduling score calculation. If the prediction execution time corresponding to computing node N1 is short, the model loading cost is 0, and the risk corresponding to the uncertainty information is small, while computing node N2, although it also meets the confidence requirements, has a higher model loading cost or a larger risk, then according to the above scheduling score calculation results, computing node N1 can be determined as the target computing node.

[0033] Furthermore, the preset reliability threshold is dynamically adjusted based on exogenous environmental variables, specifically including: Obtain the exogenous environment variables corresponding to the inference task to be scheduled. The exogenous environment variables include at least one of the following: system load status, task priority, and network status. Based on the exogenous environmental variables, the corresponding confidence threshold adjustment amount is determined, and the preset confidence threshold is updated based on the confidence threshold adjustment amount to obtain the adjusted confidence threshold. The confidence level of each computing node is re-compared with the adjusted confidence level threshold to determine the candidate scheduling nodes.

[0034] Specifically, in order to make the selection process of candidate scheduling nodes adaptable to different task scenarios and cluster operation status, the preset confidence threshold is not fixed, but dynamically adjusted according to the exogenous environmental variables corresponding to the inference task to be scheduled, so as to obtain an adjusted confidence threshold applicable to the current scheduling scenario, and the confidence of each computing node is re-compared based on the adjusted confidence threshold to determine the candidate scheduling nodes. First, the exogenous environment variables corresponding to the inference task to be scheduled are obtained. The exogenous environment variables include at least one of system load status, task priority and network status. The system load status is used to characterize the busyness of the overall or local computing resources of the current ARM cluster. The task priority is used to characterize the processing priority of the inference task to be scheduled during the scheduling process. The network status is used to characterize the network communication conditions corresponding to the inference task to be scheduled during the transmission or scheduling process. By introducing exogenous environment variables, the pre-set confidence threshold can no longer be used as a fixed judgment condition, but can be adaptively adjusted in combination with the current scheduling environment. After obtaining the exogenous environmental variables, the corresponding confidence threshold adjustment amount is determined based on the exogenous environmental variables, and the preset confidence threshold is updated based on the confidence threshold adjustment amount to obtain the adjusted confidence threshold. In this embodiment, the adjusted confidence threshold can be expressed as: ; in, This indicates a preset confidence threshold. This indicates the adjustment amount for the confidence threshold. This represents the adjusted confidence threshold. Furthermore, the adjustment amount for the confidence threshold can be determined by exogenous environmental variables, for example, it can be expressed as: ; in, Indicates the system load status. Indicates task priority. Indicates network status. This represents the mapping relationship between exogenous environmental variables and the confidence threshold adjustment amount, so that the preset confidence threshold can be corrected according to the current scheduling scenario, so that the adopted confidence threshold matches the current environmental conditions; After obtaining the adjusted confidence threshold, the confidence level of each computing node is compared with the adjusted confidence threshold again to determine the candidate scheduling nodes. In other words, in this embodiment, the candidate scheduling nodes are not directly determined based on the initial preset confidence threshold, but are obtained by re-screening based on the adjusted confidence threshold after combining exogenous environmental variables. This allows the screening results of candidate scheduling nodes to change with the current system load status, task priority and network status, and makes the subsequent determination process of target computing nodes more in line with the current actual scheduling needs.

[0035] For example, if the current inference task to be scheduled is a high-priority task and the overall system load of the ARM cluster is high, the corresponding confidence threshold adjustment amount can be determined according to the task priority and system load status. The preset confidence threshold is updated to obtain the adjusted confidence threshold. Then, the confidence corresponding to each computing node is compared with the adjusted confidence threshold again, and only computing nodes that meet the requirements of the adjusted confidence threshold are retained as candidate scheduling nodes for subsequent scheduling score calculation and target computing node determination. Through the above processing, the pre-set confidence threshold can be dynamically changed with exogenous environmental variables on the basis of the original confidence screening mechanism, so that the screening process of candidate scheduling nodes is consistent with the current scheduling scenario.

[0036] S7. Schedule the inference task to be scheduled to the target computing node for execution.

[0037] Furthermore, the S7 steps specifically include: Distribute the inference tasks to be scheduled to the target computing node; The target computing node is controlled to call the resident target model or perform the target model loading operation according to the corresponding model resident status. Control the target computing node to execute the scheduled inference task and obtain the actual execution result information corresponding to the scheduled inference task.

[0038] Specifically, after determining the target computing node, the inference tasks to be scheduled are scheduled to be executed on the target computing node to complete the process from scheduling decision to actual task execution, and obtain the corresponding actual execution result information to provide a data foundation for possible subsequent scheduling optimization or model updates. The task to be scheduled for inference is sent to the target computing node. After the target computing node is determined, the task data, model identifier and related execution parameters of the task to be scheduled for inference are sent to the target computing node so that the target computing node receives the inference task to be executed, realizes the conversion of the scheduling result into the actual execution process, and enables the determined target computing node to take over the task to be scheduled for inference. After the inference task to be scheduled is sent to the target computing node, the corresponding model invocation or loading operation is executed according to the model resident status of the target computing node. Specifically, when the target model is already resident in the target computing node and is in a reusable state, the resident target model is directly invoked to execute the inference task. When the target model is not resident or is resident but is not in a reusable state, the target model loading operation is executed to load the target model into the target computing node before inference processing. Through the above processing, the model resident status can be specifically applied in the actual execution stage, thereby ensuring the consistency between scheduling decisions and the execution process. After completing the model invocation or model loading, the target computing node is controlled to execute the scheduled inference task and obtain the actual execution result information corresponding to the scheduled inference task. The actual execution result information may include the actual execution time, resource consumption, or task execution completion status, etc. On the one hand, it can be used to verify the accuracy of the aforementioned execution result prediction and confidence calculation, and on the other hand, it can also provide reference data for subsequent scheduling strategy optimization. For example, for a certain inference task to be scheduled, once the target computing node N1 is determined, the task is sent to computing node N1. If the target model is already resident in computing node N1 and is in a reusable state, the model is directly called to execute the inference task. If it is not resident or is not reusable, the model loading operation is performed first and then inference is performed. After the task is completed, the actual execution time and execution status information of the task on computing node N1 can be obtained and used as a reference for the subsequent scheduling process. Thus, the target computing node determined based on confidence and scheduling score can be implemented in the actual execution process, so that the entire scheduling method forms a complete closed loop from task information acquisition, scheduling decision to task execution.

[0039] Please see the appendix Figure 2 A computing power scheduling system for an ARM cluster server, the system comprising: The task information acquisition module acquires the task information of the inference task to be scheduled and the node status information of each computing node in the ARM cluster. The model residency analysis module determines the model residency status of the inference task to be scheduled on each computing node based on the task information and the node status information of each computing node, and determines the corresponding model loading cost. The effective computing power prediction module predicts the effective computing power of each computing node within a preset future time window based on the node status information of each computing node. The execution result prediction and uncertainty assessment module predicts the execution results of the inference task to be scheduled on each computing node based on the model loading cost and effective computing power, and obtains the uncertainty information of the corresponding prediction results. The confidence determination module determines the confidence level of the inference task to be scheduled on each computing node based on the execution results and its uncertainty information. The scheduling decision module determines the target computing node based on the confidence level and scheduling score of each computing node, and schedules the inference tasks to be scheduled to the target computing node for execution.

[0040] Specifically, the task information acquisition module is used to acquire the task information of the inference task to be scheduled and the node status information of each computing node in the ARM cluster. The model residency analysis module is used to determine the model residency status of the inference task to be scheduled on each computing node based on the task information and the node status information of each computing node, and to determine the corresponding model loading cost. The effective computing power prediction module is used to predict the effective computing power of each computing node within a preset future time window based on the node status information of each computing node. The execution result prediction and uncertainty assessment module is used to predict the execution result of the inference task to be scheduled on each computing node based on the model loading cost and effective computing power, and to acquire the uncertainty information of the corresponding prediction result. The confidence determination module is used to determine the confidence level of the inference task to be scheduled on each computing node to meet the preset performance constraints based on the execution result and its uncertainty information. The scheduling decision module is used to determine the target computing node based on the confidence level and scheduling score of each computing node, and to schedule the inference task to be scheduled to the target computing node for execution. Through the coordinated cooperation of the above modules, the complete processing process of model residency status analysis, effective computing power prediction, execution result assessment, confidence determination, and target computing node selection for the inference task to be scheduled in the ARM cluster server is realized.

[0041] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A computing power scheduling method for an ARM cluster server, characterized in that, include: S1. Obtain the task information of the inference task to be scheduled, as well as the node status information of each computing node in the ARM cluster; S2. Based on the task information and the node status information of each computing node, determine the model dwell status of the inference task to be scheduled on each computing node, and determine the corresponding model loading cost. S3. Based on the node status information of each computing node, predict the effective computing power of each computing node within a preset future time window; S4. Based on the model loading cost and the effective computing power, predict the execution result of the scheduled inference task on each computing node, and obtain the uncertainty information of the corresponding prediction result; S5. Based on the execution results and their uncertainty information, determine the confidence level of the inference task to be scheduled on each computing node to satisfy the preset performance constraints; S6. Based on the confidence level and scheduling score corresponding to each computing node, determine the target computing node, wherein the scheduling score is at least related to the execution result, model loading cost and uncertainty information; S7. Schedule the inference task to be scheduled to the target computing node for execution.

2. The computing power scheduling method for an ARM cluster server according to claim 1, characterized in that, The S1 step specifically includes: Obtain task information for the inference task to be scheduled, including task identifier, model identifier, task priority, and preset performance constraints; Obtain node status information of each computing node in the ARM cluster, including processor utilization, frequency status, temperature status, memory resource status, and model residency information; Based on the task information and the node status information of each computing node, a set of candidate computing nodes corresponding to the inference task to be scheduled is established.

3. The computing power scheduling method for an ARM cluster server according to claim 1, characterized in that, The S2 step specifically includes: Based on the model identifier in the task information, query the model residency information corresponding to each computing node to determine whether each computing node has resided the target model corresponding to the inference task to be scheduled. If it is determined that the target model has been resident in each computing node, the target model is further determined to be in a reusable state in the corresponding computing node based on the version information and resource usage status of the target model, so as to obtain the model resident status corresponding to each computing node. For computing nodes whose model residency status indicates that the target model is not resided or is not in a reusable state, the model loading cost corresponding to each computing node is determined based on the amount of data loaded by the target model and the data input / output bandwidth of the corresponding computing node.

4. The computing power scheduling method for an ARM cluster server according to claim 1, characterized in that, The S3 step specifically includes: Based on the node status information of each computing node, node features that characterize the running status of the computing node are extracted. The node features include processor utilization, frequency status, temperature status, memory bandwidth usage status, and NUMA access status. The node features are input into a preset effective computing power prediction model to predict the available computing power of each computing node within a preset future time window, so as to obtain the effective computing power prediction value corresponding to each computing node. The predicted effective computing power value is used as the schedulable computing power index for each computing node within the preset future time window.

5. The computing power scheduling method for an ARM cluster server according to claim 1, characterized in that, The S4 step specifically includes: Based on the model loading cost and effective computing power prediction value corresponding to each computing node, the execution result prediction value of the inference task to be scheduled on each computing node is calculated respectively, and the execution result prediction value includes the predicted execution time. The predicted execution result is input into the uncertainty assessment model to estimate the uncertainty of the predicted execution result, so as to obtain the uncertainty information corresponding to each computing node. The uncertainty assessment model is any one or more of the following: heteroscedasticity regression model, quantile regression model, or MC Dropout-based neural network model.

6. The computing power scheduling method for an ARM cluster server according to claim 1, characterized in that, The S5 step specifically includes: Based on the predicted execution results and uncertainty information of each computing node, the distribution information of the execution results of the inference task to be scheduled on each computing node is constructed. The distribution information of execution results corresponding to each computing node is compared with the preset performance constraints to determine the probability value of the scheduled inference task on each computing node satisfying the preset performance constraints. The probability value corresponding to each computing node is determined as the confidence level corresponding to each computing node.

7. The computing power scheduling method for an ARM cluster server according to claim 1, characterized in that, The S6 step specifically includes: The confidence level of each computing node is compared with a preset confidence threshold, and computing nodes with a confidence level greater than or equal to the preset confidence threshold are determined as candidate scheduling nodes; For each candidate scheduling node, a corresponding scheduling score is calculated based on the predicted execution result, model loading cost, and uncertainty information. The candidate scheduling node with the best scheduling score is selected as the target computing node.

8. The computing power scheduling method for an ARM cluster server according to claim 7, characterized in that, The preset reliability threshold is dynamically adjusted based on exogenous environmental variables, specifically including: Obtain the exogenous environment variables corresponding to the inference task to be scheduled, wherein the exogenous environment variables include at least one of system load status, task priority, and network status; Based on the exogenous environmental variables, determine the corresponding confidence threshold adjustment amount, and update the preset confidence threshold based on the confidence threshold adjustment amount to obtain the adjusted confidence threshold; The confidence level of each computing node is re-compared with the adjusted confidence level threshold to determine the candidate scheduling nodes.

9. The computing power scheduling method for an ARM cluster server according to claim 1, characterized in that, The S7 step specifically includes: The inference task to be scheduled is sent to the target computing node; The target computing node is controlled to call the resident target model or perform the loading operation of the target model according to the corresponding model resident status. Control the target computing node to execute the scheduled inference task, and obtain the actual execution result information corresponding to the scheduled inference task.

10. A computing power scheduling system for an ARM cluster server, characterized in that, A computing power scheduling method for an ARM cluster server according to any one of claims 1-9, the system comprising: The task information acquisition module acquires the task information of the inference task to be scheduled and the node status information of each computing node in the ARM cluster. The model residency analysis module determines the model residency status of the inference task to be scheduled on each computing node based on the task information and the node status information of each computing node, and determines the corresponding model loading cost. The effective computing power prediction module predicts the effective computing power of each computing node within a preset future time window based on the node status information of each computing node. The execution result prediction and uncertainty assessment module, based on the model loading cost and the effective computing power, predicts the execution result of the scheduled inference task on each of the computing nodes, and obtains the uncertainty information of the corresponding prediction result; The confidence determination module determines the confidence level of the inference task to be scheduled on each computing node based on the execution result and its uncertainty information. The scheduling decision module determines the target computing node based on the confidence level and scheduling score of each computing node, and schedules the inference task to be scheduled to the target computing node for execution.

Images