A management node task scheduling method and system of a heterogeneous computing power scheduling system

Abstract

The application provides a management node task arrangement method and system of a heterogeneous computing power scheduling system, belongs to the technical field of distributed computing and cluster resource scheduling, and synchronously acquires computing power performance data, topological position and running state data of a heterogeneous node, historical load data of a current task and attribute data of a target task, and constructs a resource descriptor. A topological graph is constructed with communication overhead as a weight, and a scoring function is used to score the heterogeneous node. Task fingerprint data generated based on the attribute data is input into a decision tree model to obtain a predicted running duration. Before a peak period, the preemption cost is calculated according to the running duration and checkpoint data, a preoccupied task is determined, and a resource recycling instruction is generated. After the resources are released, the resource descriptor is updated and the available resource capacity is extracted, and finally, an execution node is determined and a start instruction is issued. The application ensures that the target task is assigned to a node with sufficient resources and optimal communication cost for execution.

Description

Technical Field

[0001] This application belongs to the field of distributed computing and cluster resource scheduling technology, and in particular relates to a method and system for scheduling management node tasks in a heterogeneous computing power scheduling system. Background Technology

[0002] In heterogeneous computing power scheduling, task orchestration by the management node refers to the process in a computing cluster composed of multiple types of accelerator cards, where the management node uniformly receives task requests, perceives the global resource status, and assigns tasks to appropriate computing nodes for execution based on the matching relationship between resource supply and task demand. With the continuous growth in computing power demands from scenarios such as large-scale model training, online inference services, and scientific computing, the accuracy and real-time performance of management node task orchestration have become key factors determining the overall efficiency of the computing power platform.

[0003] Existing task orchestration methods for management nodes typically abstract accelerator cards as integer resources with a single dimension, triggering scheduling decisions at fixed intervals to allocate tasks to nodes with sufficient resource availability. Some methods introduce task-device matching mechanisms based on deep reinforcement learning, improving overall throughput by modeling the correspondence between task features and device capabilities; however, these methods still rely on offline training strategies, limiting their responsiveness to real-time resource status changes. Furthermore, in scenarios involving large-scale training and online inference deployments, all of these methods primarily focus on batch processing jobs, lacking the ability to predict load fluctuations.

[0004] However, existing methods only abstract accelerator cards by quantity, and cannot distinguish the actual differences in computing power, storage bandwidth and other aspects between different architectures and generations of accelerator cards. This results in high-value computing resources being occupied by low-value tasks, and the overall resource utilization efficiency is low. At the same time, existing methods lack modeling of the physical network connectivity of the cluster. The task orchestration results often involve cross-network domain communication, which significantly reduces the execution efficiency of communication-intensive tasks.

[0005] Furthermore, existing methods cannot predict peak periods based on load trends and intervene in resource reclamation in advance. During peak traffic periods, resource contention between online inference and training tasks cannot be effectively coordinated, leading to a significant increase in response latency for online inference services. Therefore, existing technologies suffer from technical problems such as low utilization of heterogeneous computing resources, high communication overhead for large-scale training, and difficulty in guaranteeing online inference service latency, resulting in insufficient task orchestration for management nodes. Summary of the Invention

[0006] The purpose of this application is to provide a method, apparatus and storage medium for orchestrating management node tasks in a heterogeneous computing power scheduling system, so as to solve the problem of insufficient management node task orchestration in the prior art.

[0007] To address the aforementioned technical problems, in a first aspect, this application provides a method for orchestrating tasks on the management nodes of a heterogeneous computing power scheduling system, comprising: Synchronously acquire computing power performance data, topology location and running status data of heterogeneous nodes, historical load data of the current task and attribute data of the target task; Based on computing power performance data, topology location, and operational status data, a set of multivariate resource descriptors is constructed; Based on resource descriptors, a topology graph is constructed with heterogeneous nodes as nodes, communication links between nodes as edges, and communication overhead as edge weights. Based on the topology graph, a pre-defined multi-dimensional scoring function is used to calculate the score of each heterogeneous node. The task fingerprint data, determined by hashing the attribute data, is input into the trained decision tree model to obtain the predicted runtime. Before the peak period determined by historical load data, the preemption cost is calculated based on the predicted runtime and the checkpoint data of the current task. Based on the preemption cost and the preset threshold range, the preempted task and the corresponding resource reclamation instruction are determined in the current task. After the heterogeneous node corresponding to the preempted task responds to the resource reclamation command and releases resources, the updated resource descriptor is obtained and the available resource capacity is extracted. Based on attribute data, available resource capacity, and scores, the execution node is determined among heterogeneous nodes, and a start command for executing the target task is sent to the execution node.

[0008] Optionally, the computing power performance data includes the original computing power value, precision weighting coefficient, memory capacity, memory bandwidth, interconnect bandwidth and instruction compatibility identifier of each heterogeneous node, and the operating status data includes the real-time utilization rate, device temperature and error count of each heterogeneous node. Based on computing performance data, topology location, and operational status data, a set of multivariate resource descriptors is constructed, including: The first loss amount is obtained by multiplying the equipment temperature by the preset temperature loss coefficient; the second loss amount is obtained by multiplying the error count by the preset count loss coefficient; and the health coefficient is obtained by calculating the difference between the preset benchmark value and the first and second loss amounts. The utilization rate coefficient is obtained by calculating the average real-time utilization rate within the preset sliding time window. The target computing power value is obtained by multiplying the original computing power value, the precision weight coefficient, the utilization rate coefficient, and the health coefficient. Based on the target computing power, memory capacity, memory bandwidth, interconnect bandwidth, instruction compatibility flag, topology location, real-time utilization, device temperature, and error count, a resource descriptor is constructed for each heterogeneous node.

[0009] Optionally, based on resource descriptors, a topology graph is constructed with heterogeneous nodes as nodes, communication links between nodes as edges, and communication overhead as edge weights. Based on the topology graph, a pre-defined multi-dimensional scoring function is used to calculate the score of each heterogeneous node, including: The communication overhead of each communication link is obtained by weighting and summing the reciprocal of the interconnect bandwidth in the resource descriptor with the transmission hop count and transmission delay of the corresponding communication link. A topology graph is constructed with heterogeneous nodes as nodes, communication links between nodes as edges, and communication overhead as edge weights. The communication overhead component is obtained by summing the communication overhead of all communication links connected to each heterogeneous node in the topology graph. The resource balancing component is obtained by calculating the ratio of the allocated resources of each heterogeneous node to the total allocated resources. The resource penalty component is obtained by calculating the ratio of the remaining resources of each heterogeneous node after executing the target task to the total remaining resources. The communication overhead component, resource balancing component, and resource penalty component are substituted into the multidimensional scoring function for weighted summation to obtain the score for each heterogeneous node.

[0010] Optionally, the attribute data includes the container image identifier, command parameters, dataset identifier, and device type identifier of the target task. The trained decision tree model is trained based on a gradient boosting mechanism using training samples including historical task fingerprint data and corresponding real historical runtime. The task fingerprint data, determined by hashing the attribute data, is input into the trained decision tree model to obtain the predicted runtime, including: The task fingerprint data is obtained by hashing and combining the container image identifier, command parameters, dataset identifier, and device type identifier. The task fingerprint data is input into multiple decision trees in the decision tree model. Starting from the root node of each decision tree, the feature value of each dimension in the task fingerprint data is compared with the splitting threshold of the corresponding traversed node, and the branch is selected layer by layer until the leaf node is reached. The prediction increment value of the leaf node is extracted, and all prediction increment values ​​are accumulated to obtain the prediction runtime.

[0011] Optionally, before the peak period determined based on historical load data, a preemption cost is calculated based on the predicted runtime and checkpoint data of the current task. Based on the preemption cost and a preset threshold range, the preempted task and its corresponding resource reclamation instruction are determined within the current task, including: Calculate the average load of historical load data for each time period within a preset time window, extrapolate the predicted load for future time periods based on the growth rate of the average load, and determine the future time periods where the predicted load is greater than the preset load threshold as peak periods. When the current time is earlier than the start time of the peak period, the remaining runtime is obtained by the difference between the predicted runtime and the cumulative runtime of the current task from its start to the current time. The checkpoint validity is obtained by calculating the reciprocal of the time interval between the current time and the generation time of the latest checkpoint in the checkpoint data. The remaining runtime and checkpoint validity are weighted and summed to obtain the preemption cost of the current task. From all current tasks, the current tasks whose preemption cost is within a preset threshold range are identified as preempted tasks, and a resource reclamation instruction is generated.

[0012] Optionally, after the heterogeneous node corresponding to the preempted task responds to the resource reclamation command and releases resources, the updated resource descriptor is obtained and the available resource capacity is extracted, including: After the heterogeneous node corresponding to the preempted task responds to the resource reclamation command and releases resources, the real-time utilization rate, device temperature and error count of each heterogeneous node are reacquired. The updated resource descriptor for each heterogeneous node is obtained by updating the values ​​of the corresponding fields in the resource descriptor based on the real-time utilization, device temperature, and error count. Extract the target computing power value and video memory capacity from the updated resource descriptor. Obtain the available computing power based on the difference between the target computing power value and the current allocated computing power. Obtain the available video memory based on the difference between the video memory capacity and the current allocated video memory. Summarize the available computing power and available video memory to determine the available resource capacity of each heterogeneous node.

[0013] Optionally, the attribute data also includes the target task's required computing power and required video memory, and the available resource capacity includes the available computing power and available video memory for each heterogeneous node; Based on attribute data, available resource capacity, and scores, the execution node is determined among heterogeneous nodes, and a start command for executing the target task is sent to the execution node, including: Heterogeneous nodes whose available computing power in the available resource capacity is greater than or equal to the required computing power and whose available video memory in the available resource capacity is greater than or equal to the required video memory are identified as candidate nodes. The candidate node with the highest score among all candidate nodes is selected as the execution node. The container image identifier, command parameters, dataset identifier, required computing power, and required video memory are encapsulated into a startup instruction and sent to the execution node to execute the target task.

[0014] Secondly, this application provides a task orchestration system for the management node of a heterogeneous computing power scheduling system, comprising: The acquisition module is used to synchronously acquire computing power performance data, topology location and running status data of heterogeneous nodes, historical load data of the current task and attribute data of the target task; The building module is used to construct a set of multi-dimensional resource descriptors based on computing power performance data, topology location, and operating status data; The construction module is also used to construct a topology graph based on resource descriptors, with heterogeneous nodes as nodes, communication links between nodes as edges, and communication overhead as edge weights, and to calculate the score of each heterogeneous node based on the topology graph using a preset multidimensional scoring function. The generation module is used to input the task fingerprint data determined by hashing based on attribute data into the trained decision tree model to obtain the predicted runtime. The generation module is also used to calculate the preemption cost based on the predicted runtime and checkpoint data of the current task before the peak period determined by historical load data, and to determine the preempted task and the corresponding resource reclamation instruction in the current task according to the preemption cost and the preset threshold range. The generation module is also used to obtain the updated resource descriptor and extract the available resource capacity after the heterogeneous node corresponding to the preempted task responds to the resource reclamation command and releases resources; The sending module is used to determine the execution node among heterogeneous nodes based on attribute data, available resource capacity and score, and send the start command for executing the target task to the execution node.

[0015] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor, used to execute the computer program, implements the steps of the management node task orchestration method of the heterogeneous computing power scheduling system as described in the first aspect above.

[0016] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the management node task orchestration method of the heterogeneous computing power scheduling system described in the first aspect above.

[0017] The heterogeneous computing power scheduling system management node task orchestration method provided in this application first acquires the computing power performance data, topology location and running status data of heterogeneous nodes synchronously, and constructs a multi-dimensional resource descriptor. This enables the management node to perceive the actual differences in computing power, storage bandwidth and other dimensions of different architectures and generational accelerators, thereby achieving precise matching with hardware capabilities when allocating tasks, avoiding high-value computing power resources being occupied by low-value tasks, and improving the overall resource utilization efficiency.

[0018] Subsequently, a topology graph is constructed with communication overhead as the weight, and a multidimensional scoring function is used to score each heterogeneous node. Communication-intensive tasks are prioritized and arranged to the node combination with the lowest communication cost, thereby reducing cross-network domain communication overhead and improving the execution efficiency of large-scale training tasks.

[0019] Next, by predicting peak periods based on historical load data, resource reclamation and capacity updates are completed before the peak arrives based on predicted runtime and checkpoint data. This allows for advance coordination of resource allocation between online inference and training tasks, ensuring that the response latency of the online inference service during peak traffic periods is not affected by the congestion caused by training tasks. In summary, this application achieves a closed-loop process from accurate resource awareness and topology-aware orchestration to peak prediction and preemptive intervention, effectively solving the technical problem of insufficient task orchestration at management nodes in existing technologies.

[0020] Furthermore, this application calculates the average load and its growth rate of historical load data within a preset time window to extrapolate the predicted load for future periods and identify peak periods, enabling management nodes to have the ability to anticipate business traffic trends and changing the situation where existing methods only respond passively after traffic peaks arrive.

[0021] Assuming the current time is earlier than the start time of the peak period, the preemption cost is calculated by combining the remaining runtime of the task with the checkpoint validity. This ensures that the task with the least interruption loss is prioritized for reclamation, thus avoiding significant losses in training task progress due to blind preemption.

[0022] By selecting tasks whose preemption cost falls within a preset threshold range as preemptible tasks, resources are released before peak periods arrive, reserving sufficient computing and storage resources for online inference tasks. In summary, this application effectively ensures the stability of resource supply and response latency for online inference services during peak traffic periods through the coordinated early identification of peak periods and minimum-cost preemption decisions. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a task orchestration method for a management node of a heterogeneous computing power scheduling system provided in this application embodiment; Figure 2 A flowchart illustrating a method for constructing a resource descriptor provided in an embodiment of this application; Figure 3 This application provides a schematic diagram of the structure of a topology graph. Figure 4This application provides a schematic diagram of the structure of a management node task orchestration system for a heterogeneous computing power scheduling system, as shown in the embodiments of this application. Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0025] Existing management node task orchestration methods face three structural defects in heterogeneous computing scenarios: First, the resource abstraction of accelerator cards remains at the quantitative level, failing to perceive the substantial differences in computing power, storage bandwidth, etc., between different architectures and generations of accelerator cards, causing high-value computing resources to be frequently occupied by low-priority tasks, making it difficult to improve overall resource utilization efficiency.

[0026] Secondly, the orchestration decision-making process lacks modeling of the cluster's physical network connectivity. Tasks are often assigned to combinations of nodes with long communication paths, resulting in communication-intensive tasks incurring significant cross-network domain communication overhead during execution, which significantly inhibits training efficiency. Thirdly, existing methods are primarily designed for batch processing jobs and lack the ability to predict business load trends. Resource contention between online inference and training tasks during peak traffic periods cannot be coordinated in advance, thus significantly increasing the response latency of online inference services.

[0027] To address the aforementioned issues, this application proposes a task orchestration method for management nodes in heterogeneous computing power scheduling. The core idea is to integrate four information links—resource awareness, topology modeling, runtime prediction, and load forecasting—into the orchestration decision-making process. Specifically, by synchronously collecting computing performance data, topology location, and operational status data of heterogeneous nodes, a resource descriptor covering multiple dimensions such as computing power, storage capacity, and communication bandwidth is constructed. This enables an accurate characterization of the true capabilities of accelerator cards with different architectures, resolving the resource mismatch problem caused by a single abstraction.

[0028] Building upon this foundation, a topology graph is constructed using inter-node communication overhead as a weight, and candidate nodes are evaluated using a multi-dimensional scoring function. Tasks are prioritized and orchestrated to the node combinations with the lowest communication costs, eliminating the additional overhead caused by cross-network domain communication. Simultaneously, by hashing the attribute data of the target task and inputting it into a decision tree model to predict runtime, and combining historical load data to identify future peak periods, preemption costs are calculated and resources are reclaimed before the peak arrives based on the remaining task runtime and checkpoint data. This ensures that online inference tasks receive sufficient computing and storage resources during peak traffic periods, resolving the technical problem of insufficient task orchestration for management nodes in existing technologies.

[0029] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] To address the problems of existing technologies, embodiments of this application provide a method, apparatus, device, computer storage medium, and computer program product for orchestrating management node tasks in a heterogeneous computing power scheduling system. The method for orchestrating management node tasks in a heterogeneous computing power scheduling system provided in this application embodiment will be described first.

[0031] Figure 1 This illustration shows a flowchart of a task orchestration method for a management node in a heterogeneous computing power scheduling system according to an embodiment of this application. Figure 1 As shown, the method includes: S101. Synchronously acquire computing power performance data, topology location and running status data of heterogeneous nodes, historical load data of the current task and attribute data of the target task.

[0032] Heterogeneous nodes refer to computing servers in a cluster that are equipped with different architectures or generations of accelerator cards, and these heterogeneous nodes exhibit significant differences in computing power, storage capacity, communication bandwidth, and instruction compatibility. Computing performance data refers to a set of values ​​describing the static hardware capabilities of heterogeneous nodes, including each heterogeneous node's original computing power value, precision weighting coefficient, memory capacity, memory bandwidth, interconnect bandwidth, and instruction compatibility identifier.

[0033] The raw computing power value refers to the peak floating-point operation capability of the accelerator card under nominal conditions. The precision weighting coefficient refers to the computing power conversion ratio corresponding to different numerical calculation precision formats such as FP32, FP16, BF16, etc. The instruction compatibility identifier refers to the identifier of the set of operation instructions supported by the accelerator card and the corresponding runtime version.

[0034] Topology location refers to the location description of heterogeneous nodes in the cluster physical network and includes the non-consistent memory access region number to which the node belongs, the high-speed peripheral expansion bus root complex number, the rack number, the connected leaf switch number, and the connected spine switch number.

[0035] Operational status data refers to a set of indicators reflecting the current dynamic operation of heterogeneous nodes, including the real-time utilization, device temperature, and error count of each heterogeneous node. Current tasks refer to tasks that are already running in the cluster, while historical load data refers to the time-series record of the number of queries per second for the current task within a historical time period, used to reflect the changing patterns of business traffic over time.

[0036] The target task refers to the new task to be orchestrated and scheduled. The attribute data refers to a set of parameters that describe the running requirements of the target task, including container image identifier, command parameters, dataset identifier, device type identifier, required computing power, and required video memory.

[0037] In scenarios involving large-scale model training and online inference in a hybrid deployment, when the management node receives an orchestration request for the target task, it first synchronously collects the computing performance data, topology location, and operational status data of all heterogeneous nodes in the entire cluster. Specifically, during the startup phase, each heterogeneous node uses a local adapter plugin to call the device enumeration interface to scan each accelerator card on the node, read the static hardware parameters of each accelerator card, and report them through the communication channel between the management node and the compute nodes, thus obtaining the data for each heterogeneous node. The computing performance data corresponding to the accelerator card on the device ,in This is the original computing power value. For accuracy weighting coefficients, For video memory capacity, For video memory bandwidth, For interconnect bandwidth, This is an instruction compatibility identifier.

[0038] During the startup phase, each heterogeneous node uses its local adapter plugin to call the topology reporting interface to read the processor region, peripheral expansion bus root complex, and physical slot information of each accelerator card. This information, combined with the node's network access location within the cluster, is then reported via the communication channel to obtain the information for each heterogeneous node. topological location ,in This is the region number for non-consistent memory access. This is the root complex number for the high-speed peripheral expansion bus. For the rack number, Number the leaf switches. Number the spine switch.

[0039] Each heterogeneous node continuously collects real-time operating metrics of its accelerator card via periodic heartbeats and pushes them to the management node through a communication channel, thus obtaining the data for each heterogeneous node. Latest running status data ,in For real-time utilization, For equipment temperature, This is the error count.

[0040] Meanwhile, the management node reads the query count per second records of the same online inference services running in the cluster during the current task, which are stored in the local time-series cache at fixed sampling intervals throughout the historical running period. By aggregating cluster-level metrics, it obtains the macro-historical load data of the current task. ,in Indicates the first The average number of queries per second over a given time period. Additionally, the management node extracts attribute data of the target task from orchestration requests. ,in For container image identification, For command parameters, Identify the dataset. For equipment type identification, To meet the demand for computing power, To meet the demand for video memory.

[0041] S102. Based on computing power performance data, topology location and operating status data, construct a set of multi-dimensional resource descriptors.

[0042] Optionally, the computing performance data in step S102 includes the original computing power value, precision weighting coefficient, memory capacity, memory bandwidth, interconnect bandwidth and instruction compatibility identifier of each heterogeneous node, and the operating status data includes the real-time utilization rate, device temperature and error count of each heterogeneous node.

[0043] Step S102, which involves constructing a set of multivariate resource descriptors based on computing power performance data, topology location, and operational status data, may specifically include: Figure 2 A flowchart illustrating a method for constructing a resource descriptor according to an embodiment of this application is shown. Figure 2 As shown, the method includes: S1021. Calculate the first loss amount by multiplying the equipment temperature by the preset temperature loss coefficient, calculate the second loss amount by multiplying the error count by the preset count loss coefficient, and calculate the health coefficient by calculating the difference between the preset benchmark value and the first and second loss amounts.

[0044] First loss amount This refers to the quantified value of the computing power loss caused by equipment temperature exceeding the normal operating range. Second loss amount. This refers to a quantified value of the computing power loss caused by accumulated hardware error events on the accelerator card. Preset temperature loss coefficient. With preset counting loss coefficient This refers to the pre-calibrated loss sensitivity parameters based on different accelerator card types, as shown in Table 1 below: Table 1: Comparison Table of Temperature Loss Coefficient and Count Loss Coefficient As shown in Table 1, the specific values ​​of each coefficient in Table 1 are based on the cluster history of various accelerator cards. The statistical average value was determined by linear regression analysis of the telemetry data on the performance degradation under full load operation within one month, and all coefficients are positive. and The larger the value, the more sensitive the accelerator card is to the loss of computing power due to temperature rise or error events. In actual deployment, it can be calibrated in conjunction with the hardware specification manual and written into the configuration file. The management node can take effect after reading it during the startup phase without restarting the core process.

[0045] Health Index This refers to a normalized, quantified value that comprehensively reflects the current actual usability of the accelerator card. (Preset baseline value) This refers to the initial reference value of the health coefficient when both the accelerator card temperature and error count are in ideal conditions; its value is usually set at... to Between , , The configuration can be customized according to the task scenario type, as shown in Table 2 below: Table 2: Preset Baseline Value Comparison Table As shown in Table 2, the baseline values ​​corresponding to the three types of scenarios in Table 2 satisfy... A higher baseline value means stricter requirements for the health status of the accelerator card. In high-reliability inference scenarios, even a small loss may cause the health coefficient to fall below the usable threshold, thus putting the node at a disadvantage in subsequent scoring stages.

[0046] In a scenario where large-scale model training and online inference are deployed in a hybrid manner, following the operational status data acquired by S101, the management node first processes each heterogeneous node. Calculate the two loss amounts separately. The operating status data is as follows: According to each heterogeneous node The type of accelerator card used can be found in Table 1. , The first loss amount was calculated. Second loss amount .

[0047] Next, based on the current task scenario type, the corresponding preset baseline value is obtained from Table 2. Finally, to prevent serious logical errors such as negative health values ​​due to drastic temperature anomalies or a large number of errors in a short period, which could lead to negative downstream computing power assessments, a boundary protection mechanism is introduced to obtain the health coefficient. .

[0048] S1022. Calculate the average real-time utilization rate within the preset sliding time window to obtain the utilization rate coefficient. Based on the product of the original computing power value, the precision weight coefficient, the utilization rate coefficient, and the health coefficient, obtain the target computing power value.

[0049] Utilization coefficient This refers to an estimate of the effective utilization level obtained by averaging historical samples of the real-time utilization of heterogeneous nodes within a preset sliding time window. To avoid the paradox of idle node computing power being incorrectly evaluated as zero due to the product formula, the real-time utilization rate here... The extraction is based on the node idle rate or the hardware base state resident ratio, thereby reflecting the relative available ratio of the node actually retained under recent workload.

[0050] The preset sliding time window refers to a fixed-length historical utilization sampling sequence maintained by the management node, and the window length can be specified through a configuration file. The target computing power value refers to the normalized effective computing power estimate obtained after comprehensively considering the original computing power of the accelerator card, precision conversion, historical effective utilization, and health status.

[0051] During the implementation of the solution, following the health coefficient obtained in S1021, the management node manages each heterogeneous node. Calculate the utilization coefficient and the target computing power value separately. Specifically, the management node maintains the historical sampling sequence of utilization for each node, with a window length of [value missing]. Then the node The utilization coefficient is ,in For the first Node during the next sampling Real-time utilization rate.

[0052] Next, according to , , and Get each heterogeneous node Target computing power ,in This is the original computing power value. For accuracy weighting coefficients, This is the utilization rate coefficient. This represents the health score.

[0053] S1023. Based on the target computing power, memory capacity, memory bandwidth, interconnect bandwidth, instruction compatibility flag, topology location, real-time utilization, device temperature, and error count, construct the resource descriptor for each heterogeneous node.

[0054] resource descriptor This refers to unifying the static hardware capabilities, dynamic operating status, and physical topology location information of heterogeneous nodes into a single structured data object. This object includes nine dimensions: target computing power, memory capacity, memory bandwidth, interconnect bandwidth, instruction compatibility flag, topology location, real-time utilization, device temperature, and error count. The resource descriptor is stored as a structure array in the management node's memory and includes a built-in version number field. The version number is atomically incremented with each update for consistency checks during concurrent reads and writes.

[0055] During the implementation of the scheme, following the calculation results of S1021 and S1022, the management node manages each heterogeneous node. The above-mentioned dimensions are aggregated into a resource descriptor. in The target computing power value calculated for S1022, , , , Computing performance data obtained from S101 , The topological location obtained for S101 , , Operational status data obtained from S101 .

[0056] This embodiment obtains a target computing power value that reflects the true usable capability of the accelerator card, and then aggregates the target computing power value with multi-dimensional fields such as memory capacity, interconnect bandwidth, and topology location into a unified resource descriptor, so as to accurately characterize the capability differences of accelerator cards of different architectures and generations.

[0057] S103. Based on resource descriptors, construct a topology graph with heterogeneous nodes as nodes, communication links between nodes as edges, and communication overhead as edge weights. Calculate the score of each heterogeneous node based on the topology graph using a preset multidimensional scoring function.

[0058] Optionally, step S103, based on resource descriptors, constructs a topology graph with heterogeneous nodes as nodes, communication links between nodes as edges, and communication overhead as edge weights. The process of calculating the score of each heterogeneous node based on the topology graph using a preset multi-dimensional scoring function may specifically include: S1031. The communication overhead of each communication link is obtained by weighted summing the reciprocal of the interconnection bandwidth in the resource descriptor with the transmission hop count and transmission delay of the corresponding communication link. A topology graph is constructed with heterogeneous nodes as nodes, communication links between nodes as edges, and communication overhead as edge weights.

[0059] Communication links refer to the physical or logical connection paths used for data transmission between heterogeneous nodes in a cluster, and include three types: high-speed interconnect links between accelerator cards within a node, peripheral expansion bus links between processors and accelerator cards within a node, and network exchange links between nodes.

[0060] Hop count refers to the number of network forwarding nodes that data passes through between two heterogeneous nodes and reflects the length of the communication path. Transmission delay. This refers to the end-to-end time, measured in milliseconds, required for data to travel from the source node to the target node via a communication link. Communication overhead is a quantified value that comprehensively reflects the communication cost between two heterogeneous nodes.

[0061] During the implementation of the solution, following the resource descriptors obtained in S1023, the management node first retrieves the resource descriptors from each heterogeneous node. Extracting interconnect bandwidth and combined with topological location Leaf switch number in With spine switch number Calculate the number of hops between any two nodes If both nodes are accelerator cards spanning the PCIe bus within the same computing device, then If the two nodes belong to the same leaf switch, then If they belong to different leaf switches under the same spine switch, then If the switches belong to different spine switches across network domains, then the upper limit value is fixed. .

[0062] Then for each pair of nodes Calculate communication overhead ,in This refers to the network interconnection bandwidth between nodes, and for communication links within the same node, it is adaptively replaced with the aforementioned internal interconnection bandwidth. , For transmission delay, It is a minimum-max normalization function used to uniformly map the inverse of bandwidth, integer hop count, and millisecond-level latency, which have absolute physical dimensions, to... Within the dimensionless interval, , , These are the preset weight coefficients corresponding to the above normalized values, and the sum of the three is 1.

[0063] Finally, a weighted undirected graph is constructed using the heterogeneous nodes of the entire cluster as vertices, the communication links between nodes as edges, and the communication cost as the edge weight. ,in For a set of nodes, For a set of communication links, This is a set of communication overhead. Figure 3 A schematic diagram of the topology provided in one embodiment of this application is shown. Figure 3 As shown, this topology graph is constructed with heterogeneous nodes as vertices, communication links between nodes as edges, and the communication cost of each communication link as the weight of the corresponding edge. The first heterogeneous node in the graph... Second heterogeneous node Third heterogeneous node and the fourth heterogeneous node Corresponding to the set of nodes respectively heterogeneous nodes in , , , .

[0064] First communication link Second communication link Third communication link and the fourth communication link Corresponding to the edge set Middle connection and , and , and , and The first communication link Second communication link Third communication link and the fourth communication link The edge weights are respectively the communication overhead sets. In , , , Each communication overhead is calculated as follows: Calculated.

[0065] In this embodiment, the first heterogeneous node With the second heterogeneous node The number of hops between two switches belonging to the same leaf switch First communication link edge weight The value with the smallest value and the shortest corresponding side length is found in the graph; the third heterogeneous node. With the first heterogeneous node Second heterogeneous node The number of transmission hops for different leaf switches belonging to the same spine switch Second communication link edge weight and third communication link edge weight The value is centered in the graph and the corresponding side length is centered as well.

[0066] Fourth heterogeneous node With the third heterogeneous node The number of transmission hops corresponding to different spine switches Take the larger value, fourth communication link edge weight The node with the largest value and the longest corresponding side in the graph indicates the third heterogeneous node. With the fourth heterogeneous node The communication cost between cross-spine switches is the highest in the graph. Based on this, the management node labels the communication cost of each pair of connected heterogeneous nodes in the graph as the weight of the corresponding edge, resulting in the following: Figure 3 The topology diagram shown .

[0067] S1032. Summing up the communication overhead of all communication links connected to each heterogeneous node in the topology graph to obtain the communication overhead component, obtaining the resource balancing component based on the ratio of the allocated resources of each heterogeneous node to the total allocated resources, and calculating the ratio of the remaining resources of each heterogeneous node after executing the target task to the total remaining resources to obtain the resource penalty component.

[0068] Communication overhead components This refers to the sum of communication overhead of the communication links between a heterogeneous node and all other nodes in the topology diagram, and is used to reflect the total communication burden of that node in the cluster network. The smaller the value, the lower the communication cost between the node and other nodes in the topology, and the more suitable it is for carrying communication-intensive tasks.

[0069] Resource Equalization It refers to the ratio of the allocated resources of a heterogeneous node to the total allocated resources of the cluster, and is used to measure the degree of balance in the current resource allocation. The closer the distribution is to the average value of each node, the more balanced the allocation. Resource penalty component. It refers to the ratio of the remaining resources of a heterogeneous node after undertaking the target task to the total remaining resources of the cluster, and is used to quantify the degree of impact on cluster resource fragmentation after undertaking the target task.

[0070] During the implementation of the scheme, the topology diagram constructed following S1031 was continued. The management node manages each heterogeneous node. Calculate the three components separately. First, for the nodes... The communication overhead component is obtained by summing the communication overhead of all connected communication links. .

[0071] Next, from the resource descriptor Extract the allocated computing power and allocated GPU memory, calculate the ratio of the node's allocated resources to its total available resources, and take its negative mapping to obtain the resource balancing component. Specifically, if the node's total computing power is capped at... The computing power already allocated to it is Then the computing power distribution balance is The same process is applied to the memory, and then the arithmetic mean is taken.

[0072] Finally, from the attribute data Extract the required computing power With demand for video memory compute nodes The remaining computing power and GPU memory after fulfilling the target task are used to calculate the resource penalty component. Considering that in the optimal adaptation strategy for cluster scheduling, in order to prevent the fragmentation of complete ultra-large computing power nodes, the smaller the remaining space, the higher the degree of task fit with the gaps of that node.

[0073] Specifically, if node The currently available computing power is Available video memory is Then the remaining computing power after bearing the load is The remaining video memory is Calculate the ratio of the remaining value to the original total idle state of the node, and apply the inverse mapping function to obtain... .

[0074] S1033. Substitute the communication overhead component, resource balancing component, and resource penalty component into the multidimensional scoring function for weighted summation to obtain the score of each heterogeneous node.

[0075] Multidimensional scoring function It refers to a function that takes communication overhead component, resource balancing component and resource penalty component as input, assigns preset weights to each component and then sums them to output a comprehensive score for each heterogeneous node.

[0076] Because each component has a rigorous dimension normalization and negative transformation design, a higher score indicates that the node has a lower overall communication cost in the cluster topology, a more balanced distribution of its internal resources, and effectively avoids the fragmentation of large resources after undertaking the target task. In other words, the node with the highest overall score is the most suitable as the execution node for this task. The calculation formula for the multidimensional scoring function is as follows: .

[0077] in The nodes calculated for S1032 The communication overhead component, To balance the distribution of resources, As a resource penalty weight, , , These are the preset weight coefficients for the three components and satisfy the following conditions: .

[0078] , , The value of reflects the relative importance of different dimensions in the current task queue, and can be configured differently according to the task queue type: for communication-intensive large-scale training tasks, it can be appropriately increased. The proportion of [something] should be prioritized to ensure communication efficiency; for online inference tasks, it can be appropriately increased. The weighting is adjusted to ensure a balanced allocation of resources, and the weighting configuration can be dynamically adjusted through a configuration file without restarting the core process.

[0079] During the implementation of the solution, the first step is to select the weight combination corresponding to the current large-scale training task queue. ,satisfy Then the management node will manage each node. corresponding Substituting into the multidimensional scoring function, each node is calculated. Score .

[0080] This embodiment constructs a topology graph reflecting the connectivity of the cluster's physical network, precisely scheduling communication-intensive training tasks to nodes with the lowest communication costs and the most balanced resource allocation. This not only effectively reduces cross-network domain communication overhead but also improves the execution efficiency of large-scale training tasks.

[0081] S104. Input the task fingerprint data determined by hashing the attribute data into the trained decision tree model to obtain the predicted runtime.

[0082] Optionally, the attribute data in step S104 includes the container image identifier, command parameters, dataset identifier, and device type identifier of the target task. The trained decision tree model is trained based on a gradient boosting mechanism using training samples including historical task fingerprint data and corresponding real historical runtime.

[0083] Step S104, which involves inputting the task fingerprint data determined by hashing the attribute data into the trained decision tree model to obtain the predicted runtime, can specifically include: S1041. By hashing and encoding the container image identifier, command parameters, dataset identifier, and device type identifier respectively, and then combining them, the task fingerprint data is obtained.

[0084] Task fingerprint data Task fingerprint data refers to a fixed-length vector obtained by hashing and encoding the identifiers of each dimension in the attribute data of the target task, and combining them to uniquely represent the combination of operational features of the target task. The core function of task fingerprint data is to uniformly map attribute fields of different formats and lengths into numerical vectors of equal length so that decision tree models can process orchestration requests from different types of target tasks with a unified input format.

[0085] Hash encoding refers to the process of mapping a string-type identifier field to a fixed-width integer value using a hash function. The same input consistently produces the same output, and fingerprint data for similar tasks is consistent and reproducible. In this embodiment, to ensure computational efficiency during high-frequency scheduling and maintain an extremely low hash collision rate, a hash algorithm with a category-based bucketing mechanism or support for locally sensitive features, such as the structured modified MurmurHash3, is used for feature mapping.

[0086] During the implementation of the solution, the attribute data extracted from the orchestration request by S101 is continued. The management node first identifies the container image. Command parameters Dataset identifier Equipment type identification Each of the four fields is hashed.

[0087] Specifically, the MurmurHash3 preset hash function is called for each field to map the field content to... The bit-width unsigned integer hash value is obtained as follows: , , , ,in A hash value for identifying the container image. The hash value of the command parameter. The hash value that identifies the dataset. The hash value identifies the device type. Then, the four hash values ​​are concatenated and combined in a fixed order to obtain the task fingerprint data of the target task. .

[0088] S1042. Input the task fingerprint data into the multiple decision trees included in the decision tree model. Starting from the root node of each decision tree, compare the feature value of each dimension in the task fingerprint data with the splitting threshold of the corresponding traversed node and select branches layer by layer until the leaf node is reached. Extract the prediction increment value of the leaf node and accumulate all prediction increment values ​​to obtain the prediction runtime.

[0089] Decision tree model This refers to an ensemble prediction model trained based on a gradient boosting mechanism, which is composed of multiple decision trees sequentially combined. Let the decision tree be denoted as Training samples This refers to a data pair consisting of historical task fingerprint data and the corresponding actual historical runtime, where... For the first Task fingerprint data of historical tasks, Its corresponding actual historical runtime, This represents the total number of training samples.

[0090] The split threshold refers to the feature value determined by minimizing the residual during the training phase of each intermediate node in the decision tree to judge the boundary, and is used to guide the selection of the left or right branch layer by layer in the inference phase based on the task fingerprint data. Predicted incremental value. It refers to the amount of correction to the current cumulative prediction value stored in the leaf nodes of a single decision tree, and the sum of the prediction increment values ​​of all decision trees is the final prediction runtime.

[0091] During the training phase, the decision tree model Training samples based on gradient boosting mechanism Sequential iterative training is performed. Specifically, this embodiment uses mean squared error as the loss function. And set the maximum depth of a single decision tree to be The learning rate is The first decision tree Using the initial predicted value as a baseline, the residuals are fitted to the true historical runtime of the training samples to determine the splitting threshold for each intermediate node and the predicted increment value for each leaf node. .

[0092] Each decision tree thereafter before The residual, representing the difference between the cumulative predicted value and the true value of each tree, serves as the learning objective. By minimizing the loss function, the residual error is gradually fitted, and the splitting threshold and leaf node prediction increment value are determined sequentially. The training process involves monitoring the loss function value on independent validation sets. When the validation set loss is continuous... The wheel stops descending or reaches the set maximum number of iterations. The early stop condition is triggered when a tree is planted.

[0093] After iteration, the model The leaf nodes store residual correction values ​​that cover the feature space of historical tasks, enabling the model to locate semantically matching leaf nodes and output accurate prediction increment values ​​for new tasks with similar feature distributions through the layer-by-layer branching of the tree structure.

[0094] During the implementation of the scheme, the task fingerprint data obtained in S1041 is continued. The management node will Input the trained decision tree model sequentially Included A decision tree. First, for the first... Decision Tree Starting from the root node, retrieve the task fingerprint data. The feature value of the current node in the corresponding dimension is compared with the split threshold stored in that node: if the feature value is less than the split threshold, the node proceeds to the left child node; otherwise, it proceeds to the right child node. This process is repeated layer by layer downwards until a leaf node is reached, at which point the predicted increment value stored in that leaf node is extracted. .

[0095] Subsequently Repeat the above traversal process for each decision tree to obtain a sequence of predicted increment values. Next, all predicted increments are summed to obtain the predicted runtime of the target task. This embodiment effectively reduces cross-network domain communication overhead, achieving a synergistic improvement in the efficient utilization of heterogeneous computing resources and the accuracy of task orchestration.

[0096] S105. Calculate the preemption cost based on the predicted runtime and checkpoint data of the current task before the peak period determined by historical load data, and determine the preempted task and corresponding resource reclamation instruction in the current task according to the preemption cost and the preset threshold range.

[0097] Optionally, step S105, which calculates the preemption cost based on the predicted runtime and checkpoint data of the current task before the peak period determined by historical load data, and determines the preempted task and corresponding resource reclamation instruction in the current task based on the preemption cost and a preset threshold range, may specifically include: S1051. Calculate the average load of historical load data in each time period within a preset time window, estimate the predicted load of future time periods based on the growth rate of the average load, and determine the future time period when the predicted load is greater than the preset load threshold as the peak period.

[0098] Average load This refers to the average number of queries per second in historical load data within a preset time window, used to smooth out short-term fluctuations in historical traffic sequences and reflect the typical business load level during that period. Load growth rate It refers to the rate of change between the average load in adjacent time periods and is used to depict the overall growth trend of business traffic.

[0099] Predicted load This refers to the estimated number of queries for a future time period, extrapolated from historical average load and load growth rate. For predicting the time step. Peak period. This refers to the future time period during which the predicted load exceeds a preset load threshold. The preset load threshold can be configured differently based on the service type, as shown in Table 3 below: Table 3: Preset Load Threshold Comparison Table As shown in Table 3, the preset load thresholds for the three types of services meet the requirements. The higher the threshold, the more likely the service will trigger peak judgment under higher traffic pressure. In actual deployment, it can be injected through configuration file in combination with historical traffic baseline. The management node can take effect after reading it during the startup phase.

[0100] During the implementation of the solution, the historical load data obtained by S101 will be continued. The management node first calculates the average load for each time period within the preset time window. That is, the first one in the window The average number of queries per second over a given time period is taken.

[0101] Next, the absolute change in average load over adjacent time periods, i.e., the load increment rate per unit time step, is calculated and its moving average is taken within the window to obtain the load increment rate. Based on the average load over the most recent period Starting from, according to Linear extrapolation is performed to obtain the predicted load sequence for each future time step. Finally, the corresponding preset load threshold is obtained from Table 3 based on the current business type. The predicted load sequence will satisfy the following conditions: The earliest time step was determined as the peak period. start time .

[0102] S1052. When the current time is earlier than the start time of the peak period, the remaining runtime is obtained by the difference between the predicted runtime and the cumulative runtime of the current task from its start to the current time. The checkpoint validity is obtained by calculating the reciprocal of the time interval between the current time and the generation time of the latest checkpoint in the checkpoint data.

[0103] Cumulative runtime This refers to the length of time the current task has been running since its start time, and is used to reflect the execution progress of the current task. Remaining runtime It refers to the difference between the predicted runtime and the cumulative runtime, and is used to estimate the resource time still required for the current task to complete. The larger the value, the greater the loss of computational progress caused by interrupting the task.

[0104] Checkpoint data refers to the intermediate execution status records periodically saved during the current task's operation, including the generation time of the latest checkpoint. This includes the corresponding task execution progress information and is used to support resumption of execution from the most recent checkpoint after a task is preempted. Checkpoint validity. It refers to a quantitative value that reflects the freshness of the latest checkpoint from the current time. The larger the value, the closer the latest checkpoint is to the current moment, meaning that if a task is preempted, it can resume from a newer progress and the loss from interruption is smaller.

[0105] During the implementation of the plan, the peak period start time determined by S1051 will be continued. The management node first determines the current time. Is it earlier than Only The calculation of subsequent preemption costs is triggered at that time. For each current task, the management node reads the task's start time. Calculate cumulative runtime .

[0106] Next, a zero lower limit is set to obtain the remaining runtime. Subsequently, the generation time of the latest checkpoint is extracted from the checkpoint data of the current task. Calculate the time interval Take its reciprocal to obtain the checkpoint validity. .

[0107] S1053. The remaining runtime and checkpoint timeliness are weighted and summed to obtain the preemption cost of the current task. The current task whose preemption cost is within the preset threshold range is identified as the preempted task, and a resource reclamation instruction is generated.

[0108] Cost of seizing This refers to a quantitative value that comprehensively reflects the computational progress loss and checkpoint recovery cost caused by preempting a current task. The smaller the value, the lower the overall cost of preempting the task, meaning the task has a shorter remaining execution time and the latest checkpoint is closer to the current time, allowing for resource recovery with minimal progress loss after interruption.

[0109] Resource reclamation commands refer to control signals issued by the management node to the heterogeneous node where the preempted task resides, including commands to terminate the task process and clear occupied storage space. The preset threshold range refers to the range of preemption costs used to filter preempted tasks, and can be configured differently according to the urgency of resource reclamation, as shown in Table 4 below: Table 4: Preset Threshold Range Comparison Table As shown in Table 4, the upper bound of the threshold intervals mentioned above is determined by statistically extracting percentiles from the dimensionless cost distribution calculated from a massive number of interrupted tasks in history, which are then normalized to the distribution scale scalars at the 30th, 60th, and 100th percentiles of the cost value. The smallest upper bound of the threshold interval corresponding to high urgency recovery indicates that only tasks with extremely low preemption costs are allowed when the peak period is approaching, to ensure the timeliness of resource recovery; the threshold interval corresponding to low urgency recovery is wider, allowing tasks with moderate costs to be included in the preemption candidate set.

[0110] During the implementation of the scheme, the remaining runtime of each current task calculated in S1052 is used as a continuation. With checkpoint validity The management node first uses a preset weighting coefficient corresponding to the remaining runtime. Preset weighting coefficients corresponding to the timeliness of checkpoints Calculate the preemption cost for each current task. Obtain the preemption cost sequence for all current tasks. ,in This represents the total number of tasks currently in progress.

[0111] Next, based on the current urgency of resource recovery, the corresponding preset threshold range is obtained from Table 4. Select tasks from all current tasks that meet the requirements, in ascending order of preemption cost. The current task is used as the set of preempted tasks. Finally, regarding Each preempted task generates a resource reclamation instruction, including instructions to terminate the task process and clear the occupied storage space. These instructions are sent to the corresponding heterogeneous nodes via the communication channel for execution, thus completing the pre-release of resources before the peak period.

[0112] This embodiment identifies peak traffic in advance, accurately selects the preempted tasks with the least interruption loss, and completes resource recovery in advance. This ensures that online inference tasks can obtain sufficient computing power and storage resources during peak traffic periods, effectively guaranteeing the response latency stability of online inference services.

[0113] S106. After the heterogeneous node corresponding to the preempted task responds to the resource reclamation command and releases resources, obtain the updated resource descriptor and extract the available resource capacity.

[0114] Optionally, step S106, after the heterogeneous node corresponding to the preempted task responds to the resource reclamation command and releases resources, the process of obtaining the updated resource descriptor and extracting the available resource capacity may specifically include: S1061. After the heterogeneous node corresponding to the preempted task responds to the resource reclamation command and releases resources, the real-time utilization rate, device temperature and error count of each heterogeneous node are reacquired.

[0115] The resource release completion signal is a status confirmation message returned to the management node via the communication channel after the heterogeneous node where the preempted task is located has executed the resource reclamation instruction, that is, completed the termination of the task process and cleared the occupied storage space. It is used to notify the management node that the physical computing power and video memory resources of the node have been restored to a usable state.

[0116] After receiving the resource release completion signal, the management node needs to re-collect the latest operating status data of the node to ensure that the real-time utilization rate is used as the basis for subsequent resource descriptor updates. Equipment temperature and error count It reflects the actual operational status after resource recycling.

[0117] During the implementation of the solution, following the resource reclamation command issued by S1053, the management node continuously monitors the resource release completion signals reported by each heterogeneous node through the communication channel. First, after receiving the resource release completion signal returned by the heterogeneous node corresponding to the preempted task, the management node triggers a process to re-collect the running status data of all heterogeneous nodes in the entire cluster.

[0118] Specifically, upon receiving a re-collection request, the local adaptation plugin of each heterogeneous node immediately reads the current real-time operating metrics of the accelerator card and pushes them to the management node via the communication channel, thus obtaining the data for each heterogeneous node. Latest real-time utilization rate after resource release Equipment temperature and error count .

[0119] S1062. Update the values ​​of the corresponding fields in the resource descriptor according to the real-time utilization, equipment temperature and error count to obtain the updated resource descriptor for each heterogeneous node.

[0120] Updated resource descriptor This refers to recalculating the health coefficient, utilization coefficient, and target computing power value based on the operational status data re-collected in S1061, following the calculation process from S1021 to S1023, and replacing the resource descriptor with the updated values. The new version resource descriptor is obtained after the corresponding field. The built-in version number field of the resource descriptor is updated atomically during each update for consistency verification during concurrent reads and writes.

[0121] During the implementation of the plan, data was re-collected from S1061. , , The management node first calculates according to the method in S1021, based on the node... The type of accelerator card used can be found in Table 1. , Recalculate the first loss amount Second loss amount And combined with the preset benchmark values ​​obtained from Table 2 Calculate the updated health coefficient .

[0122] Next, following the calculation method in S1022, the latest sampled value is included within the preset sliding time window. The average of the real-time utilization series is taken to obtain the updated utilization coefficient. Then calculate the updated target computing power value. .

[0123] Finally, , , , Replace resource descriptor The corresponding field, the rest of the fields , , , , Keeping the resource descriptors unchanged, we obtain the updated resource descriptors for each heterogeneous node. The version number field is then atomically incremented for use in step S1063.

[0124] S1063. Extract the target computing power value and video memory capacity from the updated resource descriptor. Obtain the available computing power based on the difference between the target computing power value and the current allocated computing power. Obtain the available video memory based on the difference between the video memory capacity and the current allocated video memory. Summarize the available computing power and available video memory to the available resource capacity of each heterogeneous node.

[0125] Available computing power This refers to the difference between the updated target computing power and the allocated computing power of a heterogeneous node at the current moment, and is used to reflect the actual computing power surplus available for new tasks after resource reclamation. Available GPU memory. It refers to the difference between the video memory capacity of a heterogeneous node and the allocated video memory, and is used to reflect the remaining video memory available for new tasks on that node.

[0126] Allocated computing power This refers to the total computing power currently used by all running tasks on this node at the current moment. Allocated GPU memory. This refers to the total video memory currently used by all running tasks on this node at the current moment. Available resource capacity. This refers to summarizing available computing power and available video memory to describe the current resource status of a heterogeneous node that is available for the target task.

[0127] During the implementation of the scheme, the updated resource descriptor obtained from S1062 is used. The management node first obtains data from each heterogeneous node. Extract the updated target computing power value With video memory capacity Next, the nodes are read from the task allocation records maintained by the management node. Current allocated computing power With allocated video memory Calculate the available computing power separately With available video memory Finally, and Summarized into nodes Available resource capacity .

[0128] This embodiment obtains available computing power and available video memory by subtracting the allocated computing power and allocated video memory from the updated target computing power value and video memory capacity, respectively, to ensure that the available resource capacity on which subsequent execution nodes are determined accurately reflects the real state after resource reclamation.

[0129] S107. Based on attribute data, available resource capacity, and score, determine the execution node among heterogeneous nodes, and send a start command to the execution node to execute the target task.

[0130] Optionally, the attribute data in step S107 may also include the target task’s required computing power and required video memory, and the available resource capacity may include the available computing power and available video memory of each heterogeneous node.

[0131] Step S107 determines the execution node among heterogeneous nodes based on attribute data, available resource capacity, and score, and sends a start command to the execution node to execute the target task, including: S1071. Select heterogeneous nodes whose available computing power in the available resource capacity is greater than or equal to the required computing power and whose available video memory in the available resource capacity is greater than or equal to the required video memory as candidate nodes.

[0132] Candidate nodes refer to heterogeneous nodes in the entire cluster whose available computing power and available GPU memory are not less than the target task's required computing power and GPU memory requirements. Required computing power With demand for video memory Attribute data obtained from S101 The corresponding fields in the table represent the minimum computing power and minimum video memory required to run the target task.

[0133] During the implementation of the scheme, the available resource capacity of each heterogeneous node obtained in S1063 is combined with the attribute data extracted in S101. The management node extracts the required computing power from the attribute data. With demand for video memory traverse each heterogeneous node Available resource capacity Determine whether the conditions are met simultaneously. and Heterogeneous nodes that meet the conditions will be included in the candidate node set. .

[0134] S1072. Determine the candidate node with the highest score from all candidate nodes as the execution node, encapsulate the container image identifier, command parameters, dataset identifier, required computing power, and required video memory into a startup instruction, and send the startup instruction to the execution node to execute the target task.

[0135] Execution Node This refers to the heterogeneous node with the highest overall score selected from the candidate node set based on the score calculated according to S1033, which will ultimately be used to carry out the execution of the target task. The startup command is a control message issued by the management node to the execution node to start the target task's runtime environment and initiate the task process, including the container image identifier. Command parameters Dataset identifier Required computing power and demand for video memory The five parameters and the execution node complete the runtime environment initialization and task process startup according to the startup command.

[0136] During the implementation of the scheme, the candidate node set obtained from S1071 is followed by... Based on the scores of each node calculated in S1033, the management node first... Retrieve the score corresponding to each candidate node in the middle. Sort the nodes by score from highest to lowest, and select the candidate node with the highest score as the execution node. .

[0137] Next, the management node retrieves the attribute data. Extract , , , , Five parameters are encapsulated into a startup command. via communication channel to the execution node Send start command Execution node After receiving the startup command, according to Pull the runtime environment image, according to Start the task process, according to and Complete the isolation and allocation of computing power and GPU memory, and complete the closed loop of orchestration for the target task.

[0138] This embodiment filters out a set of candidate nodes that simultaneously meet the resource supply conditions, and then selects the node with the highest score in the multi-dimensional scoring function from the candidate nodes as the execution node. The container image identifier, command parameters, dataset identifier, required computing power, and required video memory are encapsulated into a startup instruction and issued, realizing a complete orchestration closed loop from resource sufficiency verification to topology optimal node selection to task startup.

[0139] Figure 4 This is a schematic diagram illustrating a specific implementation of a task orchestration system for a management node of a heterogeneous computing power scheduling system provided in this application. (Refer to...) Figure 4 The system may include: The acquisition module 410 is used to synchronously acquire computing power performance data, topology location and running status data of heterogeneous nodes, historical load data of the current task and attribute data of the target task; Module 420 is used to construct a set of multi-dimensional resource descriptors based on computing power performance data, topology location, and operating status data; The construction module 420 is also used to construct a topology graph based on resource descriptors, with heterogeneous nodes as nodes, communication links between nodes as edges, and communication overhead as edge weights, and to calculate the score of each heterogeneous node based on the topology graph using a preset multidimensional scoring function. The generation module 430 is used to input the task fingerprint data determined by hashing based on the attribute data into the trained decision tree model to obtain the predicted runtime. The generation module 430 is also used to calculate the preemption cost based on the predicted runtime and the checkpoint data of the current task before the peak period determined by historical load data, and to determine the preempted task and the corresponding resource reclamation instruction in the current task according to the preemption cost and the preset threshold range. The generation module 430 is also used to obtain the updated resource descriptor and extract the available resource capacity after the heterogeneous node corresponding to the preempted task responds to the resource reclamation instruction and releases the resources. The sending module 440 is used to determine the execution node among the heterogeneous nodes based on attribute data, available resource capacity and score, and send a start command to the execution node to execute the target task.

[0140] The management node task orchestration system of the heterogeneous computing power scheduling system in this application embodiment is used to implement the aforementioned management node task orchestration method of the heterogeneous computing power scheduling system. Therefore, the specific implementation of the management node task orchestration system of the heterogeneous computing power scheduling system can be found in the embodiment section of the management node task orchestration method of the heterogeneous computing power scheduling system above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0141] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application is shown.

[0142] The electronic device may include a processor 510 and a memory 520 storing computer program instructions.

[0143] Specifically, the processor 510 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0144] Memory 520 may include mass storage for data or instructions. For example, and not limitingly, memory 520 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 520 may include removable or non-removable (or fixed) media. Where appropriate, memory 520 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 520 is non-volatile solid-state memory.

[0145] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this disclosure.

[0146] The processor 510 reads and executes computer program instructions stored in the memory 520 to implement any of the management node task orchestration methods of the heterogeneous computing power scheduling system in the above embodiments.

[0147] In one example, the electronic device may also include a communication interface 530 and a bus 540. Wherein, such as Figure 5 As shown, the processor 510, memory 520, and communication interface 530 are connected through bus 540 and complete communication with each other.

[0148] The communication interface 530 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0149] Bus 540 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 540 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0150] The electronic device can execute the management node task orchestration method of the heterogeneous computing power scheduling system in the embodiments of this application, thereby realizing the management node task orchestration method of the heterogeneous computing power scheduling system described in conjunction with the accompanying drawings.

[0151] Furthermore, in conjunction with the management node task orchestration method of the heterogeneous computing power scheduling system in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the management node task orchestration methods of the heterogeneous computing power scheduling system in the above embodiments.

[0152] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0153] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0154] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0155] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0156] The foregoing has provided a detailed description of the task orchestration method, apparatus, and storage medium for a heterogeneous computing power scheduling system's management nodes. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of these embodiments are merely illustrative of the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for task orchestration of management nodes in a heterogeneous computing power scheduling system, characterized in that, include: Synchronously acquire computing power performance data, topology location and running status data of heterogeneous nodes, historical load data of the current task and attribute data of the target task; Based on the computing power performance data, the topology location, and the operating status data, a set of multi-dimensional resource descriptors is constructed; Based on the resource descriptor, a topology graph is constructed with the heterogeneous nodes as nodes, the communication links between nodes as edges, and the communication overhead as the edge weights. Based on the topology graph, a preset multidimensional scoring function is used to calculate the score of each heterogeneous node. The task fingerprint data, determined by hashing the attribute data, is input into the trained decision tree model to obtain the predicted runtime. Before the peak period determined by the historical load data, the preemption cost is calculated based on the predicted runtime and the checkpoint data of the current task. Based on the preemption cost and the preset threshold range, the preempted task and the corresponding resource reclamation instruction are determined in the current task. After the heterogeneous node corresponding to the preempted task responds to the resource reclamation command and releases resources, the updated resource descriptor is obtained and the available resource capacity is extracted. Based on the attribute data, the available resource capacity, and the score, an execution node is determined among the heterogeneous nodes, and a start command for executing the target task is sent to the execution node.

2. The method according to claim 1, characterized in that, The computing power performance data includes the original computing power value, precision weighting coefficient, video memory capacity, video memory bandwidth, interconnect bandwidth and instruction compatibility identifier of each heterogeneous node; the operating status data includes the real-time utilization rate, device temperature and error count of each heterogeneous node. Based on the computing power performance data, the topology location, and the operating status data, a set of multi-dimensional resource descriptors is constructed, including: The first loss amount is obtained by multiplying the equipment temperature by the preset temperature loss coefficient; the second loss amount is obtained by multiplying the error count by the preset count loss coefficient; and the health coefficient is obtained by calculating the difference between the preset benchmark value and the first and second loss amounts. The utilization rate coefficient is obtained by calculating the average value of the real-time utilization rate within the preset sliding time window. The target computing power value is obtained by multiplying the original computing power value, the precision weight coefficient, the utilization rate coefficient, and the health coefficient. Based on the target computing power, the video memory capacity, the video memory bandwidth, the interconnect bandwidth, the instruction compatibility identifier, the topology location, the real-time utilization rate, the device temperature, and the error count, a resource descriptor for each heterogeneous node is constructed.

3. The method according to claim 2, characterized in that, The process involves constructing a topology graph based on the resource descriptor, using heterogeneous nodes as nodes, communication links between nodes as edges, and communication overhead as edge weights. Then, based on this topology graph, a pre-defined multi-dimensional scoring function is used to calculate the score for each heterogeneous node, including: The communication overhead of each communication link is obtained by weighted summing the reciprocal of the interconnect bandwidth in the resource descriptor with the transmission hop count and transmission delay of the corresponding communication link. The topology graph is constructed with the heterogeneous nodes as nodes, the communication links between nodes as edges, and the communication overhead as edge weights. The communication overhead component is obtained by summing the communication overhead of all communication links connected to each heterogeneous node in the topology graph. The resource balancing component is obtained by calculating the ratio of the allocated resources of each heterogeneous node to the total allocated resources. The resource penalty component is obtained by calculating the ratio of the remaining resources of each heterogeneous node after executing the target task to the total remaining resources. The communication overhead component, the resource balancing component, and the resource penalty component are substituted into the multidimensional scoring function for weighted summation to obtain the score of each heterogeneous node.

4. The method according to claim 1, characterized in that, The attribute data includes the container image identifier, command parameters, dataset identifier, and device type identifier of the target task. The trained decision tree model is trained based on a gradient boosting mechanism using training samples including historical task fingerprint data and corresponding real historical runtime. The step of inputting the task fingerprint data determined by hashing the attribute data into the trained decision tree model to obtain the predicted runtime includes: Task fingerprint data is obtained by hashing and encoding the container image identifier, the command parameters, the dataset identifier, and the device type identifier, and then combining them. The task fingerprint data is input into multiple decision trees included in the decision tree model. Starting from the root node of each decision tree, the feature value of each dimension in the task fingerprint data is compared with the split threshold of the corresponding traversed node, and branches are selected layer by layer until the leaf node is reached. The prediction increment value of the leaf node is extracted, and all the prediction increment values ​​are accumulated to obtain the prediction runtime.

5. The method according to claim 1, characterized in that, The step of calculating the preemption cost based on the predicted runtime and checkpoint data of the current task before the peak period determined by the historical load data, and determining the preempted task and corresponding resource reclamation instruction in the current task according to the preemption cost and a preset threshold range, includes: Calculate the average load of the historical load data for each time period within a preset time window, and estimate the predicted load for future time periods based on the growth rate of the average load. Determine the future time periods where the predicted load is greater than a preset load threshold as peak periods. When the current time is earlier than the start time of the peak period, the remaining runtime is obtained by the difference between the predicted runtime and the cumulative runtime of the current task from its start to the current time. The checkpoint validity is obtained by calculating the reciprocal of the time interval between the current time and the generation time of the latest checkpoint in the checkpoint data. The remaining runtime and the checkpoint timeliness are weighted and summed to obtain the preemption cost of the current task. From all the current tasks, the current task whose preemption cost is within the preset threshold range is identified as the preempted task, and the resource reclamation instruction is generated.

6. The method according to claim 2, characterized in that, After the heterogeneous node corresponding to the preempted task responds to the resource reclamation command and releases resources, the process of obtaining the updated resource descriptor and extracting the available resource capacity includes: After the heterogeneous node corresponding to the preempted task responds to the resource reclamation command and releases resources, the real-time utilization rate, the device temperature, and the error count of each heterogeneous node are reacquired. The updated resource descriptor for each heterogeneous node is obtained by updating the values ​​of the corresponding fields in the resource descriptor based on the real-time utilization, the device temperature, and the error count. Extract the target computing power value and the video memory capacity from the updated resource descriptor, obtain the available computing power based on the difference between the target computing power value and the current allocated computing power, obtain the available video memory based on the difference between the video memory capacity and the current allocated video memory, and summarize the available computing power and the available video memory into the available resource capacity of each heterogeneous node.

7. The method according to claim 4, characterized in that, The attribute data also includes the computing power and video memory requirements of the target task, and the available resource capacity includes the available computing power and available video memory of each heterogeneous node; The process of determining an execution node among the heterogeneous nodes based on the attribute data, the available resource capacity, and the score, and sending a start command to the execution node for executing the target task, includes: The heterogeneous nodes whose available computing power in the available resource capacity is greater than or equal to the required computing power and whose available video memory in the available resource capacity is greater than or equal to the required video memory are identified as candidate nodes; The candidate node with the highest score among all the candidate nodes is determined as the execution node. The container image identifier, the command parameters, the dataset identifier, the required computing power, and the required video memory are encapsulated into the startup instruction and sent to the execution node to execute the target task.

8. A task orchestration system for the management node of a heterogeneous computing power scheduling system, characterized in that, include: The acquisition module is used to synchronously acquire computing power performance data, topology location and running status data of heterogeneous nodes, historical load data of the current task and attribute data of the target task; A construction module is used to construct a set of multi-dimensional resource descriptors based on the computing power performance data, the topology location, and the operating status data; The construction module is also used to construct a topology graph based on the resource descriptor, with the heterogeneous nodes as nodes, the communication links between nodes as edges, and the communication overhead as the edge weights, and to calculate the score of each heterogeneous node based on the topology graph using a preset multidimensional scoring function. The generation module is used to input the task fingerprint data determined by hashing the attribute data into the trained decision tree model to obtain the predicted runtime. The generation module is also used to calculate the preemption cost based on the predicted runtime and the checkpoint data of the current task before the peak period determined according to the historical load data, and to determine the preempted task and the corresponding resource reclamation instruction in the current task according to the preemption cost and the preset threshold range. The generation module is also used to obtain the updated resource descriptor and extract the available resource capacity after the heterogeneous node corresponding to the preempted task responds to the resource reclamation instruction and releases resources; The sending module is used to determine the execution node among the heterogeneous nodes based on the attribute data, the available resource capacity and the score, and send a start command to the execution node for executing the target task.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, used to execute the computer program to implement the steps of the management node task orchestration method of the heterogeneous computing power scheduling system as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the management node task orchestration method of the heterogeneous computing power scheduling system as described in any one of claims 1 to 7.

Images