Heterogeneous resource scheduling method and system and storage medium

Abstract

The invention discloses a heterogeneous resource scheduling method and system and a storage medium, and relates to the technical field of computers. The heterogeneous resource scheduling method comprises the steps of obtaining a plurality of computing power clusters based on a current computing task; obtaining a first computing power cluster from each computing power cluster; for each first computing power cluster, acquiring a corresponding communication efficiency value; obtaining a target computing power cluster based on each communication efficiency value; and based on the target computing power cluster, executing the current computing task. The method has the core advantages that the limitation that the computing power cluster is constructed only through the resource quota in a traditional method is broken through, the physical interconnection topology characteristics of the computing power node bottom layer are fully considered, the cross-device communication bottleneck can be avoided, the linear speed-up ratio reduction risk is further reduced, and the problem of computing power islands is fundamentally solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a heterogeneous resource scheduling method, system, and storage medium. Background Technology

[0002] Federated heterogeneous resource scheduling refers to a core collaborative management and control technology for distributed computing scenarios. Its management targets are computing, storage, and network resources belonging to multiple independent autonomous management domains. Because these resources exhibit significant heterogeneity in computing architecture, hardware type, storage configuration, and network infrastructure, the core objective of federated heterogeneous resource scheduling is to, under the premise of strictly ensuring the autonomy, data security, and compliance requirements of each management domain, schedule various computing tasks to the optimal architecture nodes or cross-domain resources (i.e., computing nodes in the following text) on demand through unified resource abstraction modeling, a global collaborative decision-making mechanism, and dynamic adaptation allocation strategies. Ultimately, this achieves overall optimization of global resource utilization, task execution efficiency, and system reliability.

[0003] However, existing federated heterogeneous resource scheduling methods have the following technical drawbacks: They construct computing clusters using scalar counting of resource quotas, completely neglecting the underlying physical interconnect topology characteristics between computing nodes. This deficiency is particularly prominent in scenarios with extremely high communication efficiency requirements, such as large model training. For example, if the selected computing nodes are physically distributed discretely, requiring communication across high-speed Peripheral Component Interconnect Express (PCIe) switches, the computing cluster's communication bandwidth will be limited by the cross-node network transmission capacity. This will cause the linear speedup ratio of the computing cluster to decrease by more than 50%, severely impacting task execution efficiency and easily leading to the "computing island" problem where resources cannot be efficiently coordinated, thus restricting the overall performance of heterogeneous resources. Summary of the Invention

[0004] The purpose of this application is to provide a heterogeneous resource scheduling method, system, and storage medium to solve the technical problem that existing heterogeneous resource scheduling methods are prone to affecting task execution efficiency due to data transmission efficiency.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] Firstly, this application proposes a technical solution for a heterogeneous resource scheduling method, which is applied to an artificial intelligence computing power service platform, the artificial intelligence computing power service platform including multiple computing power nodes; the heterogeneous resource scheduling method includes: Based on the current computing task, multiple computing power clusters are obtained; each computing power cluster includes multiple first computing power nodes; the first computing power node is any computing power node among the various computing power nodes; A first computing power cluster is obtained from each computing power cluster; the first computing power cluster is the computing power cluster from each computing power cluster for which no corresponding communication efficiency value has been obtained; the communication efficiency value is at least used to characterize the degree of physical connection between each first computing power node in the first computing power cluster; For each first computing power cluster, the corresponding communication efficiency value is obtained; the communication efficiency value corresponds one-to-one with the computing power cluster. Based on various communication efficiency values, obtain the target computing power cluster; Based on the target computing power cluster, execute the current computing task.

[0007] As a specific solution in the technical solution of this application, the step of obtaining the communication efficiency value based on the first computing power cluster includes: Traverse each first computing node in the first computing power cluster to obtain the second computing node and all third computing nodes; the second computing node is any computing node among the first computing nodes that has not obtained the corresponding physical topology value; the third computing node is any computing node among the first computing nodes except the second computing node. For each second computing node, perform the following steps until each second computing node obtains the corresponding communication distance value: Based on the second computing node and each third computing node, obtain the physical topology distance value corresponding to each third computing node; Based on each physical topology distance value, a communication distance value is obtained; the communication distance value is used at least to characterize the physical topology distance between the second computing node and each of the third computing nodes; If each second computing node obtains its corresponding communication distance value, then the communication efficiency value is obtained based on each communication distance value.

[0008] As a specific solution in this application, obtaining the communication efficiency value based on various communication distance values ​​includes: Based on the second computing node and each third computing node, obtain the time consumption quantization value corresponding to each third computing node; the time consumption quantization value is at least used to characterize the time it takes for data to be transmitted from the second computing node to the corresponding third computing node; The communication efficiency value is obtained based on the quantized values ​​of each time consumption and each communication distance value.

[0009] As a specific solution in the technical solution of this application, the time quantification value includes any one or a combination of bandwidth, latency, hop count, signal attenuation rate, non-uniform memory access distance and set communication primitive theory completion time.

[0010] As a specific solution in this application, the communication efficiency value is positively correlated with the degree of physical connection between each first computing node in the first computing power cluster; the step of obtaining the target computing power cluster based on each communication efficiency value includes: Based on each communication efficiency value, a first communication efficiency value is obtained; the first communication efficiency value is the maximum value among all communication efficiency values. Based on the first communication efficiency value, a second computing power cluster is obtained; the second computing power cluster is the computing power cluster corresponding to the first communication efficiency value. The second computing power cluster is used as the target computing power cluster.

[0011] As a specific solution in this application, the communication efficiency value is positively correlated with the degree of physical connection between each first computing power node in the first computing power cluster; each computing power cluster includes a corresponding computing power pointer, which at least indicates whether the corresponding computing power cluster is busy or idle; obtaining the target computing power cluster based on each communication efficiency value includes: Multiple second communication efficiency values ​​are obtained from each communication efficiency value; the second communication efficiency value is the communication efficiency value corresponding to any idle computing cluster in each computing cluster; A third communication efficiency value is obtained based on each of the second communication efficiency values; the third communication efficiency value is the maximum value among the second communication efficiency values. Based on the third communication efficiency value, a third computing power cluster is obtained; the third computing power cluster is the computing power cluster corresponding to the third communication efficiency value. If the computing power pointer status corresponding to the third computing power cluster is idle, then the third computing power cluster is used as the target computing power cluster, and the computing power pointer status of the third computing power cluster is adjusted to busy. If the computing power pointer status corresponding to the third computing power cluster is busy, then multiple new second communication efficiency values ​​are obtained again until the target computing power cluster is obtained.

[0012] As a specific solution in this application, the step of executing the current computing task based on the target computing power cluster includes: Traverse all execution instructions contained in the current computing task to obtain the first instruction to be executed; the first instruction is any instruction required to complete the current computing task. For each first instruction, the following steps are performed until all execution instructions contained in the current computing task have been executed: The first instruction is mapped to the second instruction based on a unified heterogeneous instruction set; the second instruction is a vendor-specific driver instruction corresponding to the fourth computing power node; the fourth computing power node is a computing power node in the target computing power cluster configured to execute the first instruction; The second instruction is sent to the fourth computing node to drive the fourth computing node to execute the corresponding operation in the second instruction in order to complete the execution of the current computing task.

[0013] As a specific solution in this application, after executing the current computing task based on the target computing power cluster, the method further includes: Based on the target computing power cluster, a fourth communication efficiency value and a fifth communication efficiency value are obtained; the fourth communication efficiency value is the communication efficiency value of the target computing power cluster before the current computing task is executed; the fifth communication efficiency value is the communication efficiency value of the target computing power cluster at any time point after the target computing power cluster is obtained and before the current computing task is completed. Based on the fourth and fifth communication efficiency values, anomaly values ​​are obtained; the anomaly values ​​are used to characterize at least the probability of anomalies in the target computing power cluster. If the abnormal value is greater than or equal to the preset value, the target computing power cluster is reset.

[0014] Secondly, this application proposes a technical solution for a heterogeneous resource scheduling system, which is applied to an artificial intelligence computing power service platform, the artificial intelligence computing power service platform including multiple computing power nodes; the heterogeneous resource scheduling system includes: The processor is used to acquire multiple computing power clusters based on the current computing task; the current computing task is acquired in advance; each computing power cluster includes multiple first computing power nodes; the first computing power node is any computing power node among the various computing power nodes; In addition, a first computing power cluster is obtained from each computing power cluster; the first computing power cluster is the computing power cluster from each computing power cluster for which a corresponding communication efficiency value has not been obtained; the communication efficiency value is at least used to characterize the degree of physical connection between each first computing power node in the first computing power cluster. Furthermore, for each first computing power cluster, a corresponding communication efficiency value is obtained; the communication efficiency value corresponds one-to-one with the computing power cluster. Furthermore, based on various communication efficiency values, the target computing power cluster is obtained; A scheduler is used to execute the current computing task based on the target computing power cluster.

[0015] Thirdly, this application proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the heterogeneous resource scheduling method as described in any one of the first aspects.

[0016] Compared with the prior art, the beneficial effects of this application are: This application obtains multiple computing power clusters by combining the current computing task, and acquires communication efficiency values ​​that characterize the degree of physical connection between computing power nodes in each cluster. Then, it selects a target computing power cluster to execute the current computing task based on these communication efficiency values. Its core advantage lies in overcoming the limitations of traditional methods that only construct computing power clusters based on resource quotas. By fully considering the underlying physical interconnection topology characteristics of computing power nodes, it can avoid cross-device communication bottlenecks, thereby reducing the risk of linear speedup degradation and fundamentally solving the "computing power island" problem. Simultaneously, by prioritizing the selection of target computing power clusters with tightly connected computing power nodes and optimal communication efficiency, it can improve data transmission rate and collaborative performance, especially suitable for application scenarios with high communication requirements such as large model training, achieving dual optimization of global resource utilization and task execution efficiency. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the physical topology of a computing cluster proposed in an embodiment of this application; Figure 2 This is a schematic diagram of the physical topology of another computing cluster proposed in the embodiments of this application; Figure 3 This is a flowchart illustrating a heterogeneous resource scheduling method proposed in an embodiment of this application. Figure 4 This is a schematic diagram of the structure of a heterogeneous resource scheduling system proposed in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a server proposed in an embodiment of this application. Detailed Implementation

[0018] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects (e.g., the first computing node and the second computing node represent different computing nodes, and so on), and are not necessarily used to describe a specific order or sequence. It should be understood that such names can be used interchangeably where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules appearing in the embodiments of this application is merely a logical division; in actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not performed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through interfaces, indirect couplings between modules, or electrical or other similar forms of communication connections, none of which are limited in the embodiments of this application. Moreover, the modules or sub-modules described as separate components may or may not be physically separate, may or may not be physical modules, or may be distributed across multiple circuit modules. Some or all of these modules can be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0019] To address the technical problem presented in the background section that existing heterogeneous resource scheduling methods are prone to impacting task execution efficiency due to data transmission efficiency, this application proposes an embodiment of a heterogeneous resource scheduling method. This method is applied to an Artificial Intelligence (AI) computing power service platform, which includes multiple computing power nodes.

[0020] It is important to note that in this application, the AI ​​computing power service platform refers to a distributed service architecture that integrates multiple computing power nodes belonging to different independent autonomous management domains, capable of providing diverse heterogeneous resources such as computing, storage, and networking, and possessing unified resource abstraction modeling, global collaborative decision-making, and dynamic adaptive allocation capabilities. This platform not only supports centralized management and scheduling of various heterogeneous resources but also strictly guarantees the autonomy, data security, and compliance requirements of each management domain. Furthermore, it can adapt to complex computing scenarios with extremely high requirements for communication efficiency and resource collaboration capabilities, such as large-scale model training, providing an underlying support environment for heterogeneous resource scheduling. AI computing power service platforms with such functions are mature technologies and will not be elaborated upon further here.

[0021] It is important to note that this application does not impose any restrictions on the application scenarios of the heterogeneous resource scheduling method. That is, the heterogeneous resource scheduling method described below can be applied to any type of AI computing power service platform. For example, an AI computing power service platform can be a dedicated platform for large-scale model training for research institutions; it can also be a general-purpose AI computing platform for internet companies; it can also be a secure and compliant AI computing power platform for the government sector; and it can also be a lightweight AI service platform for edge computing scenarios, etc. It is crucial to understand that the technical solution of this application is particularly suitable for federated learning application scenarios.

[0022] It is important to note that in this application, a computing node refers to a physical device or virtual unit that possesses independent computing capabilities and can perform data computation, storage, and network interaction functions. For example, a computing node can be a physical server equipped with computing chips such as a Central Processing Unit (CPU) and a Graphics Processing Unit (GPU), providing hardware support for high-density computing tasks such as large model training and data mining; it can also be a virtual computing instance built based on cloud computing technology, achieving elastic allocation and on-demand use of computing power through resource virtualization technology; it can also be an embedded computing device in an edge computing scenario, deployed at edge nodes close to the data generation end to meet low-latency data processing requirements; it can also be a storage-computing integrated node in a distributed storage computing power cluster, providing data storage services while possessing local data preprocessing capabilities; or it can be a customized computing unit in a dedicated computing power cluster, with hardware optimized for specific algorithms or business scenarios to improve the processing efficiency of specific tasks.

[0023] It's important to understand that existing heterogeneous resource scheduling methods that use scalar counting of resource quotas to build computing clusters refer to constructing clusters that meet the resource requirements of computing tasks solely by statistically analyzing and allocating quantifiable scalar indicators such as the number of computing nodes. This approach only focuses on whether the quantity of resources meets the standard, completely ignoring the underlying physical characteristics of the computing cluster, such as the physical connections between nodes, topological relationships, and network transmission characteristics. It fails to consider the geographical distribution of computing nodes or assess the path complexity and efficiency of data transmission between nodes, ultimately leading to computing clusters that may suffer from scattered node distribution and poor communication links. For example: Figure 1 This is a schematic diagram of the physical topology of a computing power cluster proposed in an embodiment of this application. It includes computing power node 1, computing power node 2, computing power node 3 and computing power node 4, and computing power node 1, computing power node 2, computing power node 3 and computing power node 4 form a closed ring topology in sequence (hereinafter referred to as the first embodiment). Figure 2This is a schematic diagram of the physical topology of another computing cluster proposed in this application embodiment, which includes computing node five, computing node six, computing node seven, and computing node eight, and computing node five, computing node six, computing node seven, and computing node eight form a chain-like topology (hereinafter referred to as the second embodiment). Compared with the second embodiment, if a computing task requires four computing nodes and has high requirements for the real-time performance and stability of data transmission between computing nodes, then the ring-closed topology in the first embodiment will exhibit better communication performance. This is because each computing node in the ring-closed topology directly establishes a physical connection with the two adjacent computing nodes, and the data transmission path does not need to pass through many intermediate forwarding nodes, which can effectively reduce the signal attenuation rate and transmission delay; at the same time, the ring structure has bidirectional communication capability, and when a single link experiences temporary fluctuations, data can be transmitted through the reverse path to ensure the continuity of communication. In the chain-like topology of the second embodiment, the computing nodes at both ends (i.e., computing node five and computing node eight) can only communicate directly with their adjacent nodes on one side. Communication between the nodes at both ends relies on forwarding from upstream and downstream nodes. This not only results in longer transmission paths and more hops, but also easily creates communication bottlenecks at intermediate nodes, leading to decreased bandwidth utilization. Especially in applications involving large-scale parallel data transmission, this difference directly affects the overall execution efficiency of the computing task. Based on this, as... Figure 3 As shown, the heterogeneous resource scheduling method in this application embodiment includes steps 100 to 500.

[0024] Step 100: Based on the current computing task, obtain multiple computing power clusters.

[0025] In this embodiment, the current computing task can be obtained in advance. The current computing task refers to the specific computing requirements submitted by the user through the AI ​​computing power service platform, or automatically generated by the AI ​​computing power service platform according to a preset business process. It requires the use of computing resources for data processing, model training, algorithm computation, and other operations. Furthermore, the task explicitly includes key information such as the execution instructions required to complete the objective, data input specifications, performance indicator requirements, and result output format. For example, the current computing task could be a large-scale model training task in the field of artificial intelligence; a big data analysis and processing task; an intelligent inference task related to computer vision; a multilingual translation or text generation task in the field of natural language processing; or a numerical simulation task in the field of scientific computing, etc.

[0026] In this embodiment, a computing power cluster refers to a resource set formed by integrating several computing power nodes (i.e., the first computing power nodes in the following text) selected from multiple computing power nodes of an artificial intelligence computing power service platform based on the resource requirements of the current computing task, according to preset assembly rules. This set can collaboratively complete data computation, storage interaction, and task execution. Its core feature is that the various computing power nodes within the set have a clear cooperative relationship and can achieve data transmission and instruction interaction through underlying physical connections or network communication. That is, in the embodiments of this application, each computing power cluster includes multiple first computing power nodes, where each first computing power node is any computing power node among the various computing power nodes in the artificial intelligence computing power service platform. In the application scenario of federated heterogeneous resource scheduling, obtaining one or more computing power clusters capable of executing the current computing task is a mature technology and will not be elaborated here.

[0027] Step 200: Obtain the first computing cluster from the various computing clusters.

[0028] In this embodiment, the first computing power cluster is any computing power cluster among the various computing power clusters that has not obtained a corresponding communication efficiency value. The communication efficiency value is at least used to characterize the degree of physical connection between the various first computing power nodes in the first computing power cluster.

[0029] In this embodiment, the purpose of obtaining the first computing power cluster from various computing power clusters is to systematically screen the communication efficiency-related status of all candidate computing power clusters, ensuring that each computing power cluster can be included in the calculation range of communication efficiency values, and accurately locating computing power clusters that have not yet completed communication efficiency value evaluation. This not only lays a data foundation for subsequent selection of target computing power clusters based on communication efficiency values, but also avoids scheduling decision biases caused by omitting the communication efficiency evaluation of some computing power clusters, thereby ensuring that the finally selected target computing power cluster has superior communication performance and effectively improves the execution efficiency of the current computing task.

[0030] In this embodiment, the tightness of the physical connection between the first computing nodes in the first computing power cluster refers to the quantitative representation of any one or a combination of attributes such as the rationality of the connection layout of each first computing node in the underlying physical topology, the number of hops in the data transmission path, and the stability and reliability of the communication link (e.g., data transmission delay or signal attenuation rate, as mentioned below). Its core reflects the number of hops in establishing physical connections between the first computing nodes, the level of signal attenuation, and the magnitude of data transmission delay. It can intuitively demonstrate the quality of the basic communication conditions when the computing nodes within the computing power cluster work collaboratively. A higher tightness means higher data transmission efficiency and stronger stability between the computing nodes, which is more conducive to improving the overall execution efficiency of computing tasks.

[0031] Step 300: For each first computing power cluster, obtain the corresponding communication efficiency value.

[0032] In this embodiment, the communication efficiency value corresponds one-to-one with the computing power cluster.

[0033] It is important to understand that in this embodiment, the communication efficiency value can be obtained in any reasonable way, as long as the communication efficiency value can characterize the degree of physical connection between the first computing power nodes in the first computing power cluster. For example, the communication efficiency value can be the average of the various time-consuming quantization values ​​mentioned below, or the average of the various communication distance values, etc. That is to say, in this embodiment, there is no restriction on the correspondence between the communication efficiency value and the degree of physical connection between the first computing power nodes in the first computing power cluster. For example, the communication efficiency value and the degree of physical connection between the first computing power nodes in the first computing power cluster can be positively correlated or negatively correlated. If the communication efficiency value and the degree of physical connection between the first computing power nodes are positively correlated, then the larger the communication efficiency value, the greater the degree of physical connection between the first computing power nodes; if the communication efficiency value and the degree of physical connection between the first computing power nodes are negatively correlated, then the smaller the communication efficiency value, the greater the degree of physical connection between the first computing power nodes.

[0034] To save space, in the subsequent embodiments of this application, only the positive correlation between the communication efficiency value and the degree of physical connection between each first computing power node is used as an example to explain in detail the calculation method of the communication efficiency value, the selection logic of the target computing power cluster, and the execution details of the related process. This example is only to make the explanation of the technical solution clearer and easier to understand and the logic more coherent. It does not mean that the technical solution protected by this application is limited to this kind of correlation. Other forms of communication efficiency value correlation that can accurately characterize the degree of physical connection between computing power nodes are also within the protection scope of this application.

[0035] In one embodiment of this application, step 300, which involves obtaining a communication efficiency value based on the first computing power cluster, includes steps 310 to 340.

[0036] Step 310: Traverse each first computing node in the first computing power cluster to obtain the second computing power node and all third computing power nodes.

[0037] In this embodiment, the second computing node is any computing node among the first computing nodes that has not obtained the corresponding physical topology value. The third computing node is any computing node among the first computing nodes other than the second computing node. Figure 1Taking the computing power cluster in the example, if the second computing power node is computing power node one, then computing power node two, computing power node three, and computing power node four are all third computing power nodes; if the second computing power node is computing power node two, then computing power node one, computing power node three, and computing power node four are all third computing power nodes, and so on. No further enumeration is required.

[0038] For each second computing node, steps 320 and 330 are executed until each second computing node obtains the corresponding communication distance value.

[0039] In this embodiment, the communication distance value is used at least to characterize the physical topology distance between the second computing node and each of the third computing nodes. Specifically, the physical topology distance between two computing nodes refers to a comprehensive quantitative representation of the data transmission attributes of the actual communication path traversed by the two computing nodes during data transmission within the underlying physical network architecture of the computing cluster. This distance reflects the complexity and ease of data interaction between the two computing nodes. Specifically, it can be the number of network devices (e.g., switches and routers) that data passes through during its journey from one computing node to another, or it can be the actual physical length of the communication link.

[0040] Step 320: Based on the second computing node and each third computing node, obtain the physical topology distance value corresponding to each third computing node.

[0041] As mentioned above, in this embodiment, the physical topology distance value can be the number of network devices, the actual physical length of the communication link, etc.

[0042] In one specific embodiment of this application, the physical topology distance value can be the minimum number of computing nodes required for data transmission between two computing nodes. Figure 2 Taking the computing power cluster in the example, since computing power node 5 and computing power node 6 are directly connected, the physical topological distance between them is 1; since computing power node 5 and computing power node 8 are indirectly connected through computing power node 6 and computing power node 7, the physical topological distance between them is 3. In other words, in this embodiment, the greater the number of computing power nodes separating two computing power nodes, the larger the physical topological distance between them.

[0043] Step 330: Obtain the communication distance value based on the distance values ​​of each physical topology.

[0044] In this embodiment, the communication distance value can be obtained based on each physical topology distance value using any reasonable method, as long as the obtained communication distance value can characterize the physical topology distance between the second computing node and each of the third computing nodes. For example, the communication distance value can be the average of each physical topology distance value; or, the communication distance value can be the variance of each physical topology distance value, etc.

[0045] Step 340: After each second computing node has obtained the corresponding communication distance value, the communication efficiency value is obtained based on each communication distance value.

[0046] In this embodiment, the communication efficiency value can be obtained based on each communication distance value using any reasonable method, as long as the obtained communication efficiency value can characterize the degree of physical connection between each first computing power node in the first computing power cluster. For example, in order to make the communication efficiency value positively correlated with the degree of physical connection between each first computing power node, in one embodiment of this application, the calculation formula for obtaining the communication efficiency value based on each communication distance value can be as follows:

[0047] in, This represents the communication efficiency value of the first computing power cluster; This indicates the number of each first-power node in the first computing power cluster. It is a positive integer greater than or equal to 2; This represents the communication distance value corresponding to the i-th first computing node in the first computing power cluster, where i is greater than or equal to 1 and less than or equal to 2. In this embodiment, the larger the communication distance value, the smaller the communication efficiency value, which means the tighter the physical connection between the first computing power nodes; conversely, the smaller the communication distance value, the larger the communication efficiency value, which means the tighter the physical connection between the first computing power nodes.

[0048] In another embodiment of this application, the formula for calculating the communication efficiency value based on each communication distance value can be as follows:

[0049] in, This represents the communication efficiency value of the first computing power cluster; This indicates the number of each first-power node in the first computing power cluster. It is a positive integer greater than or equal to 2; This represents the communication distance value corresponding to the i-th first computing node in the first computing power cluster, where i is greater than or equal to 1 and less than or equal to 2. ; This represents an exponential function with the natural constant e as its base. In this embodiment, the larger the communication distance values, the smaller the communication efficiency value, which means the tighter the physical connection between the first computing power nodes; conversely, the smaller the communication distance values, the larger the communication efficiency value, which means the tighter the physical connection between the first computing power nodes.

[0050] It's important to note that, theoretically, a smaller communication distance between two computing nodes would result in fewer hops in the data transmission path, less signal attenuation, and potentially lower communication time. However, actual communication efficiency is also affected by transmission performance metrics such as bandwidth and latency. Therefore, relying solely on communication distance is insufficient to fully characterize the true communication capabilities of a computing cluster. For example, ... Figure 1 and Figure 2 As shown, without considering transmission performance indicators such as bandwidth and latency, due to Figure 1 In a closed-loop topology, adjacent computing nodes are directly connected; therefore, in Figure 1 In a computing cluster, some data transmission paths (such as the data transmission path between computing node one and computing node four) are shorter and do not require additional forwarding nodes, resulting in a smaller overall communication distance and theoretically higher communication efficiency. Figure 2 In a chain-like topology, the two end computing nodes (i.e., computing node five and computing node eight) need to forward data through an intermediate node, resulting in a longer transmission path, greater communication distance, and theoretically lower communication efficiency. However, in practical applications, such as... Figure 1 As shown, the data transmission performance between computing node one and computing node two is poor. Figure 1 and Figure 2 The solid lines indicate strong data transmission performance between two computing nodes; the dashed lines indicate weak data transmission performance between two computing nodes. The data transmission performance between computing node three and computing node two is also poor. This type of transmission bottleneck between computing nodes can directly lower the overall performance. Figure 1 The communication efficiency of the entire computing cluster; and such Figure 2 As shown, although the overall communication distance of the chain topology is greater, the high bandwidth and low latency transmission characteristics of computing nodes five, six, seven, and eight may result in a higher actual data interaction rate for the entire computing cluster compared to... Figure 1 The computing power cluster in the middle.

[0051] It needs to be clear that, in Figure 1 and Figure 2 In this diagram, if the data transmission performance between two computing nodes is strong, it is represented by a solid line; if the data transmission performance between two computing nodes is weak, it is represented by a dashed line. For example, Figure 1If the line connecting computing power node 1 and computing power node 2 is a dashed line, it indicates that the bandwidth between these two computing power nodes is low, the latency is high, or the signal attenuation is significant. Data transmission is prone to stuttering or packet loss, and the overall transmission performance is poor. Figure 2 The solid line connecting computing power node five and computing power node six indicates that these two computing power nodes have the advantages of high bandwidth or low latency transmission, minimal signal attenuation, strong data transmission stability, and can complete data interaction quickly and efficiently, with overall transmission performance at a superior level.

[0052] It is important to note that Figure 1 and Figure 2 The embodiments presented herein are merely illustrative examples to facilitate understanding of the heterogeneous resource scheduling method proposed in this application by those skilled in the art, and do not imply that the physical topology of the computing cluster can be limited to such embodiments in actual applications. Figure 1 The circular closed topology shown and as Figure 2 The two specific forms of chain topology shown are not necessarily the same as those of computing power clusters using ring topology, and it is not to say that all computing power clusters using ring topology will necessarily have better communication efficiency than those using chain topology. This will not be discussed further later.

[0053] In summary, it is difficult to accurately characterize the true communication capability of a computing power cluster using only individual communication distance values. To obtain a communication efficiency value that can more accurately characterize the true communication capability of a computing power cluster, in one embodiment of this application, step 340, obtaining the communication efficiency value based on each communication distance value, includes steps 341 and 342.

[0054] Step 341: Based on the second computing node and each third computing node, obtain the time consumption quantization value corresponding to each third computing node.

[0055] In this embodiment, the time-consuming quantization value is used at least to characterize the time it takes for data to be transmitted from the second computing node to the corresponding third computing node. That is, in this embodiment, the time-consuming quantization value can be any value that can characterize the time it takes for data to be transmitted from the second computing node to the corresponding third computing node. For example, the time-consuming quantization value can be any one of bandwidth, latency, hop count, signal attenuation rate, non-uniform memory access distance, and theoretical completion time of set communication primitives.

[0056] It is important to note that, to eliminate the influence of units, the quantified time consumption values ​​can also be the results of standardization of the aforementioned indicators. The standardization process can be tailored to the characteristics of different indicators. For example, for indicators with significant differences in bandwidth and latency, Z-score standardization or Min-Max standardization can be used to map all quantified time consumption values ​​to the same data range. This avoids deviations in communication efficiency calculations due to differences in the units of measurement of the indicators themselves, ensuring that the final communication efficiency value more objectively and accurately reflects the true communication capabilities of the computing cluster. Data standardization is a mature technology in the computer field and will not be elaborated upon here.

[0057] Of course, in other embodiments of this application, the time consumption quantification value can also be a combination of the above-mentioned indicators. For example, delay and signal attenuation rate are core indicators that directly affect communication quality. The smaller the value of both, the better the communication quality. Therefore, the standardized delay and signal attenuation rate can be summed or averaged as the time consumption quantification value. Other reasonable combinations of indicators will not be listed and elaborated here.

[0058] It is important to understand that in this embodiment, the time quantization value can be positively or negatively correlated with the superiority of communication quality (i.e., the time it takes for data to be transmitted from the second computing node to the corresponding third computing node). In this embodiment, the shorter the data transmission time, the better the communication quality; conversely, the longer the data transmission time, the worse the communication quality. If the time quantization value is positively correlated with the superiority of communication quality, then the larger the time quantization value, the better the communication quality and the shorter the data transmission time; conversely, if the time quantization value is negatively correlated with the superiority of communication quality, then the larger the time quantization value, the worse the communication quality and the longer the data transmission time. To avoid redundancy, the following detailed explanation will only consider the case where the time quantization value is positively correlated with the superiority of communication quality. The implementation logic for other correlations can be derived by referring to this example.

[0059] Step 342: Obtain the communication efficiency value based on each time consumption quantization value and each communication distance value.

[0060] As mentioned above, in this embodiment, the larger the quantized time consumption values, the better the communication quality between the first computing nodes in the first computing power cluster, meaning the corresponding communication efficiency value should be higher; conversely, the larger the communication distance values, the greater the communication distance between the first computing nodes in the first computing power cluster, meaning the corresponding communication efficiency value should be lower. In other words, in the embodiments of this application, any reasonable method can be used to obtain the communication efficiency value based on the quantized time consumption values ​​and the communication distance values, as long as the communication efficiency value is positively correlated with the quantized time consumption values ​​and negatively correlated with the communication distance values. For example, in a specific embodiment of this application, step 342, the formula for calculating the communication efficiency value based on the quantized time consumption values ​​and the communication distance values, can be as follows:

[0061] in, This represents the communication efficiency value of the first computing power cluster; This indicates the number of each first-power node in the first computing power cluster. It is a positive integer greater than or equal to 2; This represents the communication distance value corresponding to the i-th first computing node in the first computing power cluster, where i is greater than or equal to 1 and less than or equal to 2. ; This represents the time quantization value corresponding to the i-th first computing node in the first computing power cluster.

[0062] In another embodiment of this application, step 342, the calculation formula for obtaining the communication efficiency value based on each time consumption quantization value and each communication distance value, can be as follows:

[0063] in, This represents the communication efficiency value of the first computing power cluster; This indicates the number of each first-power node in the first computing power cluster. It is a positive integer greater than or equal to 2; This represents the communication distance value corresponding to the i-th first computing node in the first computing power cluster, where i is greater than or equal to 1 and less than or equal to 2. ; This represents the quantized time consumption value corresponding to the i-th first computing node in the first computing power cluster; This represents an exponential function with the natural constant e as its base. It can be any positive number close to 0, such as 0.1 or 0.01, to avoid the phenomenon that the sum of the time-consuming quantization values ​​in the denominator is 0 in extreme cases, which would lead to abnormal results in the calculation of communication efficiency value.

[0064] This embodiment derives communication efficiency values ​​by quantifying various time consumption values ​​and communication distance values. This approach considers both the tightness of the physical topology connections between computing nodes and incorporates actual data transmission performance indicators, achieving a multi-dimensional and accurate characterization of the computing cluster's communication capabilities. This method breaks away from the limitations of relying solely on a single topology or performance indicator, effectively avoiding misjudgments such as "close physical distance but poor transmission performance" or "excellent transmission indicators but scattered topology." It ensures that communication efficiency values ​​better align with the data interaction needs of real-world applications, providing a more reliable basis for selecting superior target computing clusters, and ultimately guaranteeing the execution efficiency and stability of computing tasks from the underlying communication level.

[0065] Step 400: Obtain the target computing power cluster based on each communication efficiency value.

[0066] In the embodiments of this application, the target computing power cluster can be obtained based on various communication efficiency values ​​in any reasonable manner. For example, the various communication efficiency values ​​can be sorted in descending order, and several computing power clusters at the top of the sorting results can be selected as candidate computing power clusters. The specific number of these candidates can be flexibly set according to the complexity of the current computing task, the scale of resource requirements, and the overall load of the artificial intelligence computing power service platform, such as 3, 10, or other suitable numbers. Then, a candidate computing power cluster can be randomly selected from the candidate computing power clusters as the target computing power cluster. Alternatively, the real-time resource utilization rate of the candidate computing power clusters (e.g., CPU utilization, memory usage ratio, GPU memory usage, and network bandwidth usage) and / or the historical task execution stability of the candidate computing power clusters (e.g., the average completion rate of historical tasks and the number of abnormal interruptions during historical task execution) can be combined to conduct a secondary screening of the initially selected candidate computing power clusters, and finally determine the target computing power cluster that best meets the execution requirements of the current computing task. Of course, the method of obtaining the target computing power cluster can also be as shown in Embodiments 1 and 2 below.

[0067] Example 1 of obtaining the target computing power cluster In this embodiment, the communication efficiency value is positively correlated with the degree of physical connection between each first computing node in the first computing power cluster.

[0068] In this embodiment, step 400, which obtains the target computing power cluster based on each communication efficiency value, includes steps 410 to 430.

[0069] Step 410: Obtain the first communication efficiency value based on each communication efficiency value.

[0070] In this embodiment, the first communication efficiency value is the maximum value among all communication efficiency values. In the field of computer science, obtaining the maximum value (i.e., the first communication efficiency value) among multiple values ​​is a mature technology, and will not be elaborated here.

[0071] Step 420: Based on the first communication efficiency value, obtain the second computing power cluster.

[0072] In this embodiment, the second computing power cluster is the computing power cluster corresponding to the first communication efficiency value.

[0073] Step 430: Use the second computing power cluster as the target computing power cluster.

[0074] In other words, in this embodiment, the computing power cluster with the highest communication efficiency value is selected as the target computing power cluster. The core logic is that the communication efficiency value is positively correlated with the tightness of the physical connection between computing power nodes. This computing power cluster has tighter node interconnection and a better data transmission path. Furthermore, this selection method can minimize cross-node communication latency and bandwidth loss, effectively avoid the "computing power island" problem, and thus ensure that the current computing task has efficient collaborative capabilities during execution, significantly improving the overall task processing efficiency.

[0075] It is important to note that the heterogeneous resource scheduling method proposed in this application is mainly applied to high-concurrency computing scenarios in artificial intelligence computing power service platforms. These scenarios often involve a large number of simultaneous computing tasks and intense resource competition. In application scenarios with a surge in computing tasks, if all tasks select the computing power cluster with the highest communication efficiency, the resources of that cluster may be instantly saturated, leading to severe overload. This could not only significantly reduce the task execution efficiency of that cluster but also cause other computing power clusters to become idle, creating a new "resource waste" dilemma. Furthermore, it could trigger a chain reaction of task queuing congestion and a surge in response latency, ultimately contradicting the overall optimal scheduling goal of global resource utilization and task execution efficiency. Therefore, the method for obtaining the target computing power cluster can be found in Embodiment 2 below.

[0076] Example 2 of obtaining the target computing power cluster In this embodiment, the communication efficiency value is positively correlated with the degree of physical connection between the first computing nodes in the first computing power cluster. Furthermore, each computing power cluster includes a corresponding computing power pointer, which at least indicates whether the corresponding computing power cluster is busy or idle.

[0077] In this embodiment, the purpose of setting a computing power pointer for each computing power cluster is to accurately and in real-time identify the resource occupancy status of the corresponding computing power cluster, providing a crucial status reference for subsequent selection of target computing power clusters based on communication efficiency values. This setting allows for quick determination of whether a high-communication-efficiency computing power cluster is available, avoiding the scheduling of computing tasks to already busy clusters, thereby reducing task queuing time and ensuring balanced utilization of global computing power resources. This guarantees the execution efficiency of the current computing task while preventing resource idleness or overload.

[0078] In this embodiment, the computing power pointer can be a set of binary identification codes, which use a specific binary value to clearly represent the idle or busy state of the computing power cluster through a preset encoding rule (e.g., 0 indicates that the computing power cluster is idle and 1 indicates that the computing power cluster is busy). The computing power pointer can also be a status register, which uses a specific value or semaphore stored in the register to reflect the resource occupancy of the computing power cluster. The computing power pointer can also be a program code segment with status feedback function, which can monitor the task execution progress of the computing power cluster in real time and output the corresponding status identification information synchronously.

[0079] In this embodiment, step 400, which obtains the target computing power cluster based on each communication efficiency value, includes steps 440 to 470.

[0080] Step 440: Obtain multiple second communication efficiency values ​​from the various communication efficiency values.

[0081] In this embodiment, the second communication efficiency value is the communication efficiency value corresponding to any idle computing cluster in each computing cluster.

[0082] Step 450: Obtain the third communication efficiency value based on each of the second communication efficiency values.

[0083] In this embodiment, the third communication efficiency value is the maximum value among the various second communication efficiency values.

[0084] Step 460: Based on the third communication efficiency value, obtain the third computing power cluster.

[0085] In this embodiment, the third computing power cluster is the computing power cluster corresponding to the third communication efficiency value.

[0086] Step 470: If the computing power pointer state corresponding to the third computing power cluster is idle, then the third computing power cluster is taken as the target computing power cluster; and the computing power pointer state of the third computing power cluster is adjusted to busy; if the computing power pointer state corresponding to the third computing power cluster is busy, then multiple new second communication efficiency values ​​are obtained again until the target computing power cluster is obtained.

[0087] This embodiment achieves real-time and accurate marking of the busy / idle status of computing power clusters by setting computing power pointers, constructing a two-layer scheduling logic of "communication efficiency priority + status adaptation". When selecting target computing power clusters, the communication efficiency value is first used as the core sorting criterion to prioritize and lock the candidate computing power clusters with the best performance. Then, the availability of the clusters is quickly verified by the computing power pointer to avoid scheduling tasks to overloaded computing power clusters. For busy high-communication-efficiency computing power clusters, the task is postponed to the second-best available computing power cluster through a traversal marking mechanism. This not only ensures the communication efficiency basis for task execution but also achieves a balanced load of global computing power resources. It effectively avoids the resource waste problem of a single computing power cluster being overloaded while other computing power clusters are idle, and significantly reduces task queuing time, further improving the overall flexibility and reliability of federated heterogeneous resource scheduling.

[0088] It is important to note that in federated heterogeneous resource scheduling, when prioritizing target computing clusters based on communication efficiency values, it is inevitable to encounter situations where all high-communication-efficiency computing clusters are busy. To prevent tasks from getting stuck in infinite waiting and to ensure the continuous progress of the scheduling process, a flexible fallback mechanism needs to be established. In one embodiment of this application, if all computing clusters corresponding to all communication efficiency values ​​are busy, multiple computing clusters are reacquired; or, a target computing cluster is selected based on preset rules; the preset rules include selecting the computing cluster with the shortest remaining task processing time among the busy clusters or selecting the computing cluster with the lowest resource utilization rate among the busy clusters as the target computing cluster.

[0089] In this embodiment, reacquiring multiple computing power clusters refers to re-scanning and filtering all computing power nodes in the AI ​​computing power service platform based on the original resource requirements of the current computing task, and re-building multiple computing power clusters that meet the task execution requirements. For example, during the previous round of acquiring computing power clusters, a large number of computing power nodes were not included in the cluster building scope because they were busy carrying other computing tasks. During the reacquisition process, these computing power nodes may have completed their original tasks and returned to an idle state. Therefore, these idle computing power nodes can be included in the cluster building scope. Combined with the real-time status of the computing power nodes, multiple new computing power clusters composed of more different computing power nodes can be formed, providing more candidate clusters for the current computing task, avoiding the task from waiting indefinitely because the original computing power clusters are all busy, and ensuring the continuous progress of the scheduling process and the efficient execution of the task.

[0090] This concludes the description of Example 2 for acquiring the target computing power cluster.

[0091] Step 500: Execute the current computing task based on the target computing power cluster.

[0092] It is important to understand that in the field of computer science, executing a computing task (i.e., the current computing task) based on a certain computing power cluster (i.e., the target computing power cluster) is a mature technology, which will not be elaborated here.

[0093] It is important to note that different chip manufacturers (e.g., Zhonghao Xinying, NVIDIA, and Huawei) each equip themselves with proprietary topology description interfaces (e.g., DLC-SMI, NVML, and DSMI) and private driver instruction sets, and there is a lack of unified standards for these interfaces and instruction sets. Existing scheduling systems often use a "hard-coded" approach to directly adapt to various chips (i.e., the interface adaptation rules for different chips are written into the core code of the scheduling system). This means that every time a new type of chip (i.e., heterogeneous computing nodes) is added or replaced in the AI ​​computing power service platform, the core code of the scheduling system needs to be modified and the scheduling logic reconstructed, severely restricting the rapid management and scheduling adaptation efficiency of new chips. Therefore, in one embodiment of this application, step 500, based on the target computing power cluster, executes the current computing task, including steps 510 to 530.

[0094] Step 510: Traverse all execution instructions contained in the current computing task to obtain the first instruction to be executed.

[0095] In this embodiment, the first instruction can be any instruction required to complete the current computing task. For example, the first instruction can be a data reading instruction, used to retrieve the original dataset required for the current computing task from the target storage node; it can be a data preprocessing instruction, used to perform operations such as cleaning, format conversion, feature extraction, and normalization on the original data; it can be a model training instruction, used to call the heterogeneous computing resources of the target computing power cluster to perform forward propagation and backward gradient update of the neural network model; it can be an intermediate result storage instruction, used to securely store the intermediate data generated during the computing process to a designated storage unit; it can be a result verification instruction, used to verify the accuracy, completeness, and compliance of the final computing result; or it can be a result output instruction, used to feed back the verified computing result to the user terminal or related system according to a preset format.

[0096] For each first instruction, steps 520 and 530 are executed until all execution instructions contained in the current computing task have been executed.

[0097] Step 520: Map the first instruction to the second instruction based on the unified heterogeneous instruction set.

[0098] In this embodiment, the second instruction is a vendor-specific driver instruction corresponding to the fourth computing node. The fourth computing node is the computing node in the target computing cluster configured to execute the first instruction. Here, the vendor-specific driver instruction refers to a driver instruction developed exclusively by the computing node hardware manufacturer for its own product characteristics, and only compatible with the specific computing hardware of that manufacturer. The vendor-specific driver instruction can directly interact with the underlying hardware architecture of the computing node, precisely calling upon the hardware's computing resources, storage resources, and communication resources, thereby realizing specific operational logic or device control functions. It is typically only effective within the hardware ecosystem of that manufacturer and is incompatible with hardware driver instructions from other manufacturers.

[0099] In this embodiment, the unified heterogeneous instruction set is pre-set. This unified heterogeneous instruction set refers to a general instruction set formed by abstracting, standardizing, and uniformly encapsulating various vendor-specific driver instructions for computing power nodes (e.g., physical servers or virtual computing units equipped with different brands of GPUs and CPUs) from different vendors and architectures within an AI computing power service platform. This set includes standardized instruction interfaces and mapping rules covering core operations such as data transmission, computation execution, and resource scheduling. Its core function is to break down the compatibility barriers between different vendor-specific driver instruction sets. It establishes a precise and efficient mapping relationship between the general execution instructions (i.e., the first instruction) of upper-layer computing tasks and the exclusive driver instructions (i.e., the second instruction) of different underlying computing power nodes. This allows the AI ​​computing power service platform to quickly manage and adapt to new computing power nodes by simply expanding the corresponding mapping rules in the unified heterogeneous instruction set, without modifying the core code of the scheduling system or reconstructing the scheduling logic when adding or replacing new heterogeneous computing power nodes.

[0100] Step 530: Send the second instruction to the fourth computing node to drive the fourth computing node to execute the corresponding operation in the second instruction in order to complete the execution of the current computing task.

[0101] This embodiment successfully breaks down the compatibility barriers between proprietary driver instruction sets from different chip manufacturers by setting a unified heterogeneous instruction set, and constructs a standardized mapping channel between upper-layer general execution instructions and lower-layer proprietary driver instructions. This setting not only accurately converts the first instruction of the current computing task into a second instruction that can be recognized and executed by the fourth computing node in the target computing power cluster, ensuring the smooth operation of computing tasks in heterogeneous hardware environments, but also simplifies the management process of heterogeneous computing power nodes. When the AI ​​computing power service platform adds or replaces heterogeneous computing power nodes, it does not need to modify the core code of the scheduling system or reconstruct the scheduling logic. It only needs to extend the corresponding mapping rules in the unified heterogeneous instruction set to quickly adapt to and schedule new computing power nodes, thereby improving the AI ​​computing power service platform's compatibility with heterogeneous resources and scheduling flexibility.

[0102] It is important to note that in the "center-edge" architecture, edge computing nodes inherently suffer from latency in status feedback. Furthermore, existing computing task execution relies on a static workflow orchestration system, meaning the execution path is fixed and cannot be dynamically reconstructed based on hardware health conditions during operation. This results in fault self-healing requiring manual intervention, with recovery times reaching several hours. Even though hardware errors trigger device reset operations such as clearing Error-Correcting Codes (ECC) and resetting links, if the physical reset fails, the system can only mark the computing node as "permanently faulty" and isolate it. This still fails to fundamentally solve the problem of unhealthy computing nodes repeatedly entering the scheduling pool, thus affecting task execution efficiency.

[0103] It's important to understand that the "center-edge" architecture is a distributed computing architecture. Its core principle is the division of labor and collaboration between central and edge nodes, adapting to application scenarios such as data processing and resource scheduling. The central node, acting as the core hub, possesses powerful computing power, storage, and global management capabilities, responsible for overall scheduling, data aggregation and analysis, and global decision-making. Edge nodes, deployed closer to the data generation point or the user side, have relatively streamlined computing power but offer rapid response, handling local data preprocessing, real-time task execution, and status collection and feedback. The core advantages of this architecture are reduced data transmission latency, reduced load on the central node, and ensured rapid response for local tasks. It is commonly found in applications such as AI computing power scheduling and IoT data processing. In other words, the "center-edge" architecture is a mature technology and will not be elaborated upon here.

[0104] To avoid the problem of sub-healthy computing power nodes repeatedly entering the scheduling pool and thus affecting task execution efficiency, in one embodiment of this application, after step 500, which is based on the target computing power cluster and executes the current computing task, the method further includes steps 600 to 800.

[0105] Step 600: Based on the target computing power cluster, obtain the fourth communication efficiency value and the fifth communication efficiency value.

[0106] In this embodiment, the fourth communication efficiency value is the communication efficiency value of the target computing power cluster before it executes the current computing task. The fifth communication efficiency value is the communication efficiency value of the target computing power cluster at any time point after obtaining the target computing power cluster and before the completion of the current computing task. That is, the fourth communication efficiency value is the communication efficiency value of the target computing power cluster obtained in step 300; the fifth communication efficiency value is the communication efficiency value of the target computing power cluster obtained after obtaining the target computing power cluster by repeating the same calculation logic and process of obtaining the communication efficiency value in step 300 at any one or more preset monitoring time points (for example, setting a monitoring time point every 5 seconds or every 10 seconds), or by triggering calculation according to real-time status monitoring requirements (for example, triggering calculation if the current computing task times out).

[0107] Step 700: Based on the fourth communication efficiency value and the fifth communication efficiency value, obtain the outlier value.

[0108] In this embodiment, the outlier is used at least to characterize the probability of an anomaly in the target computing power cluster. As mentioned above, a higher communication efficiency value indicates better communication quality between the computing power nodes in the target computing power cluster; conversely, a lower communication efficiency value indicates poorer communication quality between the computing power nodes in the target computing power cluster.

[0109] In other words, if the communication efficiency value of the target computing cluster does not change much before and after the target computing cluster executes the current computing task, it means that the physical connection status and communication quality between the computing nodes in the target computing cluster remain stable without any obvious abnormal fluctuations, and the target computing cluster is in a normal working state, capable of continuously providing stable communication support for subsequent possible computing tasks. If the communication efficiency value of the target computing cluster changes significantly, it means that the physical connection status or communication quality between the computing nodes in the target computing cluster has become significantly abnormal, and the target computing cluster is likely in a sub-healthy or faulty state. If not handled in time, it may affect the normal execution efficiency of subsequent computing tasks, or even cause task execution to be interrupted.

[0110] In other words, in this embodiment, the difference between the fourth communication efficiency value and the fifth communication efficiency value can be used as an outlier. For example, the outlier can be the difference or ratio between the fourth and fifth communication efficiency values. In this embodiment, the larger the outlier, the greater the probability that the target computing power cluster is abnormal; the smaller the outlier, the smaller the probability that the target computing power cluster is abnormal.

[0111] Step 800: If the abnormal value is greater than or equal to the preset value, then reset the target computing power cluster.

[0112] In this embodiment, preset values ​​can be set according to needs or experience. For example, if the abnormal value is the difference between the fourth communication efficiency value and the fifth communication efficiency value, the preset value can be 0.4 or 0.5, etc.; if the abnormal value is the ratio between the fourth communication efficiency value and the fifth communication efficiency value, the preset value can be 3 or 4, etc.

[0113] In this embodiment, resetting the target computing power cluster refers to performing a systematic state recovery and fault diagnosis operation on the target computing power cluster. Specifically, this includes: disabling the current scheduling permissions of all computing power nodes in the target computing power cluster and interrupting abnormal communication links; performing hardware self-checks on each computing power node in the target computing power cluster (e.g., verifying CPU and GPU running status, memory integrity, and network interface connectivity), isolating or replacing nodes confirmed to have hardware faults; resetting the communication configuration parameters within the target computing power cluster, clearing link caches and error logs, and reinitializing the physical connections and data transmission protocols between nodes; re-evaluating the communication efficiency value of the cluster after the reset according to the calculation logic of the communication efficiency value in step 300, and verifying whether it has recovered to the normal threshold range; finally, updating the computing power pointer status corresponding to the target computing power cluster. If the communication efficiency value meets the requirements after the reset, the computing power pointer is adjusted to an idle state and re-included in the scheduling pool. If it still does not meet the requirements, it is marked as a state to be maintained and removed from the scheduling queue to prevent it from participating in task scheduling again and affecting the overall execution efficiency.

[0114] It is important to note that during the reset of the target computing cluster, in order to avoid interruption, data loss, or progress delays in the current computing task due to the reset, the currently executing computing task in the target computing cluster can be migrated to another computing cluster for execution before resetting the target computing cluster; or, the execution of the current computing task can be temporarily paused and the real-time running status and intermediate data can be saved. After the target computing cluster is reset and the communication efficiency returns to the normal threshold, the task can be restarted based on the saved status to continue execution.

[0115] This embodiment can proactively detect whether the target computing power cluster is abnormal by monitoring the communication efficiency value of the target computing power cluster in real time. If the target computing power cluster is abnormal, a systematic reset will be performed on the target computing power cluster. This can not only check the sub-health status of hardware and potential communication failures, and avoid sub-healthy nodes repeatedly entering the scheduling pool, but also reduce the dependence of fault self-healing on manual intervention, significantly shorten the recovery time, and ensure the stable operation and global scheduling efficiency of the heterogeneous resource scheduling system.

[0116] The heterogeneous resource scheduling method proposed in this application obtains multiple computing power clusters by combining the current computing task, and acquires communication efficiency values ​​that characterize the degree of physical connection between computing power nodes in each cluster. Then, it selects a target computing power cluster to execute the current computing task based on the communication efficiency values. Its core advantage lies in overcoming the limitations of traditional methods that only construct computing power clusters based on resource quotas. By fully considering the underlying physical interconnection topology characteristics of computing power nodes, it can avoid cross-device communication bottlenecks, thereby reducing the risk of linear speedup decline and fundamentally solving the "computing power island" problem. Simultaneously, by prioritizing the selection of target computing power clusters with tightly connected computing power nodes and optimal communication efficiency, it can improve data transmission rate and collaborative performance, especially suitable for application scenarios with high communication requirements such as large model training, achieving dual optimization of global resource utilization and task execution efficiency.

[0117] Having introduced the heterogeneous resource scheduling method proposed in the embodiments of this application, the following describes an embodiment of a heterogeneous resource scheduling system proposed in this application. This heterogeneous resource scheduling system is applied to an artificial intelligence computing power service platform, which includes multiple computing power nodes. For example... Figure 4 As shown, the heterogeneous resource scheduling system includes: The processor 11 is configured to acquire multiple computing power clusters based on the current computing task; the current computing task is acquired in advance; each computing power cluster includes multiple first computing power nodes; the first computing power node is any computing power node among the various computing power nodes; In addition, a first computing power cluster is obtained from each computing power cluster; the first computing power cluster is the computing power cluster from each computing power cluster for which a corresponding communication efficiency value has not been obtained; the communication efficiency value is at least used to characterize the degree of physical connection between each first computing power node in the first computing power cluster. Furthermore, for each first computing power cluster, a corresponding communication efficiency value is obtained; the communication efficiency value corresponds one-to-one with the computing power cluster. Furthermore, based on various communication efficiency values, the target computing power cluster is obtained; Scheduler 12 is used to execute the current computing task based on the target computing power cluster.

[0118] As a specific embodiment of this application, the processor 11 is further configured to traverse each first computing node in the first computing power cluster to obtain a second computing node and all third computing nodes; the second computing node is any computing node among the first computing nodes that has not obtained a corresponding physical topology value; the third computing node is any computing node among the first computing nodes other than the second computing node. Furthermore, for each second computing node, the following steps are performed until each second computing node obtains its corresponding communication distance value: Based on the second computing node and each third computing node, obtain the physical topology distance value corresponding to each third computing node; Based on each physical topology distance value, a communication distance value is obtained; the communication distance value is used at least to characterize the physical topology distance between the second computing node and each of the third computing nodes; Furthermore, if each second computing node obtains a corresponding communication distance value, then the communication efficiency value is obtained based on each communication distance value.

[0119] As a specific embodiment of this application, the processor 11 is further configured to obtain a time consumption quantization value corresponding to each of the third computing power nodes based on the second computing power node and each of the third computing power nodes; the time consumption quantization value is at least used to characterize the time taken for data to be transmitted from the second computing power node to the corresponding third computing power node; Furthermore, the communication efficiency value is obtained based on each time consumption quantization value and each communication distance value.

[0120] As a specific embodiment of this application, the time quantification value includes any one or a combination of bandwidth, latency, hop count, signal attenuation rate, non-uniform memory access distance, and set communication primitive theory completion time.

[0121] As a specific embodiment of this application, the communication efficiency value is positively correlated with the degree of physical connection between each first computing node in the first computing power cluster; the processor 11 is further configured to obtain a first communication efficiency value based on each communication efficiency value; the first communication efficiency value is the maximum value among the various communication efficiency values; Furthermore, based on the first communication efficiency value, a second computing power cluster is obtained; the second computing power cluster is the computing power cluster corresponding to the first communication efficiency value. And, the second computing power cluster is used as the target computing power cluster.

[0122] As a specific embodiment of this application, the communication efficiency value is positively correlated with the degree of physical connection between each first computing node in the first computing power cluster; each computing power cluster includes a corresponding computing power pointer, which is used to indicate at least whether the corresponding computing power cluster is busy or idle; the processor 11 is further configured to obtain multiple second communication efficiency values ​​from the various communication efficiency values; the second communication efficiency values ​​are the communication efficiency values ​​corresponding to any idle computing power cluster in each computing power cluster; Furthermore, based on each of the second communication efficiency values, a third communication efficiency value is obtained; the third communication efficiency value is the maximum value among the various second communication efficiency values. Furthermore, based on the third communication efficiency value, a third computing power cluster is obtained; the third computing power cluster is the computing power cluster corresponding to the third communication efficiency value. Furthermore, if the computing power pointer status corresponding to the third computing power cluster is idle, then the third computing power cluster is used as the target computing power cluster, and the computing power pointer status of the third computing power cluster is adjusted to busy. Furthermore, if the computing power pointer status corresponding to the third computing power cluster is busy, then multiple new second communication efficiency values ​​are reacquired until the target computing power cluster is obtained.

[0123] As a specific embodiment of this application, the processor 11 is further configured to traverse all execution instructions contained in the current computing task to obtain a first instruction to be executed; the first instruction is any instruction required to complete the current computing task. Furthermore, for each first instruction, the following steps are performed until all execution instructions contained in the current computing task have been executed: The first instruction is mapped to the second instruction based on a unified heterogeneous instruction set; the second instruction is a vendor-specific driver instruction corresponding to the fourth computing power node; the fourth computing power node is a computing power node in the target computing power cluster configured to execute the first instruction; The second instruction is sent to the fourth computing node to drive the fourth computing node to execute the corresponding operation in the second instruction in order to complete the execution of the current computing task.

[0124] As a specific embodiment of this application, the processor 11 is further configured to obtain a fourth communication efficiency value and a fifth communication efficiency value based on the target computing power cluster; the fourth communication efficiency value is the communication efficiency value of the target computing power cluster before the target computing power cluster executes the current computing task; the fifth communication efficiency value is the communication efficiency value of the target computing power cluster at any time point after obtaining the target computing power cluster and before the current computing task is completed. Furthermore, based on the fourth and fifth communication efficiency values, anomaly values ​​are obtained; the anomaly values ​​are used at least to characterize the probability of anomalies in the target computing power cluster. Furthermore, if the abnormal value is greater than or equal to a preset value, the target computing power cluster is reset.

[0125] The heterogeneous resource scheduling system proposed in this application obtains multiple computing power clusters by combining the current computing task, and acquires communication efficiency values ​​that characterize the degree of physical connection between computing power nodes in each cluster. Then, it selects target computing power clusters to execute the current computing task based on these communication efficiency values. Its core advantage lies in overcoming the limitations of traditional methods that only construct computing power clusters based on resource quotas. By fully considering the underlying physical interconnection topology characteristics of computing power nodes, it can avoid cross-device communication bottlenecks, thereby reducing the risk of linear speedup decline and fundamentally solving the "computing power island" problem. Simultaneously, by prioritizing target computing power clusters with tightly connected computing power nodes and optimal communication efficiency, it can improve data transmission rate and collaborative performance, especially suitable for application scenarios with high communication requirements such as large model training, achieving dual optimization of global resource utilization and task execution efficiency.

[0126] Having introduced the heterogeneous resource scheduling system proposed in the embodiments of this application, the following describes an embodiment of a computer-readable storage medium proposed in this application. Specifically, the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the heterogeneous resource scheduling method as described in any of the above embodiments.

[0127] This application also provides a server; please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1100 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1122 (e.g., one or more processors) and memory 1132, and one or more storage media 1130 (e.g., one or more mass storage devices) for storing application programs 1142 or data 1144. The memory 1132 and storage media 1130 may be temporary or persistent storage. The program stored in the storage media 1130 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the CPU 1122 may be configured to communicate with the storage media 1130 and execute the series of instruction operations in the storage media 1130 on the server 1100.

[0128] Server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0129] The steps performed by the server in the above embodiments can be based on this Figure 5 The structure of server 1100 shown. For example, as in the above embodiment, by Figure 4 The steps performed by the processor 11 or scheduler 12 shown can be based on this Figure 5 The server structure shown.

[0130] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0133] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0135] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0136] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0137] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A heterogeneous resource scheduling method, applied to an artificial intelligence computing power service platform, wherein the artificial intelligence computing power service platform includes multiple computing power nodes; characterized in that, The heterogeneous resource scheduling method includes: Based on the current computing task, multiple computing power clusters are obtained; each computing power cluster includes multiple first computing power nodes; the first computing power node is any computing power node among the various computing power nodes; A first computing power cluster is obtained from each computing power cluster; the first computing power cluster is the computing power cluster from each computing power cluster for which no corresponding communication efficiency value has been obtained; the communication efficiency value is at least used to characterize the degree of physical connection between each first computing power node in the first computing power cluster; For each first computing power cluster, the corresponding communication efficiency value is obtained; the communication efficiency value corresponds one-to-one with the computing power cluster. Based on various communication efficiency values, obtain the target computing power cluster; Based on the target computing power cluster, execute the current computing task.

2. The heterogeneous resource scheduling method according to claim 1, characterized in that, The step of obtaining the communication efficiency value based on the first computing power cluster includes: Traverse each first computing node in the first computing power cluster to obtain the second computing node and all third computing nodes; the second computing node is any computing node among the first computing nodes that has not obtained the corresponding physical topology value; the third computing node is any computing node among the first computing nodes except the second computing node. For each second computing node, perform the following steps until each second computing node obtains the corresponding communication distance value: Based on the second computing node and each third computing node, obtain the physical topology distance value corresponding to each third computing node; Based on each physical topology distance value, a communication distance value is obtained; the communication distance value is used at least to characterize the physical topology distance between the second computing node and each of the third computing nodes; If each second computing node obtains its corresponding communication distance value, then the communication efficiency value is obtained based on each communication distance value.

3. The heterogeneous resource scheduling method according to claim 2, characterized in that, The process of obtaining the communication efficiency value based on each communication distance value includes: Based on the second computing node and each third computing node, obtain the time consumption quantization value corresponding to each third computing node; the time consumption quantization value is at least used to characterize the time it takes for data to be transmitted from the second computing node to the corresponding third computing node; The communication efficiency value is obtained based on the quantized values ​​of each time consumption and each communication distance value.

4. The heterogeneous resource scheduling method according to claim 3, characterized in that, The time quantification values ​​include any one or a combination of bandwidth, latency, hop count, signal attenuation rate, non-uniform memory access distance, and theoretical completion time of set communication primitives.

5. The heterogeneous resource scheduling method according to any one of claims 1 to 4, characterized in that, The communication efficiency value is positively correlated with the degree of physical connection between each first computing node in the first computing power cluster; obtaining the target computing power cluster based on each communication efficiency value includes: Based on each communication efficiency value, a first communication efficiency value is obtained; the first communication efficiency value is the maximum value among all communication efficiency values. Based on the first communication efficiency value, a second computing power cluster is obtained; the second computing power cluster is the computing power cluster corresponding to the first communication efficiency value. The second computing power cluster is used as the target computing power cluster.

6. The heterogeneous resource scheduling method according to any one of claims 1 to 4, characterized in that, The communication efficiency value is positively correlated with the degree of physical connection between each first computing node in the first computing power cluster; each computing power cluster includes a corresponding computing power pointer, which is used to indicate at least whether the corresponding computing power cluster is busy or idle. The process of obtaining the target computing power cluster based on various communication efficiency values ​​includes: Multiple second communication efficiency values ​​are obtained from each communication efficiency value; the second communication efficiency value is the communication efficiency value corresponding to any idle computing cluster in each computing cluster; A third communication efficiency value is obtained based on each of the second communication efficiency values; the third communication efficiency value is the maximum value among the second communication efficiency values. Based on the third communication efficiency value, a third computing power cluster is obtained; the third computing power cluster is the computing power cluster corresponding to the third communication efficiency value. If the computing power pointer status corresponding to the third computing power cluster is idle, then the third computing power cluster is used as the target computing power cluster, and the computing power pointer status of the third computing power cluster is adjusted to busy. If the computing power pointer status corresponding to the third computing power cluster is busy, then multiple new second communication efficiency values ​​are obtained again until the target computing power cluster is obtained.

7. The heterogeneous resource scheduling method according to any one of claims 1 to 4, characterized in that, The execution of the current computing task based on the target computing power cluster includes: Traverse all execution instructions contained in the current computing task to obtain the first instruction to be executed; the first instruction is any instruction required to complete the current computing task. For each first instruction, the following steps are performed until all execution instructions contained in the current computing task have been executed: The first instruction is mapped to the second instruction based on a unified heterogeneous instruction set; the second instruction is a vendor-specific driver instruction corresponding to the fourth computing power node; the fourth computing power node is a computing power node in the target computing power cluster configured to execute the first instruction; The second instruction is sent to the fourth computing node to drive the fourth computing node to execute the corresponding operation in the second instruction in order to complete the execution of the current computing task.

8. The heterogeneous resource scheduling method according to any one of claims 1 to 4, characterized in that, After executing the current computing task based on the target computing power cluster, the method further includes: Based on the target computing power cluster, a fourth communication efficiency value and a fifth communication efficiency value are obtained; the fourth communication efficiency value is the communication efficiency value of the target computing power cluster before the current computing task is executed; the fifth communication efficiency value is the communication efficiency value of the target computing power cluster at any time point after the target computing power cluster is obtained and before the current computing task is completed. Based on the fourth and fifth communication efficiency values, anomaly values ​​are obtained; the anomaly values ​​are used to characterize at least the probability of anomalies in the target computing power cluster. If the abnormal value is greater than or equal to the preset value, the target computing power cluster is reset.

9. A heterogeneous resource scheduling system applied to an artificial intelligence computing power service platform, wherein the artificial intelligence computing power service platform includes multiple computing power nodes; characterized in that, The heterogeneous resource scheduling system includes: The processor is used to acquire multiple computing power clusters based on the current computing task; the current computing task is acquired in advance; each computing power cluster includes multiple first computing power nodes; the first computing power node is any computing power node among the various computing power nodes; In addition, a first computing power cluster is obtained from each computing power cluster; the first computing power cluster is the computing power cluster from each computing power cluster for which a corresponding communication efficiency value has not been obtained; the communication efficiency value is at least used to characterize the degree of physical connection between each first computing power node in the first computing power cluster. Furthermore, for each first computing power cluster, a corresponding communication efficiency value is obtained; the communication efficiency value corresponds one-to-one with the computing power cluster. Furthermore, based on various communication efficiency values, the target computing power cluster is obtained; A scheduler is used to execute the current computing task based on the target computing power cluster.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the heterogeneous resource scheduling method as described in any one of claims 1 to 8.

Images