Heterogeneous computing multi-dimensional weighted scheduling method for railway engineering multi-modal AI task

Abstract

The application discloses a heterogeneous computing power multi-dimensional weighted scheduling method and system for railway engineering AI tasks, comprising: a scheduling execution module receives an AI task and obtains a directed acyclic graph thereof; nodes are filtered according to the hard constraint of video memory capacity to obtain a candidate node set; a three-dimensional scheduling score is calculated for each candidate node, including a hardware matching degree score, a dynamic load perception score and a topological locality score; a cluster global physical state vector is obtained and scheduling weights of each dimension are dynamically generated; the three-dimensional scheduling scores are weighted and summed using the scheduling weights, and the optimal node is selected to execute scheduling. The hardware matching degree score contains a cross-computing stack migration cost factor, which is generated by comparing the operator compatibility of the directed acyclic graph and the heterogeneous computing runtime library of the target node. The application can accurately avoid the computing power degradation caused by the heterogeneous ecological barrier, eradicate the implicit resource congestion and network storm, and improve the stability of the cluster under high-frequency concurrency.

Description

Technical Field

[0001] This invention belongs to the technical field of computing resource scheduling, and in particular to a heterogeneous computing power multidimensional weighted scheduling method and system for AI tasks in railway engineering. Background Technology

[0002] In the current intelligent construction of railway engineering, computing centers typically employ heterogeneous computing clusters (such as coexisting NVIDIA CUDA architecture GPUs and Huawei Ascend CANN architecture NPUs) to process multimodal AI tasks with highly industry-specific characteristics in parallel. Unlike general internet AI tasks, railway engineering scenarios exhibit significant extreme data characteristics: First, railway digital twin and BIM tasks involve ultra-large-scale 3D models and multi-level spatial indexes, which are extremely sensitive to the absolute capacity and continuity of GPU memory; second, multi-source heterogeneous perception tasks such as petabyte-level laser point cloud analysis, catenary visual inspection, track inspection videos, and infrared images generated by full-line railway scanning involve the transfer of massive amounts of Tensor data between nodes, exhibiting asynchronous high-throughput characteristics and being highly dependent on PCIe bus and network bandwidth; finally, in the railway inspection edge-cloud collaborative scenario, real-time fault alarms place stringent requirements on end-to-end inference latency.

[0003] Currently, the industry primarily relies on native container orchestration schedulers (such as the Kubernetes Default Scheduler) or conventional heuristic resource management algorithms for task node allocation and scheduling. However, faced with complex heterogeneous ecosystems and multimodal data characteristics, existing conventional computing power scheduling methods suffer from the following three distinct technical shortcomings in their execution logic:

[0004] First, existing schedulers, when filtering and matching resources, typically only compare the requested GPU memory capacity of a task with the remaining GPU memory capacity of the target node, completely failing to perceive the underlying operator ecosystem compatibility. For example, current technology cannot predict how many incompatible operators will need to be rolled back to CPU execution if a task compiled based on the NVIDIA CUDA ecosystem is forcibly scheduled to a Huawei Ascend CANN architecture node, let alone quantify the resulting heterogeneous compilation and data copy-back overhead. This logic fails to establish a cross-architecture ecosystem awareness mechanism at the underlying level.

[0005] Second, existing technologies commonly use simple "network hop count" or "physical distance tree" as the basis for calculating communication costs when evaluating the node network topology of distributed tasks. This evaluation method is detached from the actual load characteristics of the business. The data throughput of multimodal tasks in the railway industry varies greatly. For example, petabyte-level point cloud processing is extremely dependent on bus bandwidth, while some computationally intensive tasks are not sensitive to bandwidth, and the physical link bandwidth within the cluster varies significantly. Existing scheduling algorithms cannot calculate the absolute communication time cost by "feature data volume / actual bandwidth," which makes it easy for high-throughput tasks to be misassigned to low-bandwidth nodes, causing severe I / O blocking.

[0006] Third, existing multi-dimensional scheduling evaluation systems, when comprehensively considering node resources, network, and load for scoring, often rely on fixed rules or static parameters for the weighting of various indicators. This rigid mechanism cannot capture transient changes in the global physical state of the cluster. For example, when the cluster faces extreme "global memory fragmentation" or "local network storms," ​​the static scheduler cannot adaptively adjust the scheduling weights of each dimension through a state feedback mechanism, directly leading to memory overflow or large-scale suspension of large model tasks under extreme conditions.

[0007] In summary, this invention aims to address the technical challenges of existing scheduling algorithms in scheduling multimodal AI tasks in railway engineering within heterogeneous computing power clusters. These challenges include resource allocation failures and low execution efficiency due to single matching dimensions, distorted communication evaluation, and static weight allocation. Summary of the Invention

[0008] Based on the problems mentioned above, this invention proposes a heterogeneous computing power multidimensional weighted scheduling method and system for AI tasks in railway engineering, the specific scheme of which is as follows:

[0009] A heterogeneous computing power multidimensional weighted scheduling method for AI tasks in railway engineering includes the following steps:

[0010] S1: The scheduling and execution module receives AI tasks and obtains their directed acyclic graph;

[0011] S2: The scheduling and execution module filters heterogeneous nodes in the cluster based on the hard constraint of video memory capacity, and removes nodes whose remaining video memory capacity is less than the task request amount, thus obtaining a set of candidate nodes.

[0012] S3: The scheduling execution module calculates a three-dimensional scheduling score for each node in the candidate node set; the three-dimensional scheduling score includes: hardware matching score, dynamic load awareness score, and topology locality score;

[0013] The hardware matching score includes a cross-computing stack migration cost factor, which is generated by comparing the operator compatibility of the directed acyclic graph with the heterogeneous computing runtime library of the target node.

[0014] S4: The scheduling execution module obtains the cluster global physical state vector and dynamically generates scheduling weights for each dimension based on the global physical state vector;

[0015] S5: The scheduling execution module uses the scheduling weight to perform a weighted summation of the three-dimensional scheduling scores, selects the node with the highest comprehensive score as the target scheduling node, and issues a scheduling instruction to execute container placement.

[0016] Preferably, in step S3, the cross-computing stack migration cost factor The generation method is as follows: count the number of operators in the directed acyclic graph that cannot be natively supported by the target node hardware and need to be rolled back to the CPU for execution. Total number of operators Combined with the bus copy delay coefficient of the target node Calculate using the following formula:

[0017]

[0018] in, Data was collected through polling using hardware physical probes deployed on each heterogeneous computing node.

[0019] Preferably, in step S3, the dynamic load perception score is determined by the modal anti-interference efficiency factor. and the memory fragmentation index of the target node The fusion calculation yields the following formula:

[0020] ;

[0021] in, ;

[0022] For tasks to resources Sensitivity weights, For the target node in resources Real-time resource utilization Includes bus, Video memory bandwidth, cache;

[0023] ;

[0024] For the target node Number of idle GPUs within each NUMA / Socket domain;

[0025] Modality-driven fragmentation sensitivity for the target node;

[0026] ;

[0027]

[0028] To analyze the tensor parallel or pipeline parallel communication frequency feature values ​​extracted from the directed acyclic graph;

[0029] This represents the topology penalty coefficient for heterogeneous nodes. This is an adjustable topology penalty scaling factor for the project.

[0030] Preferably, in step S3, the topological locality score is calculated as follows:

[0031]

[0032] in, The feature data volume of the AI ​​task is obtained by the scheduling and execution module when receiving the task by parsing the task declaration file or querying the dataset metadata;

[0033] The scheduling execution module obtains and caches the actual physical link bandwidth of the shortest path between candidate nodes and related nodes in advance through a hardware topology detection tool during the cluster initialization phase.

[0034] The network latency sensitivity penalty factor is determined based on the task modality, with a larger value for high real-time tasks and a smaller value for offline batch processing tasks.

[0035] Preferably, in step S4, the dynamic generation method of the scheduling weight is as follows:

[0036] The scheduling and execution module obtains the cluster's global memory fragmentation rate. Global I / O congestion Global network bandwidth saturation ;

[0037] The scheduling and execution module determines the basic demand base based on the task modality label mapping. ;

[0038] Calculate the hardware compatibility weight using the following formula. Load-aware weights Topological locality weights :

[0039]

[0040]

[0041]

[0042] The formula for calculating the overall score is as follows:

[0043]

[0044] in, Score the hardware compatibility. For dynamic load perception score, The score is for topological locality.

[0045] The calculation formula is:

[0046] in, The current AI task requests the following video memory capacity: For the target server node The current actual total free video memory capacity, For the architecture's dynamic performance coefficient, This is the cost factor for migration across computing stacks.

[0047] Preferably, the cluster's global memory fragmentation rate Take the memory fragmentation index of all target nodes The maximum value in; the global I / O congestion level. Take the average PCIe bandwidth utilization of all candidate nodes; the global network bandwidth saturation. The bandwidth utilization of cross-node physical links is obtained by collecting data through container network interfaces and switch SNMP protocols.

[0048] Another objective of this invention is to provide a heterogeneous computing power multidimensional weighted scheduling system for railway engineering AI tasks, used to execute the aforementioned heterogeneous computing power multidimensional weighted scheduling method for railway engineering AI tasks, comprising:

[0049] DAG parsing and operator comparison module: used to intercept the directed acyclic graph of AI tasks and cross-compare it with the heterogeneous computing runtime library of the target node, and output the number of operator backoffs;

[0050] Heterogeneous telemetry acquisition module: used to collect the hardware resource utilization and bus copy latency of each node through hardware physical probes;

[0051] Global Status Aggregation Module: Used to summarize telemetry data from each node and generate global memory fragmentation rate, global I / O congestion level, and global network bandwidth saturation.

[0052] The environmental feedback dynamic weighting module is used to generate scheduling weights for each dimension based on the global state vector and task mode.

[0053] Multi-dimensional scoring module: used to calculate hardware matching score, dynamic load awareness score, topology locality score and comprehensive score;

[0054] Scheduling execution module: Used to select the node with the highest overall score and issue scheduling instructions.

[0055] Preferably, in the multidimensional scoring module:

[0056] The calculation of the hardware matching score includes a cross-computing stack migration cost factor, which is generated based on the number of operator backoffs output by the DAG parsing and operator comparison module and the bus copy delay coefficient acquired by the heterogeneous telemetry acquisition module.

[0057] The calculation of the dynamic load perception score includes the memory fragmentation index of the target node and the modality-driven fragmentation sensitivity determined based on the directed acyclic graph of the AI ​​task.

[0058] The calculation of the topological locality score includes the ratio of the amount of task feature data to the actual physical link bandwidth obtained by the heterogeneous telemetry acquisition module.

[0059] Preferably, the heterogeneous telemetry acquisition module includes one or more of the following components:

[0060] Hardware physical probes deployed on each heterogeneous computing node are used to poll and collect bus copy latency coefficients at millisecond intervals.

[0061] The underlying monitoring tools deployed on each heterogeneous computing node are used to collect PCIe bus bandwidth utilization, HBM memory bandwidth utilization, and L2 cache utilization.

[0062] The hardware physical probes include one or more of the following types: kernel-mode eBPF probe, NVIDIA DCGM monitoring service, Ascend-smi monitoring service, and PCIe Telemetry probe.

[0063] Preferably, the scheduling execution module, the DAG parsing and operator comparison module, the heterogeneous telemetry acquisition module, the global state aggregation module, the environmental feedback dynamic weighting module, and the multidimensional scoring module are integrated into the Kubernetes scheduler extension plugin and deployed in the form of Scheduler Extender.

[0064] The beneficial effects of this invention are:

[0065] 1. Introduce a cross-computing stack migration cost factor. Through operator compatibility comparison, the heterogeneous migration loss is transformed into a mathematical penalty, which guides the task to achieve a closed loop within the homogeneous stack and realizes accurate adaptation of heterogeneous resources.

[0066] 2. Construct an absolute communication cost model based on the amount of feature data and the actual physical link bandwidth, and combine it with a modal-resource cross-validation mechanism to effectively block I / O blocking caused by the transmission of massive data across low-speed links.

[0067] 3. By dynamically generating scheduling weights through the collection of global state vectors of the cluster, and combining them with the analysis of directed acyclic graphs to dynamically adjust fragmentation sensitivity, the inter-card communication tasks can fill fragmented video memory, and large model tasks are preferentially allocated to contiguous video memory, achieving optimal fault-tolerant allocation at the edge of computing power exhaustion. Attached Figure Description

[0068] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. However, it should be understood that these drawings are designed for illustrative purposes only and are not intended to limit the scope of the present invention. Furthermore, unless specifically indicated, these drawings are intended only to conceptually illustrate the structural construction described herein and are not necessarily drawn to scale.

[0069] Figure 1 This is the main flowchart of the heterogeneous computing power dynamic multidimensional weighted scheduling of the present invention;

[0070] Figure 2 This is a diagram of the cross-computing stack migration cost evaluation engine and cross-layer linkage computing architecture of the present invention;

[0071] Figure 3 This is a diagram of the dynamic load sensing and modal cross-validation calculation structure of the present invention;

[0072] Figure 4 This is the structure diagram of the dynamic weighted control based on global state environmental feedback of the present invention;

[0073] Figure 5 This is a schematic diagram of the heterogeneous computing power dynamic scheduling system of the present invention. Detailed Implementation

[0074] First, it should be noted that the specific structure, features, and advantages of the present invention will be described in detail below by way of examples. However, all descriptions are for illustrative purposes only and should not be construed as limiting the present invention in any way. Furthermore, any single technical feature described or implied in the embodiments mentioned herein, or any single technical feature shown or implied in the accompanying drawings, can still be arbitrarily combined or deleted among these technical features to obtain more other embodiments of the present invention that may not be directly mentioned herein. Additionally, for the sake of simplifying the drawings, the same or similar technical features may be indicated only in one place in the same drawing.

[0075] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "setting," "connection," "fixing," "screw connection," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two components or the interaction between two components. Unless otherwise explicitly limited, those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0076] The following is in conjunction with the appendix Figure 1 -Appendix Figure 5 This invention will be described in detail.

[0077] Example 1:

[0078] As attached Figure 1 As shown, this embodiment provides a heterogeneous computing power multidimensional weighted scheduling method for AI tasks in railway engineering. This method is executed by the scheduling execution module and specifically includes the following steps:

[0079] S1: The scheduling and execution module receives AI tasks and obtains their directed acyclic graph (DAG) and modal labels. The modal labels are used to identify the task type, including but not limited to: railway large language model inference, PB-level laser point cloud processing, and 3D BIM rendering of railway hub stations.

[0080] S2: The scheduling and execution module filters heterogeneous nodes in the cluster based on the hard constraint of video memory capacity. It traverses all heterogeneous nodes in the cluster, reads the current free video memory capacity of each node, and eliminates nodes whose free video memory capacity is less than the task request size. The nodes generate a candidate node set. .

[0081] S3: The scheduling execution module calculates a three-dimensional scheduling score for each node in the candidate node set; the three-dimensional scheduling score includes: hardware matching score, dynamic load awareness score, and topology locality score;

[0082] Hardware matching score quantifies the physical fit between task memory requirements and node idle resources, and introduces a penalty for migration costs across computing stacks. Dynamic load awareness score quantifies the current implicit congestion and resource fragmentation levels of candidate nodes. Topological locality score evaluates the communication cost of distributed tasks.

[0083] The hardware matching score includes a cross-computing stack migration cost factor, which is generated by comparing the operator compatibility of the directed acyclic graph with the heterogeneous computing runtime library of the target node.

[0084] S4: The scheduling execution module obtains the cluster global physical state vector and dynamically generates scheduling weights for each dimension based on the global physical state vector;

[0085] S5: The scheduling execution module uses the scheduling weight to perform a weighted summation of the three-dimensional scheduling scores, selects the node with the highest comprehensive score as the target scheduling node, and issues a scheduling instruction to execute container placement.

[0086] In this application, a candidate node refers to any node in the set of nodes to be evaluated after hard filtering; a target node refers to the candidate node currently being evaluated when calculating the three-dimensional scheduling score.

[0087] Furthermore, in the embodiments, the cross-computing stack migration cost factor in step S3 can also be considered. The generation method is as follows: count the number of operators in the directed acyclic graph that cannot be natively supported by the target node hardware and need to be rolled back to the CPU for execution. Total number of operators Combined with the bus copy delay coefficient of the target node Calculate using the following formula:

[0088]

[0089] in, Data was collected through polling using hardware physical probes deployed on each heterogeneous computing node.

[0090] In this embodiment, the cross-computing stack migration cost factor in the hardware matching score is described in detail. The calculation method.

[0091] As attached Figure 2 As shown, the architecture of this invention consists of three core physical layers from top to bottom: the top layer is the business closed loop and scheduling control plane, the middle layer is the cross-layer parsing and compilation linkage plane, and the bottom layer is the heterogeneous physical telemetry plane.

[0092] (1) DAG analysis and total operator statistics:

[0093] The DAG parsing and operator comparison module intercepts AI task resource requests at the Kubernetes scheduling execution plane, extracts the directed acyclic graph (DAG) operator graph of the task (such as ONNX or Torch FX format), sends the DAG to the compilation linkage layer, and parses it to obtain the total number of operators. .

[0094] (2) Operator compatibility cross-comparison:

[0095] The DAG parsing and operator comparison module performs cross-validation by externally calling the native compilation libraries of the target heterogeneous nodes (such as the NVIDIA CUDA Runtime operator dictionary or the Huawei Ascend CANN Runtime operator dictionary) to accurately separate and count the number of operators that cannot be natively supported by the target chip and must be rolled back to the host CPU for execution. .

[0096] (3) Physical probe acquisition bus copy delay:

[0097] The heterogeneous telemetry acquisition module uses hardware physical probes deployed on each heterogeneous computing node (such as mounting an operating system kernel-mode eBPF probe, or calling native monitoring services from chip manufacturers such as NVIDIA DCGM) to perform real-time polling at a high-frequency sampling period of milliseconds (e.g., 100ms), collecting and reporting the bus copy latency coefficient. .

[0098] (4) Calculation and Feedback:

[0099] The multidimensional scoring module integrates the above parameters and calculates the absolute migration cost using the following formula:

[0100]

[0101] This cost is pushed to the top-level multi-dimensional scoring module in real time through a global penalty constraint feedback loop, thus breaking down the technical barrier between the compilation layer and the scheduling layer.

[0102] Furthermore, in the embodiments, it can also be considered that in step S3, the dynamic load perception score is determined by the modal anti-interference efficiency factor. and the memory fragmentation index of the target node The fusion calculation yields the following formula:

[0103] ;

[0104] in, ;

[0105] For tasks to resources Sensitivity weights, For the target node in resources Real-time resource utilization Includes bus, Video memory bandwidth, cache;

[0106] ;

[0107] For the target node Number of idle GPUs within each NUMA / Socket domain;

[0108] Modality-driven fragmentation sensitivity for the target node;

[0109] ;

[0110]

[0111] To analyze the tensor parallel or pipeline parallel communication frequency feature values ​​extracted from the directed acyclic graph;

[0112] This represents the topology penalty coefficient for heterogeneous nodes. This is an adjustable topology penalty scaling factor for the project.

[0113] In this embodiment, as shown in the appendix Figure 3 As shown, the entire calculation process is divided into business modality feature extraction on the left branch and real-time telemetry of nodes on the right branch.

[0114] (1) Modal anti-interference efficiency factor Calculation:

[0115] The multidimensional scoring module will obtain the modal resource sensitivity vector. Real-time occupancy rate of physical resources acquired at high frequency through heterogeneous telemetry acquisition modules The anti-interference factor is calculated by the multi-dimensional scoring module according to the following formula. :

[0116]

[0117] in, The sensitivity weights of the task mode to a specific physical resource r. This represents the real-time telemetry occupancy rate of the target node on physical resource r.

[0118] For example, in the railway large language model inference task, its sensitivity weight to HBM memory bandwidth is... Take the higher value, for example =0.9); for petabyte-scale laser point cloud processing tasks, its sensitivity weight to PCIe bus throughput is... Take the higher value, for example =e.g., 0.85.

[0119] (2) Memory fragmentation index Calculation:

[0120] The multi-dimensional scoring module calculates the dispersion of idle GPUs within the target node across different physical locations, such as different CPU sockets or PCIe domains, to obtain the system fragmentation index. The calculation formula is:

[0121]

[0122] in This represents the number of currently available idle GPUs within the k-th NUMA / Socket domain.

[0123] NUMA (Non-Uniform Memory Access) refers to a local resource domain in a multiprocessor server, consisting of several CPUs, memory controllers, and their directly connected GPUs. Data access between different NUMA domains typically requires cross-bus or cross-processor interconnects, resulting in higher communication latency. A socket refers to a physical CPU slot in a server. In practical heterogeneous computing servers, one socket usually corresponds to one NUMA domain and connects to several GPU accelerator cards. This invention defines a set of GPUs within the same socket or the same NUMA domain as a local resource domain, used to evaluate the dispersion of GPU resources across different physical domains.

[0124] In general, the index When all idle GPUs are concentrated in the same socket, =0 (zero fragmentation); when idle GPUs are scattered across multiple different sockets. Approaching 1 (high fragmentation). Since cross-domain communication can cause severe QPI / UPI bus latency, this formula accurately converts the discrete state of this physical topology into a quantized exponent.

[0125] (3) Modal-driven fragmentation sensitivity Calculation:

[0126] The DAG parsing and operator comparison module extracts tensor parallel or pipelined parallel communication frequency features from the directed acyclic graph of the parsing task. Combined with the topology penalty coefficient of heterogeneous nodes Calculate using the following formula :

[0127]

[0128] For large model fine-tuning tasks involving multi-card collaboration Take the higher value; for independent vision tasks that do not require inter-card communication, Approaching 0.

[0129] This represents the topology penalty coefficient for heterogeneous nodes. The adjustable topology penalty scaling factor for the project is set to a range of values. ).

[0130] When the task is executed completely isomorphically natively on the target node ( ), At this point, the task only suffers from the basic fragmentation penalty caused by its own DAG communication frequency; however, once cross-ecosystem scheduling occurs, such as operator incompatibility and fallback to the CPU, rise, It will be linearly amplified.

[0131] Not all tasks are afraid of memory fragmentation. For independent concurrent image recognition tasks in railway scenarios, even if memory fragmentation is severe ( Approaching 1), since there is no communication requirement between its tasks, the system calculates... Exponential decay failure allows tasks to be packed into fragmented GPU memory, greatly improving resource utilization. For fine-tuning large models requiring multi-GPU collaboration, If the value is extremely high, the system will trigger a severe exponential penalty, forcing it to search for contiguous video memory.

[0132] (3) Fusion computing:

[0133] The multidimensional scoring module calculates the dynamic load perception score using the following exponential decay formula:

[0134] Furthermore, in the embodiments, the calculation method of the topological locality score in step S3 can be considered as follows:

[0135]

[0136] in, The feature data volume of the AI ​​task is obtained by the scheduling and execution module when receiving the task by parsing the task declaration file or querying the dataset metadata;

[0137] The scheduling execution module obtains and caches the actual physical link bandwidth of the shortest path between candidate nodes and related nodes in advance through a hardware topology detection tool during the cluster initialization phase.

[0138] The network latency sensitivity penalty factor is determined based on the task modality, with a larger value for high real-time tasks and a smaller value for offline batch processing tasks.

[0139] In this embodiment, the multidimensional scoring module calculates the topological locality score according to the following formula. :

[0140]

[0141] in:

[0142] The scheduling and execution module obtains the characteristic data volume of the AI ​​task by parsing the task declaration file or querying the dataset metadata when receiving the task. To obtain and cache the actual physical link bandwidth of the shortest path between candidate nodes and related nodes, the scheduling execution module uses hardware topology detection tools, such as nvidia-smi topo-m and hwloc, during the cluster initialization phase.

[0143] The penalty factor is determined based on the task modality. For high real-time tasks, such as railway inspection edge-cloud collaborative fault alarms, a larger value is used, ranging from 5 to 10. For offline batch processing tasks, such as offline 3D reconstruction of historical point cloud data, a smaller value is used, ranging from 0.1 to 1. The scheduling execution module is allowed to sacrifice some communication performance for distributed allocation across low-speed switches in order to achieve higher global computing power card utilization.

[0144] Furthermore, in the embodiments, the dynamic generation method of the scheduling weight in step S4 can be considered as follows:

[0145] The scheduling and execution module obtains the cluster's global memory fragmentation rate. Global I / O congestion Global network bandwidth saturation ;

[0146] The scheduling and execution module determines the basic demand base based on the task modality label mapping. ;

[0147] Calculate the hardware compatibility weight using the following formula. Load-aware weights Topological locality weights :

[0148]

[0149]

[0150]

[0151] The formula for calculating the overall score is as follows:

[0152]

[0153] in, Score the hardware compatibility. For dynamic load perception score, The score is for topological locality.

[0154] The calculation formula is:

[0155]

[0156] in, The current AI task requests the following video memory capacity: For the target server node The current actual total free video memory capacity, For the architecture's dynamic performance coefficient, This is the cost factor for migration across computing stacks.

[0157] Furthermore, the cluster's global memory fragmentation rate can also be considered in the embodiments. Take the memory fragmentation index of all target nodes The maximum value in; the global I / O congestion level. Take the average PCIe bandwidth utilization of all candidate nodes; the global network bandwidth saturation. The bandwidth utilization of cross-node physical links is obtained by collecting data through container network interfaces and switch SNMP protocols.

[0158] In this embodiment, as shown in the appendix Figure 4 As shown, the system first collects the current global memory fragmentation rate of the computing cluster in real time through the global state aggregation module. Global I / O congestion Global network bandwidth saturation This forms the global state vector.

[0159] in, Take the memory fragmentation index of all candidate nodes The maximum value in, Take the average PCIe bandwidth utilization of all candidate nodes. The cross-node physical link bandwidth utilization is obtained by collecting data through the container network interface and the switch's SNMP protocol. Subsequently, this state vector is fed back to the environment's dynamic weighting module and compared with the pre-parsed task modality's basic requirement base. Perform Hadamard product and Softmax nonlinear exponent normalization calculations, and output the hardware matching degree weights according to the formula. Load-aware weights Topological locality weights This dynamic weighting mechanism enables the system to intelligently rebalance weights under extreme conditions such as when cluster resources are on the verge of collapse. For example, when the cluster faces extreme memory shortages, The weight of hardware compatibility increases dramatically, and the system automatically and exponentially increases it. Prioritize ensuring that the task does not experience memory overflow.

[0160] Example 2:

[0161] A heterogeneous computing power multidimensional weighted scheduling system for railway engineering AI tasks, used to execute the aforementioned heterogeneous computing power multidimensional weighted scheduling method for railway engineering AI tasks, includes:

[0162] DAG parsing and operator comparison module: used to intercept the directed acyclic graph of AI tasks and cross-compare it with the heterogeneous computing runtime library of the target node, and output the number of operator backoffs;

[0163] Heterogeneous telemetry acquisition module: used to collect the hardware resource utilization and bus copy latency of each node through hardware physical probes;

[0164] Global Status Aggregation Module: Used to summarize telemetry data from each node and generate global memory fragmentation rate, global I / O congestion level, and global network bandwidth saturation.

[0165] The environmental feedback dynamic weighting module is used to generate scheduling weights for each dimension based on the global state vector and task mode.

[0166] Multi-dimensional scoring module: used to calculate hardware matching score, dynamic load awareness score, topology locality score and comprehensive score;

[0167] Scheduling execution module: Used to select the node with the highest overall score and issue scheduling instructions.

[0168] This embodiment provides a heterogeneous computing power multidimensional weighted scheduling system for AI tasks in railway engineering. This system is used to execute the method described in Embodiment 1.

[0169] As attached Figure 5 As shown, this system is divided into five physical / logical layers from bottom to top:

[0170] First layer: Heterogeneous physical resource layer

[0171] It includes coexisting hardware such as NVIDIA GPUs, Huawei Ascend NPUs, and CPU clusters. The NVIDIA GPUs are interconnected via NVLink based on CUDA Runtime, while the Huawei Ascend NPUs are interconnected via HCCL based on CANN Runtime.

[0172] Second layer: Multi-source data perception and fusion layer

[0173] A DAG operator graph interceptor and a heterogeneous hardware unified telemetry agent are deployed. The DAG operator graph interceptor is used to intercept directed acyclic graphs (DAGs) for AI tasks; the heterogeneous hardware unified telemetry agent is used to collect hardware resource utilization and bus copy latency of each node through hardware physical probes. This layer corresponds to the functional implementation of the DAG parsing and operator comparison module and the heterogeneous telemetry acquisition module.

[0174] Third layer: Strategy core computation layer

[0175] This layer specifically implements the DAG parsing and operator comparison module and the multi-dimensional scoring module. In addition, this layer is also responsible for summarizing and aggregating the raw telemetry data collected from each node in the second layer to generate global memory fragmentation rate, global I / O congestion, and global network bandwidth saturation, thus realizing the core function of the global state aggregation module.

[0176] Fourth layer: Control and decision-making layer

[0177] It consists of an environmental feedback dynamic weighting module and a multi-dimensional scoring module. The environmental feedback dynamic weighting module is used to generate scheduling weights for each dimension based on the global state vector; the multi-dimensional scoring module is used to calculate hardware matching score, dynamic load awareness score, topology locality score, and comprehensive score.

[0178] Fifth layer: Business closed loop and execution layer

[0179] This layer corresponds to the scheduling and execution module, which interfaces with the native Kubernetes architecture via API to transform the optimal scoring decision at the lower level into actual container orchestration and placement instructions, thereby achieving closed-loop management of software and hardware collaboration.

[0180] The above-mentioned DAG parsing and operator comparison module, heterogeneous telemetry acquisition module, global state aggregation module, environmental feedback dynamic weighting module, multi-dimensional scoring module, and scheduling execution module are integrated into the Kubernetes scheduler extension plugin and deployed in the form of Scheduler Extender.

[0181] Furthermore, in the embodiments, the multidimensional scoring module can also be considered as follows:

[0182] The calculation of the hardware matching score includes a cross-computing stack migration cost factor, which is generated based on the number of operator backoffs output by the DAG parsing and operator comparison module and the bus copy delay coefficient acquired by the heterogeneous telemetry acquisition module.

[0183] The calculation of the dynamic load perception score includes the memory fragmentation index of the target node and the modality-driven fragmentation sensitivity determined based on the directed acyclic graph of the AI ​​task.

[0184] The calculation of the topological locality score includes the ratio of the amount of task feature data to the actual physical link bandwidth obtained by the heterogeneous telemetry acquisition module.

[0185] Furthermore, in the embodiments, the heterogeneous telemetry acquisition module may include one or more of the following components:

[0186] Hardware physical probes deployed on each heterogeneous computing node are used to poll and collect bus copy latency coefficients at millisecond intervals.

[0187] Low-level monitoring tools deployed on each heterogeneous computing node, such as NVIDIA DCGM, Ascend-smi, and PCIe Telemetry probes, are used to collect PCIe bus bandwidth utilization, HBM memory bandwidth utilization, and L2 cache utilization.

[0188] The hardware physical probes include one or more of the following types: kernel-mode eBPF probe, NVIDIA DCGM monitoring service, Ascend-smi monitoring service, and PCIe Telemetry probe.

[0189] Furthermore, in the embodiments, the scheduling execution module, the DAG parsing and operator comparison module, the heterogeneous telemetry acquisition module, the global state aggregation module, the environmental feedback dynamic weighting module, and the multidimensional scoring module can be integrated into the Kubernetes scheduler extension plugin and deployed in the form of Scheduler Extender.

[0190] Application Example 1:

[0191] To verify the actual performance of the above methods and systems, this application example constructs a typical railway edge-cloud collaborative heterogeneous computing power cluster and conducts high-frequency concurrent tidal stress tests.

[0192] High-frequency concurrent tidal stress tests were performed in the following hardware and software environment.

[0193] I. Hardware and software environment configuration

[0194] Heterogeneous computing cluster scale: It comprises 12 high-performance computing nodes. Eight nodes are equipped with NVIDIA A100 GPUs (8 GPUs per node, interconnected via high-speed NVLink within the node); four nodes are equipped with Huawei Ascend 910B NPUs (8 GPUs per node, interconnected via HCCL within the node). The inter-node network uses 100G RDMA Ethernet, and the internal host bus is PCIe 4.0. The basic scheduling framework deploys a standard Kubernetes cluster, and the system of this invention is mounted as a SchedulerExtender plugin.

[0195] II. Multimodal Test Task Pool

[0196] A total of 10,000 railway-specific AI tasks were deployed to the cluster at high frequency and concurrently, including three modalities:

[0197] Railway-specific large language model inference (LLM): extremely consumes continuous high-bandwidth video memory (HBM), accounting for 40%, 4000.

[0198] PB-level laser point cloud processing along railway lines: heavily reliant on PCIe bus bandwidth and cross-node I / O throughput, accounting for 30%, 3000 nodes.

[0199] 3D BIM rendering of major railway hub stations: computationally intensive, requiring a balance between computing power and video memory, accounting for 30%, 3000 units.

[0200] The above task distribution ratio is not random, but is set as a fixed ratio of 4:3:3 based on the typical railway edge-cloud collaborative business load.

[0201] III. Control Group and Evaluation Indicators

[0202] Control group A: Kubernetes default scheduler (K8sDefault);

[0203] Control group B: Volcano scheduler, currently the mainstream cloud-native AI batch processing scheduling framework in the industry, with certain task queuing and basic topology awareness capabilities.

[0204] Experimental group (Proposed): The multidimensional dynamic weighted scheduling algorithm proposed in this invention.

[0205] Evaluation metrics include: total cluster completion time (Makespan), average global accelerator card utilization, memory overflow (OOM) crash rate, cross-architecture operator fallback rate, network / bus congestion rate for high-throughput tasks, and average end-to-end task response latency.

[0206] IV. Test Results

[0207] Under the same load test of 10,000 concurrent tasks, the method of this invention achieved significant technical advantages compared with the control group:

[0208] Under the same load test of 10,000 concurrent tasks, the specific performance of each algorithm is shown in the table below:

[0209] Table 1

[0210] Total Cluster Completion Time (Makespan) 14.5 hours 11.2 hours 10.1 hours Shortened by 30.3% Global accelerator card average utilization 42.60% 61.30% 78.00% An increase of 35.4 percentage points Task memory overflow (OOM) crash rate 18.40% 11.70% 4.00% Decreased by 14.4 percentage points Cross-architecture operator fallback ratio 31.80% 29.50% Less than 1% Reduced by more than 30 percentage points High-throughput task network / bus congestion rate 27.20% 14.10% 9.30% A decrease of 17.9 percentage points Average response time for end-to-end tasks 4.2 seconds 3.5 seconds 2.6 seconds Shortened by 38.1%

[0211] Experimental results show that:

[0212] (1) In the control group, because K8s and Volcano can only perceive "memory availability" but not "underlying compilation ecosystem," a large number of tasks natively compiled based on CUDA were blindly assigned to the Ascend 910B node. This resulted in nearly 31.8% of tasks experiencing architectural incompatibility, forcing many operators to fall back to host CPU computation, causing severe PCIe data copy-back latency. However, this invention introduces a cross-computing stack migration cost factor (…). During the scheduling and scoring phase, a severe mathematical penalty was imposed on cross-architecture mismatches that caused fallbacks. Prioritized task guidance achieved closed loops within the homogeneous stack, effectively reducing heterogeneous computing power translation losses and increasing the overall cluster utilization to 78%. The cross-architecture operator fallback rate decreased from 31.8% to below 1%, effectively avoiding the decline in execution efficiency caused by architecture mismatches.

[0213] (2) During peak congestion periods when large model inference and BIM rendering occur simultaneously, cluster memory becomes highly fragmented. Traditional schedulers, due to their static weight fixation, still cram large model tasks into fragmented memory, resulting in 18.4% of tasks experiencing OOM (Out of Memory) crashes. This invention sensitively captures the global fragmentation risk through a dynamic environment feedback function and, combined with modal features, adaptively increases the weight of large model tasks on contiguous memory. At the same time, it dynamically fills the fragmented space with independent visual tasks that do not require inter-card communication, reducing the OOM rate to 4%.

[0214] (3) The control group often misallocated the network hops to cross-rack nodes connected only by ordinary PCIe, causing a network storm instantly. The present invention uses an absolute bandwidth division penalty model to accurately identify the high throughput requirements of point cloud tasks and force them to converge to nodes in the same rack with NVLink or HCCL high-speed buses, reducing the congestion rate from 27.2% to 9.3% and the end-to-end latency to 2.6 seconds, meeting the high real-time requirements of railway edge-cloud collaborative scenarios.

[0215] The above embodiments have provided a detailed description of the present invention, but the content described is only a preferred embodiment of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A heterogeneous computing power multidimensional weighted scheduling method for AI tasks in railway engineering, characterized in that, Includes the following steps: S1: The scheduling and execution module receives AI tasks and obtains their directed acyclic graph; S2: The scheduling and execution module filters heterogeneous nodes in the cluster based on the hard constraint of video memory capacity, and removes nodes whose remaining video memory capacity is less than the task request amount, thus obtaining a set of candidate nodes. S3: The scheduling execution module calculates a three-dimensional scheduling score for each node in the candidate node set; The three-dimensional scheduling score includes: hardware matching score, dynamic load awareness score, and topology locality score. The hardware matching score includes a cross-computing stack migration cost factor, which is generated by comparing the operator compatibility of the directed acyclic graph with the heterogeneous computing runtime library of the target node. S4: The scheduling execution module obtains the cluster global physical state vector and dynamically generates scheduling weights for each dimension based on the global physical state vector; S5: The scheduling execution module uses the scheduling weight to perform a weighted summation of the three-dimensional scheduling scores, selects the node with the highest comprehensive score as the target scheduling node, and issues a scheduling instruction to execute container placement.

2. The heterogeneous computing power multidimensional weighted scheduling method for AI tasks in railway engineering according to claim 1, characterized in that, In step S3, the cross-computing stack migration cost factor The generation method is as follows: count the number of operators in the directed acyclic graph that cannot be natively supported by the target node hardware and need to be rolled back to the CPU for execution. Total number of operators Combined with the bus copy delay coefficient of the target node Calculate using the following formula: ； in, Data was collected through polling using hardware physical probes deployed on each heterogeneous computing node.

3. The heterogeneous computing power multidimensional weighted scheduling method for AI tasks in railway engineering according to claim 1, characterized in that, In step S3, the dynamic load perception score is determined by the modal anti-interference efficiency factor. and the memory fragmentation index of the target node The fusion calculation yields the following formula: ； in, ; For tasks to resources Sensitivity weights For the target node in resources Real-time resource utilization Includes bus, Video memory bandwidth, cache; ； For the target node Number of idle GPUs within each NUMA / Socket domain; Modality-driven fragmentation sensitivity for the target node; ； ； To analyze the tensor parallel or pipeline parallel communication frequency feature values ​​extracted from the directed acyclic graph; This represents the topology penalty coefficient for heterogeneous nodes. This is an adjustable topology penalty scaling factor for the project.

4. The heterogeneous computing power multidimensional weighted scheduling method for AI tasks in railway engineering according to claim 1, characterized in that, In step S3, the topological locality score is calculated as follows: ； in, The feature data volume of the AI ​​task is obtained by the scheduling and execution module when receiving the task by parsing the task declaration file or querying the dataset metadata; The scheduling execution module obtains and caches the actual physical link bandwidth of the shortest path between candidate nodes and related nodes in advance through a hardware topology detection tool during the cluster initialization phase. The network latency sensitivity penalty factor is determined based on the task modality, with a larger value for high real-time tasks and a smaller value for offline batch processing tasks.

5. The heterogeneous computing power multidimensional weighted scheduling method for AI tasks in railway engineering according to claim 1, characterized in that, In step S4, the dynamic generation method of the scheduling weight is as follows: The scheduling and execution module obtains the cluster's global memory fragmentation rate. Global I / O congestion Global network bandwidth saturation ; The scheduling and execution module determines the basic demand base based on the task modality label mapping. ; Calculate the hardware compatibility weight using the following formula. Load-aware weights Topological locality weights : ； ； ； The formula for calculating the overall score is as follows: ； in, Score the hardware compatibility. For dynamic load perception score, The score is for topological locality. The calculation formula is: ； in, The current AI task requests the following video memory capacity: For the target server node The current actual total free video memory capacity, For the architecture's dynamic performance coefficient, This is the cost factor for migration across computing stacks.

6. The heterogeneous computing power multidimensional weighted scheduling method for AI tasks in railway engineering according to claim 5, characterized in that: The cluster's global memory fragmentation rate Take the memory fragmentation index of all target nodes The maximum value in; the global I / O congestion level. Take the average PCIe bandwidth utilization of all candidate nodes; the global network bandwidth saturation. The bandwidth utilization of cross-node physical links is obtained by collecting data through container network interfaces and switch SNMP protocols.

7. A heterogeneous computing power multidimensional weighted scheduling system for railway engineering AI tasks, used to execute the heterogeneous computing power multidimensional weighted scheduling method for railway engineering AI tasks as described in any one of claims 1-6, characterized in that, include: DAG parsing and operator comparison module: used to intercept the directed acyclic graph of AI tasks and cross-compare it with the heterogeneous computing runtime library of the target node, and output the number of operator backoffs; Heterogeneous telemetry acquisition module: used to collect the hardware resource utilization and bus copy latency of each node through hardware physical probes; Global Status Aggregation Module: Used to summarize telemetry data from each node and generate global memory fragmentation rate, global I / O congestion level, and global network bandwidth saturation. The environmental feedback dynamic weighting module is used to generate scheduling weights for each dimension based on the global state vector and task mode. Multi-dimensional scoring module: used to calculate hardware matching score, dynamic load awareness score, topology locality score and comprehensive score; Scheduling execution module: Used to select the node with the highest overall score and issue scheduling instructions.

8. The heterogeneous computing power multidimensional weighted scheduling system for AI tasks in railway engineering according to claim 7, characterized in that, In the multidimensional scoring module: The calculation of the hardware matching score includes a cross-computing stack migration cost factor, which is generated based on the number of operator backoffs output by the DAG parsing and operator comparison module and the bus copy delay coefficient acquired by the heterogeneous telemetry acquisition module. The calculation of the dynamic load perception score includes the memory fragmentation index of the target node and the modality-driven fragmentation sensitivity determined based on the directed acyclic graph of the AI ​​task. The calculation of the topological locality score includes the ratio of the amount of task feature data to the actual physical link bandwidth obtained by the heterogeneous telemetry acquisition module.

9. The heterogeneous computing power multidimensional weighted scheduling system for AI tasks in railway engineering according to claim 7, characterized in that, The heterogeneous telemetry acquisition module includes one or more of the following components: Hardware physical probes deployed on each heterogeneous computing node are used to poll and collect bus copy latency coefficients at millisecond intervals. The underlying monitoring tools deployed on each heterogeneous computing node are used to collect PCIe bus bandwidth utilization, HBM memory bandwidth utilization, and L2 cache utilization. The hardware physical probes include one or more of the following types: kernel-mode eBPF probe, NVIDIA DCGM monitoring service, Ascend-smi monitoring service, and PCIe Telemetry probe.

10. The heterogeneous computing power multidimensional weighted scheduling system for railway engineering AI tasks according to claim 7, characterized in that: The scheduling execution module, the DAG parsing and operator comparison module, the heterogeneous telemetry acquisition module, the global state aggregation module, the environmental feedback dynamic weighting module, and the multidimensional scoring module are integrated into the Kubernetes scheduler extension plugin and deployed in the form of Scheduler Extender.

Images