Artificial intelligence platform computing power resource scheduling management method and system

Abstract

The invention discloses an artificial intelligence platform computing power resource scheduling management method and system, and relates to the technical field of artificial intelligence computing power scheduling, and the method comprises the specific steps: hardware topology collection traverses heterogeneous resources, and generates and updates a graph; the collaborative strategy decision is based on the atlas, and optimal precision and parallel combination are screened; load balancing deploying split tasks, matching hardware capability and planning a transmission path; executing and monitoring multi-dimensional real-time tracking; the quantization deviation is dynamically adapted and adjusted, and the strategy is timely optimized; according to the invention, through hardware topology acquisition and collaborative strategy decision, a computing power resource optimal configuration scheme is formed, and the training task starting efficiency and stability are improved; load balancing deployment and dynamic adaptation adjustment are combined, resource balancing allocation and whole-process dynamic optimization are achieved, the problems of resource imbalance and response lag are effectively solved, the computing power utilization efficiency is improved to the maximum extent, and efficient and stable training is guaranteed.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of artificial intelligence computing power scheduling, and particularly relates to an artificial intelligence platform computing power resource scheduling management method and system. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the complexity of training tasks is continuously improved, the demand for computing power resources is continuously increased, and a heterogeneous hardware architecture gradually becomes a mainstream configuration of an artificial intelligence platform. Such a platform usually integrates multiple hardware devices of different types, and various types of hardware have significant differences in computing capability, storage capacity, communication characteristics, and the like. The topology between devices is also more complex. In order to ensure efficient progress of artificial intelligence training tasks, computing power resource scheduling management becomes a key link, and it is necessary to fully grasp the real-time state and correlation of hardware resources, reasonably match precision configuration and parallel strategy, and balance task allocation, data transmission and storage optimization to cope with dynamically changing load demand and hardware state in the training process. Under this background, how to realize comprehensive perception of heterogeneous hardware resources, accurate strategy formulation, efficient load deployment and dynamic adaptive adjustment has become a core requirement for improving the utilization efficiency of artificial intelligence platform computing power and ensuring the stability of training, and has also promoted the development of computing power resource scheduling management technology in a more refined and intelligent direction.

[0003] The traditional artificial intelligence platform computing power resource scheduling management method has many deficiencies in dealing with heterogeneous hardware environments and complex training requirements. In the aspect of hardware information processing, it is often difficult to comprehensively collect the basic parameters of heterogeneous hardware and the topology data between devices, or the data is not updated in time, which leads to a lack of complete and accurate hardware state support for scheduling decisions, and it is difficult to fully adapt to the characteristics of different hardware. In the strategy formulation link, the combination optimization of mixed precision and mixed parallel is insufficient, and it is mostly dependent on experience configuration, which cannot realize the deep adaptation of precision requirements and hardware performance, and is prone to low computing efficiency or precision loss exceeding the standard. In the load deployment process, the matching degree of task splitting and hardware capability is low, and the data transmission path planning and storage rule design lack sufficient consideration of the hardware topology, which is prone to cause resource load imbalance, increase data transmission time consumption and storage conflict risk. In addition, the traditional method has weak dynamic adjustment capability in the training process, and cannot optimize the scheduling scheme in time according to the training stage, hardware state change and load fluctuation, which leads to problems such as waste of computing power, precision deviation or unstable operation in the later training stage, and cannot meet the efficient and stable artificial intelligence training requirements. SUMMARY

[0004] The present application aims to make up for the deficiencies of the prior art, and provides an artificial intelligence platform computing resource scheduling management method and system, which generates a graph through hardware topology collection, determines the optimal combination of mixed precision and parallelism using a collaborative adaptation quantification algorithm, and realizes load balancing deployment. The method covers hardware perception, strategy decision, task splitting and allocation, real-time monitoring and dynamic adaptation adjustment, ensuring efficient use of computing power. The system includes collection, decision, deployment, monitoring and adjustment modules, supports multi-interface collection and three-level alarm, can dynamically adjust the trigger threshold according to the training stage, and optimizes the calculation efficiency and accuracy.

[0005] To solve the above technical problems, the present application provides the following technical solutions: on the one hand, an artificial intelligence platform computing resource scheduling management method, which comprises:

[0006] Hardware topology collection: through a hardware perception agent program, the heterogeneous hardware resources of the artificial intelligence platform are traversed, the hardware basic parameters and the inter-device topology structure data are collected, the hardware topology graph is generated, and the data is stored and updated in real time through a database;

[0007] Collaborative strategy decision: based on the hardware topology graph, the optimal combination of mixed precision and mixed parallelism is screened through a collaborative adaptation quantification algorithm, and at the same time, the precision configuration mode, the parallel strategy combination and the switching trigger condition are determined, and the associated parameters are output;

[0008] Load balancing deployment step: according to the optimal collaborative scheduling strategy, the training task is split, the subtask parameters and the topology associated parameters are extracted, the associated parameters are reused, the subtask load characteristics are calculated using a topology perception load characteristic algorithm, the task is allocated through a hardware capability-load characteristic matching technology, the data transmission path is planned, and the data localization storage rule is established;

[0009] Execution and monitoring: the task allocation table is issued under the multi-interface, the data distribution and calculation are started, and the multi-dimensional real-time monitoring is started, the hardware, communication, training and algorithm associated indicators are collected, the state data stream is formed after flow processing, and the indicators are monitored using the threshold values;

[0010] Dynamic adaptation adjustment: the reference parameters are extracted, the reference trigger threshold is set, the deviation is quantified using a dynamic adjustment trigger threshold algorithm, it is judged whether the adjustment is triggered, if the adjustment is triggered, the training state is saved, the scheme is regenerated and optimized in the collaborative strategy decision step, and the data is recorded.

[0011] Further, in the hardware topology collection, the hardware-aware agent supports multi-interface collaborative collection, and the hardware basic parameters include a CPU model, a core number, NUMA node attribution, a GPU model, a computing capability, and a video memory capacity, an NPU model, an algorithmic capacity specification, and supported quantization precision, wherein the CPU parameters are collected through a CPUID instruction and a numactl tool, the GPU parameters are collected through a GPU driver interface, and the NPU parameters are collected through an NPU SDK; and the inter-device topology structure data includes inter-device connection modes, communication bandwidth, and delay, NUMA node distribution, and GPU interconnection architecture, wherein the connection modes and the communication parameters are collected through bus scanning and network detection tools, and the node distribution and the interconnection architecture are collected through system file analysis and driver interface query.

[0012] Further, in the hardware topology collection, the hardware topology graph is constructed in a graph structure data format, a node in the graph corresponds to a single hardware device or a NUMA node, and node attributes include a hardware type, a model, a computing capability, a storage capacity, an energy consumption parameter, a real-time running state, a fault history record, a cumulative running time length, and a maintenance period, an edge in the graph corresponds to a physical connection relationship between devices, and edge attributes include a connection protocol, a communication bandwidth, a delay value, a link error rate, a link occupancy rate, and historical communication peak data, and the graph is stored in a special database.

[0013] Further, in the collaborative strategy decision, a mathematical expression of a collaborative adaptation quantification algorithm is as follows: wherein S is a collaborative adaptation degree, a is a hardware topology weight coefficient, H is a hardware topology adaptation coefficient, G is a precision-parallel collaborative gain, β is a computing efficiency weight coefficient, n is a number of mixed precision modes, is a hardware acceleration coefficient of the i-th precision mode, P is a computing proportion corresponding to the i-th precision mode, is a compatibility coefficient of the i-th precision mode and a parallel strategy, γ is a precision loss penalty coefficient, and Δε is a precision loss rate of mixed precision relative to full precision. i

[0014] ​​​Further, in the collaborative strategy decision, the precision configuration mode is determined by hardware precision adaptation technology, CPU adapts FP32 precision, GPU adapts FP16 precision, NPU adapts INT8 precision, the core link adopts FP32+FP16 mixed configuration when the convergence precision requirement is less than or equal to 2%, adopts FP16 precision when the convergence precision requirement is between 2% and 5%, and adopts INT8+FP16 mixed configuration when the convergence precision requirement is greater than 5%; the parallel strategy combination is determined by parameter scale-hardware matching technology, data parallel is used when the model parameter is less than 100 billion, data parallel+tensor parallel is used when the model parameter is between 100 billion and 1000 billion, and three-mode combination is used when the model parameter is greater than 1000 billion; the number of parallel slices is consistent with the number of hardware; the switching trigger condition is determined by state threshold monitoring technology, and the switching is performed according to the preset device selection rule when any one of the following conditions is met: the number of iterations reaches 70% of the total budget, the change rate of iteration loss is less than or equal to 0.01% for 1000 consecutive iterations, the hardware utilization rate is greater than or equal to 95% or less than or equal to 60% for 3 seconds.

[0015] Further, in the load balancing deployment, the current task is split according to the optimal collaborative scheduling strategy by using hierarchical splitting technology, the splitting dimension is determined according to the mixed parallel strategy type, the training data is split according to the number of hardware nodes in the data parallel dimension, the model parameters are split according to the number of GPU cores in the tensor parallel dimension, and the model modules are split according to the network layer structure in the model parallel dimension; the splitting proportion is allocated according to the computing capacity and storage capacity of each hardware device; the subtask parameters and topological correlation parameters are extracted by using double-source extraction technology, the subtask parameters are extracted from the training task configuration and splitting result, including the computation amount, data transmission amount, precision requirement and completion time limit of each subtask, and the topological correlation parameters are extracted from the hardware topology map, including the communication bandwidth, delay value and data locality identification of the transmission link corresponding to the subtask, and the computing capacity, storage capacity and interface type of the allocation target device.

[0016] Further, in the load balancing deployment, the expression of the topology-aware load characteristic algorithm is: wherein, R ′ is the optimized topology-aware communication-computing ratio, k is the number of training subtasks, D t is the data transmission amount of the tthsubtask, L t is the transmission link delay of the tthsubtask, L loc,t is the data locality gain coefficient of the tthsubtask, F t is the original computation amount of the tthsubtask, k p,t is the precision acceleration coefficient corresponding to the tthsubtask, H dov,t is the hardware efficiency coefficient of the tthsubtask allocation device, and λ is the parallel strategy correction factor.

[0017] Further, in the load balancing deployment, the task is allocated by hardware capability-load characteristic matching technology, based on the degree of adaptation of sub-task load characteristics and hardware device core capability, high communication-computing ratio sub-task is allocated to hardware cluster with communication bandwidth ≥200GB / s, high computing-communication ratio sub-task is allocated to hardware device with computing power ≥1000TOPS, and medium load characteristic sub-task is allocated to remaining hardware; the data transmission path is planned by link priority sorting technology, the transmission link is sorted from high to low according to communication bandwidth and from low to high according to delay value, and the same NUMA node link or GPU interconnection architecture link is preferentially selected; the data localization storage rule established by the storage location binding technology includes storage location binding, capacity reservation, data classification storage and migration mechanism, the storage location binding establishes fixed association between input data, intermediate calculation result and output data of sub-task and local storage or distributed storage partition of the allocated device, the capacity reserves storage space according to 1.2 times of the total amount of sub-task data, the data classification storage stores input data according to training batch partition, stores model shard preferentially in local video memory of the allocated device, temporarily stores intermediate result in local memory or cache, and synchronously stores output data to distributed storage for backup, the migration mechanism provides that when the local storage occupancy rate reaches 85%, the intermediate result not called by the current calculation is migrated to the distributed storage of the same NUMA node, and the migration process is only performed in the gap between sub-task calculations and does not interrupt the current training process.

[0018] Further, in the execution and monitoring, the index threshold adopts three-level hierarchical alarm rules, the first-level alarm trigger condition is hardware device failure and training accuracy error ≥5%, the notification mode is short message combined with platform pop-up window, the response time is ≤10 seconds, and the alarm information includes fault device ID, error value, occurrence time and emergency treatment suggestion; the second-level alarm trigger condition is hardware load imbalance ≥15% and cross-device communication delay exceeding standard ≥50%, the notification mode is email combined with platform notification, the response time is ≤30 seconds, and the alarm information includes imbalance device identification, delay value, link information and adjustment direction; the third-level alarm trigger condition is that the index fluctuation of hardware utilization rate, communication bandwidth utilization rate and training iteration speed is ≥20%, the notification mode is platform notification, and the alarm information includes index name, current value, fluctuation range and monitoring period.

[0019] Further, in the dynamic adaptation adjustment, the reference trigger threshold in the dynamic adaptation adjustment step is dynamically set according to the training stage. In the initial training stage, that is, when the iteration number is less than or equal to 30% of the total budget, the reference trigger threshold is 0.2; in the middle training stage, that is, when 30% < iteration number < 70%, the reference trigger threshold is 0.18; and in the later training stage, that is, when the iteration number is greater than 70%, the reference trigger threshold is 0.15. The process of determining whether to trigger adjustment is as follows: according to the current training iteration number, the reference trigger threshold of the corresponding stage is determined, and then the current quantization bias value is calculated by the dynamic adjustment trigger threshold algorithm. If the current quantization bias value is greater than or equal to the reference trigger threshold of the corresponding stage, it is determined that adjustment is triggered.

[0020] Further, in the dynamic adaptation adjustment, the mathematical expression of the dynamic adjustment trigger threshold algorithm is: wherein, T is the dynamic adjustment trigger threshold, ω1, ω2, ω3 are weight coefficients, S0 is the initial optimal collaborative adaptation degree, ΔS is the collaborative adaptation degree bias, ΔR ′ In order to optimize the communication-computation ratio in real time, R ′ 0 is the initial optimized communication-computation ratio, ΔL is the real-time loss function value, and L0 is the target convergence loss.

[0021] On the other hand, an artificial intelligence platform computing resource scheduling management system, the system comprises:

[0022] A hardware topology collection module is configured with a hardware perception agent program, which is used to traverse heterogeneous hardware resources, collect hardware basic parameters and device topology structure data, generate, store and update hardware topology atlas in real time;

[0023] A collaborative strategy decision module is in communication connection with the hardware topology collection module, calls a collaborative adaptation quantification algorithm, filters the optimal combination of mixed precision and mixed parallelism based on the hardware topology atlas, determines the precision configuration mode, parallel strategy combination and switching trigger condition, and outputs the associated parameters;

[0024] A load balancing deployment module is in communication connection with the collaborative strategy decision module, adopts hierarchical splitting technology to split training tasks, extracts subtask parameters and topology associated parameters through double-source extraction technology, calls a topology-aware load characteristic algorithm to calculate the load characteristics of the subtasks, allocates tasks based on hardware capability-load characteristic matching technology, plans data transmission paths through link priority sorting technology, and establishes data localization storage rules through storage location binding technology;

[0025] An execution and monitoring module is in communication connection with the load balancing deployment module, is configured with a multi-interface task issuing unit, a data distribution and calculation starting unit, a multi-dimensional monitoring unit and a three-level alarm unit, issues a task allocation table, starts data distribution and calculation, collects and processes associated indicators, and triggers hierarchical alarm;

[0026] The dynamic adaptive adjustment module is in communication connection with the execution and monitoring module and the collaborative strategy decision module respectively, and is used for extracting the benchmark parameter, dynamically setting the benchmark trigger threshold, calling the dynamic adjustment trigger threshold algorithm to quantize the deviation, judging whether adjustment is triggered or not, saving the training state and triggering the collaborative strategy decision module to regenerate the scheme if adjustment is triggered.

[0027] Compared with the prior art, the artificial intelligence platform computing resource scheduling management method and system have the following beneficial effects:

[0028] Firstly, the present application comprehensively captures heterogeneous hardware resource information and inter-device associated data through the hardware topology collection module, constructs a complete hardware topology map and updates it in real time, provides comprehensive and accurate hardware foundation support for scheduling decision, and uses the adaptive measurement quantization algorithm relying on the collaborative strategy decision module to realize mixed precision and mixed parallel deep adaptation, so that the precision configuration mode and parallel strategy combination fully match the hardware characteristics, while balancing the calculation efficiency and precision requirement. This precise strategy making based on the actual state of hardware can fully tap the performance potential of different hardware, avoid efficiency loss caused by resource mismatch, effectively control precision loss, form an optimal configuration for computing resources in the initial scheduling stage, significantly improve the startup efficiency and running foundation stability of artificial intelligence training tasks, and lay a solid foundation for subsequent efficient calculation.

[0029] Secondly, the present application adopts scientific task splitting and parameter extraction method through the load balancing deployment module, accurately identifies the load characteristics of subtasks by combining the topology-aware load characteristic algorithm, optimizes the data transmission path and establishes the local storage rule through the accurate matching of hardware capability and load characteristics, reduces the data transmission time consumption and storage conflict, realizes efficient task deployment and balanced resource allocation, and matches the dynamic adaptive adjustment module to dynamically optimize the benchmark threshold according to the training stage, triggers the strategy adjustment in time through quantizing the deviation, ensures that the hardware state change, load fluctuation and training progress demand can be quickly responded in the training process, and continuously optimizes the scheduling scheme. This full-process dynamic adaptation and accurate deployment method effectively solves the problems of resource imbalance and response lag in traditional scheduling, guarantees the stable progress of the training process, maximizes the comprehensive utilization efficiency of computing resources, and enables the artificial intelligence platform to maintain an efficient running state in different training scenarios.

[0030] Other advantages, objects, and features of the present application will be in part apparent and in part pointed out hereinafter in the specification, and it is to be understood that changes in the specification can be made by persons skilled in the art without departing from the spirit and scope of the present application, which will be exemplified by the following examples. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other accompanying drawings can also be obtained by those skilled in the art without creative labor on the premise of the accompanying drawings.

[0032] Figure 1 For artificial intelligence platform computing resource scheduling management method flow framework diagram;

[0033] Figure 2 For artificial intelligence platform computing resource scheduling management system module connection diagram. DETAILED DESCRIPTION

[0034] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined object of the application, the specific embodiments, structures, features and effects according to the present application will be described in detail below in combination with the accompanying drawings and preferred embodiments.

[0035] Embodiment one:

[0036] Small-scale image classification model training scene.

[0037] In the Internet technology company, a small-scale image classification AI model training project is carried out. The model is used to identify the category of e-commerce platform commodity pictures. The parameter scale is 800,000, the convergence precision requirement is ≤2%, and the training platform includes CPU, GPU, NPU heterogeneous hardware resources. The artificial intelligence platform computing resource scheduling management method of the present application is used to complete the training task, and the training process is ensured to be efficient and adaptive to hardware resources, and the precision and speed balance is accurately controlled, as shown in Figure 1

[0038] ​Hardware topology collection: Start the hardware-aware agent program built-in the platform, and traverse all heterogeneous hardware resources through the multi-interface collaborative collection mode. This multi-interface collaborative collection mode can comprehensively cover the parameter acquisition needs of different types of hardware and avoid the limitations of single-interface collection. For CPU, the CPUID instruction and numactl tool are used to collect its model, core number and NUMA node attribution, to ensure the accuracy and integrity of CPU parameter collection. For GPU, the GPU driver interface is used to collect its model, computing power and video memory capacity, to directly interface with the hardware driver to ensure data real-time. For NPU, the NPUSDK is used to collect its model, computing power specification and supported quantization precision, to meet the exclusive data acquisition standard of NPU hardware. At the same time, the connection mode, communication bandwidth and delay between devices are collected through bus scanning and network detection tools, and the topology structure data such as NUMA node distribution and GPU interconnection architecture are obtained through system file analysis and driver interface query, to comprehensively master the connection state and transmission capacity between devices. Based on the collected information, a hardware topology graph is generated. The graph adopts a graph structure data format, with nodes corresponding to individual hardware devices or NUMA nodes and edges corresponding to the physical connection between devices. The graph is stored in a special database and updated in real time, so that subsequent strategy decision-making can be based on the latest and most comprehensive hardware and topology information, avoiding strategy mismatch caused by data lag.

[0039] Collaborative strategy decision-making: Call the collaborative adaptation quantification algorithm to select the optimal combination of mixed precision and mixed parallelism based on the hardware topology graph. The mathematical expression of the collaborative adaptation quantification algorithm is: where S is the collaborative adaptation, a is the hardware topology weight coefficient, H is the hardware topology adaptation coefficient, G is the precision-parallelism collaborative gain, β is the computing efficiency weight coefficient, n is the number of mixed precision modes, is the hardware acceleration coefficient of the i-th precision mode, P i is the computing proportion corresponding to the i-th precision mode, The compatibility coefficient of the i-th precision mode and the parallel strategy, γ is the precision loss penalty coefficient, and Δ∈ is the precision loss rate of mixed precision relative to full precision. The algorithm can comprehensively consider the hardware adaptability, calculation efficiency and precision loss, and realize the strategy selection under the multi-objective balance. Because the model convergence precision requirement is ≤2%, the hardware precision adaptation technology is used to determine that the core link adopts FP32+FP16 mixed configuration. This configuration not only meets the high-precision training demand, but also improves the calculation efficiency with the cooperation of different precisions; the model parameter is 800 million<10 billion, and the parameter size-hardware matching technology is used to determine that the parallel strategy is data parallel, and the parallel slice number is consistent with the hardware number, so that the parallel calculation can fully utilize the existing hardware resources and avoid resource idling or overload. The state threshold monitoring technology is used to set the switching trigger condition: the iteration number reaches 70% of the total budget, the continuous 1000 iteration loss change rate is ≤0.01%, the hardware utilization rate is continuously ≥95% or ≤60% for 3 seconds, and any condition meets the pre-device selection rule to switch, so as to ensure that when the training progress, model convergence state or hardware running state changes, the strategy can be adjusted in time to maintain the training effect and efficiency, and finally the associated parameters of the precision configuration, parallel strategy and switching condition are output, providing a clear basis for subsequent task deployment.

[0040] Load balancing deployment: according to the optimal cooperative scheduling strategy, the hierarchical splitting technology is used to split the training data according to the number of hardware nodes, and the splitting proportion is allocated combined with the calculation capacity and storage capacity of each hardware device, so that the task splitting can be adapted to the actual bearing capacity of the hardware, and the load of some devices can be avoided to be too heavy or too light. The double-source extraction technology is used to extract the sub-task parameters and topological association parameters. The sub-task parameters are directly derived from the training task configuration and splitting result, and the topological association parameters are derived from the hardware topology map. The complementary of double-source data ensures the comprehensiveness of the parameters, provides sufficient basis for load calculation and task allocation, and at the same time, the reuse of associated parameters reduces data redundancy processing and improves process efficiency. After the reuse of associated parameters, the topology-aware load characteristic algorithm is called to calculate the load characteristics of the sub-tasks. The expression of the topology-aware load characteristic algorithm is: Wherein, R ′ is the optimized topology-aware communication-computation ratio, k is the number of training sub-tasks, D t is the data transmission amount of the t-th sub-task, L t is the transmission link delay of the t-th sub-task, L loc,t is the data locality gain coefficient of the t-th sub-task, F t is the original calculation amount of the t-th sub-task, k p,t is the precision acceleration coefficient corresponding to the t-th sub-task, and H dov,tThe hardware efficiency coefficient of the device is allocated to the t-th subtask, and λ is the parallel strategy correction factor. The algorithm can accurately depict the computing and communication requirements of the subtask by combining the hardware topology information, and avoid the deviation of load evaluation caused by deviating from the topology environment. Based on the hardware capacity-load characteristic matching technology, the task is allocated: the high communication-computation ratio subtask is allocated to the hardware cluster with communication bandwidth ≥200GB / s, which fully utilizes the high bandwidth resource to reduce the transmission time; the high computation-communication ratio subtask is allocated to the hardware device with computing power ≥1000TOPS, which relies on strong computing power to quickly complete the computing task; the medium load subtask is allocated to the remaining hardware, which realizes the global efficient utilization of hardware resources. Through the link priority sorting technology, the transmission path is planned by preferentially selecting the link within the same NUMA node or the GPU interconnection architecture, which has the advantages of high communication bandwidth and low delay, and can significantly improve the data transmission efficiency; through the storage location binding technology, the rule is established: binding the storage location to form a fixed association between data and devices, reducing the data migration frequency; reserving a capacity of 1.2 times the total data amount to avoid training interruption caused by insufficient storage; classifying and storing data to meet the access frequency and storage needs of different data; when the local storage occupancy rate reaches 85%, the uncalled intermediate results are migrated to the same NUMA node distributed storage during the computing gap, which not only ensures the current computing, but also releases the local storage space to maintain the balance of storage.

[0041] Execution and monitoring: Through the multi-interface task allocation table, the multi-interface design adapts to the receiving needs of different hardware devices, ensuring that task instructions are quickly and accurately delivered to each device, and smoothly starting the data distribution and computing process. Multi-dimensional real-time monitoring is enabled, and hardware, communication, training and algorithm-related indicators are collected comprehensively. After stream processing, state data flow is formed. Stream processing can convert raw indicators into usable data in real time, avoiding monitoring lag caused by data accumulation. A three-level hierarchical alarm mechanism is enabled: when the hardware failure or training accuracy error ≥5%, the first-level alarm is triggered, and relevant personnel are notified through SMS + platform pop-up window, with a response time ≤10 seconds, ensuring that serious abnormalities can be quickly detected and handled, reducing losses; when the hardware load imbalance ≥15% or the cross-device communication delay exceeds the standard ≥50%, the second-level alarm is triggered, and the information is pushed through email + platform notification, with a response time ≤30 seconds, reminding personnel to adjust resource allocation or optimize the link; when the hardware utilization rate and other indicators fluctuate ≥20%, the third-level alarm is triggered, and the platform notification is used to inform personnel in real time, and the alarm information contains key data and processing suggestions, providing clear guidance for abnormal disposal.

[0042] Dynamic adaptive adjustment: extract reference parameters, set reference trigger threshold according to training stage: initial value is 0.2, at this time the training is in the exploration stage, the threshold is set relatively loose to avoid frequent adjustment affecting the stability of training; the medium value is 0.18, the training gradually enters the stable stage, the threshold is appropriately tightened to improve the strategy accuracy; the later value is 0.15, close to the training completion, the threshold is further strict to ensure the model convergence effect. The dynamic adjustment trigger threshold algorithm quantifies the deviation, and the mathematical expression of the dynamic adjustment trigger threshold algorithm is: Wherein, T is the dynamic adjustment trigger threshold, ω1, ω2, ω3 are weight coefficients, S0 is the initial optimal synergistic adaptation degree, ΔS is the synergistic adaptation degree deviation, ΔR ′ For real-time optimization of communication-computation ratio, R ′ 0 is the initial optimized communication-computation ratio, ΔL is the real-time loss function value, L0 is the target convergence loss, this algorithm can comprehensively consider the key factors such as synergistic adaptation degree deviation, communication-computation ratio change and loss function fluctuation, accurately measure the deviation degree of the current training state and the optimal state, and avoid adjustment mistakes caused by single index judgment. According to the corresponding stage threshold value determined by the current iteration number, if the quantized deviation is greater than or equal to the threshold value, the adjustment is triggered, the training state is saved to prevent data loss, and the synergistic strategy decision step is returned to generate a new scheme, so that the strategy can adapt to the current training state and hardware environment changes, optimize the subsequent process and record the adjustment data, providing experience reference for subsequent similar training tasks.

[0043] In summary, in the small-scale image classification model training scene, for the demand of 800,000 parameters and convergence accuracy ≤2%, the method of the application realizes efficient power scheduling through five steps. Hardware topology collection helps to obtain heterogeneous hardware parameters and topology data through a multi-interface proxy program, and generates a real-time updated atlas to lay a data foundation for subsequent decision-making; synergistic strategy decision calls synergistic adaptation quantization algorithm to select optimal accuracy and parallel combination, which meets the model demand and hardware capability; load balancing deployment realizes accurate matching of tasks and hardware through hierarchical splitting, double-source extraction and topology perception algorithm, optimizes data transmission and storage; execution and monitoring rely on multi-dimensional monitoring and three-level alarm to respond to abnormalities in time; dynamic adaptive adjustment sets threshold according to stage, calls exclusive algorithm to quantify deviation and flexibly optimizes strategy. The whole method realizes efficient utilization of hardware resources, balance of training accuracy and efficiency, and ensures stable training progress.

[0044] Embodiment two:

[0045] Enterprise product quality inspection image model training scene.

[0046] In a manufacturing enterprise, an internal artificial intelligence training platform is built for product quality inspection image defect recognition AI model training. The model parameter scale is 600,000, and the convergence accuracy requirement is 3%. The artificial intelligence platform computing resource scheduling management system of the application is deployed on the platform. Through the collaborative work of each module, efficient scheduling of heterogeneous hardware computing power, precise control of the training process, and rapid response to abnormal situations are realized, ensuring stable and efficient model training, such as Figure 2 as shown.

[0047] Hardware topology collection module: configure hardware perception agent program with multiple interface collaboration. The multi-interface design can adapt to the parameter collection needs of different heterogeneous hardware such as CPU, GPU, NPU, etc., ensuring that there are no dead angles in the collection process. After the system starts, it automatically traverses all hardware devices in the platform, collects CPU model, core number, NUMA node attribution, etc. through CPUID instruction + numactl tool, and uses hardware native tool to ensure the authenticity of CPU data; collects GPU computing power, video memory capacity, etc. through GPU driver interface, directly interfaces with hardware bottom layer to realize real-time data acquisition; collects NPU computing power specifications, supported quantization precision, etc. through NPU SDK, which meets the exclusive data collection standard of NPU. At the same time, through bus scanning and network detection tools, collect data such as connection mode between devices, communication bandwidth, and delay, and through system file analysis and driver interface query, obtain topology information such as NUMA node distribution and GPU interconnection architecture, fully capture the connection state and transmission performance between hardware. Generate a hardware topology graph in graph structure format, with nodes containing complete attributes such as hardware type, model, and computing power, and edges containing key information such as connection protocol and communication bandwidth, and store them in a dedicated database and update the related data of nodes and edges in real time, providing real-time, comprehensive, and accurate hardware and topology data support for subsequent modules, avoiding scheduling deviations caused by incomplete or lagging data.

[0048] Collaborative strategy decision module: maintain real-time communication with the hardware topology collection module to ensure timely access to updated hardware topology graphs, allowing strategy decisions to be based on the latest hardware environment information. Call the collaborative adaptation quantification algorithm to select the optimal combination of mixed precision and mixed parallelism. The mathematical expression of the collaborative adaptation quantification algorithm is: The algorithm can comprehensively balance the hardware topology adaptability, computing efficiency and precision loss, and realize multi-dimensional optimal balance. Because the model convergence accuracy is 3%, the hardware precision adaptation technology is used to determine that the core link adopts FP16 precision, which maximizes the computing efficiency under the premise of meeting the accuracy requirements; the model parameter is 600 million<10 billion, the parameter size-hardware matching technology is used to determine the parallel strategy as data parallel, and the parallel slice number is consistent with the hardware number, so that each hardware device can undertake adaptive parallel tasks, and the parallel computing capability of the hardware cluster is fully utilized. The switching trigger condition is set to ensure that the strategy switching can be triggered in time when the training progress, model convergence state or hardware running state changes significantly, and the associated parameters of precision configuration, parallel strategy and switching condition are output to the load balancing deployment module to provide clear and feasible execution basis for task splitting and deployment.

[0049] The load balancing deployment module receives the associated parameters from the collaborative strategy decision module, and uses the optimal strategy as a guide to carry out task deployment work. The hierarchical splitting technology is used to split the training task according to the number of hardware nodes, and the splitting proportion is allocated according to the computing capacity and storage capacity of each hardware device, so that the task splitting can accurately match the hardware carrying capacity, and the resource waste caused by the overload of some devices and the idling of some devices is avoided. The double-source extraction technology is used to extract the sub-task parameters and topology associated parameters, the sub-task parameters are derived from the training task configuration and splitting result, and the topology associated parameters are derived from the hardware topology map. The complementary of double-source data ensures the comprehensiveness of the parameters, and provides sufficient basis for load calculation and task allocation. The reuse of associated parameters reduces data processing, improves process efficiency, and then calls the topology-aware load characteristic algorithm to calculate the load characteristics of the sub-task. The expression of the topology-aware load characteristic algorithm is: The algorithm can accurately describe the computing and communication requirements of the sub-task combined with the topology information, so that the load evaluation is more in line with the actual running scene. According to the hardware capacity-load characteristic matching rule, the task is allocated, the high communication-computation ratio sub-task is adapted to the high-bandwidth hardware cluster, the high computation-communication ratio sub-task is adapted to the high-computing hardware device, and the medium-load sub-task is allocated to the remaining hardware, so as to realize the accurate matching of hardware resources and task requirements; the link priority sorting technology is used to plan the transmission path, and the high-bandwidth and low-latency link is preferentially selected, which significantly improves the data transmission efficiency; the storage location binding technology is used to establish the data localization storage rule, including storage location binding, 1.2 times capacity reservation according to data total amount, data classification storage and migration mechanism, to ensure efficient and stable data storage and access, avoid the influence of storage problems on training progress, and at the same time, the migration mechanism is only executed during the computing gap and does not interrupt the training process, which guarantees the continuity of training.

[0050] The execution and monitoring module includes a multi-interface task issuing unit, a data distribution and calculation starting unit, a multi-dimensional monitoring unit, and a three-level alarm unit. The units have clear division of labor and work cooperatively. The multi-interface task issuing unit issues a task allocation table through an interface adapted to different hardware to ensure that task instructions can be quickly and accurately delivered to each device. The data distribution and calculation starting unit starts the data distribution and calculation process in order according to the allocation table to ensure smooth development of the training task. The multi-dimensional monitoring unit comprehensively collects hardware, communication, training, and algorithm-related indicators, converts them into state data streams through stream processing, and captures the system running state and training progress in real time. The three-level alarm unit triggers different levels of alarms according to rules: a first-level alarm when a hardware failure or training accuracy error is greater than or equal to 5%, a third-level alarm when an indicator fluctuation is greater than or equal to 20%, and alarm information including fault device ID, error value, adjustment direction, and other key content to provide clear disposal guidance for relevant personnel, ensuring that abnormal conditions can be discovered and responded to quickly, reducing the impact on training tasks.

[0051] The dynamic adaptation adjustment module communicates with the execution and monitoring module and the collaborative strategy decision module, obtains index data from the execution and monitoring module in real time, extracts baseline parameters, and dynamically sets baseline trigger thresholds according to the training stage. The thresholds are set to fit the characteristics of different training stages to avoid delayed or excessive adjustments caused by a single threshold. The dynamic adjustment trigger threshold algorithm is used to quantify deviations, and the mathematical expression of the dynamic adjustment trigger threshold algorithm is: The algorithm can comprehensively adapt to deviations in key indicators such as communication-computation ratio changes and loss function fluctuations to accurately measure the degree of deviation between the current training state and the optimal state. By comparing the current stage threshold, it is determined whether to trigger adjustment. If adjustment is triggered, the training state is saved to prevent data loss, and the collaborative strategy decision module is notified to regenerate the optimal solution. Subsequent modules execute according to the new solution to ensure that the strategy can adapt to changes in training state and hardware environment in real time. At the same time, data such as adjustment reasons, parameter changes, and adjustment effects are recorded to provide valuable practical experience for subsequent strategy optimization of similar training tasks and continuously improve the scheduling management capability of the system.

[0052] In summary, in the enterprise product quality inspection image model training scene, for the demand of 600,000 parameters and 3% convergence accuracy, the system of the application achieves the computing power control target through the cooperation of five modules. The hardware topology collection module acquires and updates hardware and topology data to provide basic support for the system; the cooperative strategy decision module outputs the optimal strategy based on the graph and the exclusive algorithm to guide the task deployment; the load balancing deployment module splits the task according to the strategy, matches the hardware, and optimizes the data flow; the execution and monitoring module realizes task distribution, monitoring and hierarchical alarm through multiple units, and quickly handles exceptions; the dynamic adaptive adjustment module links other modules, sets thresholds and quantifies deviations according to stages, and triggers strategy iteration. The modules of the system are closely connected, realize efficient computing power scheduling and accurate training process, and ensure stable and efficient completion of model training.

[0053] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content to obtain equivalent embodiments with equivalent changes, without departing from the technical solution of the present application. Any modification, change, equivalent change and modification of the above embodiments made according to the technical essence of the present application are still within the scope of the technical solution of the present application.

Claims

1. A method for scheduling and managing computing resources on an artificial intelligence platform, characterized in that, The method includes: Hardware topology acquisition: Through the hardware awareness agent program, the heterogeneous hardware resources of the artificial intelligence platform are traversed to collect basic hardware parameters and topology data between devices, generate a hardware topology map, and store and update it in real time through a database. Collaborative strategy decision-making: Based on the hardware topology map, the optimal combination of hybrid precision and hybrid parallelism is selected through a collaborative adaptation quantification algorithm. At the same time, the precision configuration mode, parallel strategy combination and switching trigger conditions are determined, and the associated parameters are output. Load balancing deployment steps: Split training tasks according to the optimal collaborative scheduling strategy, extract sub-task parameters and topology-related parameters, reuse the related parameters, use the topology-aware load characteristic algorithm to calculate the load characteristics of sub-tasks, allocate tasks through hardware capability-load characteristic matching technology, plan data transmission paths, and establish data local storage rules. Execution and Monitoring: The task allocation table is distributed through multiple interfaces to start data distribution and computation, and multi-dimensional real-time monitoring is enabled. Hardware, communication, training and algorithm-related indicators are collected and formed into a status data stream after streaming processing. The indicator thresholds are used for monitoring. Dynamic adaptation and adjustment: Extract benchmark parameters, set benchmark trigger thresholds, use dynamic adjustment trigger threshold algorithm to quantify deviation, determine whether to trigger adjustment, if triggered, save training state, return to collaborative strategy decision step to regenerate scheme and optimize execution, and record data.

2. The method for scheduling and managing computing resources on an artificial intelligence platform according to claim 1, characterized in that, In the hardware topology acquisition, the hardware awareness agent program supports multi-interface collaborative acquisition. The basic hardware parameters include the CPU model, number of cores, and NUMA node affiliation; the GPU model, computing power, and memory capacity; and the NPU model, computing power specifications, and supported quantization precision. CPU parameters are acquired through the CPUID instruction and the numactl tool, GPU parameters are acquired through the GPU driver interface, and NPU parameters are acquired through NPUSDK. The device topology data includes the device connection method, communication bandwidth, latency, NUMA node distribution, and GPU interconnect architecture. The connection method and communication parameters are acquired through bus scanning and network probing tools, while the node distribution and interconnect architecture are acquired through system file parsing and driver interface querying.

3. The method for scheduling and managing computing resources on an artificial intelligence platform according to claim 1, characterized in that, In the hardware topology acquisition, the hardware topology map is constructed using a graph data format. Nodes in the map correspond to individual hardware devices or NUMA nodes. Node attributes include hardware type, model, computing power, storage capacity, energy consumption parameters, real-time operating status, fault history, cumulative runtime, and maintenance cycle. Edges in the map correspond to the physical connection relationships between devices. Edge attributes include connection protocol, communication bandwidth, latency, link error rate, link occupancy rate, and historical communication peak data, which are stored in a dedicated database.

4. The method for scheduling and managing computing resources on an artificial intelligence platform according to claim 1, characterized in that, In the collaborative strategy decision-making process, the mathematical expression of the collaborative adaptation quantification algorithm is: Where S is the cooperative adaptation degree, α is the hardware topology weight coefficient, H is the hardware topology adaptation coefficient, G is the precision-parallel cooperative gain, β is the computational efficiency weight coefficient, and n is the number of mixed precision modes. Let P be the hardware acceleration coefficient for the i-th precision mode. i This represents the computational percentage corresponding to the i-th precision mode. denoted as the compatibility coefficient between the i-th precision mode and the parallel strategy, γ as the precision loss penalty coefficient, and Δ∈ as the precision loss rate of the mixed precision relative to the full precision.

5. The method for scheduling and managing computing resources on an artificial intelligence platform according to claim 1, characterized in that, In the load balancing deployment, the current task is split according to the optimal collaborative scheduling strategy using a layered splitting technique. First, the splitting dimension is determined according to the hybrid parallel strategy type. The data parallel dimension splits the training data according to the number of hardware nodes, the tensor parallel dimension splits the model parameters according to the number of GPU cores, and the model parallel dimension splits the model modules according to the network layer structure. Then, the splitting ratio is allocated according to the computing power and storage capacity of each hardware device. The extraction of subtask parameters and topology association parameters adopts a dual-source extraction technique. Subtask parameters are extracted from the training task configuration and splitting results, including the computational load, data transmission load, accuracy requirements, and completion time limit of each subtask. Topology association parameters are extracted from the hardware topology map, including the communication bandwidth, latency value, data locality identifier of the transmission link corresponding to the subtask, and the computing power, storage capacity, and interface type of the target device.

6. The method for scheduling and managing computing resources on an artificial intelligence platform according to claim 1, characterized in that, In the load balancing deployment, the expression for the topology-aware load characteristic algorithm is: Among them, R ′ The optimized topology-aware communication-computation ratio is given by k, where k is the number of training subtasks, and D is the number of training subtasks. t Let L be the data transfer amount of the t-th subtask. t Let L be the transmission link delay for the t-th subtask. loc,t Let F be the data locality gain coefficient for the t-th subtask. t Let k be the original computational cost of the t-th subtask. p,t H represents the accuracy acceleration factor corresponding to the t-th subtask. dov,t The hardware efficiency coefficient for allocating equipment to the t-th subtask is given by λ, where λ is the parallel strategy correction factor.

7. The method for scheduling and managing computing resources on an artificial intelligence platform according to claim 1, characterized in that, In the execution and monitoring process, the indicator thresholds adopt a three-level hierarchical alarm rule. The first-level alarm trigger condition is hardware equipment failure or training accuracy error ≥5%, and the notification method is SMS combined with platform pop-up window, with a response time ≤10 seconds. The alarm information includes the faulty device ID, error value, occurrence time, and emergency handling suggestions. The second-level alarm trigger condition is hardware load imbalance ≥15% or cross-device communication latency exceeding the standard ≥50%, and the notification method is email combined with platform notification, with a response time ≤30 seconds. The alarm information includes the imbalanced device identifier, latency value, link information, and adjustment direction. The third-level alarm trigger condition is fluctuation of indicators such as hardware utilization, communication bandwidth utilization, and training iteration speed ≥20%, and the notification method is platform notification. The alarm information includes indicator name, current value, fluctuation range, and monitoring period.

8. The method for scheduling and managing computing resources of an artificial intelligence platform according to claim 1, characterized in that, In the dynamic adaptation adjustment, the baseline trigger threshold in the dynamic adaptation adjustment step is dynamically set according to the training stage. In the early stage of training, when the number of iterations is ≤ 30% of the total budget, the baseline trigger threshold is 0.2; in the middle stage of training, when 30% < number of iterations ≤ 70%, the baseline trigger threshold is 0.18; and in the later stage of training, when the number of iterations > 70%, the baseline trigger threshold is 0.

15. The process for determining whether to trigger the adjustment is as follows: determine the baseline trigger threshold for the corresponding stage based on the current number of training iterations, and then calculate the current quantization deviation value through the dynamic adjustment trigger threshold algorithm. If the current quantization deviation value is greater than or equal to the baseline trigger threshold for the corresponding stage, then it is determined that the adjustment is triggered.

9. The method for scheduling and managing computing resources on an artificial intelligence platform according to claim 1, characterized in that, In the dynamic adaptation adjustment, the mathematical expression for the dynamic adjustment trigger threshold algorithm is: Where T is the dynamically adjusted trigger threshold, ω1, ω2, and ω3 are weight coefficients, S0 is the initial optimal co-fit degree, ΔS is the co-fit degree deviation, and ΔR is the weight coefficient. ′ To optimize the communication-to-computation ratio in real time, R ′ 0 represents the initial optimized communication-computation ratio, ΔL represents the real-time loss function value, and L0 represents the target convergence loss.

10. A computing resource scheduling and management system for an artificial intelligence platform, the system being applicable to the computing resource scheduling and management method for an artificial intelligence platform as described in any one of claims 1-9, characterized in that, The system includes: Hardware topology acquisition module: It is equipped with a hardware awareness agent program, which is used to traverse heterogeneous hardware resources, collect basic hardware parameters and topology data between devices, and generate, store and update the hardware topology map in real time. Collaborative strategy decision-making module: It communicates with the hardware topology acquisition module, calls the collaborative adaptation quantification algorithm, selects the optimal combination of hybrid precision and hybrid parallelism based on the hardware topology map, determines the precision configuration mode, parallel strategy combination and switching trigger conditions, and outputs the associated parameters. Load balancing deployment module: It communicates with the collaborative strategy decision module, uses layered splitting technology to split training tasks, extracts sub-task parameters and topology association parameters through dual-source extraction technology, calls the topology-aware load characteristic algorithm to calculate the load characteristics of sub-tasks, allocates tasks based on hardware capability-load characteristic matching technology, plans data transmission paths through link priority sorting technology, and establishes data local storage rules through storage location binding technology. Execution and Monitoring Module: Communicates with the load balancing deployment module and is configured with a multi-interface task distribution unit, a data distribution and calculation startup unit, a multi-dimensional monitoring unit, and a three-level alarm unit. It distributes task allocation tables, starts data distribution and calculation, collects and processes related indicators, and triggers hierarchical alarms. Dynamic Adaptation and Adjustment Module: It communicates with the Execution and Monitoring Module and the Collaborative Strategy Decision Module respectively. It is used to extract benchmark parameters, dynamically set benchmark trigger thresholds, call the dynamic adjustment trigger threshold algorithm to quantify the deviation, determine whether to trigger adjustment, and if triggered, save the training state and trigger the Collaborative Strategy Decision Module to regenerate the scheme.

Images