Large model scheduling method and device based on NPU computing power

Abstract

The invention relates to a large model scheduling method and device based on NPU computing power, and the method specifically comprises the steps: splitting a large model into a plurality of subtasks according to layers, and analyzing the dependency relationship between the subtasks; performing operation demand prediction on the large model and the subtasks to obtain an operation amount factor and a memory demand factor; training a computing power evaluation model, and predicting real-time computing power evaluation data of each NPU node; selecting a scheduling strategy, calculating the initial priority of each to-be-scheduled sub-task according to the scheduling strategy, and allocating each sub-task to an optimal NPU node for execution by using a greedy algorithm in combination with resource demand information and NPU computing power evaluation data; and monitoring the operation condition of the subtask on each NPU node in real time, and when the execution of the subtask is abnormal, triggering an alarm and taking countermeasures. Compared with the prior art, the method has the advantages that the NPU resource utilization rate and the computing power scheduling efficiency are improved, and the equipment safety is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a large model scheduling method and device based on NPU computing power. Background Art

[0002] With the rapid development of artificial intelligence technology, large models are increasingly used in various fields. However, large models have huge computing requirements and extremely high requirements for hardware resources. Traditionally, GPUs are usually used for large model training and calculation. GPUs are usually general-purpose parallel processors used to handle a variety of tasks such as graphics rendering. However, with the increasing complexity and large-scale of deep learning models, the computing power of GPUs can no longer meet the demand. At the same time, GPUs also have some disadvantages, such as high power consumption, large space occupation, and lack of versatility. In addition, since GPUs are general-purpose parallel processors, their design is not optimized specifically for neural network calculations, so they may be inefficient when processing neural network models.

[0003] Traditional CPUs and GPUs cannot meet the computing requirements of large models, while NPU (neural processing unit), as a processor specially designed for neural network computing, has the advantages of high efficiency and low power consumption, making it an ideal choice for large model computing. However, how to achieve efficient scheduling between large models and NPU and improve resource utilization is an urgent problem to be solved.

[0004] The existing NPU large model scheduling solution architecture is as follows Figure 1 As shown in Figure 1, the model developers submit requirements and deploy the development, while the integration engineers are responsible for the intermediate resource allocation. Specifically, the existing scheduling scheme methods are as follows: Figure 2 As shown, the model developer estimates the number of NPU resources based on the large model to be developed, and then submits the resource requirements of the large model development design plan to the integration engineer. The integration engineer finds a suitable NPU host in the resource pool and allocates it to the large model developer. Finally, the model developer performs training and development work on the large model on the allocated NPU host.

[0005] Since the resource allocation of large models requires manual evaluation of the large model design plan by integration engineers, and after the evaluation is passed, manual selection of suitable NPU hosts in the resource pool for allocation is required, the scheduling process is cumbersome and time-consuming, resulting in low efficiency in large model scheduling and affecting the progress of large model product development.

[0006] Also, since the NPU host is allocated exclusively and cannot share resources with other large models, a large amount of resources are often idle, resulting in low NPU resource utilization and resource waste.

[0007] In response to the above problems, Chinese patent application CN116974768A proposes an invention patent application for a computing power scheduling method based on deep learning, which optimizes computing power scheduling through deep learning technology to achieve dynamic adjustment and efficient resource utilization. Its computing power scheduling system includes: data acquisition module, deep learning module, and strategy construction module, which are responsible for data acquisition, training resource allocation strategy using deep learning model, and optimizing node performance and resource allocation strategy, respectively, to achieve intelligent management and optimization of computing power resources and improve NPU utilization. However, in actual applications, the efficiency of computing power calculation and scheduling optimization using deep learning models is still not very high. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a large model scheduling method and device based on NPU computing power. An adaptive scheduling strategy is adopted to dynamically schedule according to the computing requirements of the model and the computing power resources of the neural network processor, thereby improving the NPU resource utilization and computing power scheduling efficiency, and real-time monitoring of the subtask operation status. When an abnormal situation occurs, an alarm is immediately issued to take countermeasures to ensure equipment safety.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] A large model scheduling method based on NPU computing power includes the following steps:

[0011] Perform structural analysis on the large model, split the large model into multiple subtasks by layer, and analyze the dependencies between each subtask;

[0012] Predict the computational requirements of large models and subtasks to obtain resource requirement information, including operation count (OC) and memory count (MC).

[0013] Use the trained computing power evaluation model to predict the computing power evaluation data of each NPU node of the device to be evaluated;

[0014] Select a scheduling strategy, calculate the initial priority of each subtask to be scheduled according to the scheduling strategy, combine resource demand information with the computing power evaluation data of the NPU node, and use a greedy algorithm to assign each subtask to the optimal NPU node for execution;

[0015] Monitor the running status of subtasks on each NPU node in real time. When abnormal situation is found in the execution of subtasks, trigger alarm and take countermeasures.

[0016] Furthermore, the structural analysis includes analysis of the overall structure of the model, the types of each layer of the model, the number of model parameters and the connection relationship between the computing units.

[0017] Furthermore, the computing demand prediction includes:

[0018] Quantify the amount of computation for each layer in the large model, including the number of floating-point operations per second (FLOPs) of the convolutional layer and the number of multiplication and addition operations of the fully connected layer, and store them as computation factor;

[0019] The temporary and persistent memory required by large models at runtime is quantified and stored as a memory requirement factor.

[0020] Furthermore, the training steps of the computing power evaluation model include:

[0021] By acquiring hardware information, performing benchmark tests, and performing load analysis on the device to be evaluated, the basic computing power indicators, actual performance data, and operation characteristics and mode information of the model of the device to be evaluated are obtained to form a feature independent variable set;

[0022] Based on the online monitoring data, record the computing performance of the device to be evaluated under different workloads, including response time, throughput, and NPU utilization, and establish a mapping relationship between the obtained data and the feature independent variable set to form a dependent variable set;

[0023] A gradient boosting decision tree model (GBDT) is established, and the feature independent variable set and dependent variable set are input to iteratively train the gradient boosting decision tree model, and finally a computing power evaluation model is obtained.

[0024] Furthermore, the real-time computing power evaluation data of the NPU node includes response time, throughput and NPU utilization.

[0025] Furthermore, the scheduling strategies include First Come First Served (FCFS), Shortest Job First (SJF), priority scheduling, preemptive scheduling, fair share scheduling, maximizing throughput and workload balancing.

[0026] Furthermore, the step of using a greedy algorithm to assign each subtask to the optimal NPU node for execution includes:

[0027] According to the dependencies between each subtask, the subtasks are organized into queues, and the first subtasks of all queues are sorted from high to low according to the initial priority;

[0028] Based on the real-time computing power evaluation data of each NPU node and the resource requirement information of the first subtask in all queues, the resource matching score between the node and the subtask to be executed is calculated;

[0029] According to the resource matching scores of the first subtasks in all queues, schedule them to the NPU node with the highest resource matching score in descending order of priority;

[0030] Update the subtask queue, and the remaining subtasks are queued again according to the dependency relationship, and sort the first subtasks in all queues from high to low according to the initial priority;

[0031] After the subtask queue is updated, the real-time computing power evaluation data of each NPU node and the resource requirement information of each subtask are re-obtained, and the above steps are repeated until all subtasks are scheduled.

[0032] Furthermore, the step of calculating resource matching scores includes:

[0033] Calculate the remaining capacity of the NPU node using the following formula:

[0034] RC j =C j ×(1-U j );

[0035] Where j is the number of nodes, RC j is the remaining capacity of the jth NPU node, C j is the total memory capacity of the jth NPU node, U j The real-time computing power evaluation data of the j-th NPU node;

[0036] Calculate the resource matching score between the NPU node and the subtask. The formula is as follows:

[0037]

[0038] Among them, i is the number of subtasks, RM ij is the resource matching score between node j and subtask i, w npu is the computational load factor weight of the NPU node, OC i is the computational factor of the ith subtask, w mem is the memory requirement factor weight of the NPU node, MC i is the memory requirement factor of the i-th subtask.

[0039] Furthermore, the abnormal situation includes timeout and resource overflow, and the response measures include rescheduling, resource recovery and parameter tuning.

[0040] 10. A large model scheduling device based on NPU computing power, comprising:

[0041] The model analysis module parses the large model file package, performs structural analysis on the large model, splits the large model into multiple subtasks by layer, analyzes the dependencies between each subtask, predicts the computing requirements of the large model and subtasks, and obtains resource demand information, including computing factor and memory demand factor;

[0042] The computing power evaluation module trains the computing power evaluation model and uses the trained computing power evaluation model to predict the real-time computing power evaluation data of each NPU node;

[0043] The scheduling decision module selects the scheduling strategy, calculates the initial priority of each subtask to be scheduled according to the scheduling strategy, combines the resource demand information with the NPU computing power evaluation data, and uses the greedy algorithm to assign each subtask to the optimal NPU node for execution;

[0044] The execution feedback module monitors the running status of subtasks on each NPU node and feeds back the collected execution data to the scheduling decision module. When abnormal situation is found in the execution of subtasks, an alarm is triggered and countermeasures are taken.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. The present invention uses an intelligent scheduling algorithm and an adaptive scheduling strategy to split a large model into multiple subtasks, allocate NPU computing power to the subtasks, and achieve efficient scheduling and optimal resource utilization of the large model on the NPU computing power, thereby improving the NPU resource utilization rate;

[0047] 2. The present invention uses the gradient boosting decision tree model to evaluate computing power, and the greedy algorithm assigns each subtask to the optimal NPU node for execution. The scheduling process is simple, saving scheduling time and improving the scheduling efficiency of large models;

[0048] 3. The present invention evaluates computing power through a gradient boosting decision tree model, which can process a large number of eigenvalues, is not prone to overfitting, runs fast while providing computing power evaluation data, has strong interpretability, and has high accuracy;

[0049] 4. The present invention uses a greedy algorithm combined with a selected scheduling scheme to perform computing power scheduling, which is intuitive and easy to implement. After scheduling once, it can optimize the scheduling again according to the updated real-time computing power evaluation data of the NPU node and the resource demand information of each subtask, which is efficient and ensures more reasonable scheduling;

[0050] 5. While scheduling the NPU computing power, the present invention also monitors the running status of subtasks on each NPU node in real time, and immediately issues an alarm and takes countermeasures when an abnormal situation occurs, thereby ensuring the safety of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is the architecture diagram of the existing NPU large model scheduling solution;

[0052] Figure 2 This is a diagram of the existing NPU large model scheduling solution method;

[0053] Figure 3 This is a diagram of a large model scheduling method based on NPU computing power of the present invention;

[0054] Figure 4 This is the architecture diagram of the large model file package;

[0055] Figure 5 Model splitting and dependency diagram;

[0056] Figure 6 Schedule execution graph for subtasks;

[0057] Figure 7 This is a diagram of a large model scheduling device based on NPU computing power in the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0059] Example 1

[0060] This embodiment discloses a large model scheduling method based on NPU computing power, such as Figure 3 As shown, the specific steps include:

[0061] S1: Perform structural analysis on the large model, split the large model into multiple subtasks by layer, analyze the dependencies between each subtask, predict the computing requirements of the large model and subtasks, obtain resource requirement information, including operation count (OC) and memory count (MC), train the computing power evaluation model, and use the trained computing power evaluation model to predict the real-time computing power evaluation data of each NPU node;

[0062] S2, select a scheduling strategy, calculate the initial priority of each subtask to be scheduled according to the scheduling strategy, combine resource demand information with NPU computing power evaluation data, and use a greedy algorithm to assign each subtask to the optimal NPU node for execution;

[0063] S3 monitors the running status of subtasks on each NPU node in real time. When abnormal situation is found in the execution of subtasks, an alarm is triggered and countermeasures are taken.

[0064] Furthermore, the structural analysis includes analysis of the overall structure of the model, the types of each layer of the model, the number of model parameters and the connection relationship between the computing units.

[0065] First, load and parse the model. Use appropriate tools to load the large model file package (the file package contains the model description file and the model hierarchical calculation code), and obtain the overall topological structure of the model by parsing the model description.

[0066] Large model file package such as Figure 4 As shown in the figure, the model description of a large model file package usually contains information such as the overall architecture of the model, the connection relationship between each layer, and the number of parameters. It provides a high-level understanding of the entire model, including:

[0067] The types of each layer in the model refer to the types of each layer in the neural network model, including convolutional layer, pooling layer, fully connected layer, etc. Each layer type has different functions and characteristics, and is used to process input data and extract features in different ways. For example:

[0068] Convolutional layer: used to perform convolution operations on input data and extract spatial features.

[0069] Pooling layer: used to reduce the dimension and downsample the output of the convolutional layer to reduce the number of parameters and the amount of calculation.

[0070] Fully connected layer: used to convert the output of the previous layer into a one-dimensional vector and perform a fully connected operation with the weights to achieve the final classification or regression prediction tasks;

[0071] The number of model parameters refers to the number of parameters in each layer, including weight parameters and bias parameters. These parameters are learned by the model during training and are used to adjust the model's response to input data and prediction of output results. The number of parameters directly affects the complexity and training difficulty of the model, and also determines the model's performance and generalization capabilities.

[0072] The connection relationship of the computing units describes the connection mode and pattern between each computing unit (neuron). In convolutional neural networks, these connection relationships are usually determined by parameters such as the size and stride of the convolution kernel. Specifically, the connection relationship includes the connection mode between each computing unit and its adjacent computing units, as well as the connection relationship between computing units in different layers. These connection relationships determine how information is transmitted and processed in the network.

[0073] By introducing in detail the types of model layers, the number of model parameters, and the connection relationships of computing units in the model description data structure, we can better understand and analyze the structure and function of the neural network model, and provide reference and guidance for the design, training, and tuning of the model.

[0074] Furthermore, model splitting and dependency analysis include:

[0075] By utilizing the multi-task parallel computing characteristics of NPU hardware, large models can be reasonably divided into multiple subtasks, such as Figure 5 Each subtask shown contains a layer of the model. By splitting by layer, each subtask can be executed independently on the NPU, making full use of hardware resources and improving computing efficiency. At the same time, this splitting method can also be flexibly adjusted according to the constraints of hardware resources and the structural characteristics of the model to achieve better performance and resource utilization.

[0076] After the split is completed, the dependencies between the layers within the model are analyzed to ensure that these dependencies are followed during the subtask scheduling process to avoid unnecessary waiting and blocking. The split subtasks are sorted according to the dependencies and a subtask execution queue is established.

[0077] Furthermore, the computing demand prediction includes:

[0078] The amount of computation of each layer in the model is accurately quantified and recorded as the computation factor, which mainly includes the FLOPs (floating-point operations per second) of the convolutional layer and the number of multiplication and addition operations of the fully connected layer.

[0079] FLOPs calculation of convolutional layer:

[0080] For the convolutional layer, FLOPs can be calculated by the following formula:

[0081] FLOPs = K*N*M*C*H*W

[0082] Among them, K is the number of convolution kernels, N is the height of the output feature map, M is the width of the output feature map, C is the number of channels of the input feature map, H is the height of the convolution kernel, and W is the width of the convolution kernel.

[0083] Its computational factor OC=FLOPs.

[0084] Calculation of the number of multiplication and addition operations of the fully connected layer:

[0085] For the fully connected layer, the number of multiplication and addition operations can be calculated by the following formula:

[0086] Multiply_Adds=N_input*N_output

[0087] Where N_input is the size of the input feature vector and N_output is the size of the output feature vector.

[0088] Its operation factor OC=Multiply_Adds.

[0089] Calculate the amount of temporary memory and persistent memory required by the model at runtime, including the impact of activation functions and caching mechanisms on memory consumption.

[0090] Calculation of activation function:

[0091] The calculation of the activation function usually requires temporary memory to store intermediate results, depending on the type of activation function and the size of the input data, namely:

[0092] M_act=output_size*batch_size

[0093] Where output_size is the output size of the activation function and batch_size is the batch size.

[0094] Calculation of cache mechanism:

[0095] When the model is running, a cache mechanism may be used to store intermediate results to improve computing efficiency. The cache mechanism may occupy additional persistent memory, namely:

[0096] M_cache=N_cache*S_cache*B_cache

[0097] Where N_cache represents the number of intermediate results stored in the cache, S_cache represents the size of each intermediate result, and B_cache represents the batch size of the intermediate results stored in the cache.

[0098] Taking all the above factors into consideration, we can design an algorithm formula to evaluate the memory requirement factor, as follows:

[0099] Memory requirement factor MC = ∑(M_act+M_cache).

[0100] Furthermore, the step of training the computing power evaluation model includes:

[0101] By acquiring hardware information, performing benchmark tests, and performing load analysis on the device to be evaluated, the basic computing power indicators, actual performance data, and operation characteristics and mode information of the model of the device to be evaluated are obtained to form a feature independent variable set;

[0102] By using online monitoring data, the computing performance of the device to be evaluated under different workloads is recorded, including response time, throughput, and NPU utilization, and a mapping relationship is established between the obtained data and the feature independent variable set to form a dependent variable set;

[0103] A gradient boosting decision tree model (GBDT) is established, and the feature independent variable set and dependent variable set are input to iteratively train the gradient boosting decision tree model, and finally a computing power evaluation model is obtained.

[0104] The hardware information is obtained by using the monitoring tools provided by the system or third-party software, command line tools, and developer tools provided by the manufacturer to obtain the basic computing power indicators of the device to be evaluated, including the contents of Table 1.

[0105] Table 1 Basic computing power indicator example table

[0106] Basic computing power indicators Example values unit Number of CPU cores 8 indivual CPU frequency 3.6 GHz Number of CPU threads 16 indivual Number of stream processors in the NPU 5120 indivual NPU memory size 16 GB NPU core frequency 1.8 GHz Theoretical peak FLOPS of NPU 16.5 TFLOPS

[0107] The actual performance of the device to be evaluated is obtained through benchmark testing, using standardized benchmark tools such as SPEC CPU, SPEC MLPerf, and SPC-1 for performance testing of storage systems, and AIBenchmark and DeepBench for NPU for actual performance testing. The specific data are shown in Table 2.

[0108] Table 2 Actual performance test example table

[0109] Performance test indicators Example values unit CPU performance score (SPEC CPU) 1000 Fraction Deep Learning Performance Scores (SPEC MLPerf) 500 Fraction Storage System Performance Score (SPC-1) 2000 IOPS NPU performance score (AI Benchmark) 3000 Fraction NPU floating point computing performance score (DeepBench) 1500 Fraction

[0110] The operation characteristics and modes of the model include parallelism, memory access mode, IO requirements, etc., which are obtained through load analysis to predict whether the hardware concurrency and bandwidth can be fully utilized during actual operation. The specific contents are shown in Table 3.

[0111] Table 3 Load analysis data example table

[0112] Model operation characteristics and modes Example values unit Parallelism 100 indivual Memory access patterns High concurrent read and write IO Requirements 500 MB / s

[0113] Furthermore, the computing power evaluation model is a gradient boosting decision tree (GBDT) model, which is an integrated learning method. The specific training steps are as follows:

[0114] Considering the dynamic allocation and scheduling mechanism of system resources, we use online monitoring data to evaluate the computing performance of the system under different workloads in real time or offline, including response time, throughput and NPU utilization, which are recorded as Y1, Y2, and Y3 respectively. The basic computing power indicators, actual performance data, and computing characteristics and mode information data of the device to be evaluated are used as feature independent variables, and recorded as X1, X2, ...X n , where n is the total number of features.

[0115] The basic idea of ​​GBDT is to gradually improve the prediction ability of the model by iteratively training weak learners (usually decision trees). This solution uses the GBDT model to predict the system response time, throughput, and utilization data based on the collected hardware information, benchmarks, and load analysis data. Assume that we have M basic weak learners h m(x),m=1,2,...,M, our goal is to build a more powerful model by combining them:

[0116]

[0117] Initially, our model F(x) is a simple initial estimate, usually a constant is chosen as the initial value. Then, we iteratively train each weak learner h m (x), each training will adjust the model F(x) to reduce the residual r i , that is, the observed value y i Compared with the predicted value F(x i ) between the two.

[0118] Specifically, in the mth iteration, we fit the residual r im =y i -F m-1 (x i ), that is, the previous round model F m-1 (x i ) for sample x i The difference between the predicted value and the actual value is used to train the current weak learner h m (x). Then, we add the prediction results of the current learner to the model F(x) to obtain the new model F m (x) = F m-1 (x)+h m (x).

[0119] After completing the training of the GBDT model, to use the model for prediction, you only need to bring the input sample feature vector x into the final integrated model F(x) to get the predicted value, that is:

[0120]

[0121] Furthermore, the scheduling strategies include first-come-first-served, short job first, priority scheduling, preemptive scheduling, fair share scheduling, maximum throughput and workload balancing. After selecting the scheduling strategy, the system automatically calculates the initial priority of each task according to the scheduling strategy. The calculation method is shown in Table 4 below.

[0122] Table 4 Scheduling strategy calculation example table

[0123]

[0124] Furthermore, the step of using the greedy algorithm to assign each subtask to the optimal NPU node is as follows: Figure 6 As shown, including:

[0125] According to the dependencies between each subtask, the subtasks are organized into queues, and the first tasks in all queues are sorted from high to low according to the initial priority;

[0126] Based on the real-time computing power evaluation data of each NPU node and the resource requirement information of each subtask, the resource matching score between the node and the subtask to be executed is calculated;

[0127] According to the resource matching score of each subtask, schedule it to the NPU node with the highest resource matching score in order of priority, and update the subtask queue;

[0128] After the subtask queue is updated, the computing power evaluation model is used to re-evaluate the computing power of each NPU, and the above steps are repeated until all task queues are completed.

[0129] After calculating the priority of the task according to the scheduling strategy selected by the user, this solution combines the model analysis results and the real-time NPU computing power prediction results, and uses a greedy algorithm to establish a resource scheduling model.

[0130] Suppose we have a set of subtasks T = T1, T2, ..., T n , each subtask has a priority P i and resource requirements (computation factor OC i and the memory requirement factor MC i We also have a set of NPU nodes N = {N1, N2, ..., N m}, each node has NPU computing power evaluation (response time R j , throughput T j 、NPU utilization U j ). The goal of the scheduling model is to assign each subtask to the most appropriate NPU node for execution. We can use the following steps to build a scheduling model:

[0131] Step 1: According to the dependencies between each subtask, the subtasks are organized into queues, and the first task in all queues is ranked according to the initial priority P. i Sort from high to low.

[0132] Step 2: Based on the real-time performance evaluation of each NPU node R j , throughput T j 、NPU utilization U j As well as the computational load factor OC and memory requirement factor RC of each task, the resource matching score between node j and task i to be executed is calculated. Based on the node NPU utilization U calculated in real time j , assuming the total memory capacity of the node is C j , calculate its remaining capacity, the formula is

[0133] RC j =C j ×(1-U j )

[0134] Where j is the number of nodes, RC j is the remaining capacity of the jth NPU node, C j is the total memory capacity of the jth NPU node, U j The real-time computing power evaluation data of the j-th NPU node;

[0135] The resource matching score between node j and task i is:

[0136]

[0137] Among them, i is the number of subtasks, RM ij is the resource matching score between node j and subtask i, w npu is the computational load factor weight of the NPU node, OC i is the computational factor of the ith subtask, w mem is the memory requirement factor weight of the NPU node, MC i is the memory requirement factor of the i-th subtask.

[0138] Step 3: For each task, try to schedule it to the NPU node with the highest resource matching score in priority order according to its resource matching score.

[0139] Step 4: After the queue is updated, update the computing power evaluation of the NPU node and repeat steps 1-3 to perform the next scheduling until all task queues are completed.

[0140] Furthermore, the running status of subtasks on each NPU node is monitored in real time. When abnormal situations occur in the execution of subtasks, alarms are triggered and countermeasures are taken. Specifically, the following are performed:

[0141] Real-time monitoring of the running status of large model tasks on each NPU node, including key performance indicators such as task execution progress, computing load, memory usage, bandwidth usage, latency, etc. At the same time, hardware-level data is collected, such as NPU temperature, power consumption, and possible hardware failures or resource bottlenecks.

[0142] Establish a task status tracking mechanism to ensure that the detailed status of each task from submission to completion can be accurately understood. Generate execution reports regularly to record the historical trends and current status of various indicators to facilitate analysis of system performance and optimization space.

[0143] Establish a task status tracking mechanism to ensure that the detailed status of each task from submission to completion can be accurately understood. Generate execution reports regularly to record the historical trends and current status of various indicators to facilitate analysis of system performance and optimization space.

[0144] The collected execution data is fed back to the scheduling decision module as an important basis for adjusting the subsequent subtask allocation strategy.

[0145] Example 2

[0146] This embodiment discloses a large model scheduling device based on NPU computing power. The specific framework is as follows: Figure 7 As shown, including:

[0147] The model analysis module parses the large model file package, performs structural analysis on the large model, splits the large model into multiple subtasks by layer, analyzes the dependencies between each subtask, predicts the computing requirements of the large model and subtasks, and obtains resource demand information, including computing factor and memory demand factor;

[0148] The computing power evaluation module trains the computing power evaluation model and uses the trained computing power evaluation model to predict the real-time computing power evaluation data of each NPU node;

[0149] The scheduling decision module selects the scheduling strategy, calculates the initial priority of each subtask to be scheduled according to the scheduling strategy, combines the resource demand information with the NPU computing power evaluation data, and uses the greedy algorithm to assign each subtask to the optimal NPU node for execution;

[0150] The execution feedback module monitors the running status of subtasks on each NPU node and feeds back the collected execution data to the scheduling decision module. When abnormal situation is found in the execution of subtasks, an alarm is triggered and countermeasures are taken.

[0151] The specific details of the above modules can be understood by referring to the relevant descriptions and effects in Example 1.

[0152] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A large model scheduling method based on NPU computing power, characterized in that: The following steps are involved: Perform structural analysis on the large model, split the large model into multiple subtasks by layer, and analyze the dependencies between each subtask; Predict the computing requirements of large models and subtasks to obtain resource requirement information, including computing factor and memory requirement factor; Use the trained computing power evaluation model to predict the computing power evaluation data of each NPU node of the device to be evaluated; Select a scheduling strategy, calculate the initial priority of each subtask based on the scheduling strategy, combine resource demand information with NPU node computing power evaluation data, and use a greedy algorithm to assign each subtask to the optimal NPU node for execution; Monitor the running status of subtasks on each NPU node in real time. When abnormal situation is found in the execution of subtasks, trigger alarm and take countermeasures.

2. According to the large model scheduling method based on NPU computing power according to claim 1, it is characterized in that: The structural analysis includes analysis of the overall structure of the model, the types of each layer of the model, the number of model parameters and the connection relationship of the calculation units.

3. According to the large model scheduling method based on NPU computing power according to claim 1, it is characterized in that: The computing demand forecast includes: Quantify the computational load of the subtasks, including the number of floating-point operations per second of the convolutional layer and the number of multiplication and addition operations of the fully connected layer, and store them as computational load factors; The temporary and persistent memory required by large models and subtasks at runtime are quantified and stored as memory requirement factors.

4. The large model scheduling method based on NPU computing power according to claim 1 is characterized in that: The training steps of the computing power evaluation model include: By acquiring hardware information, performing benchmark tests, and performing load analysis on the device to be evaluated, we can obtain the basic computing power indicators, actual performance data, and computing characteristics and mode information of the large model of the device to be evaluated, and form a set of feature independent variables. Based on the online monitoring data, record the computing performance of the device to be evaluated under different workloads, including response time, throughput, and NPU utilization, and establish a mapping relationship between the obtained data and the feature independent variable set to form a dependent variable set; A gradient boosting decision tree model is established, and the feature independent variable set and dependent variable set are input to iteratively train the gradient boosting decision tree model, and finally a computing power evaluation model is obtained.

5. The large model scheduling method based on NPU computing power according to claim 1 is characterized in that: The real-time computing power evaluation data of the NPU node includes response time, throughput and NPU utilization.

6. The large model scheduling method based on NPU computing power according to claim 1 is characterized in that: The scheduling strategies include first-come-first-served, short job first, priority scheduling, preemptive scheduling, fair share scheduling, maximum throughput and workload balancing.

7. The large model scheduling method based on NPU computing power according to claim 1 is characterized in that: The steps of using a greedy algorithm to assign each subtask to the optimal NPU node for execution include: According to the dependencies between each subtask, the subtasks are organized into queues, and the first subtasks of all queues are sorted from high to low according to the initial priority; Based on the real-time computing power evaluation data of each NPU node and the resource requirement information of the first subtask in all queues, the resource matching score between the node and the subtask to be executed is calculated; According to the resource matching scores of the first subtasks in all queues, schedule them to the NPU node with the highest resource matching score in descending order of priority; Update the subtask queue, and the remaining subtasks are queued again according to the dependency relationship, and sort the first subtasks in all queues from high to low according to the initial priority; After the subtask queue is updated, the real-time computing power evaluation data of each NPU node and the resource requirement information of each subtask are re-obtained, and the above steps are repeated until all subtasks are scheduled.

8. The large model scheduling method based on NPU computing power according to claim 7 is characterized in that: The computing resource matching score includes: Calculate the remaining capacity of the NPU node using the following formula: RC j =C j ×(1-U j ); Where j is the number of nodes, RC j is the remaining capacity of the jth NPU node, C j is the total memory capacity of the jth NPU node, U j The real-time computing power evaluation data of the j-th NPU node; Calculate the resource matching score between the NPU node and the subtask. The formula is as follows: Among them, i is the number of subtasks, RM ij is the resource matching score between node j and subtask i, w npu is the computational load factor weight of the NPU node, OC i is the computational factor of the ith subtask, w mem is the memory requirement factor weight of the NPU node, MC i is the memory requirement factor of the i-th subtask.

9. The large model scheduling method based on NPU computing power according to claim 1 is characterized in that: The abnormal situations include timeout and resource overflow, and the corresponding measures include rescheduling, resource recovery and parameter tuning.

10. A large model scheduling device based on NPU computing power, characterized in that: include: The model analysis module parses the large model file package, performs structural analysis on the large model, splits the large model into multiple subtasks by layer, analyzes the dependencies between each subtask, predicts the computing requirements of the large model and subtasks, and obtains resource demand information, including computing factor and memory demand factor; The computing power evaluation module trains the computing power evaluation model and uses the trained computing power evaluation model to predict the real-time computing power evaluation data of each NPU node; The scheduling decision module selects the scheduling strategy, calculates the initial priority of each subtask to be scheduled according to the scheduling strategy, combines the resource demand information with the NPU computing power evaluation data, and uses the greedy algorithm to assign each subtask to the optimal NPU node for execution; The execution feedback module monitors the running status of subtasks on each NPU node and feeds back the collected execution data to the scheduling decision module. When abnormal situation is found in the execution of subtasks, an alarm is triggered and countermeasures are taken.

Images