Adaptive operator scheduling method and device in NPU resource shortage environment

Abstract

The embodiment of the invention discloses an operator adaptive scheduling method and device in an NPU resource shortage environment, and is applied to the technical field of computers, and the method comprises the steps: carrying out the initial priority assignment of each operator; the operators with the in-degree being 0 are added into a ready queue according to the initial priority; presetting a priority adjustment period, if the previous operator is executed in the adjustment period, changing the operator in-degree in the branch into 0, and adding into a ready queue; recalculating the priority of each operator by using a weight adjustment algorithm, and performing adaptive adjustment on the operator ready queue based on an executable degree evaluation model according to the recalculated priority of each operator; if the adjustment period is reached, adding the operator which is being executed into the operator ready queue again, and carrying out self-adaptive adjustment on the operator ready queue; and repeating the above process until the operator ready queue is empty. In this way, ordered execution of various operators under limited resources can be guaranteed, and the resource utilization efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and further to the field of computing power scheduling, and in particular to an operator adaptive scheduling method and device under an NPU resource shortage environment. Background Art

[0002] The rationality of the operator scheduling method of the neural network processor (NPU) can directly affect the system's resource allocation and computing efficiency, and thus affect the overall performance of the system.

[0003] At present, common operator scheduling methods often go to two extremes: either they simply focus on the amount of resources required by the operator at the current moment, regardless of its waiting time in the queue, resulting in some operators with small resource requirements but too long waiting time not being executed, causing idle resource waste and overall task delays; or they only focus on the expected waiting time of the operator and ignore the actual resource demand characteristics, resulting in a mismatch between resource allocation and task demand, and also unable to achieve efficient scheduling; there are also some limited dynamic scheduling methods, which will make certain adjustments to the scheduling method based on the estimated time, but they require that an operator be fully executed before making the next round of decisions. This method cannot flexibly adapt to the rapidly changing complex operating environment. The real-time performance is greatly reduced. The estimated time it relies on remains unchanged during the execution of the operator, and cannot accurately reflect the ever-changing actual situation. It is difficult to meet the urgent needs for fast and accurate scheduling of operators in high real-time scenarios. In addition, in actual application scenarios, the computing resources of a single NPU are often faced with a tight situation. A large number of operators of different types and complexities need to be executed on limited NPU resources, and the existing operator scheduling methods do not involve reasonable scheduling methods for operators when the computing resources of a single NPU are tight. Therefore, some key operators may not be allocated resources in time and their execution may be delayed, thereby slowing down the entire task process, seriously affecting the system's operating efficiency and response speed.

[0004] It can be seen that the existing operator scheduling technology has exposed many problems that need to be solved urgently when facing complex real-life scenarios such as tight single NPU resources, comprehensive consideration of multiple factors, and high real-time requirements. Summary of the invention

[0005] In view of this, the present application discloses an operator adaptive scheduling method and device in an NPU resource-constrained environment.

[0006] According to a first aspect of the present application, a method for adaptively scheduling operators in an NPU resource-constrained environment is disclosed, the method comprising:

[0007] Assign an initial priority value to each operator according to the resource critical path and branch depth; according to the operator topological sequence, add the operators with an in-degree of 0 to the ready queue according to the initial priority, and make the operators in the queue perform calculations according to the specified time slice time;

[0008] Preset a priority adjustment period. If, within the adjustment period, the previous operator has finished execution, then the in-degree of the operators in its branch becomes 0 and they are added to the ready queue; obtain the response ratio of the executable operators in the ready queue, recalculate the resource critical path score, obtain the priority adjustment weight through the response ratio and the critical path score, and use the adjustment weight algorithm to recalculate the priority of each operator; according to the recalculated priority of each operator, adaptively adjust the operator ready queue based on the executability evaluation model;

[0009] If the adjustment period is reached and any operator has not finished execution within the time slice time, then add the operator being executed back to the operator ready queue, and readjust the operator ready queue adaptively, so that the operators in the queue perform calculations according to the specified time slice time;

[0010] Repeat the above process until the operator ready queue is empty.

[0011] In some realizable ways of the first aspect, assigning an initial priority value to each operator according to the resource critical path and branch depth includes:

[0012] Perform a breadth-first traversal assignment to each operator according to the operator branch depth; among them, the operators in the same layer are given the same priority;

[0013] Calculate the resource critical path according to the priority of each operator and the required resources, and obtain the resource values and critical path scores corresponding to different operator combinations;

[0014] Use the exponential superposition model to calculate the initial priority; the exponential superposition model is as follows:

[0015] P = A·e kB ;

[0016] where P represents the initial operator priority; A represents the number of branch layers; k is an adjustment parameter and k > 0; B is the critical path score and 0 < B < 1; k controls the influence intensity of B; e kB gradually increases as B increases.

[0017] In some realizable ways of the first aspect, obtaining the response ratio of the executable operators in the ready queue, recalculating the resource critical path score, obtaining the priority adjustment weight through the response ratio and the critical path score, and using the adjustment weight algorithm to recalculate the priority of each operator includes:

[0018] Get the response ratio of executable operators in the ready queue, and recalculate the resource critical path score based on the priority and required resources of each operator;

[0019] Obtain the priority adjustment weight through the response ratio and the critical path score: W = A'·e kB ',

[0020] Among them, B' is the response ratio, A' is the critical path score;

[0021] The weight is adjusted according to the priority, and the priority of each operator is recalculated using the weight adjustment algorithm; the weight adjustment algorithm is expressed as follows:

[0022]

[0023] Among them, W A To adjust the weight, W avg is the weighted mean, W MAX is the maximum weight, and k represents the gain factor.

[0024] In some implementations of the first aspect, the evaluation dimensions of the executable evaluation model include:

[0025] The executable degree of each operator response time, the executable degree of system memory and data dimension matching, the executable degree of NPU memory and operator matching, the executable degree of the current total number of operators, and the execution time of all executed operators.

[0026] In some implementations of the first aspect, adaptively adjusting the operator ready queue based on the executable evaluation model according to the recalculated priority of each operator includes:

[0027] Obtain the executable degree of each operator by obtaining the executable degree of each operator's response time, the executable degree of the matching of system memory and data dimensions, the executable degree of the matching of NPU memory and operators, the executable degree of the total number of current operators, and the score sum of the execution time dimensions of all executed operators;

[0028] If the executable degree of any operator is lower than the set threshold α but the priority is higher than the set threshold β, the computing efficiency of the NPU and CPU is compared, and the operator with the executable degree lower than the set threshold α but the priority higher than the set threshold β is scheduled to the operator with relatively better computing efficiency, and the waiting time of the branch operator of the scheduled operator is cleared;

[0029] If the executable degree of each operator meets the set threshold α, the priority of the operators in the ready queue is readjusted using the executable degree and priority of each operator.

[0030] In some implementations of the first aspect, the priority of the ready queue operator is readjusted using the executable degree and priority of each operator, including:

[0031] Substitute the executable degree of each operator into the priority adjustment formula to obtain the adjusted priority of each operator in the ready queue; the priority adjustment formula is as follows:

[0032] P"=MAX(P'·1.1 E-ThreadE ,100);

[0033] Among them, E is the operator executable degree; ThreadE represents the priority decreasing threshold.

[0034] According to a second aspect of the present application, an operator adaptive scheduling device in an NPU resource shortage environment is disclosed, the device comprising:

[0035] The first processing unit is used to assign an initial priority to each operator according to the resource critical path and the branch depth; according to the operator topology sequence, the operator with an in-degree of 0 is added to the ready queue according to the initial priority, so that the operators in the queue perform calculations according to the specified time slice time;

[0036] The second processing unit is used to preset a priority adjustment cycle. If the previous operator is executed within the adjustment cycle, the operator in-degree in its branch becomes 0 and is added to the ready queue; the response ratio of the executable operators in the ready queue is obtained, the resource critical path score is recalculated, the priority adjustment weight is obtained through the response ratio and the critical path score, and the priority of each operator is recalculated using the adjustment weight algorithm; according to the recalculated priority of each operator, the operator ready queue is adaptively adjusted based on the executable evaluation model;

[0037] The second processing unit is further configured to, if the adjustment period is reached and any operator has not been executed within the time slice, add the executing operator to the operator ready queue again, and re-adjust the operator ready queue adaptively so that the operators in the queue execute calculations according to the specified time slice;

[0038] The second processing unit is further used to repeat the above process until the operator ready queue is empty.

[0039] According to a third aspect of the present application, an electronic device is disclosed. The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described above.

[0040] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is disclosed, where the computer instructions are used to cause a computer to execute the method described above.

[0041] According to a fifth aspect of the present application, a computer program product is disclosed, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0042] In this application, an adjustment cycle is preset according to the critical path of resources, branch depth and operator executable degree, and the priority of operators in the operator queue is dynamically adjusted to realize dynamic scheduling of operators, taking into account the resource requirements and execution waiting time of operators, avoiding the situation where operators wait in line for a long time, ensuring that key operators can be executed in time, and improving the overall completion progress of the task; in addition, when the computing resources of a single NPU are tight, reasonable scheduling can be carried out according to the characteristics of the operator and the current resource status, ensuring the orderly execution of various operators under limited NPU resources and improving resource utilization efficiency.

[0043] It should be understood that the contents described in the Summary of the Invention are not intended to limit the key or important features of the embodiments of the present application, nor are they intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0045] Figure 1 A flowchart of an operator adaptive scheduling method under an NPU resource shortage environment provided by an embodiment of the present application is shown;

[0046] Figure 2 A schematic diagram showing an exemplary computing resource critical path score provided by an embodiment of the present application is shown;

[0047] Figure 3 A schematic diagram of an exemplary recalculation of a resource critical path score provided by an embodiment of the present application is shown;

[0048] Figure 4 A block diagram of an operator adaptive scheduling device in an NPU resource shortage environment according to an embodiment of the present application is shown;

[0049] Figure 5 A block diagram of an exemplary electronic device capable of implementing embodiments of the present application is shown. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0052] In response to the problems mentioned in the background technology, the present application discloses an operator adaptive scheduling method and device in an NPU resource-constrained environment.

[0053] Specifically, an initial priority is assigned to each operator according to the resource critical path and the branch depth; according to the operator topology sequence, the operator with an in-degree of 0 is added to the ready queue according to the initial priority, so that the operators in the queue perform calculations according to the specified time slice time; a priority adjustment period is preset, and if the previous operator is executed within the adjustment period, the in-degree of the operator in its branch becomes 0 and is added to the ready queue; the response ratio of the executable operators in the ready queue is obtained, the resource critical path score is recalculated, the priority adjustment weight is obtained through the response ratio and the critical path score, and the priority of each operator is recalculated using the adjustment weight algorithm; according to the priority of each recalculated operator, the operator ready queue is adaptively adjusted based on the executable evaluation model; if the adjustment period is reached and any operator is not executed within the time slice time, the executing operator is re-added to the operator ready queue, and the operator ready queue is adaptively adjusted again, so that the operators in the queue perform calculations according to the specified time slice time; the above process is repeated until the operator ready queue is empty.

[0054] In this way, the resource requirements and execution waiting time of the operator can be taken into account. When the computing resources of a single NPU are tight, reasonable scheduling can be carried out according to the characteristics of the operator and the current resource status to ensure the orderly execution of various operators under limited NPU resources.

[0055] The following is a more detailed description of the operator adaptive scheduling method and device under the NPU resource shortage environment provided by the present application in combination with the accompanying drawings and specific embodiments.

[0056] Figure 1shows the flowchart of the operator adaptive scheduling method in the NPU resource - constrained environment; as Figure 1 shown, the operator adaptive scheduling method 100 in the NPU resource - constrained environment may include the following steps:

[0057] S110, assign an initial priority to each operator according to the resource critical path and the branch depth; according to the operator topological sequence, add the operators with an in - degree of 0 to the ready queue according to the initial priority, and make the operators in the queue execute calculations according to the specified time - slice time.

[0058] Among them, assigning an initial priority to each operator according to the resource critical path and the branch depth includes:

[0059] Perform a breadth - first traversal assignment for each operator according to the operator branch depth; among them, the operators in the same layer are assigned the same priority;

[0060] Calculate the resource critical path according to the priority of each operator and the required resources, and obtain the resource values and critical - path scores corresponding to different operator combinations;

[0061] Use the exponential superposition model to calculate the initial priority; the exponential superposition model is as follows:

[0062] P = A·e kB ;

[0063] Among them, P represents the initial operator priority; A represents the branch layer number; k is an adjustment parameter and k > 0; B is the critical - path score and 0 < B < 1; k controls the influence intensity of B; e kB gradually increases as B increases.

[0064] Assigning an initial priority to each operator of the static model according to the resource critical path and the branch depth is because generally the available resources will gradually decrease as the program runs. Assuming that the resources are relatively sufficient at the beginning, it is necessary to process the operators with high demands first to avoid insufficient resources later.

[0065] For the initial static priority allocation, the depth importance of the model is greater than the importance of consuming resources, that is, the deeper the depth, the higher the priority, because the deeper the branch, the more resources it may require, and the critical - path score (time) for comparing the resource consumption size at the same depth is considered.

[0066] Figure 2 shows an exemplary diagram of calculating the resource critical - path score provided by the embodiment of the present application; as Figure 2 shown, this process can be as follows:

[0067] AH represents different or the same operators, the black numbers represent the longest depth of the branches, and the red numbers represent the estimated resources required by the operators.

[0068] From H to A, perform breadth-first traversal and assign values. The same layer is assigned the same priority. The depth of H is 1 and the depth of A is 5 (B has a higher priority than C and D). Calculate the resource critical path. The red indicator represents the resources (time) required for each operator. We get: ACH: 50, ABEGH: 37, ABFGH: 33, ADH: 30; then the overall critical path score of ACH is 50 / 50 = 1, the critical path score of ADH is 30 / 50 = 0.6, ABEGH = 0.75, and ABFGH = 0.66.

[0069] Using the exponential superposition model, calculate the initial priority P = A·e kB , where e kB It gradually increases with the increase of B, but the amplitude is relatively smooth; k controls the influence intensity of B. A smaller k can ensure the dominant role of A, while a larger k will increase the impact of B on P. The exponential superposition model can ensure that the importance of depth is higher than the importance of the size of consumed resources to a certain extent.

[0070] It should be noted that:

[0071] The value range of the assignment priority is (1-100). According to P = A·e kB , we can see that assuming k = 1, if P < 100, then the number of levels A should be less than the maximum number of levels MaxLevel = floor (100 / e 1 )=36, but for general network models, the operator layer depth often exceeds 36, so for those with less than 36 layers, no processing is performed, and for those with more than 36 layers, the normalization operation is used to scale the number of layers to (1-36). For example, if the total number of layers is 120 and the number of operator A layers is 7, then the normalized number of layers is converted to (7 / 120*36)=2.1.

[0072] For a computational graph, the operator type and input dimension are known, and relevant features can be obtained through the characteristics of the operator calculation and the data dimension. For example, by comparing the resources consumed by the addition operator to process the dimension (1, 256, 256) + (1, 256, 256) with the resources consumed by the convolution operator conv (stride, padding, kernel_size) to process the dimension (1, 256, 256), the relative value of the required resources can be obtained without actual reasoning, which can ensure the efficiency of static priority assignment.

[0073] S120, preset a priority adjustment cycle. If the previous operator is executed within the adjustment cycle, the operator in-degree in its branch becomes 0 and is added to the ready queue. Obtain the response ratio of the executable operators in the ready queue, recalculate the resource critical path score, obtain the priority adjustment weight through the response ratio and the critical path score, and recalculate the priority of each operator using the adjustment weight algorithm. According to the recalculated priority of each operator, the operator ready queue is adaptively adjusted based on the executable evaluation model.

[0074] in:

[0075] Get the response ratio of executable operators in the ready queue, recalculate the resource critical path score, obtain the priority adjustment weight through the response ratio and the critical path score, and recalculate the priority of each operator using the weight adjustment algorithm, including:

[0076] Get the response ratio of executable operators in the ready queue, and recalculate the resource critical path score based on the priority and required resources of each operator;

[0077] Obtain the priority adjustment weight through the response ratio and the critical path score: W = A'·e kB ', where B' is the response ratio and A' is the critical path score;

[0078] The weight is adjusted according to the priority, and the priority of each operator is recalculated using the weight adjustment algorithm; the weight adjustment algorithm is expressed as follows:

[0079]

[0080] Among them, W A To adjust the weight, W avg is the weighted mean, W MAX is the maximum weight, and k represents the gain factor.

[0081] Furthermore, the evaluation dimensions of the executable evaluation model include:

[0082] The executable degree of each operator response time, the executable degree of system memory and data dimension matching, the executable degree of NPU memory and operator matching, the executable degree of the current total number of operators, and the execution time of all executed operators.

[0083] According to the recalculated priority of each operator, the operator ready queue is adaptively adjusted based on the executable evaluation model, including:

[0084] Obtain the executable degree of each operator by obtaining the executable degree of each operator's response time, the executable degree of the matching of system memory and data dimensions, the executable degree of the matching of NPU memory and operators, the executable degree of the total number of current operators, and the score sum of the execution time dimensions of all executed operators;

[0085] If the executable degree of any operator is lower than the set threshold α but the priority is higher than the set threshold β, the computing efficiency of the NPU and CPU is compared, and the operator with the executable degree lower than the set threshold α but the priority higher than the set threshold β is scheduled to the operator with relatively better computing efficiency, and the waiting time of the branch operator of the scheduled operator is cleared;

[0086] If the executable degree of each operator meets the set threshold α, the priority of the operators in the ready queue is readjusted using the executable degree and priority of each operator.

[0087] Furthermore, the priority of the operators in the ready queue is readjusted using the executable degree and priority of each operator, including:

[0088] Recalculate the priority of the operators in the ready queue using the executable degree and priority of each operator. Substitute the executable degree of each operator into the priority adjustment formula to obtain the adjusted priority of each operator in the ready queue. The priority adjustment formula is as follows:

[0089] P"=MAX(P'·1.1 E-ThreadE ,100);

[0090] Among them, E is the operator executable degree; ThreadE represents the priority decreasing threshold.

[0091] like Figure 2 As shown, when k = 1, operator B: P = 4·e 0.75 =8.468; Similarly, operator C: 5.436; operator D: 3.644; operator E: 6.351; operator F: 5.8043; operator G (taking the maximum critical path score: 0.6, 0.75): 4.234; operator A: 13.5914.

[0092] According to the topological sorting, all operators with in-degree 0 are added to the ready queue first, and sorted by priority. When operator A is executed, all operators will have a waiting time of 10, but since A is the ancestor node of all operators, the waiting time of all child node operators is reset to 0, so the operator response ratio is 1 = (execution time + waiting time) / execution time, then:

[0093] Operator Set B C D E F G Response ratio 1 1 1 1 1 1

[0094] After executing operator A, the in-degree of operators BCD becomes 0 and they are added to the ready queue. The queue is as follows:

[0095] Operator ready queue B C D Priority 8.468 5.436 3.644

[0096] For the executable operators in the ready queue, although B has a higher priority than CD, the current environment resources and some response ratio factors need to be considered to obtain the executable degree E(O) of the current operator O, as shown below:

[0097] Operator ready queue B C D Executability 5.44 7.89 14.112

[0098] The operator executable evaluation model considers the following two aspects:

[0099] 1) NPU memory: This is a memory area specially allocated for NPU computing tasks. It is used to store the intermediate calculation results, temporary tensors and weight data of the model. NPU memory has fast access speed and is suitable for high-frequency data operations to reduce the delay of operator calculation.

[0100] 2) System memory (RAM): System memory is mainly used to store larger data blocks, such as input data, output results, and model parameters. In some cases, the model weights and data are first loaded into the system memory, and then transferred to the NPU memory for calculation as needed during the calculation process.

[0101] Data is usually exchanged between system memory and NPU memory through DMA (direct memory access) and other methods to efficiently transfer data between NPU calculations and main system processing. During the model initialization phase, the system memory may load the weights of the entire model, and then transfer part of the weights to the NPU memory as needed to perform specific operator operations. Therefore, the coordinated use of NPU memory and system memory can significantly improve the execution efficiency of operators on the NPU, avoid frequent data movement, and ensure the overall operating efficiency and resource utilization of the system.

[0102] Based on the above two directions, the following dimensions are specifically considered:

[0103] Executability of each operator response time:

[0104]

[0105] Among them, R max is the maximum response ratio; T wait is the operator waiting time; T exe is the operator execution time.

[0106] System memory and data dimension matching executable degree:

[0107]

[0108] M req is based on operator type D input (O) and memory block size and other factors to estimate the required resources; L mem The memory load rate of the current system.

[0109] NPU memory and operator matching executability:

[0110]

[0111] NPU req To estimate the required resources based on factors such as operator type and memory block size; NPU mem It is the memory load rate of the current NPU.

[0112] The total number of operators currently available for execution is:

[0113]

[0114] S exec The total number of operators currently being executed and the execution time of all operators:

[0115]

[0116] T exe is the current execution time, T req is the time required by the operator, and w1+w2+w3+w4+w5=1; then:

[0117] E(o)=

[0118] 20*(w1·f response (T wait (o),T exe (o))+w2·f mem (M free ,M req (o))+w3·f NPUmem +w4·f s (S exec )+w5·f time );

[0119] If the calculated operator executable degree is lower than the set threshold α but the priority is higher than the set threshold β, such as the following situation:

[0120] Ready Queue B C D Executability 0.1 7.89 14.112

[0121] Ready Queue B C D Priority 85 5.436 3.644

[0122] Compare the advantages and disadvantages of NPU computing efficiency and CPU computing efficiency. If the CPU computing efficiency is higher than (NPU computing time + waiting time), it is scheduled to be executed on the CPU. B is directly scheduled to the CPU, and the waiting time of branch B is cleared to 0.

[0123] If the calculated executability is within the threshold range, the priority of the queue operator is recalculated using the executability and priority: P" = MAX(P'·1.1 E-ThreadE ,100), if the executable degree is less than the threshold, the calculated priority will become smaller, as shown below:

[0124] Ready Queue B C D Executability 5.44 7.89 14.112

[0125] Ready Queue B C D Priority 8.468 5.436 3.644

[0126] After adjusting the executable evaluation model, D has a higher priority than C, as shown below:

[0127] Ready Queue B D C Priority 14.2218 13.9865 11.531

[0128] S130, if the adjustment period is reached and any operator has not been executed within the time slice, the executing operator is added back to the operator ready queue, and the operator ready queue is adaptively adjusted again so that the operators in the queue execute calculations according to the specified time slice.

[0129] S140, repeat the above process until the operator ready queue is empty.

[0130] like Figure 2 As shown in the figure, when B is executed, operator CD will generate a waiting time of 3, the waiting time of branch B is cleared to 0, and operator EF is added to the ready queue, and the response ratio is recalculated. The recalculated response ratio is as follows:

[0131] Operator Set C D E F G Response ratio 1.12 1.6 1 1 1

[0132] If the ready queue is not empty at this time, the resource critical path score is recalculated, and the priority adjustment weight is obtained through the response ratio and the critical path score, as follows: Recalculate the resource critical path score, Figure 3 FIG. 4 shows a schematic diagram of an exemplary recalculation of resource critical path scores provided in an embodiment of the present application; Figure 3 As shown, the critical path is CH: 40, so the critical path scores are CH: 40 / 40 = 1, EGH: 24 / 40 = 0.6, FGH = 20 / 40 = 0.5, DH: 20 / 40 = 0.5.

[0133] Calculate the adjustment weight of the queue priority based on the response ratio and the critical path score: W = A'·e kB ', when k = 1:

[0134] W C =1*e 1.12 =3.064,W D =0.5·e 1.6 =2.47,WE =1.630,W F =1.359.

[0135] The priority adjustment weight obtained by using the response ratio and the critical path score can be shown as follows:

[0136] Operator Set D C E F Response ratio 13.9865 11.531 6.351 5.804

[0137] Using the Adjust Weight Algorithm Recalculate the priorities as follows:

[0138] W avg =2.13075;W MAX =3.064; k=1.1.

[0139]

[0140] If the weights are adjusted as follows:

[0141]

[0142] The adjusted priorities are as follows:

[0143] D C E F 15.6648 15.3925 5.209 4.19556

[0144] Then, the above executable evaluation model is used to make the final priority adjustment, select the operator with the highest current priority, and repeat the above process until the operator ready queue is empty.

[0145] It should be noted that:

[0146] The whole process is preemptive scheduling, that is, the next high-priority operator is not executed until an operator is completely executed. Instead, the priority of each operator in the ready queue is recalculated at regular intervals. If the priority is higher than the executing operator and exceeds the set threshold ρ, the executing operator will be re-added to the ready queue (the waiting time is cleared to 0).

[0147] Assume that the execution time required for B is 20. As the execution time increases, the response ratio of CD will also increase. In order to ensure real-time dynamic preemption, assume that the priority of CD is recalculated according to the above logic every 10 seconds. If the priority of D exceeds that of B and the excess value is greater than ρ, operator D is executed and B is added to the ready queue.

[0148] According to the embodiments of the present application, the following technical effects are achieved:

[0149] It takes into account factors such as the operator response ratio and the current operating environment, and assists in the dynamic and reasonable scheduling of operators. The real-time preemptive scheduling algorithm that adjusts the operator priority according to the critical path score and executable degree ensures that each executable operator can be run within a certain period of time. To a certain extent, it ensures that the algorithm can be applied to scenarios with limited resources and high real-time requirements, reducing the waiting time of each operator, thereby improving the reasoning speed of the overall model.

[0150] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0151] The above is an introduction to the method embodiment. The following is a further explanation of the scheme described in this application through an apparatus embodiment.

[0152] Figure 4 FIG. 1 shows a block diagram of an operator adaptive scheduling device in an NPU resource shortage environment according to an embodiment of the present application. Figure 4 As shown, the operator adaptive scheduling device 400 in an NPU resource shortage environment may include:

[0153] The first processing unit 410 is used to assign an initial priority to each operator according to the resource critical path and the branch depth; according to the operator topology sequence, the operator with an in-degree of 0 is added to the ready queue according to the initial priority, so that the operators in the queue perform calculations according to the specified time slice time.

[0154] The second processing unit 420 is used to preset a priority adjustment cycle. If the previous operator is executed within the adjustment cycle, the operator in-degree in its branch becomes 0 and is added to the ready queue; the response ratio of the executable operators in the ready queue is obtained, the resource critical path score is recalculated, the priority adjustment weight is obtained through the response ratio and the critical path score, and the priority of each operator is recalculated using the adjustment weight algorithm; according to the recalculated priority of each operator, the operator ready queue is adaptively adjusted based on the executable evaluation model.

[0155] The second processing unit 420 is also used to add the executing operator to the operator ready queue again if any operator has not been executed within the time slice when the adjustment period is reached, and to adaptively adjust the operator ready queue again so that the operators in the queue perform calculations according to the specified time slice.

[0156] The second processing unit 420 is further configured to repeat the above process until the operator ready queue is empty.

[0157] Understandably, Figure 4 Each unit in the operator adaptive scheduling device 400 under the NPU resource shortage environment shown has the function of implementing each step in the operator adaptive scheduling method 100 under the NPU resource shortage environment provided in the embodiment of the present application, and can achieve its corresponding technical effect. The specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment. For the convenience and conciseness of the description, it will not be repeated here.

[0158] According to an embodiment of the present application, the present application also discloses an electronic device, a readable storage medium, and a computer program product.

[0159] Figure 5 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0160] The electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a ROM 502 or a computer program loaded from a storage unit 508 into a RAM 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An I / O interface 505 is also connected to the bus 504.

[0161] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks. The computing unit 501 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the method 100 in any other appropriate manner (e.g., by means of firmware).

[0162] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device. The program code for implementing the method of the present application may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0163] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In order to provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which a user may provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input). The systems and techniques described herein can be implemented in a computing system including a backend component (e.g., as a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a frontend component (e.g., a user computer with a graphical user interface or a web browser, through which a user can interact with the implementation of the system and technology described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet. A computer system can include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain. It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document does not limit it here.The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. An operator adaptive scheduling method under NPU resource shortage environment, characterized in that: The method comprises: Assign initial priority to each operator according to the critical path of resources and branch depth; according to the operator topology sequence, add operators with in-degree 0 to the ready queue according to the initial priority, so that the operators in the queue perform calculations according to the specified time slice; A priority adjustment cycle is preset. If the previous operator is executed within the adjustment cycle, the operator in-degree in its branch becomes 0 and is added to the ready queue. The response ratio of the executable operators in the ready queue is obtained, and the resource critical path score is recalculated. The priority adjustment weight is obtained through the response ratio and the critical path score, and the priority of each operator is recalculated using the adjustment weight algorithm. According to the recalculated priority of each operator, the operator ready queue is adaptively adjusted based on the executable evaluation model. If the adjustment period is reached and any operator has not been executed within the time slice, the executing operator will be added back to the operator ready queue, and the operator ready queue will be adaptively adjusted again so that the operators in the queue can perform calculations according to the specified time slice. Repeat the above process until the operator ready queue is empty.

2. The method according to claim 1, characterized in that The initial priority assignment for each operator according to the resource critical path and branch depth includes: According to the depth of operator branches, each operator is assigned a value in a breadth-first traversal; operators at the same layer are given the same priority; According to the priority and required resources of each operator, the resource critical path is calculated to obtain the resource values ​​and critical path scores corresponding to different operator combinations; The initial priority is calculated using an exponential superposition model; the exponential superposition model shown is as follows: P=A·e kB ; Wherein, P represents the initial operator priority; A represents the number of branch layers; k is a regulation parameter and k > 0; B is the critical path score and 0 < B < 1; k controls the influence strength of B; e kB gradually increases as B increases.

3. The method according to claim 1, characterized in that The step of obtaining the response ratio of executable operators in the ready queue, recalculating the resource critical path score, obtaining the priority adjustment weight through the response ratio and the critical path score, and recalculating the priority of each operator using the weight adjustment algorithm includes: Get the response ratio of executable operators in the ready queue, and recalculate the resource critical path score based on the priority and required resources of each operator; Obtain the priority adjustment weight through the response ratio and the critical path score: W = A'·e kB '; Among them, B' is the response ratio, A' is the critical path score; The weight is adjusted according to the priority, and the priority of each operator is recalculated using the weight adjustment algorithm; the weight adjustment algorithm is expressed as follows: Among them, W A To adjust the weight, W avg is the weighted mean, W MAX is the maximum weight, and k represents the gain factor.

4. The method according to claim 1, characterized in that: The evaluation dimensions of the executable evaluation model include: The executable degree of each operator response time, the executable degree of system memory and data dimension matching, the executable degree of NPU memory and operator matching, the executable degree of the current total number of operators, and the execution time of all executed operators.

5. The method according to claim 4, characterized in that The step of adaptively adjusting the operator ready queue based on the executable evaluation model according to the recalculated priority of each operator includes: Obtain the executable degree of each operator by obtaining the executable degree of each operator's response time, the executable degree of the matching of system memory and data dimensions, the executable degree of the matching of NPU memory and operators, the executable degree of the total number of current operators, and the score sum of the execution time dimensions of all executed operators; If the executable degree of any operator is lower than the set threshold α but the priority is higher than the set threshold β, the computing efficiency of the NPU and CPU is compared, and the operator with the executable degree lower than the set threshold α but the priority higher than the set threshold β is scheduled to the operator with relatively better computing efficiency, and the waiting time of the branch operator of the scheduled operator is cleared; If the executable degree of each operator meets the set threshold α, the priority of the operators in the ready queue is readjusted using the executable degree and priority of each operator.

6. The method according to claim 5, characterized in that The method of re-adjusting the priority of the ready queue operators by using the executable degree and priority of each operator includes: Substitute the executable degree of each operator into the priority adjustment formula to obtain the adjusted priority of each operator in the ready queue; the priority adjustment formula is as follows: P”=MAX(P'·1.1 E-ThreadE ,100); Among them, E is the operator executable degree; ThreadE represents the priority decreasing threshold.

7. An operator adaptive scheduling device under NPU resource shortage environment, characterized in that: The device comprises: The first processing unit is used to assign an initial priority to each operator according to the resource critical path and the branch depth; according to the operator topology sequence, the operator with an in-degree of 0 is added to the ready queue according to the initial priority, so that the operators in the queue perform calculations according to the specified time slice time; The second processing unit is used to preset a priority adjustment cycle. If the previous operator is executed within the adjustment cycle, the operator in-degree in its branch becomes 0 and is added to the ready queue; the response ratio of the executable operators in the ready queue is obtained, the resource critical path score is recalculated, the priority adjustment weight is obtained through the response ratio and the critical path score, and the priority of each operator is recalculated using the adjustment weight algorithm; according to the recalculated priority of each operator, the operator ready queue is adaptively adjusted based on the executable evaluation model; The second processing unit is further configured to, if the adjustment period is reached and any operator has not been executed within the time slice, add the executing operator to the operator ready queue again, and adaptively adjust the operator ready queue again, so that the operators in the queue execute calculations according to the specified time slice; The second processing unit is further configured to repeat the above process until the operator ready queue is empty.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images