Hybrid precision computing power scheduling system, electronic equipment and readable storage medium

Abstract

The invention provides a mixed precision computing power scheduling system, electronic equipment and a readable storage medium, which can realize cluster-level precision level computing power scheduling by sensing the operator precision level in an accelerator card, intercepting a computing power calling function and forwarding the computing power calling function according to a precision route based on an RDMA (Remote Direct Memory Access) technology. According to the method, the utilization rate of the AI computing power pool and the computing efficiency of the AI tasks can be effectively improved, abuse of high-precision operators by part of the AI tasks can be avoided, the operation management capability of the operators is improved, and the method has popularization value. The system can fully utilize the precision operators of the whole AI computing cluster in a scheduling layer for complex scenes of single machine and multiple cards and multiple machines and multiple cards, and optimal scheduling of operator resources with different precision is carried out among a plurality of acceleration cards of a single host and among a plurality of hosts. Besides, the system can dynamically select the optimal scheduling path in combination with the operator use condition of the bottom layer besides performing scheduling allocation according to topology, so that the execution efficiency of the task is further improved.

Description

Technical Field

[0001] The present invention relates to the field of computing power scheduling technology, and in particular to a mixed-precision computing power scheduling system, an electronic device and a readable storage medium. Background Art

[0002] In the computing scenarios of AI training and reasoning tasks, operators can be divided into operators of different precisions, such as FP32, FP16, INT8, and INT4. The accelerator cards of hardware manufacturers also tend to have different precisions according to different models. Different cards have different numbers of high and low precision.

[0003] Since high-precision computing takes up a lot of memory space and requires a lot of network transmission bandwidth, the industry will convert computing processes that do not need to maintain high-precision computing to low-precision to improve computing efficiency and save high-precision operators for use in computing processes that require more high-precision computing. The industry calls this mixed-precision computing. For example, PyTorch and TensorFlow provide automatic mixed precision (AMP) training.

[0004] The current mixed precision training in the industry has the following major defects:

[0005] 1. The scheduling allocation of precision and computing power scheduling cannot be linked:

[0006] Most frameworks, such as Pytorch, only focus on the precision allocation at the algorithm layer, and are unaware of the distribution of operators with different precisions on the underlying accelerator card, as well as their capacity and usage. This results in unreasonable utilization of accelerator card operators when multiple tasks are executed.

[0007] 2. The problem of precision abuse cannot be avoided:

[0008] Currently, most of the computing power control in the industry can only achieve the dimensions of computing power and video memory. For some applications that can be satisfied with low precision, there is a lack of effective control over the malicious use of high-precision computing power for calculations, resulting in the inability to perform computing tasks that really require high-precision computing power, or the loss of computing precision.

[0009] 3. Cross-host operator scheduling is not possible:

[0010] The effective use of mixed precision in training frameworks such as Pytorch can only be performed on a single machine and a single card, and effective precision scheduling cannot be performed across cards or machines. Summary of the invention

[0011] In view of the above problems, the present invention is proposed to provide a mixed-precision computing power scheduling system, an electronic device and a readable storage medium that overcome the above problems or at least partially solve the above problems.

[0012] The present invention provides a mixed-precision computing power scheduling system, comprising:

[0013] Mixed-precision resource pool, used to record the distribution topology, capacity information, and usage information of multiple-precision operators of each accelerator card in the computing node cluster;

[0014] The computing device plug-in is deployed on each computing node to monitor the capacity and usage information of multiple precision operators of each accelerator card on the computing node in real time, and report it to the mixed precision scheduler for storage in the mixed precision resource pool;

[0015] A mixed precision scheduler is used to receive mixed precision operator application information of AI tasks submitted by users, and obtain the real-time capacity information of multiple precision operators of each accelerator card of each computing node from the mixed precision resource pool, and allocate the operator topology of the AI ​​task according to the mixed precision operator application information and the real-time capacity information of multiple precision operators of each accelerator card of each computing node; the mixed precision operator information includes the mixed precision operator level and the number of mixed precision operators;

[0016] The computing power interception module is deployed on each computing node and is used to intercept the function call of the AI ​​task. According to the operator topology of the AI ​​task, or according to the operator topology of the AI ​​task and the real-time usage information of multiple precision operators of each accelerator card of each computing node, the function call of the AI ​​task is matched with the computing power quota of operators with different precisions to determine the target accelerator card, and the function call of the AI ​​task is forwarded to the target accelerator card.

[0017] Optionally, the mixed precision scheduler is used to:

[0018] Determine the main precision level operator according to the mixed precision operator application information; the main precision level operator is the precision level operator with the largest number of applications; the precision level operators other than the main precision level operator in the mixed precision operator application information are sub-precision level operators;

[0019] According to the real-time capacity information of various precision operators of each computing node and each accelerator card, find and allocate computing nodes that meet the number of operators requested at the main precision level;

[0020] Allocate a secondary precision level operator on the same computing node as the allocated primary precision level operator to the AI ​​task;

[0021] If the number of sub-precision level operators on the same computing node as the allocated main precision level operator is lower than the number of sub-precision level operators applied for, the difference in the number of sub-precision level operators is calculated, and a computing node with a number of sub-precision level operators that meets the difference is found and allocated to the AI ​​task.

[0022] Optionally, the mixed precision scheduler is used to:

[0023] If the computing node where the user submits the AI ​​task is a computing node that meets the number of operators requested for the main precision level, the computing node where the user submits the AI ​​task will be preferentially allocated to the AI ​​task.

[0024] Optionally, the mixed precision scheduler is used to:

[0025] Determine the main precision level operator according to the mixed precision operator application information; the main precision level operator is the precision level operator with the largest number of applications; the precision level operators other than the main precision level operator in the mixed precision operator application information are sub-precision level operators;

[0026] According to the real-time capacity information of various precision operators of each computing node and each accelerator card, find and allocate computing nodes that meet the number of operators requested at the main precision level;

[0027] Allocate the secondary precision level operators of the computing nodes other than the computing nodes where the allocated primary precision level operators are located to the AI ​​task.

[0028] Optionally, the computing power interception module is used to:

[0029] Send a query request for idle secondary precision level operators to the computing power device plug-in on the computing node where the allocated primary precision level operator is located, and receive the query result of idle secondary precision level operators;

[0030] If the idle secondary precision level operator query result is that the secondary precision level operator on the computing node where the allocated primary precision level operator is located is currently in an idle state, the secondary precision level operator on the computing node where the allocated primary precision level operator is located is allocated to the AI ​​task;

[0031] If the query result of the idle secondary precision level operator is that the secondary precision level operator on the computing node where the allocated primary precision level operator is located is currently in a busy state, then the secondary precision level operators of the computing nodes other than the computing node where the allocated primary precision level operator is located are allocated to the AI ​​task.

[0032] Optionally, the computing power interception module is used to:

[0033] If the secondary precision level operator on the computing node where the allocated primary precision level operator is located has been allocated to another AI task, then during the processing of the calculation of the AI ​​task, when a new calculation task of the other AI task is received, a suspend operation is performed on the new calculation task of the other AI task, and after the calculation of the AI ​​task is completed, the secondary precision level operator on the computing node where the allocated primary precision level operator is located is switched to the other AI task.

[0034] Optionally, the computing power interception module is used to:

[0035] Intercepting the function call of the AI ​​task and parsing the function call to obtain the requested operator precision level;

[0036] Determine whether the mixed-precision operator application information of the AI ​​task includes the operator precision level of the function call request;

[0037] If not, reject the function call;

[0038] If so, the execution of the function call is controlled according to the usage of the requested operator precision level and the application amount of the requested operator precision level in the mixed-precision operator application information.

[0039] Optionally, the computing power interception module is used to:

[0040] Calculate the computing power required to execute the function call;

[0041] Determine whether the sum of the usage of the requested operator precision level and the computing power required for the function call execution exceeds the application amount of the requested operator precision level in the mixed precision operator application information;

[0042] If not, executing the function call;

[0043] If it exceeds, the function call is placed in a queue, other function calls are executed and computing power is released until the remaining available quota of the requested operator precision level is higher than the computing power required for executing the function call, and the function call is executed.

[0044] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the electronic device executes the computer program, it loads a mixed-precision computing power scheduling system as described in any one of the embodiments of the present invention.

[0045] The present invention also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the mixed-precision computing power scheduling system as described in any one of the embodiments of the present invention is loaded.

[0046] The present invention includes the following advantages:

[0047] The mixed-precision computing power scheduling system of the present invention intercepts the computing power call function and forwards it by precision routing based on RDMA technology by sensing the operator precision level in the accelerator card, thereby realizing cluster-level precision level computing power scheduling, which can effectively improve the utilization rate of the AI ​​computing power pool and the computing efficiency of AI tasks, and can avoid the abuse of high-precision operators by some AI tasks, improve the operational management capabilities of operators, and has promotion value. The system targets complex scenarios of single-machine multi-cards and multi-machine multi-cards, and can make full use of the precision operators of the entire AI computing cluster at the scheduling layer, and perform optimal scheduling of operator resources of different precisions between multiple accelerator cards of a single host and between multiple hosts. In addition, in addition to scheduling and allocating according to the topology, the system can also dynamically select the optimal scheduling path in combination with the underlying operator usage, thereby further improving the execution efficiency of the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a structural block diagram of a mixed-precision computing power scheduling system provided by an embodiment of the present invention;

[0049] Figure 2 This is the static precision scheduling process of the mixed precision computing power scheduling system provided by an embodiment of the present invention;

[0050] Figure 3 This is the dynamic precision scheduling process of the mixed precision computing power scheduling system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Reference Figure 1 , shows a structural block diagram of a mixed-precision computing power scheduling system provided in an embodiment of the present invention, which may specifically include the following modules:

[0053] Mixed-precision resource pool, used to record the distribution topology, capacity information, and usage information of multiple-precision operators of each accelerator card in the computing node cluster;

[0054] The computing device plug-in is deployed on each computing node to monitor the capacity and usage information of multiple precision operators of each accelerator card on the computing node in real time, and report it to the mixed precision scheduler for storage in the mixed precision resource pool;

[0055] A mixed precision scheduler is used to receive mixed precision operator application information of AI tasks submitted by users, and obtain the real-time capacity information of multiple precision operators of each accelerator card of each computing node from the mixed precision resource pool, and allocate the operator topology of the AI ​​task according to the mixed precision operator application information and the real-time capacity information of multiple precision operators of each accelerator card of each computing node; the mixed precision operator information includes the mixed precision operator level and the number of mixed precision operators;

[0056] The computing power interception module is deployed on each computing node and is used to intercept the function call of the AI ​​task. According to the operator topology of the AI ​​task, or according to the operator topology of the AI ​​task and the real-time usage information of multiple precision operators of each accelerator card of each computing node, the function call of the AI ​​task is matched with the computing power quota of operators with different precisions to determine the target accelerator card, and the function call of the AI ​​task is forwarded to the target accelerator card.

[0057] The present invention implements a mixed-precision computing power scheduling method through a mixed-precision computing power scheduling system. The method senses the precision operator in the accelerator card, intercepts the computing power call function, and forwards it by precision routing based on RDMA technology. This process can automatically and accurately match operators of different precisions in the calculation process and effectively schedule them at the entire cluster level to improve the computing power utilization efficiency and operator control capability of the accelerator card, improve the utilization rate of AI task operations, and effectively avoid the abuse of high-precision operators.

[0058] The present invention deploys the following components in a cloud-native distributed training task environment:

[0059] Computing device plug-in: Deployed in each computing node, it is responsible for real-time monitoring of the status of the local accelerator card, including the distribution, capacity and usage information of operators of different precisions on it. This information is reported to the mixed precision scheduler and stored in the mixed precision resource pool for subsequent scheduling decisions.

[0060] Computing power interception module: Deployed in each computing node, this module is responsible for intercepting CUDALibrary and other library calls initiated by AI tasks, matching and determining the computing power quotas of operators with different precisions, and forwarding the calls to the specified accelerator card. Specifically, in addition to scheduling and allocating based on the topology, it can also be combined with the underlying operator usage.

[0061] Mixed-precision scheduler: This scheduler performs intelligent scheduling within the cluster based on the precision level and computing power quota requested by the AI ​​tasks submitted by the user, and uses the information in the mixed-precision resource pool to match and allocate computing power resources of corresponding precision.

[0062] Mixed-precision resource pool: records the distribution topology of different precision computing power of the entire cluster, as well as its capacity and usage information.

[0063] The hybrid precision computing power scheduling system of the present invention not only realizes the optimal scheduling of precision operator resources in complex scenarios of single-machine multi-card and multi-machine multi-card through the collaborative work of these components, but also can dynamically select the optimal scheduling path based on the underlying operator usage during the scheduling process, thereby further improving the execution efficiency of the task.

[0064] In one embodiment of the present invention, the mixed precision scheduler is used to:

[0065] Determine the main precision level operator according to the mixed precision operator application information; the main precision level operator is the precision level operator with the largest number of applications; the precision level operators other than the main precision level operator in the mixed precision operator application information are sub-precision level operators;

[0066] According to the real-time capacity information of various precision operators of each computing node and each accelerator card, find and allocate computing nodes that meet the number of operators requested at the main precision level;

[0067] Allocate a secondary precision level operator on the same computing node as the allocated primary precision level operator to the AI ​​task;

[0068] If the number of sub-precision level operators on the same computing node as the allocated main precision level operator is lower than the number of sub-precision level operators applied for, the difference in the number of sub-precision level operators is calculated, and a computing node with a number of sub-precision level operators that meets the difference is found and allocated to the AI ​​task.

[0069] In an embodiment of the present invention, the mixed precision scheduler first determines the precision level operator with the largest number of applications as the main precision level operator based on the mixed precision operator application information submitted by the AI ​​task, and the operators of the remaining precision levels are regarded as secondary precision level operators. The scheduler then prioritizes finding and allocating computing nodes that can meet the number of main precision level operator applications based on the real-time capacity information of various precision operators on each computing node and its accelerator card. On this basis, the scheduler will try to allocate the secondary precision level operators on the same computing node as the allocated main precision level operator to the AI ​​task to achieve efficient use of resources. If the number of secondary precision level operators on the same computing node is not enough to meet the application requirements, the scheduler will calculate the difference and continue to find other computing nodes to supplement the required number of secondary precision level operators to ensure that the AI ​​task can be executed smoothly.

[0070] In one embodiment of the present invention, the mixed precision scheduler is used to:

[0071] If the computing node where the user submits the AI ​​task is a computing node that meets the number of operators requested for the main precision level, the computing node where the user submits the AI ​​task will be preferentially allocated to the AI ​​task.

[0072] In the embodiment of the present invention, the mixed precision scheduler gives special consideration to the computing node where the user submits the AI ​​task when allocating computing resources. When the number of operators at the main precision level required for the task submitted by the user can be satisfied by the computing node where the user is located, the scheduler will give priority to allocating the node to the AI ​​task to reduce data transmission delay and improve computing efficiency.

[0073] In one embodiment of the present invention, the mixed precision scheduler is used to:

[0074] Determine the main precision level operator according to the mixed precision operator application information; the main precision level operator is the precision level operator with the largest number of applications; the precision level operators other than the main precision level operator in the mixed precision operator application information are sub-precision level operators;

[0075] According to the real-time capacity information of various precision operators of each computing node and each accelerator card, find and allocate computing nodes that meet the number of operators requested at the main precision level;

[0076] Allocate the secondary precision level operators of the computing nodes other than the computing nodes where the allocated primary precision level operators are located to the AI ​​task.

[0077] In an embodiment of the present invention, the mixed precision scheduler also first determines the primary precision level operator based on the mixed precision operator application information of the AI ​​task, and allocates it based on the real-time capacity information of the accelerator card of each computing node. When allocating secondary precision level operators, the scheduler is no longer limited to the same computing node to which the primary precision level operator has been allocated, but allows secondary precision level operators to be allocated to AI tasks from other computing nodes. Although this cross-node allocation strategy may increase the complexity of data transmission, it can more flexibly utilize the computing power resources in the cluster, especially when specific precision operator resources are tight on certain computing nodes, providing more execution options for AI tasks, thereby improving overall resource utilization and task execution flexibility.

[0078] In one embodiment of the present invention, the computing power interception module is used to:

[0079] Send a query request for idle secondary precision level operators to the computing power device plug-in on the computing node where the allocated primary precision level operator is located, and receive the query result of idle secondary precision level operators;

[0080] If the idle secondary precision level operator query result is that the secondary precision level operator on the computing node where the allocated primary precision level operator is located is currently in an idle state, the secondary precision level operator on the computing node where the allocated primary precision level operator is located is allocated to the AI ​​task;

[0081] If the query result of the idle secondary precision level operator is that the secondary precision level operator on the computing node where the allocated primary precision level operator is located is currently in a busy state, then the secondary precision level operators of the computing nodes other than the computing node where the allocated primary precision level operator is located are allocated to the AI ​​task.

[0082] In an embodiment of the present invention, the computing power interception module can optimize the allocation process of sub-precision level operators. Specifically, the module first sends an idle sub-precision level operator query request to the computing power device plug-in on the computing node where the main precision level operator is allocated, so as to grasp the status of these sub-precision level operators in real time. Once the query results are received, if the results show that these sub-precision level operators are currently idle, the computing power interception module will immediately assign them to the corresponding AI tasks, thereby maximizing the use of existing resources and reducing waiting time. On the contrary, if the query results show that these sub-precision level operators are busy, other computing nodes will be selected to find and assign available sub-precision level operators to AI tasks to ensure that the tasks can be carried out continuously and efficiently. This computing power allocation strategy based on real-time status query effectively improves the utilization rate of the AI ​​computing power pool and the computing efficiency of AI tasks.

[0083] In one embodiment of the present invention, the computing power interception module is used to:

[0084] If the secondary precision level operator on the computing node where the allocated primary precision level operator is located has been allocated to another AI task, then during the processing of the calculation of the AI ​​task, when a new calculation task of the other AI task is received, a suspend operation is performed on the new calculation task of the other AI task, and after the calculation of the AI ​​task is completed, the secondary precision level operator on the computing node where the allocated primary precision level operator is located is switched to the other AI task.

[0085] In an embodiment of the present invention, the computing power interception module not only pays attention to the allocation of sub-precision level operators, but also pays in-depth attention to the priority management between AI tasks. If the sub-precision level operator on the computing node where the allocated main precision level operator is located has been originally allocated to another AI task, the computing power interception module can adopt a refined management strategy. That is, although the sub-precision level operator on the computing node where the allocated main precision level operator is located has been originally allocated to another AI task, the other AI task has not yet been assigned a new computing task, making the sub-precision level operator temporarily idle, so it is first allocated to this AI task for use, and then during the calculation process of this AI task, a new computing task request of the other AI task is received, and the module will first perform a suspension operation on the new computing task to maintain the computing consistency and efficiency of the current AI task. Once the current AI task is calculated, the computing power interception module will quickly switch the allocated sub-precision level operator to the suspended other AI task and continue to execute its new computing task. This strategy ensures the reasonable allocation of resources between different AI tasks, while avoiding performance losses caused by frequent switching, and achieving smooth transition and efficient execution between AI tasks.

[0086] In one embodiment of the present invention, the computing power interception module is used to:

[0087] Intercepting the function call of the AI ​​task and parsing the function call to obtain the requested operator precision level;

[0088] Determine whether the mixed-precision operator application information of the AI ​​task includes the operator precision level of the function call request;

[0089] If not, reject the function call;

[0090] If so, the execution of the function call is controlled according to the usage of the requested operator precision level and the application amount of the requested operator precision level in the mixed-precision operator application information.

[0091] In an embodiment of the present invention, when an AI task initiates a function call, the computing power interception module will quickly intercept and deeply analyze the call to identify the requested operator precision level, and compare it with the mixed precision operator application information of the AI ​​task to confirm whether it contains the operator precision level requested by the current function call. If the level is not included in the application information, the function call is directly rejected to avoid the AI ​​task from abusing high-precision computing power. On the contrary, if the application information clearly includes the required level, the execution of the function call will be flexibly controlled based on the usage and application amount of the requested operator precision level.

[0092] In one embodiment of the present invention, the computing power interception module is used to:

[0093] Calculate the computing power required to execute the function call;

[0094] Determine whether the sum of the usage of the requested operator precision level and the computing power required for the function call execution exceeds the application amount of the requested operator precision level in the mixed precision operator application information;

[0095] If not, executing the function call;

[0096] If it exceeds, the function call is placed in a queue, other function calls are executed and computing power is released until the remaining available quota of the requested operator precision level is higher than the computing power required for executing the function call, and the function call is executed.

[0097] In an embodiment of the present invention, after determining that the application information contains the required level for function call execution, the computing power interception module can accurately calculate the computing power resources required for the function call, and further calculate the usage of the requested operator precision level and the total computing power required for the function call, and compare the total computing power with the application amount in the mixed precision operator application information. If the total computing power does not exceed the application amount, the module will immediately execute the function call to ensure the smooth progress of the task. However, once the computing power demand exceeds the application amount, the module will temporarily put the function call into the queue, and instead execute other function calls with lower computing power requirements, and continuously release computing power resources during the execution process. When the remaining available amount of the requested operator precision level is higher than the computing power required for the current function call, the module will immediately execute the function call to ensure that the task can be completed efficiently and orderly. This dynamic adjustment strategy not only effectively avoids the waste of computing power resources, but also significantly improves the execution efficiency and flexibility of AI tasks.

[0098] Reference Figure 2 , which is the static precision scheduling process of the mixed precision computing power scheduling system of the present invention.

[0099] 1. Deploy an operator device plug-in in each node. The plug-in monitors the capacity and usage of operators of different precisions on each accelerator card on the node in real time, reports it to the mixed precision scheduler, and stores it in the mixed precision resource pool.

[0100] 2. When users submit AI tasks, they can declare the number of mixed-precision operators required. The scheduler will make allocations based on the capacity information of the resource pool, with the operator with the highest required precision being the main scheduling consideration. For example, in the figure above, FP32 is the most used operator, so it will be scheduled to the node where the FP32 operator meets the requirements. The remaining operators with secondary precision will be scheduled on the same machine first. If the machine does not have enough operators, they will be scheduled to other available nodes.

[0101] 3. The AI ​​task calls the Library function library of the accelerator card for calculation. The computing power interception module intercepts the call process and forwards it according to the operator topology assigned by the scheduler, calling the calculation task of the corresponding precision of the local operation to the local accelerator card, such as FP32 in the above figure.

[0102] 4. The computing power interception module will intercept the function tasks of corresponding precision scheduled on other nodes and forward them to the corresponding machines through the RDMA network.

[0103] Reference Figure 3 , which is the dynamic precision scheduling process of the mixed precision computing power scheduling system of the present invention.

[0104] 1. AI task A calls the library function library of the accelerator card for calculation. The computing power interception module intercepts the call process and forwards it according to the operator topology assigned by the scheduler, calling the calculation task of the corresponding precision of the local operation to the local accelerator card, such as FP32 in the above figure.

[0105] 2. For operators with secondary precision, the scheduler pre-schedules them to other nodes. However, in order to achieve higher computing performance, the computing power interception module can first query the local operator device plug-in to determine whether the local machine currently has idle operators with corresponding precision, such as the FP16 operator in the above figure.

[0106] 3. If the corresponding precision operator of the local machine is currently idle, the computing power interception module will calculate the task directly on the local machine. At this time, if the AI ​​task B that should have allocated this part of the computing power also issues a computing task, the computing power interception module will suspend the function calculation, wait for the function calculation of task A to be completed, and then switch the computing power back to task B.

[0107] 4. If the corresponding precision operator of the local machine is currently busy, the computing power interception module will call the accelerator card of the corresponding host to perform calculations through RDMA remote calls.

[0108] The precision level operator authorization interception process of the mixed precision computing power scheduling system of the present invention is as follows:

[0109] 1. The computing power interception module intercepts the function computing requests of AI tasks to the accelerator card.

[0110] 2. The computing power interception module parses the precision calculation request and determines whether the task has applied for an operator of corresponding precision.

[0111] 3. If the task does not apply for an operator of this precision, the computing power interception module rejects the function call to prevent AI tasks from abusing high-precision computing power.

[0112] 4. If the task holds the quota for the precision operator, the function call is released, but the computing power occupied by the function call must be calculated to determine whether the usage of the corresponding precision operator used by the task exceeds the amount it has applied for. If it exceeds, the call is blocked, and the calculation of the parallel function is forced to queue up and wait for other function tasks to complete the call before it is released.

[0113] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to load a mixed-precision computing power scheduling system as described in any one of the embodiments of the present invention.

[0114] Specifically, the electronic device includes: a memory and a processor, the memory and the processor are connected via a bus communication, a computer program is stored in the memory, and the computer program can be run on the processor, thereby loading the mixed precision computing power scheduling system described in any one of the first aspects of the embodiments of the present invention.

[0115] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0116] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0117] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, it loads a mixed-precision computing power scheduling system as described in any one of the first aspects of the embodiments of the present invention.

[0118] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0119] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, electronic devices, storage media or computer program products. Therefore, embodiments of the present invention may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Moreover, embodiments of the present invention may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CDROM, optical storage, etc.) containing computer-usable program codes.

[0120] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0121] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0122] The above is a detailed introduction to a mixed-precision computing power scheduling system, electronic device and readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application; at the same time, for those of ordinary skill in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application. The above embodiments are only preferred embodiments cited to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art on the basis of the present invention are all within the scope of protection of the present invention.

Claims

1. A mixed precision computing power scheduling system, characterized in that: include: The mixed-precision resource pool is used to record the distribution topology, capacity information, and usage information of multiple-precision operators of each accelerator card in the computing node cluster; The computing device plug-in is deployed on each computing node to monitor the capacity and usage information of multiple precision operators of each accelerator card on the computing node in real time, and report it to the mixed precision scheduler for storage in the mixed precision resource pool; A mixed precision scheduler is used to receive mixed precision operator application information of AI tasks submitted by users, and obtain the real-time capacity information of multiple precision operators of each accelerator card of each computing node from the mixed precision resource pool, and allocate the operator topology of the AI ​​task according to the mixed precision operator application information and the real-time capacity information of multiple precision operators of each accelerator card of each computing node; the mixed precision operator information includes the mixed precision operator level and the number of mixed precision operators; The computing power interception module is deployed on each computing node and is used to intercept the function call of the AI ​​task. According to the operator topology of the AI ​​task, or according to the operator topology of the AI ​​task and the real-time usage information of multiple precision operators of each accelerator card of each computing node, the function call of the AI ​​task is matched with the computing power quota of operators with different precisions to determine the target accelerator card, and the function call of the AI ​​task is forwarded to the target accelerator card.

2. The mixed-precision computing power scheduling system according to claim 1, characterized in that: The mixed precision scheduler is used to: Determine the main precision level operator according to the mixed precision operator application information; the main precision level operator is the precision level operator with the largest number of applications; the precision level operators other than the main precision level operator in the mixed precision operator application information are sub-precision level operators; According to the real-time capacity information of various precision operators of each computing node and each accelerator card, find and allocate computing nodes that meet the number of operators requested at the main precision level; Allocate a secondary precision level operator on the same computing node as the allocated primary precision level operator to the AI ​​task; If the number of sub-precision level operators on the same computing node as the allocated main precision level operator is lower than the number of sub-precision level operators applied for, the difference in the number of sub-precision level operators is calculated, and a computing node with a number of sub-precision level operators that meets the difference is found and allocated to the AI ​​task.

3. The mixed-precision computing power scheduling system according to claim 2, characterized in that: The mixed precision scheduler is used to: If the computing node where the user submits the AI ​​task is a computing node that meets the number of operators requested for the main precision level, the computing node where the user submits the AI ​​task will be preferentially allocated to the AI ​​task.

4. The mixed-precision computing power scheduling system according to claim 1, characterized in that: The mixed precision scheduler is used to: Determine the main precision level operator according to the mixed precision operator application information; the main precision level operator is the precision level operator with the largest number of applications; the precision level operators other than the main precision level operator in the mixed precision operator application information are sub-precision level operators; According to the real-time capacity information of various precision operators of each computing node and each accelerator card, find and allocate computing nodes that meet the number of operators requested at the main precision level; Allocate the secondary precision level operators of the computing nodes other than the computing nodes where the allocated primary precision level operators are located to the AI ​​task.

5. The mixed-precision computing power scheduling system according to claim 2, 3 or 4, characterized in that: The computing power interception module is used to: Send a query request for idle secondary precision level operators to the computing power device plug-in on the computing node where the allocated primary precision level operator is located, and receive the query result of idle secondary precision level operators; If the idle secondary precision level operator query result is that the secondary precision level operator on the computing node where the allocated primary precision level operator is located is currently in an idle state, the secondary precision level operator on the computing node where the allocated primary precision level operator is located is allocated to the AI ​​task; If the query result of the idle secondary precision level operator is that the secondary precision level operator on the computing node where the allocated primary precision level operator is located is currently busy, then the secondary precision level operators of the computing nodes other than the computing node where the allocated primary precision level operator is located are allocated to the AI ​​task.

6. The mixed-precision computing power scheduling system according to claim 5, characterized in that: The computing power interception module is used to: If the secondary precision level operator on the computing node where the allocated primary precision level operator is located has been allocated to another AI task, then during the processing of the calculation of the AI ​​task, when a new calculation task of the other AI task is received, a suspend operation is performed on the new calculation task of the other AI task, and after the calculation of the AI ​​task is completed, the secondary precision level operator on the computing node where the allocated primary precision level operator is located is switched to the other AI task.

7. The mixed-precision computing power scheduling system according to claim 1, characterized in that: The computing power interception module is used to: Intercepting the function call of the AI ​​task and parsing the function call to obtain the requested operator precision level; Determine whether the mixed-precision operator application information of the AI ​​task includes the operator precision level of the function call request; If not, reject the function call; If so, the execution of the function call is controlled according to the usage of the requested operator precision level and the application amount of the requested operator precision level in the mixed-precision operator application information.

8. The mixed-precision computing power scheduling system according to claim 7, characterized in that: The computing power interception module is used to: Calculate the computing power required to execute the function call; Determine whether the sum of the usage of the requested operator precision level and the computing power required for the function call execution exceeds the application amount of the requested operator precision level in the mixed precision operator application information; If not, executing the function call; If it exceeds, the function call is placed in a queue, other function calls are executed and computing power is released until the remaining available quota of the requested operator precision level is higher than the computing power required for the function call execution, and the function call is executed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the electronic device executes the computer program, it loads the mixed-precision computing power scheduling system as described in any one of claims 1 to 8.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the mixed-precision computing power scheduling system as described in any one of claims 1 to 8 is loaded.

Images