A management system and method for intelligent computing resources

Abstract

The present invention discloses a management system and method for intelligent computing power resources, including a virtual computing power card HAL, a computing power management unit, and a computing power resource application and release unit. The virtual computing power card HAL is used to abstract the computing power of all GPU / FPGA / NPU physical computing power cards to form a computing power resource pool with computing power aggregation of multiple computing power cards; the computing power management unit is used to implement data resource management of the computing power resource pool in units of 1TFLOPS; the computing power resource application and release unit provides an API interface for different applications to implement dynamic application, dynamic release, and dynamic migration of computing power resources to reduce the number of application context switching times. The present invention implements the collaborative work and computing power aggregation of multiple computing power cards such as GPU / NPU / FPGA, thereby forming a larger computing power resource pool. Taking 1TFLOPS as the minimum unit of computing power management, the fragmentation of computing power resources is reduced, and the dynamic application and release of computing power resources are also implemented, thereby improving the efficiency of computing power cards.

Description

Technical Field

[0001] The present invention relates to the field of intelligent computing resource integration and configuration, and specifically to a management system and method for intelligent computing resources. Background Art

[0002] At present, the world is in an era of artificial intelligence, and the most important core element in the era of artificial intelligence is computing power. At the 2023 Huawei All-Connect Conference, it was proposed that computing power is the core driving force for the development of artificial intelligence. The scarcity and high cost of computing power have become the core factors restricting the development of AI. Computing power is intuitively understood as computing power. Computing power has been regarded as a quantifiable physical resource, similar to resources such as electricity and oil. The unit of measurement for electricity is degrees, the unit of measurement for oil is tons, and the unit of measurement for computing power is FLOPS (the number of floating-point operations per second). EFLOPS is equal to one hundred kilobytes (=10^18) floating-point operations per second. According to IDC data, it is estimated that the scale of my country's intelligent computing power will reach 1117EFLOPS in 2027.

[0003] The existing physical computing power is mainly divided into two categories: general computing power and intelligent computing power. General computing power is mainly provided by CPU servers, and intelligent computing power is mainly provided by computing power servers constructed by GPU / NPU / FPGA and other devices. The computing power mentioned in this invention only involves the management of intelligent computing power, not general computing power. The computing power server composed of GPU / NPU / FPGA and other devices, although the total amount of physical computing power is large, currently there are mainly the following three major problems: First, the problem of computing power aggregation of multiple computing power cards such as GPU / NPU / FPGA and multiple computing power cards working together. The hardware providers of GPU / NPU / FPGA and other hardware providers are different, and the software environments are also different. These factors make it difficult for physical computing power to form a combined force, and it cannot achieve 1+1=2. Multiple cards and multiple cards cannot work together to form computing power aggregation, so they cannot be used in application scenarios that require large-scale computing power such as AI model training / AI reasoning. The second is the problem of computing power resource fragmentation, which is mainly for application scenarios with continuous demand for computing power resources but low total computing power requirements. In this case, the computing power card cannot work at full capacity, and the use efficiency is very low, resulting in computing power waste. The third is the problem of dynamic application and release of computing power resources, which is mainly aimed at application scenarios where there is no continuous demand for computing power resources. In this case, the computing power card only works at full capacity for a period of time, and the remaining time the computing power card is in an idle state, which will also cause a lot of computing power waste.

[0004] In response to the many problems in the field of intelligent computing power mentioned above, the prior art discloses an invention patent with a publication number of CN111338756A and a patent name of "GPU pooling method, device, equipment and computer-readable storage medium", which provides a method of connecting multiple GPUs through a PCIe switch to form a GPU resource pool and manage GPU resources. This solution has two main disadvantages. First, the resource management of this solution is based on the entire GPU as a unit, and the management granularity is too large, which will cause computing power fragmentation. Second, this solution can only manage GPUs, but not components such as NPU / FPGA. In addition, the prior art also discloses an invention patent with a publication number of CN118132267A and a patent name of "A server GPU computing power integration, configuration, allocation system and method", which provides a computing power integration configuration based on a GPU server. This method has the following disadvantages: As the number of parameters of large AI models increases, there is no way to adapt to the need for computing power cards between multiple servers to work together, and there is no way to achieve the management of NPU / FPGA computing power resources.

[0005] In addition, for GPU resource pooling solutions, most of them manage resources in the form of GPU or SRIOV, binding GPU / vGPU to a specific virtual machine or container, so that the virtual machine will have permanent access to the GPU. When the virtual machine does not need to use the GPU, other virtual machines cannot use the GPU bound to other virtual machines, resulting in a waste of computing resources.

[0006] Based on this, the present invention is proposed. Summary of the invention

[0007] In order to solve the problems existing in the prior art, the present invention provides a management system and method for intelligent computing resources, which realizes the collaborative work and computing power aggregation of multiple computing cards such as GPU / NPU / FPGA, thereby forming a larger computing resource pool. 1TFLOPS is used as the minimum unit of computing power management, thereby reducing the fragmentation of computing power resources, and realizing the dynamic application and release of computing power resources, thereby improving the efficiency of computing power cards.

[0008] The technical solution adopted by the present invention to solve the above technical problems is as follows:

[0009] In a first aspect, the present invention provides a management system for intelligent computing resources, including a virtual computing card HAL, a computing power management unit, and a computing power resource application and release unit;

[0010] The virtual computing power card HAL is used to abstract the computing power of all GPU / FPGA / NPU physical computing power cards, using the Pytroch / TensorFlow application framework as the architectural perspective, without distinguishing which computing power card provides the physical computing power of GPU / NPU / FPGA, so as to form a computing power resource pool that aggregates the computing power of multiple physical computing power cards;

[0011] The computing power management unit is used to manage data resources of the aggregated computing power resource pool in units of 1 TFLOPS;

[0012] The computing power resource application and release unit is used to provide an API interface for different AI applications, including a computing power resource application module and a computing power resource release module, to realize dynamic application, dynamic release, and dynamic migration of computing power resources. Dynamic migration enables the computing power blocks allocated to the same application to be set continuously to reduce the number of application context switching times.

[0013] As a further optimized technical solution of the present invention, the computing power management unit includes 4 computing power nodes, and the 4 computing power nodes all include different types and numbers of GPU / FPGA / NPU physical computing power cards.

[0014] As a further optimized technical solution of the present invention, the four computing power nodes of the computing power management unit are respectively defined as computing power node 0, computing power node 1, computing power node 2 and computing power node 3, and the total computing power is 200T; wherein, computing power node 0 includes two GPUs with a total GPU computing power value of 18T, one NPU with a total NPU computing power value of 20T and one FPGA with a total FPGA computing power value of 25T, computing power node 1 includes two GPUs with a total GPU computing power value of 20T and two NPUs with a total NPU computing power value of 36T, computing power node 2 includes three FPGAs with a total FPGA computing power value of 23T and one NPU with a total NPU computing power value of 3T, and computing power node 3 includes two GPUs with a total GPU computing power value of 28T, one NPU with a total NPU computing power value of 15T and one FPGA with a total FPGA computing power value of 12T.

[0015] As a further optimized technical solution of the present invention, the computing power resource pool uses 1T computing power block as the unit of computing power resource management and assigns a unique computing power block number. The computing power block numbers inside each computing power node are set continuously. At the same time, the computing power blocks inside each computing power node are divided into three computing power areas, namely GPU computing power area, FPGA computing power area and NPU computing power area.

[0016] In a second aspect, the present invention further provides a method for managing intelligent computing resources, which is implemented by the management system of the intelligent computing resources and includes the following implementation steps:

[0017] Step 1: Evaluate the computing power based on the AI ​​model selected by the user to obtain the computing power evaluation value required for the AI ​​application;

[0018] Step 2: The computing power allocation module obtains the total computing power value of the computing power resource pool and the current idle computing power value from the computing power management unit, and the computing power management unit performs data resource management;

[0019] Step 3: If the computing power evaluation value is greater than the total computing power value of the computing power resource pool, the AI ​​application will exit directly and cannot run on the current computing power resource pool; if the computing power evaluation value is less than the total computing power value of the computing power resource pool and greater than the idle computing power value, then the priority of the current AI application will be used to determine whether computing power preemption has been reached. If computing power preemption has not been reached, the AI ​​application will exit. If computing power preemption has been reached, the computing power management unit will release the computing power resources currently occupied by other applications and suspend the corresponding application until it is resumed when computing power resources are sufficient;

[0020] Step 4: After the computing power management unit releases the computing power resources occupied by the current AI application, it obtains sufficient computing power resources through computing power preemption and creates a virtual computing power card HAL with sufficient computing power value;

[0021] Step 5: The virtual computing power card HAL determines which physical computing power card in which computing power node the computing power block belongs to based on the number of each computing power block in the computing power resources allocated by the computing power allocation module;

[0022] Step 6: AI applications will simultaneously drive GPU / NPU / FPU physical computing power cards through the Datadu virtual computing power card, working together to achieve computing power aggregation and complete the work tasks of AI applications;

[0023] Step 7: After the AI ​​application ends, the SD-HV virtual computing power card will call the computing power release interface in the computing power allocation unit to return the computing power block to the computing power resource pool;

[0024] Step 8: The computing resource pool now has enough idle computing resources to activate the low-priority applications that were suspended in step 3 to continue running.

[0025] As a further optimized technical solution of the present invention, the computing power evaluation value required by the AI ​​application is obtained according to the computing power evaluation, and a virtual computing power card HAL with a computing power of 50T is created, including the following steps:

[0026] Step 1: Call the computing power allocation interface in the computing power allocation unit and allocate 50T computing power. If the computing power allocation fails, exit. If the computing power allocation succeeds, proceed to the next step.

[0027] Step 2: Determine the coordinates of the computing block according to the number of the computing block, that is, which computing node and computing area the computing block belongs to; when 50T computing power is provided entirely by computing node 0 and the computing blocks are numbered [1-50], then the computing blocks [1-18] of the virtual computing card HAL are provided by the GPU, the computing blocks [19-38] are provided by the NPU, and the computing blocks [39-50] are provided by the FPGA;

[0028] Step 3: Based on the coordinate data of the computing power block and the Pytroch application framework, the virtual computing power card HAL will simultaneously activate the three computing power areas of CUDA / NPU-SDK / FPGA-SDK for scheduling, so that they can work in coordination and form computing power aggregation;

[0029] Step 4: After the virtual computing power card HAL is created, when starting the Pytroch application, specify the acceleration device as the data transfer virtual computing power card.

[0030] As a further optimized technical solution of the present invention, the computing power management unit performs the following steps to manage data resources of the computing power resource pool in units of 1 TFLOPS:

[0031] Setp1: After each computing power node is started, it registers the relevant information of the computing power node with the computing power management unit, including which computing power cards exist and the computing power value of each computing power card;

[0032] Step 2: The computing power management unit assigns a computing power block number to each computing power node according to the number of the computing power node and the type of computing power card, and divides the computing power area under the computing power node;

[0033] Step 3: The computing power management unit manages in real time how many idle computing power blocks there are in the current computing power resource pool, the computing power numbers of the idle computing power blocks, and whether each 1T computing power block is currently idle. If it is not idle, it also includes the priority information of the application currently using the computing power block;

[0034] Step 4: When the computing power management unit receives a computing power allocation request from the computing power resource application and release module, it first checks whether the current idle computing power value meets the demand. If it does, it allocates it directly. If not, it determines whether the priority of the AI ​​application reaches computing power preemption. If it reaches computing power preemption, it releases the computing power resources occupied by low-priority tasks.

[0035] As a further optimized technical solution of the present invention, the process steps of the computing power resource application module for applying for computing power allocation are as follows:

[0036] Setp1: Call the computing resource application module and send the request to the computing power management unit. The computing power management unit allocates the corresponding computing power block from the idle computing power resources and returns the corresponding computing power block number to the computing resource application module;

[0037] Step 2: After the computing power resource application module obtains computing power resources with sufficient computing power, it starts to build the Data Transfer virtual computing power card.

[0038] As a further optimized technical solution of the present invention, the process steps of the computing power resource release module for releasing computing power resources are as follows:

[0039] Step 1: The computing power resource release module releases the computing power of the corresponding computing power block according to the input computing power block number information;

[0040] Step 2: After the computing power management unit releases the computing power corresponding to the computing power block number, it finds the computing power node and computing power area to which the computing power block belongs according to the computing power block number, and inserts it into the idle computing power queue;

[0041] Step 3: Migrate idle computing resources to reduce the loss caused by context switching between AI applications.

[0042] Step 4: After the computing power migration is completed, the computing power block numbers in the computing power idle area are set continuously to reduce the number of switching times of the computing power card application context.

[0043] As a further optimized technical solution of the present invention, if the computing power of the computing power resource pool is 10T, at a certain time T1, APP1 allocates [1~3] computing power blocks, APP2 allocates [4~6] computing power blocks, APP3 allocates [7~8] computing power blocks, and [9~10] computing power blocks are idle; at time T2, APP1 exits and releases [1~3] computing power blocks. At this time, the layout of the computing power resource pool is idle computing power: [1~3], [9~10] computing power blocks, load computing power: APP2 [4~6] computing power blocks, APP3 .... [7~8] computing power blocks. When a new APP4 is added, the computing power management unit performs computing power migration. At this time, the layout of the computing power resource pool is idle computing power:

[10] computing power blocks, load computing power: APP2 [1~3] computing power blocks, APP3 [4~5] computing power blocks, APP4 [6~9] computing power blocks. The computing power scheduling process is as follows: APP2 [1~3] computing power blocks -> APP3 [4~5] computing power blocks -> APP4 [6~9] computing power blocks ->

[10] computing power blocks are idle, then the scheduling of APP4 is reduced once.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The management system of intelligent computing resources of the present invention builds a HAL computing card hardware abstraction layer by constructing a virtual computing card in an AI framework such as Pytroch / Tensorflow, thereby realizing the aggregation of GPU / NPU / FPGA physical computing power, and at the same time, the aggregated computing power is managed with 1 TFLOPS (one trillion (=10^12) floating-point operations per second) as the minimum unit, reducing the fragmentation of computing power, and realizing the dynamic acquisition and release of computing power, so that the physical computing power of computing cards such as GPU / NPU / FPGA can be fully utilized, thereby improving the efficiency of computing cards.

[0046] 2. The virtual computing power card HAL of the present invention abstracts the computing power of all physical computing power cards such as GPU / FPGA / NPU. From the perspective of application frameworks such as Pytroch / TensorFlow, only the computing power card is seen, without distinguishing which computing power card provides the physical computing power, GPU / NPU / FPGA. Therefore, the computing power aggregation of multiple computing power cards can be achieved. The computing power management unit manages data resources for the aggregated computing power resource pool in units of 1TFLOPS. The computing power resource application and release unit provides an API interface for different AI applications to realize dynamic application, dynamic release, and dynamic migration of computing power resources. Dynamic migration enables the computing power blocks allocated to the same application to be set continuously to reduce the number of application context switching times, thereby improving the efficiency of the computing power card. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is a schematic diagram of the composition of the intelligent computing power resource management system in the present invention.

[0049] Figure 2 This is a working principle diagram of the intelligent computing power resource management system in the present invention.

[0050] Figure 3 This is a comparison chart of the Pytorch framework that adds the data transfer virtual computing power card in the present invention.

[0051] Figure 4 This is a programming logic diagram for managing computing power of a single computing power node in the present invention in units of 1 TFLOPS.

[0052] Figure 5This is a programming logic diagram for managing computing power of multiple computing nodes in units of 1 TFLOPS in the present invention.

[0053] Figure 6 This is a schematic diagram of the power area division principle of the power block in the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] The following abbreviations and key terms are defined as follows:

[0056] GPU: Graphics Processing Unit, a microprocessor that specializes in image and graphics related computing on personal computers, workstations, game consoles and some mobile devices (such as tablets, smartphones, etc.);

[0057] NPU: Neural Network Processing Unit, a processor designed specifically for processing data for neural networks and machine learning;

[0058] FPGA: Field Programmable Gate Array, is a field programmable gate array;

[0059] HAL: Hardware Abstraction Layer, is the hardware abstraction layer;

[0060] CPU: Central Processing Unit, as the computing and control core of the computer system, is the final execution unit for information processing and program running;

[0061] Pytorch: A machine learning library based on the Torch library, mainly used for applications such as computer vision and natural language processing;

[0062] Tensorflow: an open source, end-to-end machine learning platform with comprehensive and flexible tools, libraries, and community resources;

[0063] TFLOPS: Tera Floating-point Operations Per Second, one trillion (=10^12) floating-point operations per second;

[0064] CUDA: A parallel computing platform and programming model developed by NVIDIA that allows developers to directly access the virtual instruction set and parallel computing elements of the GPU, mainly used to create high-performance GPU-accelerated applications;

[0065] SDK: Software Development Kit, full name software development kit, is a set of tools provided by the manufacturer to help developers build applications on a specific platform. These tools usually include compilers, debugging tools, program libraries, etc.

[0066] LibTorch - C++ library for PyTorch.

[0067] NPU-SDK: software development kit provided by NPU manufacturers;

[0068] FPGA-SDK: Software development kit provided by FPGA manufacturers. Example 1

[0069] like Figure 1 and Figure 2 As shown, this embodiment discloses a management system for intelligent computing power resources, including a virtual computing power card HAL, a computing power management unit, and a computing power resource application and release unit;

[0070] The virtual computing power card HAL abstracts the computing power of all GPU / FPGA / NPU physical computing power cards, takes the Pytroch / TensorFlow application framework as the architectural perspective, and does not distinguish which computing power card provides the physical computing power of GPU / NPU / FPGA, so as to form a computing power resource pool that aggregates the computing power of multiple physical computing power cards. The computing power management unit manages data resources in the aggregated computing power resource pool in units of 1TFLOPS. The computing power resource pool uses 1T computing power block as the unit of computing power resource management and assigns a unique computing power block number. The computing power block number inside each computing power node is set continuously, and the computing power blocks inside each computing power node are divided into three computing power areas, namely GPU computing power area, FPGA computing power area and NPU computing power area.

[0071] The computing resource application and release unit provides API interfaces for different AI applications, including a computing resource application module and a computing resource release module, to realize dynamic application, dynamic release, and dynamic migration of computing resources. Dynamic migration enables the computing power blocks allocated to the same application to be set continuously to reduce the number of application context switching times.

[0072] The computing power management unit contains 4 computing power nodes, and all of them contain different types and numbers of GPU / FPGA / NPU physical computing power cards. The 4 computing power nodes of the computing power management unit are defined as computing power node 0, computing power node 1, computing power node 2, and computing power node 3, with a total computing power of 200T; among them, computing power node 0 contains two GPUs with a total GPU computing power value of 18T, one NPU with a total NPU computing power value of 20T, and one FPGA with a total FPGA computing power value of 25T. Computing power node 1 contains two GPUs with a total GPU computing power value of 20T and two NPUs with a total NPU computing power value of 36T. Computing power node 2 contains three FPGAs with a total FPGA computing power value of 23T and one NPU with a total NPU computing power value of 3T. Computing power node 3 contains two GPUs with a total GPU computing power value of 28T, one NPU with a total NPU computing power value of 15T, and one FPGA with a total FPGA computing power value of 12T. As shown in the following table:

[0073] Example 2

[0074] like Figures 2 to 6 As shown, this embodiment discloses a management method for intelligent computing resources, which is implemented by the management system of the intelligent computing resources disclosed in Embodiment 1, and is implemented in the following steps in sequence:

[0075] 1. Evaluate the computing power based on the AI ​​model selected by the user to obtain the computing power evaluation value required by the AI ​​application. Based on the computing power evaluation, the computing power evaluation value required by the AI ​​application is obtained, and a virtual computing power card HAL with a computing power of 50T is created, including the following steps:

[0076] Step 1: Call the computing power allocation interface in the computing power allocation unit and allocate 50T computing power. If the computing power allocation fails, exit. If the computing power allocation succeeds, proceed to the next step.

[0077] Step 2: Determine the coordinates of the computing block according to the number of the computing block, that is, which computing node and computing area the computing block belongs to; when 50T computing power is provided entirely by computing node 0 and the computing blocks are numbered [1-50], then the computing blocks [1-18] of the virtual computing card HAL are provided by the GPU, the computing blocks [19-38] are provided by the NPU, and the computing blocks [39-50] are provided by the FPGA;

[0078] Step 3: Based on the coordinate data of the computing power block and the Pytroch application framework, the virtual computing power card HAL will simultaneously activate the three computing power areas of CUDA / NPU-SDK / FPGA-SDK for scheduling, so that they can work in coordination and form computing power aggregation;

[0079] Step 4: After the virtual computing power card HAL is created, when starting the Pytroch application, specify the acceleration device as the data transfer virtual computing power card.

[0080] 2. The computing power allocation module obtains the total computing power value of the computing power resource pool and the current idle computing power value from the computing power management unit, and the computing power management unit performs data resource management; the computing power management unit uses 1TFLOPS as a unit, and the working steps of data resource management for the computing power resource pool are as follows:

[0081] Setp1: After each computing power node is started, it registers the relevant information of the computing power node with the computing power management unit, including which computing power cards exist and the computing power value of each computing power card;

[0082] Step 2: The computing power management unit assigns a computing power block number to each computing power node according to the number of the computing power node and the type of computing power card, and divides the computing power area under the computing power node;

[0083] Step 3: The computing power management unit manages in real time how many idle computing power blocks there are in the current computing power resource pool, the computing power numbers of the idle computing power blocks, and whether each 1T computing power block is currently idle. If it is not idle, it also includes the priority information of the application currently using the computing power block;

[0084] Step 4: When the computing power management unit receives a computing power allocation request from the computing power resource application and release module, it first checks whether the current idle computing power value meets the demand. If it does, it allocates it directly. If not, it determines whether the priority of the AI ​​application reaches computing power preemption. If it reaches computing power preemption, it releases the computing power resources occupied by low-priority tasks.

[0085] 3. If the computing power evaluation value is greater than the total computing power value of the computing power resource pool, the AI ​​application will exit directly and cannot run on the current computing power resource pool; if the computing power evaluation value is less than the total computing power size of the computing power resource pool and greater than the idle computing power value, then the priority of the current AI application will be used to determine whether computing power preemption has been reached. If computing power preemption has not been reached, the AI ​​application will exit. If computing power preemption has been reached, the computing power management unit will release the computing power resources currently occupied by other applications and suspend the corresponding application until its operation is resumed when computing power resources are sufficient.

[0086] 4. After the computing power management unit releases the computing power resources occupied by the current AI application, it obtains sufficient computing power resources through computing power preemption and creates a virtual computing power card HAL with sufficient computing power value;

[0087] 5. The virtual computing power card HAL determines which physical computing power card in which computing power node the computing power block belongs to based on the number of each computing power block in the computing power resources allocated by the computing power allocation module;

[0088] 6. AI applications will simultaneously drive GPU / NPU / FPU physical computing power cards through the Datadu virtual computing power card, work together to achieve computing power aggregation and complete the work tasks of AI applications;

[0089] 7. After the AI ​​application ends, the Shudu virtual computing power card will call the computing power release interface in the computing power allocation unit to return the computing power block to the computing power resource pool;

[0090] 8. The computing resource pool now has enough idle computing resources to activate the low-priority applications that were suspended in step 3 to continue running.

[0091] In this embodiment, the process steps of the computing resource application module for applying for computing power allocation are as follows:

[0092] Setp1: Call the computing resource application module and send the request to the computing power management unit. The computing power management unit allocates the corresponding computing power block from the idle computing power resources and returns the corresponding computing power block number to the computing resource application module;

[0093] Step 2: After the computing power resource application module obtains computing power resources with sufficient computing power, it starts to build the Data Transfer virtual computing power card.

[0094] In addition, the process steps for the computing power resource release module to release computing power resources are as follows:

[0095] Step 1: The computing power resource release module releases the computing power of the corresponding computing power block according to the input computing power block number information;

[0096] Step 2: After the computing power management unit releases the computing power corresponding to the computing power block number, it finds the computing power node and computing power area to which the computing power block belongs according to the computing power block number, and inserts it into the idle computing power queue;

[0097] Step 3: Migrate idle computing resources to reduce the loss caused by context switching between AI applications.

[0098] Step 4: After the computing power migration is completed, the computing power block numbers in the computing power idle area are set continuously to reduce the number of switching times of the computing power card application context.

[0099] Specifically, if the computing power of the computing power resource pool is 10T, at a certain time T1, APP1 is allocated [1~3] computing power blocks, APP2 is allocated [4~6] computing power blocks, APP3 is allocated [7~8] computing power blocks, and [9~10] computing power blocks are idle; at time T2, APP1 exits and releases [1~3] computing power blocks. At this time, the layout of the computing power resource pool is idle computing power: [1~3], [9~10] computing power blocks, load computing power: APP2 [4~6] computing power blocks, APP3 [7~8] computing power blocks When a new APP4 is added, the computing power management unit performs computing power migration. At this time, the layout of the computing power resource pool is idle computing power:

[10] computing power block, load computing power: APP2 [1~3] computing power block, APP3 [4~5] computing power block, APP4 [6~9] computing power block, and the computing power scheduling process is as follows: APP2 [1~3] computing power block -> APP3 [4~5] computing power block -> APP4 [6~9] computing power block ->

[10] computing power block is idle, then the scheduling of APP4 is reduced once.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A management system for intelligent computing resources, characterized in that: It includes virtual computing power card HAL, computing power management unit, computing power resource application and release unit; The virtual computing power card HAL is used to abstract the computing power of all GPU / FPGA / NPU physical computing power cards, using the Pytroch / TensorFlow application framework as the architectural perspective, without distinguishing which computing power card provides the physical computing power of GPU / NPU / FPGA, so as to form a computing power resource pool that aggregates the computing power of multiple physical computing power cards; The computing power management unit is used to manage data resources of the aggregated computing power resource pool in units of 1 TFLOPS; The computing resource application and release unit is used to provide an API interface for different AI applications, including a computing resource application module and a computing resource release module, to realize dynamic application, dynamic release, and dynamic migration of computing resources. Dynamic migration enables the computing blocks allocated to the same application to be set continuously to reduce the number of application context switching times; The computing power management unit includes 4 computing power nodes, and each of the 4 computing power nodes includes different types and numbers of GPU / FPGA / NPU physical computing power cards; The computing resource pool uses 1T computing block as the unit of computing resource management and assigns a unique computing block number. The computing block numbers inside each computing node are set continuously. At the same time, the computing blocks inside each computing node are divided into three computing areas, namely GPU computing area, FPGA computing area and NPU computing area.

2. The management system for intelligent computing resources according to claim 1, characterized in that: The four computing nodes of the computing power management unit are defined as computing power node 0, computing power node 1, computing power node 2 and computing power node 3, and the total computing power is 200T; wherein, computing power node 0 includes two GPUs with a total GPU computing power value of 18T, one NPU with a total NPU computing power value of 20T and one FPGA with a total FPGA computing power value of 25T, computing power node 1 includes two GPUs with a total GPU computing power value of 20T and two NPUs with a total NPU computing power value of 36T, computing power node 2 includes three FPGAs with a total FPGA computing power value of 23T and one NPU with a total NPU computing power value of 3T, and computing power node 3 includes two GPUs with a total GPU computing power value of 28T, one NPU with a total NPU computing power value of 15T and one FPGA with a total FPGA computing power value of 12T.

3. A method for managing intelligent computing resources, characterized in that: The management method is implemented by the management system of the intelligent computing resources according to claim 1 or 2, and includes the following implementation steps: Step 1: Evaluate the computing power based on the AI ​​model selected by the user to obtain the computing power evaluation value required for the AI ​​application; Step 2: The management system of the intelligent computing power resources further includes a computing power allocation module, which obtains the total computing power value of the computing power resource pool and the current idle computing power value from the computing power management unit, and the computing power management unit performs data resource management; Step 3: If the computing power evaluation value is greater than the total computing power value of the computing power resource pool, the AI ​​application will exit directly and cannot run on the current computing power resource pool; if the computing power evaluation value is less than the total computing power value of the computing power resource pool and greater than the idle computing power value, then the priority of the current AI application will be used to determine whether computing power preemption has been reached. If computing power preemption has not been reached, the AI ​​application will exit. If computing power preemption has been reached, the computing power management unit will release the computing power resources currently occupied by other applications and suspend the corresponding application until it is resumed when computing power resources are sufficient; Step 4: After the computing power management unit releases the computing power resources occupied by the current AI application, it obtains sufficient computing power resources through computing power preemption and creates a virtual computing power card HAL with sufficient computing power value; Step 5: The virtual computing power card HAL determines which physical computing power card in which computing power node the computing power block belongs to based on the number of each computing power block in the computing power resources allocated by the computing power allocation module; Step 6: AI applications will simultaneously drive GPU / NPU / FPU physical computing power cards through the Datadu virtual computing power card, working together to achieve computing power aggregation and complete the work tasks of AI applications; Step 7: After the AI ​​application ends, the SD-HV virtual computing power card will call the computing power release interface in the computing power allocation unit to return the computing power block to the computing power resource pool; Step 8: The computing resource pool now has enough idle computing resources to activate the low-priority applications that were suspended in step 3 to continue running.

4. The method for managing intelligent computing resources according to claim 3, characterized in that: Based on the computing power evaluation, the computing power evaluation value required by the AI ​​application is obtained, and a virtual computing power card HAL with a computing power of 50T is created, including the following steps: Step 1: Call the computing power allocation interface in the computing power allocation unit and allocate 50T computing power. If the computing power allocation fails, exit. If the computing power allocation succeeds, proceed to the next step. Step 2: Determine the coordinates of the computing block according to the number of the computing block, that is, which computing node and computing area the computing block belongs to; when 50T computing power is provided entirely by computing node 0 and the computing blocks are numbered [1-50], then the computing blocks [1-18] of the virtual computing card HAL are provided by the GPU, the computing blocks [19-38] are provided by the NPU, and the computing blocks [39-50] are provided by the FPGA; Step 3: Based on the coordinate data of the computing power block and the Pytroch application framework, the virtual computing power card HAL will simultaneously activate the three computing power areas of CUDA / NPU-SDK / FPGA-SDK for scheduling, so that they can work in coordination and form computing power aggregation; Step 4: After the virtual computing power card HAL is created, when starting the Pytroch application, specify the acceleration device as the data transfer virtual computing power card.

5. The method for managing intelligent computing resources according to claim 3, characterized in that: The computing power management unit uses 1 TFLOPS as a unit, and the working steps of managing data resources for the computing power resource pool are as follows: Setp1: After each computing power node is started, it registers the relevant information of the computing power node with the computing power management unit, including which computing power cards exist and the computing power value of each computing power card; Step 2: The computing power management unit assigns a computing power block number to each computing power node according to the number of the computing power node and the type of computing power card, and divides the computing power area under the computing power node; Step 3: The computing power management unit manages in real time how many idle computing power blocks there are in the current computing power resource pool, the computing power numbers of the idle computing power blocks, and whether each 1T computing power block is currently idle. If it is not idle, it also includes the priority information of the application currently using the computing power block; Step 4: When the computing power management unit receives a computing power allocation request from the computing power resource application and release module, it first checks whether the current idle computing power value meets the demand. If it does, it allocates it directly. If not, it determines whether the priority of the AI ​​application reaches computing power preemption. If it reaches computing power preemption, it releases the computing power resources occupied by low-priority tasks.

6. The method for managing intelligent computing resources according to claim 5, characterized in that: The steps of the computing resource application module for computing resource allocation application are as follows: Setp1: Call the computing resource application module and send the request to the computing power management unit. The computing power management unit allocates the corresponding computing power block from the idle computing power resources and returns the corresponding computing power block number to the computing resource application module; Step 2: After the computing power resource application module obtains computing power resources with sufficient computing power, it starts to build the Data Transfer virtual computing power card.

7. The method for managing intelligent computing resources according to claim 5, characterized in that: The steps of the computing resource release module to release computing resources are as follows: Step 1: The computing power resource release module releases the computing power of the corresponding computing power block according to the input computing power block number information; Step 2: After the computing power management unit releases the computing power corresponding to the computing power block number, it finds the computing power node and computing power area to which the computing power block belongs according to the computing power block number, and inserts it into the idle computing power queue; Step 3: Migrate idle computing resources to reduce the loss caused by context switching between AI applications. Step 4: After the computing power migration is completed, the computing power block numbers in the computing power idle area are set continuously to reduce the number of switching times of the computing power card application context.

8. The method for managing intelligent computing resources according to claim 7, characterized in that: If the computing power of the computing power resource pool is 10T, at a certain time T1, APP1 allocates [1~3] computing power blocks, APP2 allocates [4~6] computing power blocks, APP3 allocates [7~8] computing power blocks, and [9~10] computing power blocks are idle; at time T2, APP1 exits and releases [1~3] computing power blocks. At this time, the layout of the computing power resource pool is idle computing power: [1~3], [9~10] computing power blocks, load computing power: APP2 [4~6] computing power blocks, APP3 [7~8] computing power blocks ,When a new APP4 is added, the computing power management unit performs computing power migration. At this time, the layout of the computing power resource pool is idle computing power: [10] computing power block, load computing power: APP2 [1~3] computing power block, APP3 [4~5] computing power block, APP4 [6~9] computing power block, the computing power scheduling process is as follows: APP2 [1~3] computing power block -> APP3 [4~5] computing power block -> APP4 [6~9] computing power block -> [10] computing power block is idle, then the scheduling of APP4 is reduced once.

Images