A computing resource processing method and device

By obtaining resource groups that support MIG mode configuration and configuring MIG solutions according to requirements, the problem of fixed resource configuration methods of AI platform is solved, the utilization rate and training efficiency of computing resources are improved, and more flexible and targeted computing resource allocation is achieved.

CN113722089BActive Publication Date: 2024-11-01LANGCHAO ELECTRONIC INFORMATION IND CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110838080.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-23
Publication Date
2024-11-01
Estimated Expiration
2041-07-23

AI Technical Summary

Technical Problem

The resource allocation method of the AI ​​platform is relatively fixed and cannot support the diverse needs of users, resulting in the inability to independently configure special computing resources such as MIG mode configuration, resulting in low computing resource utilization efficiency.

Method used

By obtaining the resource group of target computing resources that support MIG mode configuration, determining preset configuration rules based on target needs, and using different combination methods to combine the MIG mode to configure the target computing resources to enhance the special computing resource configuration of the AI ​​platform Flexibility and targeting to meet different training needs.

Benefits of technology

It improves the utilization rate and training efficiency of computing resources, enhances the flexibility and pertinence of computing resource allocation of AI platform, and can better meet different training needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113722089B_ABST
    Figure CN113722089B_ABST
Patent Text Reader

Abstract

The present application discloses a computing resource processing method and device, which is applied to an AI platform, including: obtaining a resource group including a target computing resource supporting MIG mode configuration to obtain a MIG resource group; determining a corresponding preset configuration rule according to the target demand, and configuring the target computing resource in the MIG resource group using the corresponding MIG scheme according to the preset configuration rule to obtain a configured MIG instance; wherein the preset configuration rule characterizes the similarities and differences between the MIG schemes for configuring the target computing resource in the MIG resource group; the MIG scheme includes different schemes after combining the MIG mode in different combinations; determining a preset number of target MIG instances from all MIG instances, so as to train the tasks to be trained of the AI ​​platform using the target MIG instances. The present application can enhance the flexibility and pertinence of the special computing resource configuration of the AI ​​platform to meet different training needs, thereby improving the utilization of computing resources.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a computing resource processing method and device. Background Art

[0002] Abundant computing resources are a strong support for AI platform task training. AI platform needs to configure resource groups for model training. After users are associated with resource groups, they can use the resources under the resource groups to submit training tasks. The success of resource group configuration is directly related to whether the AI ​​platform can submit training tasks normally. At present, the resource configuration method of AI platform is relatively fixed and does not support the diverse needs of users. It is impossible to realize autonomous configuration for some special computing resources, such as computing resources that support MIG mode configuration, resulting in low utilization efficiency of computing resources.

[0003] Therefore, how to provide a computing resource processing method that can achieve high utilization, flexibility and strong pertinence is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a computing resource processing method and apparatus, and also to provide corresponding equipment and storage media, which can enhance the flexibility and pertinence of the special computing resource configuration of the AI ​​platform to meet different training requirements, thereby improving the computing resource utilization. The specific scheme is as follows:

[0005] The first aspect of the present application provides a computing resource processing method, which is applied to an AI platform, including:

[0006] Obtain a resource group including a target computing resource that supports MIG mode configuration to obtain a MIG resource group;

[0007] Determine a corresponding preset configuration rule according to the target demand, and configure the target computing resources in the MIG resource group using the corresponding MIG scheme according to the preset configuration rule to obtain a configured MIG instance; wherein the preset configuration rule represents the similarities and differences between the MIG schemes for configuring the target computing resources in the MIG resource group; the MIG scheme includes different schemes after combining MIG modes in different combinations;

[0008] A preset number of target MIG instances are determined from all the MIG instances, so as to use the target MIG instances to train the tasks to be trained of the AI ​​platform.

[0009] Optionally, obtaining a resource group including a target computing resource that supports MIG mode configuration includes:

[0010] Scanning underlying computing resources, and dividing the scanned underlying computing resources into resource groups of corresponding types according to computing resource types; wherein the resource groups of the same type include the underlying computing resources of the same type;

[0011] A resource group including target computing resources supporting MIG mode configuration is determined from all the resource groups.

[0012] Optionally, determining a corresponding preset configuration rule according to a target requirement, and configuring the target computing resource in the MIG resource group using a corresponding MIG solution according to the preset configuration rule, includes:

[0013] Determine a corresponding first preset configuration rule according to the first target requirement; wherein the first preset configuration rule is to configure all the target computing resources in the MIG resource group to the same MIG solution;

[0014] All the target computing resources in the MIG resource group are configured using the same MIG solution according to the first preset configuration rule.

[0015] Optionally, determining a corresponding preset configuration rule according to a target requirement, and configuring the target computing resource in the MIG resource group using a corresponding MIG solution according to the preset configuration rule, includes:

[0016] Determine a corresponding second preset configuration rule according to the second target requirement; wherein the second preset configuration rule is to configure the target computing resource of each node in the MIG resource group to the same MIG solution, and configure different MIG solutions for the target computing resources of different nodes;

[0017] The target computing resources of the same node in the MIG resource group are configured using the same MIG solution according to the second preset configuration rule.

[0018] Optionally, determining a corresponding preset configuration rule according to a target requirement, and configuring the target computing resource in the MIG resource group using a corresponding MIG solution according to the preset configuration rule, includes:

[0019] Determine a corresponding third preset configuration rule according to the third target requirement; wherein the third preset configuration rule is to configure all the target computing resources in the MIG resource group to all different types of the MIG solutions;

[0020] All the target computing resources in the MIG resource group are configured using all different types of the MIG solutions according to the third preset configuration rule.

[0021] Optionally, determining a corresponding preset configuration rule according to a target requirement, and configuring the target computing resource in the MIG resource group using a corresponding MIG solution according to the preset configuration rule, includes:

[0022] Determine a corresponding fourth preset configuration rule according to the fourth target requirement; wherein the fourth preset configuration rule is to configure the target computing resources of the first preset node in the MIG resource group to the same MIG solution, and configure the target computing resources of the second preset node to different MIG solutions;

[0023] The target computing resources of the same node in the MIG resource group are configured using the same MIG solution according to the fourth preset configuration rule.

[0024] Optionally, after configuring the target computing resource in the MIG resource group using a corresponding MIG solution according to the preset configuration rule, the method further includes:

[0025] Determine whether the configuration is successful. If not, use the same MIG solution to reconfigure the target computing resource that failed to be configured or re-determine a new MIG solution to reconfigure the target computing resource that failed to be configured.

[0026] Optionally, determining a preset number of target MIG instances from all the MIG instances to train the tasks to be trained submitted to the AI ​​platform using the target MIG instances includes:

[0027] A preset number of target MIG instances are determined from all the MIG instances, and idle target computing resources in idle nodes including the target MIG instances are scheduled to train the tasks to be trained.

[0028] Optionally, the computing resource processing method further includes:

[0029] The target computing resource is monitored to obtain corresponding real-time status information, and the real-time status information is synchronized and displayed to the AI ​​platform.

[0030] A second aspect of the present application provides a computing resource processing device, which is applied to an AI platform, including:

[0031] An acquisition module, used for acquiring a resource group including a target computing resource supporting MIG mode configuration, so as to obtain a MIG resource group;

[0032] A configuration module, used to determine a corresponding preset configuration rule according to a target requirement, and configure the target computing resources in the MIG resource group using a corresponding MIG scheme according to the preset configuration rule to obtain a configured MIG instance; wherein the preset configuration rule characterizes the similarities and differences between the MIG schemes for configuring the target computing resources in the MIG resource group; the MIG scheme includes different schemes after combining MIG modes in different combinations;

[0033] A training module, used to determine a preset number of target MIG instances from all the MIG instances, so as to use the target MIG instances to train the training tasks submitted to the AI ​​platform

[0034] A third aspect of the present application provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned computing resource processing method.

[0035] A fourth aspect of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the aforementioned computing resource processing method is implemented.

[0036] In the present application, a resource group including target computing resources supporting MIG mode configuration is first obtained to obtain a MIG resource group; then, the corresponding preset configuration rules are determined according to the target requirements, and the target computing resources in the MIG resource group are configured using the corresponding MIG scheme according to the preset configuration rules to obtain the configured MIG instance; wherein the preset configuration rules characterize the similarities and differences between the MIG schemes for configuring the target computing resources in the MIG resource group; the MIG scheme includes different schemes after combining the MIG mode in different combinations; finally, a preset number of target MIG instances are determined from all the MIG instances to train the tasks to be trained of the AI ​​platform using the target MIG instances. The present application configures the target computing resources in the acquired MIG resource group using MIG schemes corresponding to different preset configuration rules, so as to perform task training through the configured MIG instances, thereby enhancing the flexibility and pertinence of the special computing resource configuration of the AI ​​platform to meet different training requirements, thereby improving the utilization of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 A flow chart of a computing resource processing method provided in this application;

[0039] Figure 2 Different MIG solutions provided for this application;

[0040] Figure 3 A flowchart of a specific computing resource processing method provided for this application;

[0041] Figure 4 A mig set tag provided for this application;

[0042] Figure 5 A mig result tag provided for this application;

[0043] Figure 6 A schematic diagram of a dynamic display of a configured MIG resource group provided for this application;

[0044] Figure 7 A schematic diagram of the structure of a computing resource processing device provided in this application;

[0045] Figure 8 A structural diagram of a computing resource processing electronic device provided in this application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Abundant computing resources are a strong support for AI platforms to conduct task training. For model training, AI platforms need to configure resource groups. After users are associated with resource groups, they can use the resources under the resource groups to submit training tasks. The success or failure of resource group configuration is directly related to whether the AI ​​platform can submit training tasks normally. At present, the resource configuration method of the AI ​​platform is relatively fixed and does not support the diverse needs of users. It is impossible to achieve autonomous configuration for some special computing resources, such as computing resources that support MIG mode configuration, resulting in low utilization efficiency of computing resources. In response to the above-mentioned technical defects, the present application provides a computing resource processing solution, which is applied to the AI ​​platform. By using the MIG solution corresponding to different preset configuration rules, the target computing resources in the acquired MIG resource group are configured to perform task training through the configured MIG instance, thereby enhancing the flexibility and pertinence of the special computing resource configuration of the AI ​​platform to meet different training needs, thereby improving the utilization of computing resources.

[0048] Figure 1 A flow chart of a computing resource processing method provided in an embodiment of the present application. Figure 1 As shown, the computing resource processing method is applied to the AI ​​platform, including:

[0049] S11: Acquire a resource group including target computing resources supporting MIG mode configuration to obtain a MIG resource group.

[0050] In this embodiment, a resource group including a target computing resource supporting MIG mode configuration is obtained to obtain a MIG resource group. The target resource is an A100 GPU, and the A100 GPU card supports MIG mode configuration, and the MIG resource group is a resource group including an A100 GPU card. Among them, MIG (Multi-Instance GPU) is a multi-instance GPU technology that can divide the A100 GPU into up to 7 instances, each of which is completely isolated from its own high-bandwidth video memory, cache, and computing core, and has a set of hardware resources dedicated to computing, video memory, and cache, which can provide stable and reliable service quality and effective fault isolation for workloads. Users can run multiple training tasks on the GPU card at the same time to accelerate model training and improve GPU resource utilization and training efficiency.

[0051] S12: Determine corresponding preset configuration rules according to target requirements, and configure the target computing resources in the MIG resource group using the corresponding MIG scheme according to the preset configuration rules to obtain a configured MIG instance; wherein the preset configuration rules characterize the differences and similarities between the MIG schemes for configuring the target computing resources in the MIG resource group; the MIG scheme includes different schemes after combining MIG modes in different combinations.

[0052] In this embodiment, the corresponding preset configuration rules are first determined according to the target requirements, and then the target computing resources in the MIG resource group are configured using the corresponding MIG scheme according to the preset configuration rules to obtain the configured MIG instance. The preset configuration rules represent the similarities and differences between the MIG schemes for configuring the target computing resources in the MIG resource group.

[0053] In this embodiment, the preset configuration rules include four types: (1) the first preset configuration rule is to configure all the target computing resources in the MIG resource group as the same MIG scheme; (2) the second preset configuration rule is to configure the target computing resources of each node in the MIG resource group as the same MIG scheme, and the target computing resources of different nodes are configured with different MIG schemes; (3) the third preset configuration rule is to configure all the target computing resources in the MIG resource group as different MIG schemes; (4) the fourth preset configuration rule is to configure the target computing resources of the first preset node in the MIG resource group as the same MIG scheme, and the target computing resources of the second preset node are configured with different MIG schemes. Correspondingly, step S12 can be specifically divided into four types, namely: (1) determining a corresponding first preset configuration rule according to the first target requirement; configuring all the target computing resources in the MIG resource group using the same MIG solution according to the first preset configuration rule; (2) determining a corresponding second preset configuration rule according to the second target requirement; configuring the target computing resources of the same node in the MIG resource group using the same MIG solution according to the second preset configuration rule; (3) determining a corresponding third preset configuration rule according to the third target requirement; configuring all the target computing resources in the MIG resource group using all different types of MIG solutions according to the third preset configuration rule; (4) determining a corresponding fourth preset configuration rule according to the fourth target requirement; configuring the target computing resources of the same node in the MIG resource group using the same MIG solution according to the fourth preset configuration rule.

[0054] In this embodiment, the MIG scheme includes different schemes after combining MIG modes in different combinations. NVIDIA's A100 GPU card has 5 MIG modes (mig-1g.5gb, mig-2g.10gb, mig-3g.20gb, mig-4g.20gb, mig-7g.40gb), and the 5 MIG modes have multiple combination configurations. This embodiment provides 13 MIG schemes obtained by 13 combinations, as shown in the following figure. Figure 2As shown in the figure. vGPUs are virtual GPUs that A100 GPU is divided into according to built-in rules. For example, for the first MIG solution, an A100 GPU card is configured as 7 mig-1g.5gb, and 5gb GPU fragments remain; for the second MIG solution, an A100 GPU card is configured as 1 mig-1g.5gb and 3 mig-2g.10gb, and 5gb GPU fragments remain. By analogy, 13 MIG solutions in Table 2 can be obtained.

[0055] Specifically, the embodiment of the present application provides an A100 MIG configuration that is accurate to the card level. When creating a MIG resource group, a single or multiple A100 nodes can be selected, and the A100 GPU cards of each node can be dynamically displayed according to the selection. The first preset configuration rule is to configure all A100 GPU cards of all nodes in the resource group into the same MIG solution; the second preset configuration rule is accurate to the node level, to configure each node of the resource group into a different MIG solution; the third preset configuration rule is accurate to the card level, to configure each A100 GPU card of each node of the resource group into a different MIG solution; the fourth preset configuration rule is a mixed configuration, which supports some nodes in the same resource group to be configured into the same MIG solution, and some nodes each A100 GPU card is configured into other different MIG solutions. It is not difficult to understand that the configuration complexity and difficulty increase successively. In addition, the above configuration rules can be flexibly combined and adjusted, and at the same time, the MIG mode adjustment at the card level can be implemented for the MIG resource group according to the application scenarios of the business for computing resources, model training conditions and user scale to meet a variety of business scenarios.

[0056] S13: Determine a preset number of target MIG instances from all the MIG instances, and use the target MIG instances to train the tasks to be trained of the AI ​​platform.

[0057] In this embodiment, a preset number of target MIG instances are determined from all the MIG instances, so as to use the target MIG instances to train the tasks to be trained of the AI ​​platform. After the MIG resource group is successfully configured, when submitting the training task, you can select MIG instances with different video memories and the number of MIGs according to the training situation, submit the training task and conduct the training.

[0058] It can be seen that the embodiment of the present application first obtains a resource group including target computing resources that support MIG mode configuration to obtain a MIG resource group; then determines the corresponding preset configuration rules according to the target requirements, and configures the target computing resources in the MIG resource group using the corresponding MIG scheme according to the preset configuration rules to obtain the configured MIG instance; wherein the preset configuration rules characterize the similarities and differences between the MIG schemes for configuring the target computing resources in the MIG resource group; the MIG scheme includes different schemes after combining the MIG mode in different combinations; finally, a preset number of target MIG instances are determined from all the MIG instances to train the tasks to be trained of the AI ​​platform using the target MIG instances. The embodiment of the present application configures the target computing resources in the acquired MIG resource group using MIG schemes corresponding to different preset configuration rules to perform task training through the configured MIG instances, thereby enhancing the flexibility and pertinence of the computing resource configuration of the AI ​​platform to meet different training requirements, thereby improving the utilization of computing resources.

[0059] Figure 3 A flowchart of a specific computing resource processing method provided in an embodiment of the present application. Figure 3 As shown, the computing resource processing method includes:

[0060] S21: Scan the underlying computing resources, and divide the scanned underlying computing resources into resource groups of corresponding types according to the computing resource types; wherein the resource groups of the same type include the underlying computing resources of the same type.

[0061] S22: Determine, from all the resource groups, a resource group that includes target computing resources that support MIG mode configuration.

[0062] In this embodiment, the underlying computing resources are first scanned, and the scanned underlying computing resources are divided into resource groups of corresponding types according to the computing resource types; wherein the resource groups of the same type include the underlying computing resources of the same type. Then, the resource group containing the target computing resources supporting the MIG mode configuration is determined from all the resource groups. The resource group nodes in this embodiment belong to the k8s (kubernetes) cluster management. k8s is an open source system cluster management for automated deployment, expansion and management of containerized applications. When adding a cluster, it is necessary to scan all nodes under the cluster, identify the computing resource types of the nodes, and classify the nodes into their respective resource type groups. For example, the scanned A100 GPU is divided into a special type of MIG resource group, and the nodes with MLU cards (machine learning processors, Memory Loading Unit) are automatically classified into the MLU default resource group. The MLU default resource group can dynamically appear related functions such as MLU. It can be seen that in addition to processing A100 GPU cards with special functions, this embodiment can also dynamically adapt computing resources such as different types of GPU cards and MLU cards. When adding a cluster or expanding nodes, scan the node computing resource types and automatically classify the nodes according to the computing resource types.

[0063] S23: Determine corresponding preset configuration rules according to target requirements, and configure the target computing resources in the MIG resource group using the corresponding MIG scheme according to the preset configuration rules to obtain a configured MIG instance; wherein the preset configuration rules characterize the differences and similarities between the MIG schemes for configuring the target computing resources in the MIG resource group; the MIG scheme includes different schemes after combining MIG modes in different combinations.

[0064] In this embodiment, the definition of the preset configuration rules and the MIG scheme in the above step S23 can refer to the corresponding content disclosed in the previous embodiment, and will not be repeated here. It should be noted that the AI ​​platform will perform a series of verifications when creating or updating resource groups based on the maintained information, and only the nodes that pass the verification can complete the resource adjustment. The purpose of the verification is to improve the fault tolerance and stability of the platform. Otherwise, there will be confusion in resource management and confusion in the use of resources by the underlying business, which is not conducive to the management of the AI ​​platform, the statistics and display of computing resources, and at the same time, it is not conducive to the statistics of computing resource reports. Among them, the main verification scenarios include: verifying operation permissions; verifying whether parameters are legal; verifying node computing resource types; verifying whether A100 video memory isolation and MIG mode are turned on at the same time; verifying whether offline nodes are included; verifying whether MIG mode nodes contain A100 GPU cards; in MIG mode, verify whether the number and number of A100 GPUs corresponding to the nodes in the parameters are consistent with the node GPU information in the database; when editing resource groups in MIG mode, verify whether the MIG schemes corresponding to the A100 GPU cards of the nodes have changed, and only adjust the MIG schemes for the changed A100 GPU cards, and do not adjust the MIG schemes for the unchanged GPU cards; check whether the newly added nodes have running tasks; check whether the newly added nodes have A100 GPU cards and the MIG configuration is unsuccessful.

[0065] After a series of verifications are completed, the MIG scheme of each A100 GPU card of each node is configured according to the selection, and the configuration information is filled into mig_set, and the id information of MIG scheme D corresponding to each A100 GPU card of the AI ​​platform node is updated. After the relevant updates are completed, the mig_set label is added to the Labels label of the k8s node. The mig_set label information is as follows Figure 4 As shown. The bottom layer monitors the k8s node events and finds that the node's mig_set has changed. It will determine whether it has changed based on the time stamp in mis_set, and re-check based on mig_mode and the information in config. After the check passes, the MIG configuration of the bottom layer A100 GPU card will be started. The mig_result label is added to the Labels label of the k8s node, and the mig_result label information is updated synchronously and in real time until all A100 GPU cards of the node are configured. The mig_result label information is as follows: Figure 5shown. It should be noted that the above "mig_mod" represents the mig configuration mode, where "none_mix" means that A100 GPU mig is switched from off to on, "mix_mix" means that A100 GPU mig is switched from one MIG scheme to another, "mix_retry" means that A100 GPU "none_mix" or "mix_mix" fails and is retried, "mix_none" means that A100 GPU mig is switched from on to off, and "none_retry" means that A100 GPU "mix_none" fails and is retried; "config" is a {key:value} mode, where key is the unique number of the node A100 GPU card, and value is the MIG scheme id corresponding to the A100 GPU card. The config contains all the MIG configuration schemes of the node's A100 GPU cards; "status" means the A100 GPU MIG configuration status, where 0 represents success, 1 represents configuring, and 2 represents failed; "progress" means the node A100 GPU MIG configuration progress, such as 7 / 8, which means that the node has 8 A100 GPU card, 7 GPU cards have been successfully configured; "description" is the description information. For example, the following description information indicates that the configuration of card 0 failed, and the configuration of the other cards was successful: "description":{"0":"fail","1":"success","2":"success","3":"success","4":"success","5":"success","6":"success","7":"success"}". "time" indicates the timestamp of this operation. The timestamps of mig_set and mig_result for the same operation are consistent. The timestamp setting is to prevent repeated operations and incorrect updates of the configuration status, and it also has fault tolerance functions such as timeout termination.

[0066] After each GPU card operation is completed, the mig_result tag information of k8s will be modified synchronously. The node event monitoring of the AI ​​platform business layer will monitor the status changes of mig_result, synchronously obtain mig_result information, and monitor the status changes in real time. When the status changes, the MIG configuration status of the business layer node will be updated synchronously, and the MIG configuration status and MIG resource configuration details of the entire resource group will be dynamically displayed. Dynamically display the MIG resource details that have been configured in the resource group, such as Figure 6 As shown in the figure, the process is displayed dynamically. Every time an A100 GPU card is successfully configured, it is automatically counted in the configuration details. The A100 GPU cards that fail to be configured are not counted.

[0067] S24: Determine whether the configuration is successful. If not, use the same MIG solution to reconfigure the target computing resource that failed to be configured or re-determine a new MIG solution to reconfigure the target computing resource that failed to be configured.

[0068] In this embodiment, the MIG configuration status (configuring, success, failed) of each A100 GPU card in each node of the resource group is different at the same time, and there is also a situation where the A100 MIG configuration fails. These situations need to be processed to improve the compatibility, usability, fault tolerance and reliability of the platform resource group, and ensure that the A100 GPU card configured in MIG mode can be normally scheduled and used by k8s, thereby improving the utilization rate of GPU resources and the training efficiency of training tasks.

[0069] The configuration and adjustment of MIG mode requires time. At the same time, the time for successful configuration of MIG mode varies for different nodes in the same resource group or multiple A100 GPU cards in the same node, and the configuration may fail. For the configuration of MIG mode of different A100 GPU cards in each node of the resource group, the MIG resource group of the AI ​​platform needs to have a certain fault tolerance when configuring. Correspondingly, it can include node-level fault tolerance, that is, if a node configuration fails, the failed node can be retried and the previous configuration can be re-performed. At the same time, the MIG solution can be re-selected and re-configured without affecting the nodes that have been successfully configured. It can also include card-level fault tolerance, that is, if the configuration of individual A100 GPU cards in a node fails, only the failed A100 GPU card can be retried and the previous configuration can be re-performed. At the same time, the MIG solution can be re-selected and re-configured only for the failed GPU card, without affecting the A100 GPU card that has been successfully configured.

[0070] S25: Determine a preset number of target MIG instances from all the MIG instances, and schedule idle target computing resources in idle nodes including the target MIG instances to train the tasks to be trained.

[0071] In this embodiment, a preset number of target MIG instances are determined from all the MIG instances, and the idle target computing resources in the idle nodes containing the target MIG instances are scheduled to train the tasks to be trained. Specifically, tasks can be submitted at the node level, card level, and instance level, that is, taking nodes as units, select the nodes under the resource group, submit training tasks, and the tasks will be scheduled to the nodes; taking the A100 GPU card that has been configured with the MIG instance as a unit, select the A100 GPU card under the resource group to submit training tasks, the tasks will schedule the node where the card is located, and use the MIG instance of the A100 GPU card; taking the configured MIG instance as a unit, select the MIG instance under the resource group, submit training tasks, the tasks will randomly schedule the idle nodes with the MIG instance, and randomly select the idle A100 GPU card with the MIG instance. If the selected MIG instances are all in use, they will wait in line.

[0072] In this embodiment, the target computing resource can also be monitored to obtain the corresponding real-time status information, and the real-time status information is synchronized and displayed to the AI ​​platform. That is, for special nodes with A100 GPU cards, all A100 GPU card information of the node needs to be synchronized to the AI ​​platform, specifically by scanning the underlying computing resource information, such as computing resource type, quantity (used / total), etc., video memory size (used / total), A100 GPU UUID and number, etc. If the computing resource changes, it will be updated synchronously, and the user can choose to use the corresponding computing resource according to the business scenario. For example, when the server adds or unplugs the A100 GPU card, the AI ​​platform will automatically update the maintenance information of the A100 GPU card, add or remove the record of the A100 GPU card, and maintain the changes of the A100 GPU card GPU UUID. In addition, when the AI ​​platform adds a cluster, expands the node, removes the node, and synchronizes the node information regularly, it will synchronize all the scanned computing resource information (A100 GPU, MLU, etc.) to the AI ​​platform, and perform maintenance and updates.

[0073] It can be seen that the embodiment of the present application scans the underlying computing resources, divides the scanned underlying computing resources into resource groups of corresponding types according to the computing resource types, and determines the resource group containing the target computing resources that support MIG mode configuration from all the resource groups, so that the AI ​​platform can flexibly adapt to different computing resource types and can handle special resource types. The flexible configuration of the MIG mode of each A100 GPU single card in each node of the resource group is realized, and it is compatible with computing resources such as GPU and MLU at the same time. It has multiple levels of configuration schemes, different levels of fault-tolerant scenarios, and multiple levels of task submission methods to improve the fault tolerance, compatibility, flexibility and reliability of the AI ​​platform.

[0074] See also Figure 7 As shown, the embodiment of the present application also discloses a computing resource processing device, which is applied to an AI platform, including:

[0075] An acquisition module 11 is used to acquire a resource group including a target computing resource supporting MIG mode configuration to obtain a MIG resource group;

[0076] A configuration module 12 is used to determine a corresponding preset configuration rule according to a target requirement, and configure the target computing resources in the MIG resource group using a corresponding MIG scheme according to the preset configuration rule to obtain a configured MIG instance; wherein the preset configuration rule represents the similarities and differences between the MIG schemes for configuring the target computing resources in the MIG resource group; the MIG scheme includes different schemes after combining MIG modes in different combinations;

[0077] The training module 13 is used to determine a preset number of target MIG instances from all the MIG instances, so as to use the target MIG instances to train the tasks to be trained submitted to the AI ​​platform.

[0078] It can be seen that the embodiment of the present application first obtains a resource group including target computing resources that support MIG mode configuration to obtain a MIG resource group; then determines the corresponding preset configuration rules according to the target requirements, and configures the target computing resources in the MIG resource group using the corresponding MIG scheme according to the preset configuration rules to obtain the configured MIG instance; wherein the preset configuration rules characterize the similarities and differences between the MIG schemes for configuring the target computing resources in the MIG resource group; the MIG scheme includes different schemes after combining the MIG mode in different combinations; finally, a preset number of target MIG instances are determined from all the MIG instances to train the tasks to be trained of the AI ​​platform using the target MIG instances. The embodiment of the present application configures the target computing resources in the acquired MIG resource group using MIG schemes corresponding to different preset configuration rules to perform task training through the configured MIG instances, thereby enhancing the flexibility and pertinence of the special computing resource configuration of the AI ​​platform to meet different training requirements, thereby improving the utilization of computing resources.

[0079] In some specific embodiments, the acquisition module 11 specifically includes:

[0080] A scanning unit, used for scanning the underlying computing resources, and dividing the scanned underlying computing resources into resource groups of corresponding types according to the computing resource types; wherein the resource groups of the same type include the underlying computing resources of the same type;

[0081] The determining unit is used to determine the resource group including the target computing resources supporting the MIG mode configuration from all the resource groups.

[0082] In some specific embodiments, the configuration module 12 specifically includes:

[0083] A first configuration unit, configured to determine a corresponding first preset configuration rule according to a first target requirement; wherein the first preset configuration rule is to configure all the target computing resources in the MIG resource group to the same MIG solution;

[0084] Configure all the target computing resources in the MIG resource group using the same MIG solution according to the first preset configuration rule;

[0085] A second configuration unit is used to determine a corresponding second preset configuration rule according to a second target requirement; wherein the second preset configuration rule is to configure the target computing resource of each node in the MIG resource group to the same MIG scheme, and the target computing resources of different nodes are configured with different MIG schemes;

[0086] According to the second preset configuration rule, the target computing resource of the same node in the MIG resource group is configured using the same MIG solution;

[0087] A third configuration unit, configured to determine a corresponding third preset configuration rule according to a third target requirement; wherein the third preset configuration rule is to configure all the target computing resources in the MIG resource group to all different types of the MIG solutions;

[0088] configuring all the target computing resources in the MIG resource group using all different types of the MIG solutions according to the third preset configuration rule;

[0089] A fourth configuration unit, configured to determine a corresponding fourth preset configuration rule according to a fourth target requirement; wherein the fourth preset configuration rule is to configure the target computing resources of the first preset node in the MIG resource group to the same MIG solution, and configure the target computing resources of the second preset node to different MIG solutions;

[0090] The target computing resources of the same node in the MIG resource group are configured using the same MIG solution according to the fourth preset configuration rule.

[0091] In some specific embodiments, the training module 13 is specifically used to determine a preset number of target MIG instances from all the MIG instances, and schedule idle target computing resources in idle nodes including the target MIG instances to train the tasks to be trained.

[0092] In some specific embodiments, the computing resource processing device further includes:

[0093] A retry module, used to determine whether the configuration is successful, and if not, to reconfigure the target computing resource that failed to be configured using the same MIG solution or to re-determine a new MIG solution to reconfigure the target computing resource that failed to be configured;

[0094] A synchronization module is used to monitor the target computing resource to obtain corresponding real-time status information, synchronize the real-time status information and display it to the AI ​​platform.

[0095] Furthermore, an embodiment of the present application also provides an electronic device. Figure 8 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be considered as any limitation on the scope of use of the present application.

[0096] Figure 8 The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the computing resource processing method disclosed in any of the aforementioned embodiments.

[0097] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0098] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0099] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20 to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the computing resource processing method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. The data 223 can include the tasks to be trained collected by the electronic device 20.

[0100] Furthermore, an embodiment of the present application also discloses a storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the computing resource processing method disclosed in any of the aforementioned embodiments are implemented.

[0101] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0102] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0103] The computing resource processing method, device, equipment and storage medium provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A computing resource processing method, It is characterized in that Applied to AI platforms, including: Obtain a resource group including a target computing resource that supports MIG mode configuration to obtain a MIG resource group; Determine a corresponding preset configuration rule according to the target demand, and configure the target computing resources in the MIG resource group using the corresponding MIG scheme according to the preset configuration rule to obtain a configured MIG instance; wherein the preset configuration rule represents the similarities and differences between the MIG schemes for configuring the target computing resources in the MIG resource group; the MIG scheme includes different schemes after combining MIG modes in different combinations; Determine a preset number of target MIG instances from all the MIG instances, so as to train the to-be-trained tasks of the AI ​​platform using the target MIG instances; The obtaining of a resource group including a target computing resource supporting MIG mode configuration includes: Scanning underlying computing resources, and dividing the scanned underlying computing resources into resource groups of corresponding types according to computing resource types; wherein the resource groups of the same type include the underlying computing resources of the same type; A resource group including target computing resources supporting MIG mode configuration is determined from all the resource groups.

2. The computing resource processing method according to claim 1, It is characterized in that Determining the corresponding preset configuration rules according to the target requirements, and configuring the target computing resources in the MIG resource group using the corresponding MIG solution according to the preset configuration rules, includes: Determine a corresponding first preset configuration rule according to the first target requirement; wherein the first preset configuration rule is to configure all the target computing resources in the MIG resource group to the same MIG solution; All the target computing resources in the MIG resource group are configured using the same MIG solution according to the first preset configuration rule.

3. The computing resource processing method according to claim 1, It is characterized in that Determining the corresponding preset configuration rules according to the target requirements, and configuring the target computing resources in the MIG resource group using the corresponding MIG solution according to the preset configuration rules, includes: Determine a corresponding second preset configuration rule according to the second target requirement; wherein the second preset configuration rule is to configure the target computing resource of each node in the MIG resource group to the same MIG solution, and configure different MIG solutions for the target computing resources of different nodes; The target computing resources of the same node in the MIG resource group are configured using the same MIG solution according to the second preset configuration rule.

4. The computing resource processing method according to claim 1, It is characterized in that Determining the corresponding preset configuration rules according to the target requirements, and configuring the target computing resources in the MIG resource group using the corresponding MIG solution according to the preset configuration rules, includes: Determine a corresponding third preset configuration rule according to the third target requirement; wherein the third preset configuration rule is to configure all the target computing resources in the MIG resource group to all different types of the MIG solutions; All the target computing resources in the MIG resource group are configured using all different types of the MIG solutions according to the third preset configuration rule.

5. The computing resource processing method according to claim 1, It is characterized in that Determining the corresponding preset configuration rules according to the target requirements, and configuring the target computing resources in the MIG resource group using the corresponding MIG solution according to the preset configuration rules, includes: Determine a corresponding fourth preset configuration rule according to the fourth target requirement; wherein the fourth preset configuration rule is to configure the target computing resources of the first preset node in the MIG resource group to the same MIG solution, and configure the target computing resources of the second preset node to different MIG solutions; The target computing resources of the same node in the MIG resource group are configured using the same MIG solution according to the fourth preset configuration rule.

6. The computing resource processing method according to any one of claims 1 to 5, It is characterized in that After configuring the target computing resources in the MIG resource group according to the preset configuration rule using the corresponding MIG solution, the method further includes: Determine whether the configuration is successful. If not, use the same MIG solution to reconfigure the target computing resource that failed to be configured or re-determine a new MIG solution to reconfigure the target computing resource that failed to be configured.

7. The computing resource processing method according to claim 1, It is characterized in that The step of determining a preset number of target MIG instances from all the MIG instances, and using the target MIG instances to train the tasks to be trained submitted to the AI ​​platform, includes: A preset number of target MIG instances are determined from all the MIG instances, and idle target computing resources in idle nodes including the target MIG instances are scheduled to train the tasks to be trained.

8. The computing resource processing method according to claim 1, It is characterized in that Also includes: The target computing resource is monitored to obtain corresponding real-time status information, and the real-time status information is synchronized and displayed to the AI ​​platform.

9. A computing resource processing device, It is characterized in that Applied to AI platforms, including: An acquisition module, used for acquiring a resource group including a target computing resource supporting MIG mode configuration, so as to obtain a MIG resource group; A configuration module, used to determine a corresponding preset configuration rule according to a target requirement, and configure the target computing resources in the MIG resource group using a corresponding MIG scheme according to the preset configuration rule to obtain a configured MIG instance; wherein the preset configuration rule characterizes the similarities and differences between the MIG schemes for configuring the target computing resources in the MIG resource group; the MIG scheme includes different schemes after combining MIG modes in different combinations; A training module, used to determine a preset number of target MIG instances from all the MIG instances, so as to use the target MIG instances to train the tasks to be trained submitted to the AI ​​platform; The acquisition module comprises: A scanning unit, used for scanning the underlying computing resources, and dividing the scanned underlying computing resources into resource groups of corresponding types according to the computing resource types; wherein the resource groups of the same type include the underlying computing resources of the same type; The determining unit is used to determine the resource group including the target computing resources supporting the MIG mode configuration from all the resource groups.