Resource management method, device, equipment and medium

Abstract

The present application provides a resource management method, apparatus, device and medium, which relate to the field of computer technology and are applied to a deep learning training platform. The method comprises: for multiple user groups, allocating corresponding computing resources to the multiple user groups, and setting corresponding validity periods for the computing resources allocated to the multiple user groups, respectively, so that the user groups use the computing resources to run the training tasks of the deep learning training platform before the expiration of the validity period; when the validity period of the computing resources expires, performing a recycling operation on the computing resources, and releasing the allocation relationship between the user group and the recycled computing resources, so as to improve the utilization efficiency of the computing resources in the deep learning training platform.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a resource management method, device, equipment and medium. Background Art

[0002] In the field of resource management of deep learning training platforms, the current industry generally adopts manual maintenance to recycle resources. However, this traditional approach has obvious flaws. First, manual operation is inefficient and it is difficult to respond quickly to changes in resource demand, resulting in a slow recycling process. Second, the risk of error caused by human factors makes the timing and allocation of resource recycling inaccurate, which affects the stability of the platform and user trust. In addition, as the scale of the platform expands, the human resource costs required for manual maintenance continue to rise, which is not conducive to long-term sustainable development. Finally, manual processing cannot achieve immediate response, especially when resources are tight or demand is peak, which seriously affects the user experience. Summary of the invention

[0003] In view of this, embodiments of the present application provide a resource management method, apparatus, device, and medium to overcome the above problems or at least partially solve the above problems.

[0004] A first aspect of an embodiment of the present application provides a resource management method, which is applied to a deep learning training platform, and the method includes: For multiple user groups, corresponding computing resources are allocated to the multiple user groups, and corresponding validity periods are set for the computing resources allocated to the multiple user groups, so that the user groups use the computing resources to run the training tasks of the deep learning training platform before the expiration of the validity period; When the validity period of the computing power resources ends, a recycling operation is performed on the computing power resources, and the allocation relationship between the user group and the recycled computing power resources is released.

[0005] Optionally, respectively setting corresponding validity periods for the computing resources allocated to the multiple user groups includes: A permission usage tag is set for the computing power resource; the permission usage tag is divided into a first tag, a second tag and a third tag; the validity period corresponding to the first tag indicates that the computing power resource is not subject to time restrictions, and the computing power resource with the permission usage tag of the first tag will be actively released after the user group has used it; the validity period corresponding to the second tag indicates that the computing power resource is subject to time restrictions, and the third tag indicates that the validity period of the computing power resource is the validity period of the last time the user group used the computing power resource; Adding the permission usage label and the corresponding validity period to the permission configuration information corresponding to the user group; The method also includes: before the validity period of the computing power resource expires, sending an alarm message to a plurality of user groups to which the computing power resource is allocated, so as to prompt the user groups to renew the computing power resource or adjust the usage plan of the computing power resource.

[0006] Optionally, performing a recovery operation on the computing power resource includes: Loop through the plurality of computing resource usage data corresponding to the plurality of user groups allocated with the computing resource, and determine the permission usage label recorded in the permission configuration information; When the permission usage tag of the computing power resource of the user group is the first tag, no recovery operation is performed on the computing power resource; When the permission usage tag of the computing power resource of the user group is the second tag or the third tag, comparing the validity period of the computing power resource with the current time; When the validity period of the computing power resources is not less than the current time, a recycling operation is performed on the computing power resources so that the recycled computing power resources are used for reallocation.

[0007] Optionally, before performing the recycling operation on the computing power resource, the method further includes: When the validity period of the computing power resources expires, determining whether there is a user in the user group who is using the computing power resources to run a training task on the deep learning training platform; The performing of the recycling operation on the computing power resources includes: when there is a user in the user group who is using the computing power resources to run a training task of the deep learning training platform, waiting for the training task of the user to be completed, performing the recycling operation on the computing power resources; When there is no user in the user group who is using the computing power resources to run the training task of the deep learning training platform, the computing power resources are directly recycled.

[0008] Optionally, before allocating corresponding computing resources to the multiple user groups and setting corresponding validity periods for the computing resources allocated to the multiple user groups, the method further includes: Divide all computing resources in the deep learning training platform into multiple resource groups according to different types, each resource group containing computing resources of the same type; The allocating corresponding computing resources to the multiple user groups includes: using the requirements of training tasks that need to be run by different user groups in the deep learning training platform and the permissions of different user groups to different types of computing resources as screening conditions, and determining from the multiple resource groups multiple types of computing resources that meet the screening conditions and allocating them to the user groups; The respectively setting of corresponding validity periods for the computing power resources allocated to the multiple user groups includes: setting corresponding validity periods for the multiple types of computing power resources allocated to the user group according to the estimated completion time of the training tasks that the user group needs to run in the deep learning training platform, the size of the computing power resources, the performance of the computing power resources, the historical computing power resource usage records of the user group, the remaining computing power resources in the resources, and the budget of the user group.

[0009] Optionally, before sending the warning information to the multiple user groups to which the computing resources are allocated, the method further includes: Set alarm reminder strategies and alarm information notification methods for different types of computing resources allocated to different user groups; Before the validity period of the computing power resource expires, according to the alarm reminder strategy and the notification method of the alarm information, sending corresponding alarm information to the user group to which the computing power resource is allocated; The alarm reminder strategy includes the time to start the alarm and the number of times the alarm information is sent, and the notification method of the alarm information includes in-platform station letter notification and email notification.

[0010] Optionally, the method further comprises: After allocating corresponding computing resources to the multiple user groups, allocating the training tasks created by the user groups on the deep learning training platform to the computing resources; When the training task of the user group is completed before the end of the validity period, the computing resources actively released by the user group are received.

[0011] In a second aspect of an embodiment of the present application, a resource management device is provided, which is applied to a deep learning training platform, and the device includes: An allocation module, for allocating corresponding computing resources to multiple user groups, and setting corresponding validity periods for the computing resources allocated to the multiple user groups, so that the user groups use the computing resources to run the training tasks of the deep learning training platform before the expiration of the validity periods; The recycling module is used to perform a recycling operation on the computing power resources when the validity period of the computing power resources expires, and to release the allocation relationship between the user group and the recycled computing power resources.

[0012] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in the first aspect.

[0013] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0014] Beneficial effects of this application: The present application provides a resource management method, apparatus, device and medium, which relate to the field of computer technology and are applied to a deep learning training platform. The method comprises: for multiple user groups, allocating corresponding computing resources to the multiple user groups, and setting corresponding validity periods for the computing resources allocated to the multiple user groups, respectively, so that the user groups use the computing resources to run the training tasks of the deep learning training platform before the expiration of the validity period; when the validity period of the computing resources expires, performing a recycling operation on the computing resources, and releasing the allocation relationship between the user group and the recycled computing resources.

[0015] The technical solution of the present application can help the deep learning training platform to efficiently allocate and use computing resources, ensuring that users in each user group efficiently and reasonably use the allocated computing resources before the validity period of the computing resources expires. By setting the validity period, efficient management of computing resources can be automatically achieved without human intervention, thereby improving the utilization efficiency of computing resources in the deep learning training platform, while reducing the risk of idle computing resources, and improving user experience when resources are tight or demand is peak. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0017] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 It is a flowchart of a resource management method provided by an embodiment of the present application; Figure 2 It is a schematic diagram of a framework of a resource management system provided by an embodiment of the present application; Figure 3 It is a flowchart of another resource management method provided by an embodiment of the present application; Figure 4 It is a schematic diagram of computing resource configuration information of different user groups provided by an embodiment of the present application; Figure 5 It is a schematic diagram of a framework of a resource management device provided by an embodiment of the present application; Figure 6 It is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] First, the resource management method provided in this embodiment is applied to a deep learning training platform, aiming to perform refined and intelligent resource management of the computing resources of the deep learning training platform. On the deep learning training platform, computing resources can be provided to run training tasks of users in different user groups. As the training tasks increase and the computing resources of the deep learning training platform are limited, computing resources may be tight or demand peaks may occur. Therefore, after allocating computing resources for running training tasks to different user groups, it is necessary to ensure that the computing resources of the deep learning training platform can not only be reasonably allocated on demand, but also efficiently realize recycling management to avoid resource idleness caused by some user groups occupying the allocated computing resources for a long time, thereby ensuring the resource utilization rate of the computing resources of the deep learning training platform, reducing operating costs, and enhancing user satisfaction and platform competitiveness.

[0022] Based on the above content, the present application provides a resource management method, apparatus, device and medium. The resource management method of the present application is executed through various modules in the resource management system deployed on the deep learning training platform.

[0023] Figure 1 is a flowchart of a resource management method provided by an embodiment of the present application. Figure 2 is a schematic diagram of a resource management system provided by an embodiment of the present application. Figure 3 It is a flowchart of another resource management method provided in an embodiment of the present application.

[0024] like Figure 1 As shown, an embodiment of the present application provides a resource management method, which is applied to a deep learning training platform, and the method includes: Step S101, for multiple user groups, corresponding computing resources are allocated to the multiple user groups, and corresponding validity periods are set for the computing resources allocated to the multiple user groups, so that the user groups use the computing resources to run the training tasks of the deep learning training platform before the expiration of the validity period; Step S102: When the validity period of the computing power resources ends, a recycling operation is performed on the computing power resources, and the allocation relationship between the user group and the recycled computing power resources is released.

[0025] First, if Figure 2 As shown, the resource management system includes a user group computing power resource allocation module, an alarm reminder strategy and an alarm information notification method setting module, a resource validity period setting module, an alarm module, a recycling reminder module, an alarm information sending module, a user group, and a computing power resource recycling module. The computing power resource recycling module includes a recycling sub-module and a task running sub-module. The task running sub-module is used to run scheduled tasks.

[0026] In step S101, the user group computing resource allocation module of the system will allocate computing resources (such as GPU, TPU, CPU, etc.) to multiple user groups (such as different research teams or project teams) that have training task execution requirements on the deep learning training platform. In this process, not only will the corresponding computing resources be allocated to each user group according to the needs of the user group, but the corresponding validity period will also be set for the computing resources allocated to the user group through the resource validity period setting module. The validity period is used to indicate the time limit for the use of computing resources. For example, the validity period can be: X month X day, 20XX, that is, the user group can use the computing resources before X month X day, 20XX, so that each time the computing resources obtained by the user group have a corresponding limited period for accurate management of computing resources.

[0027] For example, all A100 accelerator cards can be divided into resource group A, all V100 accelerator cards can be divided into resource group B, and these resource groups can be assigned to user groups C and D. In this way, users in user groups C and D can use the computing resources in these resource groups for training. At the same time, operators need to set the validity period of these resources according to the computing power of different resource groups, such as allowing user group C to use them for only three days or several months.

[0028] Before the computing power resources are valid for a certain period of time, all users in the user group have the right to use the computing power resources. Therefore, before the computing power resources are valid for a certain period of time, all users in the user group can use the computing power resources that are valid for a certain period of time to run the training tasks created by the user group on the deep learning training platform. For example, before the computing power resources are valid for a certain period of time, all users in the user group can use the computing power resources allocated to them for training tasks such as model training and data processing.

[0029] Finally, in step S102, once the computing power resources expire, the computing power resource recycling module of the system will recycle the computing power resources that have expired, and cancel the allocation relationship between these computing power resources and the user group, ensuring that the recycled computing power resources can be released and allocated to other user groups in need. At the same time, in this process, the recycling reminder module of the system will remind the users in the user group that the recycling operation has been executed to recycle the computing power resources.

[0030] Through the above embodiments, it is possible to help the deep learning training platform efficiently allocate and use computing resources, ensure that users in each user group use the allocated computing resources efficiently and reasonably before the expiration of the computing resources, and remind the user group of the usage plan of the allocated computing resources through alarm information before the expiration of the validity period, and finally realize the reasonable recovery and redistribution of computing resources. Such a mechanism can improve the utilization efficiency of computing resources in the deep learning training platform, while reducing the risk of idle computing resources.

[0031] Optionally, corresponding validity periods are set for the computing resources allocated to the multiple user groups, respectively, including: A permission usage tag is set for the computing power resource; the permission usage tag is divided into a first tag, a second tag and a third tag; the validity period corresponding to the first tag indicates that the computing power resource is not subject to time restrictions, and the computing power resource with the permission usage tag of the first tag will be actively released after the user group has used it; the validity period corresponding to the second tag indicates that the computing power resource is subject to time restrictions, and the third tag indicates that the validity period of the computing power resource is the validity period of the last time the user group used the computing power resource; Adding the permission usage label and the corresponding validity period to the permission configuration information corresponding to the user group; The method also includes: before the validity period of the computing power resource expires, sending an alarm message to a plurality of user groups to which the computing power resource is allocated, so as to prompt the user groups to renew the computing power resource or adjust the usage plan of the computing power resource.

[0032] Figure 4 This is a schematic diagram of computing resource configuration information for different user groups provided in one embodiment of the present application.

[0033] Specifically, in another embodiment, in the process of setting corresponding validity periods for the computing resources allocated to multiple user groups respectively, it is necessary to first set permission usage labels for the computing resources allocated to the multiple user groups respectively, so that each computing resource corresponds to a permission usage label, and the permission usage labels are divided into three types, namely, a first label, a second label and a third label.

[0034] The first label: indicates that the computing power resources are valid for a long time. The computing power resources allocated to the user group can be automatically released by the user group after the user group has finished using them. The validity period is not limited by time.

[0035] The second label: indicates that the computing power resource is short-term valid. This resource will be used within the set time limit, that is, the validity period of the computing power resource is time-limited and will be reclaimed after the validity period ends.

[0036] The third tag: indicates that the usage duration corresponding to the validity period of the computing power resource is the same as the usage duration corresponding to the validity period of the computing power resource used by the user group last time. That is, the usage duration corresponding to the validity period of the computing power resource is reused.

[0037] After setting the permission usage label and the corresponding validity period, add the permission usage label and the corresponding validity period to the permission configuration information corresponding to the user group. In this way, the administrator can easily view and manage the usage permissions and expiration status of the computing resources corresponding to each user group.

[0038] For example, suppose there is a deep learning training platform with three user groups: user group A, user group B, and user group C.

[0039] Computing resource allocation and label setting: User group A is allocated 4 GPUs, and these resources are marked with the first label (long-term validity), which is not time-limited.

[0040] User group B is allocated 2 GPUs, marked as the second label (short-term validity), the validity period indicates that the end time of the right to use the computing power resources is X month X day 20XX, and it can be used for 30 days.

[0041] User group C is allocated 3 GPUs, marked as the third label (validity period of last use). Assuming that the end time of last use is 20XX-XX-XX, and it was used for a total of 3 days, the validity period is the current time + 45 days, which corresponds to the end time of the computing power resource usage rights.

[0042] Validity period management: User group A can use these four GPUs for a long time, and after use, the user group will actively release these GPUs or use them for subsequent training tasks.

[0043] User group B must use these two GPUs within 30 days, and the system will reclaim the computing resources when they expire. If they are used up before the expiration date, these computing resources will be released automatically.

[0044] User group C uses 3 GPUs, and according to the validity period of the last use (45 days), they can use these computing resources in the next 45 days, and the system will monitor these computing resources until the end of the validity period and recycle them.

[0045] like Figure 4 As shown, user group 1 and user group 2 are respectively allocated multiple computing resources with different validity periods, among which user group 1 is allocated computing resources 1, computing resources 2, and computing resources 3, and user group 2 is allocated computing resources 4 and computing resources 5. Computing resource 1 is the first label, indicating that user group 1's permission to use computing resource 1 is long-term valid, computing resource 2 is the second label, indicating that user group 1's permission to use computing resource 2 is short-term valid, and the validity period is 2024-08-08, and computing resource 3 is the second label, indicating that user group 1's permission to use computing resource 3 is short-term valid, and the validity period is 2024-09-08. Computing resource 4 is the first label, indicating that user group 2's permission to use computing resource 4 is long-term valid, and computing resource 5 is the second label, indicating that user group 2's permission to use computing resource 5 is short-term valid, and the validity period is 2024-08-08.

[0046] And, if Figure 3 As shown in the figure, a scheduled task is set for each computing resource. The task running submodule periodically monitors the validity period of the computing resources by running the scheduled tasks. Before the expiration of the validity period, the system's alarm module will send an alarm message to all users in the user group that have been allocated computing resources, reminding the users in the user group that the corresponding computing resource usage rights are about to expire. No more reminders will be given until the computing resources are recycled and released. This allows users in the user group to renew or adjust the computing resource usage plan for the computing resources that are being used or need to be continued to use before the expiration of the validity period, ensuring the efficient use of the overall computing resources of the deep learning training platform.

[0047] Through the above embodiments, the management methods of computing resources of different user groups are distinguished by setting permission usage labels and corresponding validity periods. In this way, flexible management of computing resources of different user groups is achieved. At the same time, multiple computing resources applied for by the same user group will also be managed by different validity periods, and before the end of the validity period, the user group will be reminded of the usage plan of the allocated computing resources through alarm information, and finally the reasonable recovery and redistribution of computing resources is achieved, ensuring that user groups can reasonably use computing resources before the end of their respective validity periods, which also helps to avoid waste of computing resources and management burden.

[0048] Optionally, the performing of a recovery operation on the computing power resources specifically includes; Loop through the plurality of computing resource usage data corresponding to the plurality of user groups allocated with the computing resource, and determine the permission usage label recorded in the permission configuration information; When the permission usage tag of the computing power resource of the user group is the first tag, no recovery operation is performed on the computing power resource; When the permission usage tag of the computing power resource of the user group is the second tag or the third tag, comparing the validity period of the computing power resource with the current time; When the validity period of the computing power resources is not less than the current time, a recycling operation is performed on the computing power resources so that the recycled computing power resources are used for reallocation.

[0049] Specifically, in another embodiment, Figure 3 As shown, in the process of executing the recycling operation on the computing power resources, a loop traversal method is adopted, by looping through the multiple computing power resource usage data corresponding to the multiple user groups allocated with the computing power resources, wherein each user group corresponds to a computing power resource usage data, and the computing power resource usage data of each user group records the permission configuration information of all the computing power resources applied by the user group, so that the permission usage label recorded in the permission configuration information can be determined during the traversal process.

[0050] For each user group's computing resources, the system will check the permission usage tags of each computing resource recorded in the permission configuration information.

[0051] When the permission usage tag is determined to be the first tag, it means that the computing power resources are valid for a long time and will not be recycled in this case.

[0052] When the permission usage tag of the computing power resource of the user group is the second tag or the third tag, it is necessary to compare the validity period of the computing power resource with the current time for further judgment.

[0053] Specifically, when it is determined that the permission usage label is the second label, it means that the resource is short-term valid, indicating that the computing power resource has a clear validity period. The system will compare its validity period with the current time to determine whether the computing power resource needs to be reclaimed. If the validity period of the computing power resource is still greater than or equal to the current time (which means that the computing power resource is still in effective use), it will not be reclaimed; if the current time is greater than the validity period of the computing power resource, it means that the validity period has ended, and the recycling operation will be performed.

[0054] When it is determined that the permission usage tag is the third tag, it is necessary to determine the validity period of the computing power resource, and then determine whether the validity period is subject to time limit. If the validity period is subject to time limit, it will not be recycled in the manner of the first tag. If the validity period is not subject to time limit, it will be recycled in the manner of the second tag.

[0055] After recycling, the recycled computing resources can be redistributed to various user groups.

[0056] Through the above embodiment, based on the conditional judgment of the permission usage label and the validity period, it is ensured that only the computing power resources that have expired or are short-term valid are recycled, while the computing power resources that are long-term valid will not be disturbed. This design improves the utilization efficiency of computing power resources and reduces unnecessary waste.

[0057] Optionally, before performing the recycling operation on the computing power resource, the method further includes: When the validity period of the computing power resources expires, determining whether there is a user in the user group who is using the computing power resources to run a training task on the deep learning training platform; The performing of the recycling operation on the computing power resources includes: when there is a user in the user group who is using the computing power resources to run a training task of the deep learning training platform, waiting for the training task of the user to be completed, performing the recycling operation on the computing power resources; When there is no user in the user group who is using the computing power resources to run the training task of the deep learning training platform, the computing power resources are directly recycled.

[0058] Specifically, in another embodiment, Figure 3 As shown, in order to ensure that the training tasks currently being run by users will not be affected by the recycling of computing resources, at the end of the validity period of the computing resources, the system will first check whether there are users in each user group who are using the allocated computing resources to run the training tasks of the deep learning training platform.

[0059] When it is determined that there are users in the user group who are using computing resources for training tasks, the system will wait for these training tasks to be completed before executing the recycling operation of computing resources. This is to avoid interrupting the ongoing training tasks due to the sudden recycling of computing resources, thereby affecting the work progress and results of the user group. When it is determined that there are no users in the user group who are using these computing resources, the system will directly recycle these computing resources whose validity period has expired, and release these computing resources for use by other user groups.

[0060] Through the above embodiments, when recycling computing resources, it is necessary to consider the actual current usage of each user in the user group to protect the user's training task from being affected by the recycling operation of computing resources. In this way, the system can manage computing resources efficiently and flexibly, and realize efficient management of computing resources while ensuring the normal operation of each user group.

[0061] Optionally, before allocating corresponding computing resources to the multiple user groups and setting corresponding validity periods for the computing resources allocated to the multiple user groups, the method further includes: Divide all computing resources in the deep learning training platform into multiple resource groups according to different types, each resource group containing computing resources of the same type; The allocating corresponding computing resources to the multiple user groups includes: using the requirements of training tasks that need to be run by different user groups in the deep learning training platform and the permissions of different user groups to different types of computing resources as screening conditions, and determining from the multiple resource groups multiple types of computing resources that meet the screening conditions and allocating them to the user groups; The respectively setting of corresponding validity periods for the computing power resources allocated to the multiple user groups includes: setting corresponding validity periods for the multiple types of computing power resources allocated to the user group according to the estimated completion time of the training tasks that the user group needs to run in the deep learning training platform, the size of the computing power resources, the performance of the computing power resources, the historical computing power resource usage records of the user group, the remaining computing power resources in the resources, and the budget of the user group.

[0062] Specifically, in another embodiment, Figure 3 As shown, before allocating computing resources to user groups, all computing resources in the deep learning training platform need to be divided into multiple resource groups according to different types, and different types of computing resources are managed through different resource groups. For example, computing resources may be divided into GPU groups, CPU groups, TPU groups, etc., so that the computing resources in each resource group are of the same type.

[0063] For each user group that needs to request computing resources, when allocating computing resources, the system will filter out computing resources that meet the conditions from the multiple resource groups that were previously grouped according to the requirements of each user group's training tasks and the user group's permissions to different types of computing resources. This approach takes into account the specific training task requirements of each user group, the types of available resources, and the user group's permissions to various types of computing resources, ensuring that the computing resources provided to the user group are the most suitable for processing the user group's training tasks, thereby improving the execution effect of the training tasks.

[0064] Among them, the demand for training tasks can be determined based on factors such as the type of training tasks (for example, image recognition tasks, natural language processing tasks, data processing tasks, etc.), the accuracy required for the training tasks, the computing efficiency required for the training tasks, the completion time of the training tasks, the type of computing resources required, and the performance of the required computing resources. The user group's permissions to different types of computing resources indicate whether the user group has the authority to apply for each type of computing resources. For example, user group A can apply for ordinary performance GPU resources, but cannot apply for high-performance GPU resources, or the number of TPU resources that user group B can apply for is limited to 10, so user group B cannot apply for more than 10 TPU resources.

[0065] Furthermore, after determining the computing resources for each user group, it is necessary to set the validity period of the computing resources allocated to each user group. The setting of the validity period depends on the following factors: Estimated completion time of the training task: For example, a training task is estimated to take 10 hours to complete.

[0066] The size and performance of computing resources: If the resource is a high-performance GPU, the task may be completed in a shorter time.

[0067] Historical usage records of user groups: For example, the historical usage time of a certain user group for a certain type of computing resources in the past.

[0068] The remaining computing power resources: The system needs to reasonably set the validity period of computing power resources according to the current availability of computing power resources. For example, when the remaining computing power resources are small, the validity period should be set close to the current time to avoid insufficient computing power resources. When the remaining computing power resources are sufficient, the validity period can be set far from the current time to meet the usage needs of the user group as much as possible.

[0069] User group budget: The user group may have certain restrictions on the cost of computing resources, which will also affect the validity period of the resources.

[0070] Based on this, in one embodiment, a calculation formula for the usage time of computing resources by a user group can be provided according to the above factors affecting the validity period setting, wherein the calculation formula for the usage time of computing resources by a user group is: T=(k1*T1+k2*T2)*(w1*M1*a+w2*M2 / b) / (M3*c); Where T is the duration of computing resource usage by the user group, T1 is the estimated completion time of the training task, T2 is the historical usage duration of computing resources, k1 and k2 are the weight coefficients of T1 and T2 respectively, and k1+k2=1. When there is no historical usage duration of the computing resource by the user group, T2=T1.

[0071] M1 is the proportion of the surplus of computing power resources in the corresponding resource group, a is the proportion coefficient, w1 is the weight of the surplus of computing power resources of the user group, a is determined according to the preset computing power resource surplus proportion threshold S. When the surplus of computing power resources is less than the computing power resource surplus proportion threshold S, it means that the surplus of computing power resources is small. When the surplus of computing power resources is not less than the computing power resource surplus proportion threshold S, it means that the surplus of computing power resources is sufficient.

[0072] The proportion coefficient a=1 / S. For example, when S=50%, a=2. Therefore, when the surplus of computing power resources M1 is equal to the surplus proportion threshold S of computing power resources, that is, M1=S, then M1*a=1, indicating that the current surplus of computing power resources is just equal to the surplus proportion threshold S of computing power resources, and the current surplus of computing power resources will not increase or decrease the user group's use time of computing power resources; when M1 is less than S, it means that the current surplus of computing power resources is small, then it is necessary to reduce the user group's use time of computing power resources to avoid the situation where the surplus of computing power resources is insufficient due to long-term occupation of computing power resources; when M1 is greater than S, it means that the current surplus of computing power resources is sufficient, then the user group's use time of computing power resources is increased to improve the execution effect of the user group's training tasks.

[0073] M2 is the budget level of the user group. The budget level of the user group is divided into n levels according to the budget of the user group. The higher the budget of the user group, the higher the budget level of the user group, which increases the usage time of computing resources by the user group. On the contrary, the lower the budget of the user group, the lower the budget level of the user group, which reduces the usage time of computing resources by the user group. b is the budget level coefficient, b≤n, w2 is the weight of the budget level of the user group, w1+w2=1.

[0074] M3 is the total performance of the computing resources allocated to the user group. M3 is determined based on the size and performance of the computing resources. c is the performance coefficient. When the total performance of the computing resources allocated to the user group is better, M3 is larger, which shortens the time T that the user group uses the computing resources, thereby reducing the user group's occupation of idle computing resources. When the total performance of the computing resources allocated to the user group is worse, M3 is smaller, which lengthens the time T that the user group uses the computing resources, thereby ensuring that the user group's training tasks can run normally.

[0075] At the same time, in order to ensure that the usage time of computing resources is not less than the estimated completion time of the training task, therefore, when the calculated T is less than or equal to T1, T=T1.

[0076] Finally, after determining the usage time T of the computing power resources by the user group, the validity period of the computing power resources is calculated according to the current time.

[0077] It should be noted that coefficients such as k1, k2, w1, w2, a, and b need to be set according to actual conditions.

[0078] Through the above embodiments, through the comprehensive process of allocating computing resources on the deep learning training platform, from the type classification, screening to allocation and validity period setting of computing resources, it is ensured that each user group obtains computing resources suitable for its task and sets a reasonable usage time. This method improves the efficiency of resource utilization and ensures that the user group can successfully complete the training task. At the same time, through the calculation formula for the user group's use of computing resources, it can determine the appropriate validity period for the computing resources according to various factors, so as to ensure the user group's usage needs while achieving efficient utilization of the computing resources of the deep learning training platform.

[0079] Optionally, before sending the warning information to the multiple user groups to which the computing resources are allocated, the method further includes: Set alarm reminder strategies and alarm information notification methods for different types of computing resources allocated to different user groups; Before the validity period of the computing power resource expires, according to the alarm reminder strategy and the notification method of the alarm information, sending corresponding alarm information to the user group to which the computing power resource is allocated; The alarm reminder strategy includes the time to start the alarm and the number of times the alarm information is sent, and the notification method of the alarm information includes in-platform station letter notification and email notification.

[0080] Specifically, in another embodiment, Figure 3 As shown, in order to ensure that the user group is aware of the usage and validity period of computing resources in a timely manner, the system's alarm reminder strategy and alarm information notification method setting module can set corresponding alarm reminder strategies and alarm notification methods for different computing resources.

[0081] The alarm reminder strategy includes two items: one is the time to start the alarm, and the other is the number of times the alarm information is sent.

[0082] The time to start the alarm refers to the time before the computing power resource expires. The system will notify each user in the user group in advance that the computing power resource is about to expire. For example, the alarm reminder policy can be set to send 3 alarms 48 hours, 24 hours or 1 hour before the expiration of the validity period.

[0083] The number of times the alarm information is sent refers to how many times the system will send the alarm information according to the set alarm reminder strategy. For example, if it is set to send 3 times, the system can send alarm information to the user group at different time intervals (such as 48 hours, 24 hours and 1 hour before the expiration date), so that the user group can reasonably arrange the use plan of computing power resources or renew the computing power resources.

[0084] Alert information notification methods include in-platform message notification and email notification.

[0085] In-platform message notification: The system can send warning messages within the deep learning training platform, and users can see these warning messages after logging into the platform.

[0086] Email notification: In addition to the in-platform message notification, the system can also send alarm information to the user group via email, so that each user in the user group can view it in time.

[0087] Through the above embodiments, it is ensured that the user group can receive the alarm information in time before the expiration of the computing power resource usage period, so as to avoid affecting the normal progress of the training task due to missed notification. By clearly defining the alarm strategy and notification method, each user group can obtain accurate and timely alarm information so as to make corresponding decisions.

[0088] Optionally, the method further comprises: After allocating corresponding computing resources to the multiple user groups, allocating the training tasks created by the user groups on the deep learning training platform to the computing resources; When the training task of the user group is completed before the end of the validity period, the computing resources actively released by the user group are received.

[0089] Specifically, in another embodiment, on a deep learning training platform, the system will allocate computing resources (such as GPU, TPU, etc.) to multiple user groups. Each user group can obtain the required resources according to its own needs. After obtaining the corresponding computing resources, each user group will create their deep learning training tasks on the platform. This process involves defining models, setting hyperparameters, selecting data sets, etc., and the created training tasks will be assigned to the allocated computing resources for execution. If a user group completes their training tasks before the expiration of the computing resources, the allocated computing resources can be actively released at this time. These returned computing resources can be applied for and used by other user groups.

[0090] Through the above embodiments, the user group can obtain computing power when needed and ensure that no resources are wasted after completing the task, thereby improving overall efficiency.

[0091] Figure 5 is a schematic diagram of a resource management device provided by an embodiment of the present application. Figure 5 shown.

[0092] Based on the same inventive concept, another embodiment of the present application further provides a resource management device, which is applied to a deep learning training platform, and the device includes: An allocation module 11 is used to allocate corresponding computing resources to multiple user groups, and set corresponding validity periods for the computing resources allocated to the multiple user groups, so that the user groups use the computing resources to run the training tasks of the deep learning training platform before the expiration of the validity period; The recycling module 12 is used to perform a recycling operation on the computing power resources when the validity period of the computing power resources expires, and to release the allocation relationship between the user group and the recycled computing power resources.

[0093] Optionally, the allocation module 11 includes: A label setting unit is used to set a permission usage label for the computing power resource; the permission usage label is divided into a first label, a second label and a third label; the validity period corresponding to the first label indicates that the computing power resource is not subject to time restrictions, the computing power resource with the permission usage label of the first label will be actively released after the user group has used it, the validity period corresponding to the second label indicates that the computing power resource is subject to time restrictions, and the third label indicates that the validity period of the computing power resource is the validity period of the last time the user group used the computing power resource; An adding unit, used to add the permission usage label and the corresponding validity period to the permission configuration information corresponding to the user group; The device also includes: an alarm module, which is used to send alarm information to multiple user groups allocated with the computing power resources before the validity period of the computing power resources expires, so as to prompt the user groups to renew the computing power resources or adjust the usage plan of the computing power resources.

[0094] Optionally, the recovery module 12, comprising; A traversal unit, configured to traverse the plurality of computing resource usage data corresponding to the plurality of user groups allocated with the computing resource in a loop, and determine the permission usage label recorded in the permission configuration information; A first execution unit, configured to not perform a recycling operation on the computing power resource when the permission usage tag of the computing power resource of the user group is the first tag; A comparing unit, configured to compare the validity period of the computing power resource with the current time when the permission usage tag of the computing power resource of the user group is the second tag or the third tag; The second execution unit is used to perform a recycling operation on the computing power resources when the validity period of the computing power resources is not less than the current time, so that the recycled computing power resources can be used for reallocation.

[0095] Optionally, the device further comprises: A judgment module, used for judging whether there is a user in the user group who is using the computing power resources to run the training task of the deep learning training platform when the validity period of the computing power resources expires before executing the recycling operation on the computing power resources; The recycling module 12 includes: a waiting unit, which is used to wait for the user in the user group to complete the training task of the deep learning training platform by using the computing resources, and to perform a recycling operation on the computing resources; A direct recycling module is used to directly perform a recycling operation on the computing power resources when there is no user in the user group who is using the computing power resources to run the training task of the deep learning training platform.

[0096] Optionally, the device further comprises: A resource group classification unit, for allocating corresponding computing resources to the multiple user groups and setting corresponding validity periods for the computing resources allocated to the multiple user groups, to classify all computing resources in the deep learning training platform into multiple resource groups according to different types, each resource group containing computing resources of the same type; The allocation module 11 includes: an allocation unit, which is used to determine multiple types of computing resources that meet the screening conditions from multiple resource groups and allocate them to the user groups based on the requirements of the training tasks that different user groups need to run in the deep learning training platform and the permissions of different user groups to different types of computing resources as screening conditions; The allocation unit includes: a validity period setting unit, which is used to set corresponding validity periods for multiple types of computing resources allocated to the user group according to the estimated completion time of the training tasks that the user group needs to run in the deep learning training platform, the size of the computing resources, the performance of the computing resources, the historical computing resource usage records of the user group, the remaining computing resources in the resources, and the budget of the user group.

[0097] Optionally, the device further comprises: An alarm setting unit, used to set an alarm reminder strategy and a notification method of the alarm information for different types of computing resources allocated to different user groups before sending the alarm information to the multiple user groups allocated the computing resources; An alarm unit, configured to send corresponding alarm information to the user group to which the computing power resources are allocated, according to the alarm reminder strategy and the notification method of the alarm information before the validity period of the computing power resources expires; The alarm reminder strategy includes the time to start the alarm and the number of times the alarm information is sent, and the notification method of the alarm information includes in-platform station letter notification and email notification.

[0098] Optionally, the device further comprises: A training task allocation unit, configured to allocate the training tasks created by the user groups on the deep learning training platform to the computing resources after allocating corresponding computing resources to the multiple user groups; A receiving unit is used to receive computing resources actively released by the user group when the training task of the user group is completed before the end of the validity period.

[0099] Based on the same inventive concept, another embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the resource management method as described in any of the above embodiments.

[0100] Among them, electronic equipment refers to Figure 6 , Figure 6 is a schematic diagram of an electronic device provided by an embodiment of the present application. Figure 6 As shown, the electronic device 600 includes: a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus communication. A computer program is stored in the memory 610. The computer program can be run on the processor 620 to implement the steps in the resource management method disclosed in the above embodiment of the present application.

[0101] Based on the same inventive concept, another embodiment of the present application further provides a computer program product, including a computer program, which is executed by a processor to implement the resource management method described in any of the above embodiments.

[0102] Based on the same inventive concept, another embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein when the program is executed by a processor, the resource management method as described in any of the above embodiments is implemented.

[0103] As for the device, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0104] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0105] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0109] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0110] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0111] The resource management method, device, equipment and medium provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A resource management method, characterized in that: Applied to a deep learning training platform, the method includes: For multiple user groups, corresponding computing resources are allocated to the multiple user groups, and corresponding validity periods are set for the computing resources allocated to the multiple user groups, so that the user groups use the computing resources to run training tasks of the deep learning training platform before the expiration of the validity period; When the validity period of the computing power resources ends, a recycling operation is performed on the computing power resources, and the allocation relationship between the user group and the recycled computing power resources is released.

2. The resource management method according to claim 1, characterized in that: The respectively setting corresponding validity periods for the computing resources allocated to the multiple user groups includes: A permission usage tag is set for the computing power resource; the permission usage tag is divided into a first tag, a second tag and a third tag; the validity period corresponding to the first tag indicates that the computing power resource is not subject to time restrictions, and the computing power resource with the permission usage tag of the first tag will be actively released after the user group has used it; the validity period corresponding to the second tag indicates that the computing power resource is subject to time restrictions, and the third tag indicates that the validity period of the computing power resource is the validity period of the last time the user group used the computing power resource; Adding the permission usage label and the corresponding validity period to the permission configuration information corresponding to the user group; The method also includes: before the validity period of the computing power resource expires, sending an alarm message to a plurality of user groups to which the computing power resource is allocated, so as to prompt the user groups to renew the computing power resource or adjust the usage plan of the computing power resource.

3. The resource management method according to claim 2, characterized in that: Performing a recovery operation on the computing power resource includes: Loop through the plurality of computing resource usage data corresponding to the plurality of user groups allocated with the computing resource, and determine the permission usage label recorded in the permission configuration information; When the permission usage tag of the computing power resource of the user group is the first tag, no recovery operation is performed on the computing power resource; When the permission usage tag of the computing power resource of the user group is the second tag or the third tag, comparing the validity period of the computing power resource with the current time; When the validity period of the computing power resources is not less than the current time, a recycling operation is performed on the computing power resources so that the recycled computing power resources are used for reallocation.

4. The resource management method according to claim 1, characterized in that: Before executing the recycling operation on the computing power resource, the method further includes: When the validity period of the computing power resources expires, determining whether there is a user in the user group who is using the computing power resources to run a training task on the deep learning training platform; The performing of the recycling operation on the computing power resources includes: when there is a user in the user group who is using the computing power resources to run a training task of the deep learning training platform, waiting for the training task of the user to be completed, performing the recycling operation on the computing power resources; When there is no user in the user group who is using the computing power resources to run the training task of the deep learning training platform, the computing power resources are directly recycled.

5. The resource management method according to claim 1, characterized in that: Before allocating corresponding computing resources to the multiple user groups and setting corresponding validity periods for the computing resources allocated to the multiple user groups, the method further includes: Divide all computing resources in the deep learning training platform into multiple resource groups according to different types, each resource group containing computing resources of the same type; The allocating corresponding computing resources to the multiple user groups includes: using the requirements of training tasks that need to be run by different user groups in the deep learning training platform and the permissions of different user groups to different types of computing resources as screening conditions, and determining from the multiple resource groups multiple types of computing resources that meet the screening conditions and allocating them to the user groups; The respectively setting of corresponding validity periods for the computing power resources allocated to the multiple user groups includes: setting corresponding validity periods for the multiple types of computing power resources allocated to the user group according to the estimated completion time of the training tasks that the user group needs to run in the deep learning training platform, the size of the computing power resources, the performance of the computing power resources, the historical computing power resource usage records of the user group, the remaining computing power resources in the resources, and the budget of the user group.

6. The resource management method according to claim 2, characterized in that: Before sending the alarm information to the multiple user groups to which the computing resources are allocated, the method further includes: Set alarm reminder strategies and alarm information notification methods for different types of computing resources allocated to different user groups; Before the validity period of the computing power resource expires, according to the alarm reminder strategy and the notification method of the alarm information, sending corresponding alarm information to the user group to which the computing power resource is allocated; The alarm reminder strategy includes the time to start the alarm and the number of times the alarm information is sent, and the notification method of the alarm information includes in-platform station letter notification and email notification.

7. The resource management method according to any one of claims 1 to 6, characterized in that: The method further comprises: After allocating corresponding computing resources to the multiple user groups, allocating the training tasks created by the user groups on the deep learning training platform to the computing resources; When the training task of the user group is completed before the end of the validity period, the computing resources actively released by the user group are received.

8. A resource management device, characterized in that: Applied to a deep learning training platform, the device comprises: An allocation module, for allocating corresponding computing resources to multiple user groups, and setting corresponding validity periods for the computing resources allocated to the multiple user groups, so that the user groups use the computing resources to run the training tasks of the deep learning training platform before the expiration of the validity periods; The recycling module is used to perform a recycling operation on the computing power resources when the validity period of the computing power resources expires, and to release the allocation relationship between the user group and the recycled computing power resources.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the resource management method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the resource management method according to any one of claims 1 to 7 is implemented.

Images