Cluster GPU utilization management method and device, computer equipment and storage medium

Abstract

The application belongs to the field of software monitoring and finance, and relates to a cluster GPU utilization management method, which comprises the following steps: dividing a cluster into namespaces, setting a business GPU resource quota for each business namespace and a public GPU resource quota for a public namespace, allocating GPU resources to each business namespace, collecting resource monitoring data of the business GPU resources of each business namespace in real time, recycling the resources of an idle business namespace to the public namespace when the idle business namespace exists, and allocating the GPU resources in the public namespace to a target business namespace according to a GPU resource application, a current resource use condition and a preset resource allocation priority rule. The application also provides a cluster GPU utilization management device, a computer device and a storage medium. In addition, the application also relates to blockchain technology, and the resource monitoring data can be stored in a blockchain. The application can dynamically schedule GPU resources and maximize the utilization rate of the GPU resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software monitoring and financial technology, and particularly relates to a GPU utilization rate management method and device for a cluster, computer equipment and a storage medium. BACKGROUND

[0002] In the financial field, with the rapid development of artificial intelligence and big data technology, GPU (Graphics Processing Unit) has become a key computing resource for processing complex financial models, big data analysis, and real-time transactions. However, traditional GPU resource management methods face many challenges in financial applications.

[0003] Firstly, the stability and efficiency of computing resources are highly required in the financial field, and traditional methods often fail to maximize the utilization of GPU resources, resulting in waste of GPU resources and low efficiency. This not only increases the operating costs of financial institutions, but also limits the processing speed and accuracy of financial businesses.

[0004] To address the above problems, some solutions have been proposed in existing technologies, but there are still deficiencies in their application in the financial field. For example, the multi-type GPU cluster resource management and scheduling method and system for deep learning proposed in Chinese patent CN201910430474.7 can divide the GPU cluster into several GPU groups according to the GPU model, obtain the highest priority job in the job queue, and schedule the GPU cluster resources according to the to-be-processed job. Although this method improves the flexibility of resource scheduling to some extent, it still cannot achieve complete isolation and dynamic scheduling of GPU resources, which may pose a security risk when processing sensitive financial data, and the further improvement of resource utilization is limited. On the other hand, patent CN202211442603.4 proposes a multi-tenant-based Kubernetes cluster resource management method and system, which creates the ID and namespace of a tenant through resource quota information and pre-configured quota strategies. Although this method has made some progress in resource management and multi-tenant support, it fails to fully utilize Kubernetes technology for dynamic scheduling between different namespaces, and also fails to comprehensively manage GPU resources by combining multiple monitoring indicators, thereby failing to meet the demand for fine-grained management and efficient utilization of resources in the financial field. In addition, the cluster resource management and scheduling method proposed in patent CN202110224312.0 can divide the cluster resources into multiple resource pools to achieve task isolation. However, this method does not use namespaces to group and manage virtualized resources, resulting in a complex and tedious process of scheduling grouped resources and adjusting the number of each grouped resource, which cannot adapt to the rapidly changing business demands in the financial field.

[0005] Generally, the existing technology cannot effectively improve the utilization rate of cluster GPU while reducing the cost, and cannot realize dynamic calling through a public name space while realizing resource grouping isolation, and cannot realize resource management of the whole cluster through a combination of various monitoring indicators, thereby improving the utilization rate of GPU resources while reducing the cost. SUMMARY

[0006] Embodiments of the present application aim to provide a cluster GPU utilization management method and device, computer equipment and storage medium, to solve the technical problem that the existing technology cannot simultaneously consider cost reduction, resource grouping isolation and resource dynamic scheduling in GPU cluster resource management.

[0007] To solve the above technical problems, the embodiments of the present application provide a cluster GPU utilization management method, which adopts the following technical solutions:

[0008] Determine a namespace division mode according to a preset division condition, divide a namespace for the cluster based on the namespace division mode, and the namespace includes a plurality of business namespaces and a public namespace;

[0009] Set a business GPU resource quota for each business namespace, and obtain a public GPU resource quota for the public namespace according to the business GPU resource quota;

[0010] Allocate GPU resources for each business namespace according to the business GPU resource quota and the public GPU resource quota, and obtain business GPU resources;

[0011] Real-time collection of resource monitoring data of business GPU resources of each business namespace;

[0012] According to the resource monitoring data and a preset scheduling strategy, it is judged whether there is an idle business namespace, if there is, a resource recycling process is triggered, and the business GPU resources of the idle business namespace are recycled to the public namespace;

[0013] Receive a GPU resource application request of a target business namespace, judge whether the allocation condition is met according to the application information in the GPU resource application request, the current GPU resource usage data and the preset resource allocation priority rule, if the allocation condition is met, allocate the GPU resources in the public namespace to the target business namespace.

[0014] To solve the above technical problems, the embodiments of the present application also provide a cluster GPU utilization management device, which adopts the following technical solutions:

[0015] a division module configured to determine a namespace division manner according to a preset division condition, and divide namespaces for the cluster based on the namespace division manner, the namespaces including a plurality of service namespaces and a common namespace;

[0016] a setting module configured to set a service GPU resource quota for each of the service namespaces, and obtain a common GPU resource quota for the common namespace according to the service GPU resource quota;

[0017] a resource allocation module configured to allocate GPU resources for each of the service namespaces according to the service GPU resource quota and the common GPU resource quota, to obtain service GPU resources;

[0018] a collection module configured to collect resource monitoring data of the service GPU resources of each of the service namespaces in real time;

[0019] a resource recycling module configured to determine whether there is an idle service namespace according to the resource monitoring data and a preset scheduling strategy, and trigger a resource recycling process to recycle the service GPU resources of the idle service namespace into the common namespace if there is the idle service namespace;

[0020] a resource scheduling module configured to receive a GPU resource application request of a target service namespace, determine whether an allocation condition is met according to application information in the GPU resource application request, current GPU resource usage data, and a preset resource allocation priority rule, and allocate GPU resources in the common namespace to the target service namespace if the allocation condition is met.

[0021] To solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the technical scheme as follows:

[0022] The computer device includes a memory and a processor, the memory stores computer readable instructions, and the processor executes the computer readable instructions to realize the steps of the cluster GPU utilization management method.

[0023] To solve the above technical problems, the embodiment of the present application further provides a computer readable storage medium, which adopts the technical scheme as follows:

[0024] The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to realize the steps of the cluster GPU utilization management method.

[0025] Compared with the prior art, the present application has the following beneficial effects:

[0026] The application provides a cluster GPU utilization management method, which realizes mutual isolation of resources by dividing a cluster into namespaces according to preset division conditions; sets a business GPU resource quota for each business namespace, obtains a common GPU resource quota of a common namespace according to the business GPU resource quota, and allocates GPU resources for each business namespace according to the business GPU resource quota and the common GPU resource quota, so that resources can be reasonably allocated and the utilization rate of GPU resources can be improved; resource monitoring data of business GPU resources of each business namespace is collected in real time, and according to the resource monitoring data and a preset scheduling strategy, when there is an idle business namespace, a resource recycling process is triggered to recycle the business GPU resources of the idle business namespace into the common namespace, so that the resource quota can be dynamically adjusted, resource occupation of each namespace can be avoided, the utilization rate of GPU resources can be improved, and costs can be reduced; a GPU resource application request from a business namespace is received, and according to application information in the GPU resource application request, a current resource usage status and a preset resource allocation priority rule, it is judged whether the allocation condition is met, and when the allocation condition is met, GPU resources in the common namespace are allocated to the target business namespace of the application, so that GPU resources can be dynamically scheduled and the utilization rate of GPU resources can be maximized. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the schemes in the application, the drawings needed in the description of the embodiments of the application will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0028] Figure 1 is an exemplary system architecture diagram to which the application can be applied;

[0029] Figure 2 is a flowchart of one embodiment of the cluster GPU utilization management method according to the application;

[0030] Figure 3 is a structural schematic diagram of one embodiment of the cluster GPU utilization management device according to the application;

[0031] Figure 4 is a structural schematic diagram of one embodiment of the computer device according to the application. DETAILED DESCRIPTION

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terms used in the specification are intended to describe the particular embodiments and are not intended to limit the application; the terms "include" and "have" and their any variations used in the specification and the claims and the above description of drawings are intended to cover the non-exclusive inclusion; the terms "first", "second" and the like used in the specification and the claims and the above description of drawings are intended to distinguish different objects, not to describe a particular order.

[0033] Reference herein to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. It is expressly understood that the embodiments described herein are merely examples and are not intended to limit the scope of the application.

[0034] In order to make the person skilled in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings below.

[0035] As shown in Figure 1 The system architecture 100 can include a terminal device 101, a network 102 and a server 103, and the terminal device 101 can be a notebook computer 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0036] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0037] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing, in addition to the notebook computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an electronic book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0038] The server 103 can be a server providing various services, for example, a background server providing support for a page displayed on the terminal device 101.

[0039] It should be noted that the cluster GPU utilization management method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the cluster GPU utilization management apparatus is generally provided in the server / terminal device.

[0040] It should be understood that, Figure 1 The number of terminal devices, networks and servers in

[0041] With reference to Figure 2 , a flowchart of one embodiment of the cluster GPU utilization management method according to the present application is shown, including the following steps:

[0042] Step S201, determining a namespace division manner according to a preset division condition, dividing a namespace for the cluster based on the namespace division manner, the namespace including a plurality of service namespaces and a common namespace.

[0043] In the present embodiment, the cluster is a k8s cluster, the K8s cluster is composed of a group of physical or virtual machines, these machines are organized into a single computing resource pool and run the Kubernetes platform thereon. The main purpose of the K8s cluster is to provide high availability, scalability and fault tolerance to ensure that the application can always run normally.

[0044] The preset division condition includes division according to one or more combinations of organizational structure, business classification, function, and purpose. The namespace division manner is determined according to the preset division condition, including a namespace name, a resource type, a resource limit, and access authority. The K8s cluster is divided into n business namespace groups and a public namespace group based on the namespace division manner, wherein the n business namespaces include nameSpace1, nameSpace2, nameSpace3,..., and nameSpace n wherein n is a positive integer; and the public namespace is public nameSpace. The identification information and division condition of each namespace (including the plurality of business namespaces and the public namespace) are recorded in a resource management database. A resource quota mechanism is used to set a business GPU resource quota for each business namespace, and the quota information is stored in the resource management database.

[0045] The resource quota mechanism (Resource Quotas) is a mechanism for limiting resource usage in a namespace in Kubernetes, which can help the cluster management module (K8s Cluster) to control resource usage and avoid a single namespace or user from occupying too many resources and affecting the performance of other users or applications.

[0046] In this embodiment, the management module K8s Cluster is used to uniformly manage the n different nameSpace groups and the public nameSpace group, so as to realize resource isolation. For example, according to function division, the k8s cluster can be divided into different namespace groups such as research and development, testing, and production, such as nameSpace1 for research and development environment, nameSpace2 for testing environment, nameSpace3 for production environment, and the public nameSpace group for resource scheduling and recycling.

[0047] For example, the GPU k8s cluster is divided into a training cluster and a low-priority training and debugging function cluster according to purpose, and the cluster of the public nameSpace group is a low-priority training and debugging cluster, and the nameSpace of each group is a training purpose cluster. The cluster of the public nameSpace group can be further divided into a low-priority training cluster and a debugging function cluster according to purpose, or distinguished by namespace, for example, public nameSpace_train is a training cluster, and public nameSpace_debug is a debugging function cluster. Of course, other methods can be used to distinguish clusters with different purposes in public, such as adding the train or debug prefix in the name, which is not limited here.

[0048] In step S202, a service GPU resource quota is set for each service namespace, and a public GPU resource quota of the public namespace is obtained according to the service GPU resource quota.

[0049] Specifically, historical resource usage data and service priority information of each service namespace are obtained, and a weight coefficient of each service namespace is obtained according to the service priority information; based on the historical resource usage data and the weight coefficient, a resource usage growth rate of each service namespace in a preset time period is predicted; according to the resource usage growth rate and the weight coefficient, an optimization algorithm is used to calculate a maximum GPU quota of each service namespace; a minimum GPU quota of each service namespace is determined based on the maximum GPU quota; the maximum GPU quota and the minimum GPU quota are taken as the service GPU resource quota of the corresponding service namespace; and a public GPU resource quota is calculated according to the maximum GPU quota and the minimum GPU quota.

[0050] The historical resource usage data includes time series data such as GPU vacancy rate, CPU usage rate, and memory usage rate. According to the service priority information of each service namespace, different weight coefficients are given, and the weight coefficient of a service namespace with high priority is large. For example, according to the service priority, the service namespaces are divided into three priority levels: high, medium, and low, and the weight coefficients are 5, 3, and 1, respectively.

[0051] In this embodiment, a time series prediction algorithm is used to predict the resource usage growth rate of each service namespace in a future period (i.e., a preset time period, such as a week, a month, or half a year) according to the historical resource usage data and the weight coefficient of each service namespace. The time series prediction algorithm can use a trained LSTM time series prediction model for prediction.

[0052] In this embodiment, the step of calculating the maximum GPU quota of each service namespace using an optimization algorithm includes:

[0053] Through a clustering algorithm, the resource usage data of each service namespace is clustered, and service namespaces with similar resource usage modes are divided into the same category. In each cluster, according to the predicted resource usage growth rate and the weight coefficient, a MILP (mixed integer linear programming) model is constructed with the objective function being to minimize the total GPU quota and the constraint condition being the resource usage growth rate and the weight coefficient of each service namespace, and the optimal GPU quota allocation of the service namespace, i.e., the maximum GPU quota, is obtained by solving. The minimum GPU quota = maximum GPU quota * preset proportion, wherein the preset proportion can be set according to actual conditions, for example, the minimum GPU quota is set to 70% of the maximum GPU quota.

[0054] In the embodiment, the quota information is stored into the preset resource quota database according to a predefined data structure, wherein the quota information contains fields such as namespace identifier, maximum GPU quota, minimum GPU quota and current GPU quota. It should be understood that the GPU quota is the number of GPUs that can be applied for or used by the namespace.

[0055] The maximum GPU quota and the minimum GPU quota of each service namespace are calculated, and the common GPU resource quota of the common namespace can be inferred based on the maximum GPU quota and the minimum GPU quota.

[0056] For example, assuming that there are 3 nameSpaces, n = 3, G1_high = 16, G2_high = 14, G3_high = 10, G1_low = 16 * 70% = 11, G2_low = 14 * 70% = 9, G3_low = 10 * 70% = 7, and the common GPU resource quota of the common namespace is G p = G 总 -G_low 总 = (16 + 14 + 10) - (11 + 9 + 7) = 13.

[0057] If the calculated number of GPUs has a decimal, it can be rounded down. It should be noted that the application does not limit the method of rounding off the number of GPUs with decimals, and other methods can be used.

[0058] By comprehensively considering the historical resource usage data of the service namespaces, the service priority information and the resource usage growth rate to calculate the service GPU resource quota and the common GPU resource quota, the GPU resources can be reasonably allocated, the idle waste of GPU resources can be minimized while meeting the service demand of each service namespace, and the efficient use of GPU resources in the entire cluster can be realized.

[0059] In step S203, GPU resources are allocated to each service namespace according to the service GPU resource quota and the common GPU resource quota, and the service GPU resources are obtained.

[0060] Specifically, after receiving the application request of the service namespace for the GPU resources, the number of GPUs applied for by the service namespace is obtained according to the application request; if the number of GPUs applied for is greater than the maximum GPU quota, the application request is rejected, and the GPU resources are allocated according to the maximum GPU quota; if the number of GPUs applied for is less than the minimum GPU quota, it is determined whether the idle common GPU resource quota in the common namespace is greater than the minimum GPU quota; when the common GPU resource quota is greater than the minimum GPU quota, the idle common GPU resources in the common namespace are dispatched to the corresponding service namespace.

[0061] The application request is a request for application of GPU resources by each business namespace before the task starts.

[0062] In this embodiment, upon receiving the application request for GPU resources by the business namespace, the quota controller of the k8s cluster obtains the GPU quota information of the business namespace, and performs quota control and allocation on the GPU resources applied by the business namespace according to the GPU quota information. The application request is parsed to obtain the GPU application quantity of the business namespace; if the GPU application quantity exceeds the maximum GPU quota, the quota controller rejects the application request of the business namespace; if the GPU application quantity is lower than the minimum GPU quota, idle common GPU resources in the common namespace are scheduled and allocated to the business namespace until the GPU resources of the business namespace reach the minimum GPU quota thereof.

[0063] Before scheduling and allocating the idle common GPU resources in the common namespace to the business namespace, it is determined whether the available idle common GPU resources in the common namespace are sufficient for scheduling. If the GPU resources of the common namespace are insufficient, the resource expansion module automatically purchases GPU resources according to the usage demand, and adds the newly added GPU resources to the resource pool of the common namespace. The resource quota management module can periodically compare the resource allocation and the quota setting, and dynamically adjust the GPU resource quota of each business namespace according to the actual usage and the change of business demand, update the resource quota database, realize the elastic scaling and dynamic balance of the GPU resources, and improve the utilization rate of the GPU resources.

[0064] In some optional implementations, if the GPU resources of the common namespace are insufficient, before purchasing the GPU resources, the k8s Cluster management module is used to initiate an inspection, and GPU resources are coordinated to exit from the common namespace. The exit order can be sorted according to the priority of the tasks running in the common namespace, and the resources of the tasks with low priority are preferentially allocated to each business namespace for use. If the GPU resource utilization rate is very high, resulting in a small number of GPU in the common namespace group, additional GPU resources are purchased.

[0065] By monitoring the application request of the GPU resources of the business namespace, the resource allocation can be optimized in real time according to the change of the GPU resource usage of each business namespace, so as to improve the utilization rate of the GPU resources and avoid waste of the GPU resources.

[0066] In step S204, the resource monitoring data of the business GPU resources of each business namespace is collected in real time.

[0067] In the embodiment, the resource monitoring system collects the GPU resource usage of each business namespace in real time, including GPU utilization, CPU usage, memory usage, disk usage and other resource usage indicators as monitoring indicators, and the resource usage data corresponding to the monitoring indicators are resource monitoring data. The collected resource monitoring data is stored in the monitoring database.

[0068] Real-time collection of resource monitoring data of each namespace can realize real-time monitoring of resource usage of each namespace, which helps to dynamically optimize resource allocation and improve resource utilization.

[0069] In the embodiment, different monitoring indicators are used to manage clusters with different purposes. For example, indicator 1 is used for training clusters, such as GPU vacancy rate r1, CPU usage r2, etc. Indicator 2 is used for clusters with debugging functions, such as memory usage r3, disk usage r4, and off-duty time t1.

[0070] It should be emphasized that, in order to further ensure the privacy and security of the resource monitoring data, the above resource monitoring data can also be stored in a node of a blockchain.

[0071] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. Blockchain, in essence, is a decentralized database, which is a series of data blocks associated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity (anti-fake) of the information and generate the next block. Blockchain can include blockchain underlying platform, platform product service layer and application service layer, etc.

[0072] In step S205, according to the resource monitoring data and the preset scheduling strategy, it is judged whether there is an idle business namespace, and if there is, the resource recycling process is triggered to recycle the business GPU resources of the idle business namespace into the common namespace.

[0073] Specifically, the resource monitoring data is compared with the GPU resource quota information of the business namespace to obtain the GPU resource usage rate and the GPU resource idle rate of the business namespace; when there is a GPU resource usage rate less than a preset usage rate threshold and a GPU resource idle rate higher than a preset idle rate threshold, the corresponding business namespace is determined as an idle business namespace; priority information of tasks running in the idle business namespace is obtained, the tasks are sorted from low to high according to the priority information to obtain a task sorting result; the task sorting result is traversed from the task with the lowest priority, and it is judged whether scheduling the business GPU resources occupied by the currently traversed task to the common namespace will exceed the upper limit of the common GPU resource quota of the common namespace; if not, the business GPU resources occupied by the currently traversed task are scheduled to the common namespace, and the GPU resource usage data of the corresponding business namespace is updated; until all tasks of the idle business namespace are checked or the GPU resources of the common namespace have reached the quota upper limit, the resource recycling process is completed.

[0074] By the preset usage rate threshold and the preset idle rate threshold, it is judged whether the GPU resources of each business namespace are idle, and when the GPU resource usage rate is less than the preset usage rate threshold or the GPU resource idle rate is higher than the preset idle rate threshold, the resource recycling process is triggered. The GPU resource information of the idle business namespace, including the number and model of GPUs, is obtained, and these idle GPU resources are temporarily recycled to the resource pool of the common namespace. The dynamic scheduling system allocates the idle GPU resources from the resource pool of the common namespace to the tasks according to the priority, required GPU resource type and number of the tasks, so as to improve the GPU resource utilization rate.

[0075] In the embodiment, all idle business namespaces are determined, the idle business namespaces are sorted from low to high according to the GPU resource usage rate, and the priority information of the tasks running in the idle business namespaces is obtained, the tasks are sorted from low to high according to the priority, and the tasks are traversed from the idle business namespace with the lowest GPU resource usage rate in the order from low to high according to the priority, it is judged whether scheduling the GPU resources occupied by the current task to the common namespace will exceed the upper limit of the GPU quota of the common namespace, if not, the GPU resources occupied by the task are scheduled to the common namespace, and the GPU resource usage data of the business namespace is updated, and the scheduling record is written into a log database for subsequent analysis and optimization of resource utilization rate.

[0076] By recycling the GPU resources of the idle service namespace into the resource pool of the common namespace, the GPU resource utilization can be maximized, the resources of each service namespace can be avoided from being occupied, the GPU resource utilization of each service namespace and the overall GPU resource utilization can be improved, and the cost can be saved.

[0077] In step S206, a GPU resource application request of a target service namespace is received, and whether the allocation condition is met is judged according to the application information in the GPU resource application request, the current GPU resource usage data and the preset resource allocation priority rule. If the allocation condition is met, the GPU resources in the common namespace are allocated to the target service namespace.

[0078] Specifically, the current GPU resource usage data of the target service namespace and the application information in the GPU resource application request are obtained. The application information includes the number of currently applied GPUs, the usage duration and the priority parameter. According to the preset resource allocation priority rule and the current GPU resource usage data, the priority score of the target service namespace is calculated. It is judged whether the number of currently applied GPUs exceeds the highest GPU quota of the target service namespace. If it exceeds, the application is rejected and a prompt information is returned. It is judged whether the number of available GPUs in the common namespace meets the application requirement of the target service namespace. When the application requirement of the target service namespace is not met, the GPU resources are scheduled from the idle resources of other service namespaces to the common namespace according to the order from low to high of the priority scores. When the GPU resources in the common namespace still cannot meet the application requirement, the application is rejected and a prompt information of insufficient resources is returned. When the number of available GPUs in the common namespace meets the application requirement, the corresponding number of GPU resources are allocated from the common namespace to the target service namespace.

[0079] The target service namespace is a service namespace that currently applies for GPU resources. The GPU resource application request is a GPU resource application in the task execution process according to the actual usage and the change of business requirements. The preset resource allocation priority rule is a priority score calculation method. In this embodiment, a weighted average algorithm is adopted to calculate the priority score according to the current GPU resource usage data of the target service namespace and the priority parameter.

[0080] The number of available GPUs in the common namespace meets the application requirement of the target service namespace, which means that the number of available GPUs in the common namespace is greater than or equal to the number of GPUs applied by the target service namespace.

[0081] In some optional implementation manners, a corresponding number of GPU resources in the resource pool of the public namespace is allocated to the target business namespace of the application, while resource usage data is updated, that is, information such as the number of GPUs allocated this time, the usage duration, and the like is recorded into the statistical indicators of the corresponding business namespace and the public namespace, and information of successful GPU resource allocation is returned, including the number of GPUs allocated, the usage duration, the expiration time, and the like.

[0082] By dynamically scheduling GPU resources according to the resource allocation priority rules and the resource usage, the GPU resource utilization is maximized.

[0083] The application divides the cluster into namespaces to realize the mutual isolation of resources, allocates GPU resources to each business namespace according to the business GPU resource quota and the public GPU resource quota, can reasonably allocate resources, and improve the utilization of GPU resources, collects resource monitoring data of the business GPU resources of each business namespace in real time, according to the resource monitoring data and the preset scheduling strategy, triggers a resource recycling process when there is an idle business namespace, recycles the business GPU resources of the idle business namespace into the public namespace, can dynamically adjust the resource quota, avoids the occupation of resources by each namespace, improves the utilization of GPU resources, and reduces the cost, according to the application information in the GPU resource application request, the current GPU resource usage data, and the preset resource allocation priority rules, judges whether the allocation condition is met, and when the allocation condition is met, allocates the GPU resources in the public namespace to the target business namespace, can dynamically schedule GPU resources, and maximize the utilization of GPU resources.

[0084] In some optional implementation manners, the step of allocating the GPU resources in the public namespace to the target business namespace further comprises:

[0085] Periodically obtaining resource usage rate, quota usage rate, and business priority data of each business namespace;

[0086] According to the resource usage rate, the quota usage rate, and the business priority data, calculating the resource balance degree of each business namespace;

[0087] According to the resource balance degree, judging whether there is an unbalanced resource allocation situation;

[0088] If there is an unbalanced resource allocation situation, calculating the number of migrated GPU resources between different business namespaces according to the resource balancing algorithm;

[0089] According to the incremental migration mode, migrating the number of migrated GPU resources to the corresponding business namespace;

[0090] After the migration is completed, the resource usage and quota usage of each migrated business namespace are obtained as updated resource data;

[0091] According to the updated resource data, it is judged whether the resources of each business namespace reach a balanced state, and if not, the migration is continued until the resources reach balance.

[0092] The resource usage is the GPU usage. In this embodiment, a weighted average algorithm is used to calculate the resource balance degree of each business namespace. For example, the weights of the resource usage, the quota usage and the business priority data are 4, 3 and 3 respectively, and the resource balance degree = resource usage * 4 + quota usage * 3 + business priority * 3. If the calculated resource balance degree is greater than or equal to a preset balance threshold, it indicates that there is an imbalance in the allocation of resources in the business namespace, and a rebalancing process is triggered.

[0093] According to the resource balancing algorithm, the number of migrated GPU resources between different business namespaces is calculated. For example, 10% of the GPU resources are selected from the business namespaces with resource usage lower than 50%, quota usage lower than 60% and business priority medium or low, and are migrated to the business namespaces with resource usage higher than 80%, quota usage higher than 90% and business priority high.

[0094] In this embodiment, an incremental migration method is used, that is, a small part of GPU resources is migrated each time until the target migration number is reached, avoiding large-scale migration at one time and causing business interruption. After the resources reach balance, the migration result is synchronized to the K8s cluster management system, the GPU resource quota of the business namespace is updated, the cross-business namespace GPU resource rebalancing is completed, and the balancing process and result are recorded to provide data support for subsequent optimization of the resource balancing algorithm.

[0095] In some optional implementation manners, the resource monitoring component is used to monitor the GPU resource usage of each business namespace in real time, obtain the current GPU usage of each business namespace, and if the GPU usage of the business namespace exceeds the maximum GPU quota, an alarm is triggered and the over-limit event is recorded in the scheduling decision log.

[0096] The application realizes the migration of resources between business namespaces through resource balancing calculation, realizes dynamic balancing of resources, guarantees the consistency of the state of cluster GPU resources, and improves the resource utilization of the entire cluster.

[0097] In some optional implementation manners, the step of allocating the GPU resources in the common namespace to the target business namespace further comprises:

[0098] According to a preset idle resource monitoring period, periodically collect GPU resource usage monitoring index data of each business namespace;

[0099] Compare the GPU resource usage monitoring index data with a preset resource recovery trigger condition, and determine whether the resource recovery trigger condition is met;

[0100] When the resource recovery trigger condition is met, start a GPU resource dynamic expansion and contraction mechanism, and temporarily recover the idle GPU resources to the common namespace;

[0101] At the same time of recovering the GPU resources, update the resource management strategy of the common namespace, and adjust the resource allocation strategy according to the resource demand priority of the business namespace.

[0102] The preset idle resource monitoring period can be set according to actual conditions, for example, one week, half a month, one month, or half a year, etc., which is not limited here.

[0103] Specifically, the GPU resource usage monitoring index data of each business namespace in the k8s cluster is obtained, including GPU usage, CPU usage, memory usage, disk usage, etc.; according to the obtained GPU resource usage monitoring index data, the GPU resource utilization of each business namespace is determined to determine whether there is GPU resource vacancy or low utilization; if it is determined that the GPU usage of a certain business namespace is lower than the preset usage threshold, the resource recovery trigger condition is met, the business namespace is marked as a low utilization group, and the number of GPU resources currently used by the business namespace is obtained; by comparing the number of GPU resources marked as low utilization in the business namespace with its preset minimum GPU quota, the number of idle GPU resources that can be recovered is obtained, and the idle GPU resources are temporarily recovered to the common namespace.

[0104] At the same time of recovering the GPU resources, update the resource management strategy of the common namespace, increase the number of allocatable GPU resources, and adjust the resource allocation strategy according to the resource demand priority of the business namespace. When the actual GPU resource usage of the business namespace approaches its minimum GPU quota, dynamically schedule idle GPU resources from the common namespace to the business namespace. When scheduling GPU resources, the business namespace with high resource demand priority is preferentially satisfied. If the GPU resources of the common namespace are insufficient, GPU resources are recovered from the business namespace with low priority to the common namespace according to the usage priority. If the GPU resources of the common namespace are largely idle, according to the GPU vacancy rate threshold, part of the GPU resources are redistributed to each business namespace to improve the utilization rate of GPU resources.

[0105] By regularly monitoring the actual resource usage and resource demand changes of each business namespace, dynamic resource scheduling is realized, and resource management strategies are optimized according to the monitoring data, thereby continuously improving the GPU resource utilization.

[0106] Embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0107] Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. The program can include the processes of the above-mentioned embodiments when executed, wherein the storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0109] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0110] Further referring to Figure 3 , as an implementation of the method shown in Figure 2 , the present application provides an embodiment of a cluster GPU utilization management device, and the device embodiment is used for Figure 2The method embodiment shown corresponds to the device, which can be specifically applied to various electronic devices.

[0111] As shown in Figure 3 The cluster GPU utilization management device 300 described in the embodiment includes a division module 301, a setting module 302, a resource allocation module 303, a collection module 304, a resource recycling module 305, and a resource scheduling module 306. Among them:

[0112] The division module 301 is configured to determine a namespace division manner according to a preset division condition, divide a namespace for a cluster based on the namespace division manner, and the namespace includes a plurality of business namespaces and a public namespace;

[0113] The setting module 302 is configured to set a business GPU resource quota for each of the business namespaces, and obtain a public GPU resource quota of the public namespace according to the business GPU resource quota;

[0114] The resource allocation module 303 is configured to allocate GPU resources for each of the business namespaces according to the business GPU resource quota and the public GPU resource quota, and obtain business GPU resources;

[0115] The collection module 304 is configured to collect resource monitoring data of the business GPU resources of each of the business namespaces in real time;

[0116] The resource recycling module 305 is configured to determine whether there is an idle business namespace according to the resource monitoring data and a preset scheduling strategy, and if there is, trigger a resource recycling process to recycle the business GPU resources of the idle business namespace into the public namespace;

[0117] The resource scheduling module 306 is configured to receive a GPU resource application request of a target business namespace, determine whether an allocation condition is met according to application information in the GPU resource application request, current GPU resource usage data, and a preset resource allocation priority rule, and if the allocation condition is met, allocate GPU resources in the public namespace to the target business namespace.

[0118] It should be emphasized that, in order to further ensure the privacy and security of the resource monitoring data, the above resource monitoring data can also be stored in a node of a block chain.

[0119] Based on the cluster GPU utilization management device 300 described above, the resource isolation of each namespace is realized by dividing the cluster into namespaces; the GPU resources of each service namespace are allocated according to the service GPU resource quota and the public GPU resource quota, so that the resources can be reasonably allocated and the utilization rate of the GPU resources can be improved; the resource monitoring data of the service GPU resources of each service namespace are collected in real time, and according to the resource monitoring data and the preset scheduling strategy, when there is an idle service namespace, the resource recovery process is triggered to recover the service GPU resources of the idle service namespace to the public namespace, so that the resource quota can be dynamically adjusted, the resources of each namespace can be prevented from being occupied, the utilization rate of the GPU resources can be improved, and the cost can be reduced; according to the application information in the GPU resource application request, the current GPU resource usage data and the preset resource allocation priority rule, when the allocation condition is met, the GPU resources in the public namespace are allocated to the target service namespace, so that the GPU resources can be dynamically scheduled, and the utilization rate of the GPU resources can be maximized.

[0120] In some optional implementations, the setting module 302 includes:

[0121] The obtaining sub-module is configured to obtain historical resource usage data and service priority information of each service namespace, and obtain a weight coefficient of each service namespace according to the service priority information.

[0122] The prediction sub-module is configured to predict a resource usage growth rate of each service namespace in a preset time period based on the historical resource usage data and the weight coefficient.

[0123] The optimization calculation sub-module is configured to calculate a maximum GPU quota of each service namespace by using an optimization algorithm according to the resource usage growth rate and the weight coefficient.

[0124] The determination sub-module is configured to determine a minimum GPU quota of each service namespace based on the maximum GPU quota.

[0125] The quota sub-module is configured to take the maximum GPU quota and the minimum GPU quota as a service GPU resource quota of the corresponding service namespace.

[0126] The public resource calculation sub-module is configured to calculate a public GPU resource quota according to the maximum GPU quota and the minimum GPU quota.

[0127] The GPU resource is allocated reasonably by comprehensively considering the historical resource usage data and the business priority information of the business namespace and the resource usage growth rate to calculate the business GPU resource quota and the public GPU resource quota, idle waste of the GPU resource is minimized while meeting the business demand of each business namespace, and efficient utilization of the GPU resource in the whole cluster is realized.

[0128] In some optional implementation manners of the embodiment, the resource allocation module 303 comprises:

[0129] The receiving submodule is configured to receive an application request of the business namespace for the GPU resource, and obtain the GPU application quantity of the business namespace according to the application request.

[0130] The judging submodule is configured to, if the GPU application quantity is greater than the maximum GPU quota, reject the application request and allocate the GPU resource according to the maximum GPU quota.

[0131] The comparing submodule is configured to, if the GPU application quantity is less than the minimum GPU quota, determine whether the idle public GPU resource quota in the public namespace is greater than the minimum GPU quota.

[0132] The allocating submodule is configured to, if the public GPU resource quota is greater than the minimum GPU quota, schedule the idle public GPU resource in the public namespace to the corresponding business namespace.

[0133] The GPU resource is allocated reasonably by comprehensively considering the historical resource usage data and the business priority information of the business namespace and the resource usage growth rate to calculate the business GPU resource quota and the public GPU resource quota, idle waste of the GPU resource is minimized while meeting the business demand of each business namespace, and efficient utilization of the GPU resource in the whole cluster is realized.

[0134] In some optional implementation manners of the embodiment, the resource recycling module 305 comprises:

[0135] The comparing submodule is configured to compare the resource monitoring data with the GPU resource quota information of the business namespace to obtain the GPU resource usage rate and the GPU resource idle rate of the business namespace.

[0136] The idle resource judging submodule is configured to, when the GPU resource usage rate is less than a preset usage rate threshold value or the GPU resource idle rate is higher than a preset idle rate threshold value, determine that the corresponding business namespace is an idle business namespace.

[0137] The sorting submodule is configured to acquire priority information of tasks running in the idle service namespace, sort the tasks from low to high according to the priority information, and obtain a task sorting result.

[0138] The traversal submodule is configured to traverse the task sorting result from a task with the lowest priority, and determine whether scheduling service GPU resources occupied by the currently traversed task to the common namespace will exceed the upper limit of the common GPU resource quota of the common namespace.

[0139] The scheduling submodule is configured to, if the upper limit is not exceeded, schedule the service GPU resources occupied by the currently traversed task to the common namespace, and update GPU resource usage data of the corresponding service namespace.

[0140] The ending submodule is configured to complete the resource recycling process until all the tasks of the idle service namespace are checked or the GPU resources of the common namespace reach the upper limit of the quota.

[0141] By recycling the GPU resources of the idle service namespace into the resource pool of the common namespace, the GPU resource utilization rate can be maximized, the resources of each service namespace can be avoided from being occupied, the GPU resource utilization rate of each service namespace and the overall GPU resource utilization rate can be improved, and the cost can be saved.

[0142] In some optional implementation manners of the embodiment, the resource scheduling module 306 includes:

[0143] The information acquisition submodule is configured to acquire current GPU resource usage data of the target service namespace and application information in the GPU resource application request, the application information including a currently applied GPU quantity, a usage duration, and a priority parameter.

[0144] The priority calculation submodule is configured to calculate a priority score of the target service namespace according to the preset resource allocation priority rule and the current GPU resource usage data.

[0145] The first judgment submodule is configured to determine whether the currently applied GPU quantity exceeds a highest GPU quota of the target service namespace, and if so, reject the application and return a prompt information.

[0146] The second judgment submodule is configured to determine whether an available GPU quantity of the common namespace meets an application demand of the target service namespace.

[0147] The scheduling submodule is configured to, when the application demand of the target service namespace is not met, schedule GPU resources from idle resources of other service namespaces to the common namespace according to the order from low to high of the priority scores;

[0148] The rejection submodule is configured to, when the GPU resources in the common namespace still cannot meet the application demand, reject the application and return prompt information of insufficient resources.

[0149] The allocation submodule is configured to, when the available GPU quantity of the common namespace meets the application demand, allocate a corresponding quantity of GPU resources from the common namespace to the target service namespace.

[0150] By dynamically scheduling GPU resources according to the resource allocation priority rules and the resource usage, the GPU resource utilization is maximized.

[0151] In some optional implementations, the cluster GPU utilization management apparatus 300 further includes a resource balancing module configured to:

[0152] Periodically acquire resource usage, quota usage, and service priority data of each service namespace;

[0153] Calculate resource balancing degrees of each service namespace according to the resource usage, the quota usage, and the service priority data;

[0154] Determine whether there is an unbalanced resource allocation according to the resource balancing degrees;

[0155] If there is an unbalanced resource allocation, calculate a quantity of migrated GPU resources between different service namespaces according to a resource balancing algorithm;

[0156] Migrate the quantity of migrated GPU resources to the corresponding service namespaces in an incremental migration manner;

[0157] After the migration is completed, acquire resource usage and quota usage of each service namespace after the migration as updated resource data;

[0158] Determine whether the resources of each service namespace reach a balanced state according to the updated resource data, and if not, continue the migration until the resources reach a balanced state.

[0159] By performing resource balancing calculation on the service namespaces, the resource migration between the service namespaces is realized, the dynamic balancing of the resources is realized, the consistency of the cluster GPU resource state is ensured, and the resource utilization of the entire cluster is improved.

[0160] In some optional implementations, the cluster GPU utilization management apparatus 300 further comprises a monitoring and adjusting module configured to:

[0161] According to a preset idle resource monitoring period, periodically collect GPU resource usage monitoring index data of each service namespace;

[0162] Compare the GPU resource usage monitoring index data with a preset resource recovery trigger condition, and determine whether the resource recovery trigger condition is met;

[0163] When the resource recovery trigger condition is met, a GPU resource dynamic expansion and contraction mechanism is started to temporarily recover idle GPU resources to the common namespace;

[0164] While recovering the GPU resources, update the resource management strategy of the common namespace, and adjust the resource allocation strategy according to the resource demand priority of the service namespace.

[0165] By periodically monitoring the actual resource usage and resource demand changes of each service namespace, dynamic resource scheduling is realized, and the resource management strategy is optimized according to the monitoring data to continuously improve the GPU resource utilization.

[0166] To solve the above technical problems, the embodiment of the present application also provides a computer device. For details, please refer to Figure 4 , Figure 4 The basic structure block diagram of the computer device of the present embodiment is shown in the figure.

[0167] The computer device 4 comprises a memory 41, a processor 42 and a network interface 43 which are connected to each other through a system bus. It should be pointed out that only the computer device 4 with the memory 41, the processor 42 and the network interface 43 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), digital signal processor (DSP), embedded device, etc.

[0168] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The computer device can interact with a user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, or the like.

[0169] The memory 41 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, or the like), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, or the like. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. Of course, the memory 41 can include both an internal storage unit and an external storage device of the computer device 4. In this embodiment, the memory 41 is generally used to store an operating system and various application software installed in the computer device 4, such as computer readable instructions of the cluster GPU utilization management method, or the like. In addition, the memory 41 can also be used to temporarily store various data that has been output or will be output.

[0170] The processor 42 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer readable instructions or process data stored in the memory 41, such as computer readable instructions of the cluster GPU utilization management method.

[0171] The network interface 43 can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0172] By dividing the cluster into namespaces, the mutual isolation of resources is realized; according to the business GPU resource quota and the public GPU resource quota, the GPU resources are allocated to each business namespace, the resources can be reasonably allocated, and the utilization rate of the GPU resources is improved; the resource monitoring data of the business GPU resources of each business namespace is collected in real time, according to the resource monitoring data and a preset scheduling strategy, when there is an idle business namespace, a resource recycling process is triggered, the business GPU resources of the idle business namespace are recycled to the public namespace, the resource quota can be dynamically adjusted, the resources of each namespace are avoided to be occupied, the GPU resource utilization rate is improved, and the cost is reduced; according to the application information in the GPU resource application request, the current GPU resource use data and a preset resource allocation priority rule, when the allocation condition is met, the GPU resources in the public namespace are allocated to the target business namespace, the GPU resources can be dynamically scheduled, and the GPU resource utilization rate is maximized.

[0173] The application further provides another implementation, namely providing a computer readable storage medium, the computer readable storage medium stores computer readable instructions, the computer readable instructions can be executed by at least one processor, so that the at least one processor executes the steps of the management method of the cluster GPU utilization rate as described above.

[0174] By dividing the cluster into namespaces, the mutual isolation of resources is realized; according to the business GPU resource quota and the public GPU resource quota, the GPU resources are allocated to each business namespace, the resources can be reasonably allocated, and the utilization rate of the GPU resources is improved; the resource monitoring data of the business GPU resources of each business namespace is collected in real time, according to the resource monitoring data and a preset scheduling strategy, when there is an idle business namespace, a resource recycling process is triggered, the business GPU resources of the idle business namespace are recycled to the public namespace, the resource quota can be dynamically adjusted, the resources of each namespace are avoided to be occupied, the GPU resource utilization rate is improved, and the cost is reduced; according to the application information in the GPU resource application request, the current GPU resource use data and a preset resource allocation priority rule, when the allocation condition is met, the GPU resources in the public namespace are allocated to the target business namespace, the GPU resources can be dynamically scheduled, and the GPU resource utilization rate is maximized.

[0175] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present application.

[0176] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent replacements to some of the technical features. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A method for managing utilization of a cluster of GPUs, the method comprising: The method comprises the following steps: According to the preset division condition, determine the namespace division mode, and divide the namespace based on the namespace division mode, the namespace includes multiple business namespaces and a public namespace; Obtain the historical resource usage data and business priority information of each business namespace, and obtain the weight coefficient of each business namespace according to the business priority information; Based on the historical resource usage data and the weight coefficient, predict the resource usage growth rate of each business namespace in a preset time period; according to the resource usage growth rate and the weight coefficient, calculate the maximum GPU quota of each business namespace by using an optimization algorithm; Determine the minimum GPU quota of each business namespace based on the maximum GPU quota; take the maximum GPU quota and the minimum GPU quota as the business GPU resource quota of the corresponding business namespace; calculate the public GPU resource quota according to the maximum GPU quota and the minimum GPU quota; After receiving the application request of the business namespace for GPU resources, obtain the GPU application quantity of the business namespace according to the application request; if the GPU application quantity is greater than the maximum GPU quota, reject the application request and allocate GPU resources according to the maximum GPU quota; if the GPU application quantity is less than the minimum GPU quota, determine whether the idle public GPU resource quota in the public namespace is greater than the minimum GPU quota; when the idle public GPU resource quota is greater than the minimum GPU quota, schedule idle public GPU resources from the public namespace to the corresponding business namespace; Real-time collection of resource monitoring data of business GPU resources of each business namespace; According to the resource monitoring data and the preset scheduling strategy, determine whether there is an idle business namespace, if there is, trigger the resource recycling process, and recycle the business GPU resources of the idle business namespace to the public namespace; Receive the GPU resource application request of the target business namespace, and determine whether the allocation condition is met according to the application information in the GPU resource application request, the current GPU resource usage data and the preset resource allocation priority rule, if the allocation condition is met, allocate the GPU resources in the public namespace to the target business namespace.

2. The method of claim 1, wherein, The step of determining whether there is an idle business namespace according to the resource monitoring data and the preset scheduling strategy, if there is, triggering the resource recycling process, and recycling the business GPU resources of the idle business namespace to the public namespace, comprises: Compare the resource monitoring data with the GPU resource quota information of the business namespace to obtain the GPU resource usage rate and the GPU resource idle rate of the business namespace; When the GPU resource usage is less than a preset usage threshold or the GPU resource idle rate is higher than a preset idle rate threshold, the corresponding service namespace is determined as an idle service namespace; Priority information of tasks running in the idle service namespace is obtained, and the tasks are sorted from low to high according to the priority information to obtain a task sorting result; Starting from the task with the lowest priority, it is determined whether scheduling the service GPU resource occupied by the currently traversed task to the common namespace will exceed the upper limit of the common GPU resource quota of the common namespace; If the upper limit is not exceeded, the service GPU resource occupied by the currently traversed task is scheduled to the common namespace, and the GPU resource usage data of the corresponding service namespace is updated; Until all the tasks of the idle service namespaces are checked or the GPU resource of the common namespace reaches the quota upper limit, the resource recycling process is completed.

3. The method of claim 1, wherein, The step of determining whether the allocation condition is met according to the application information in the GPU resource application request, the current GPU resource usage data and a preset resource allocation priority rule, and if the allocation condition is met, allocating the GPU resource in the common namespace to the target service namespace includes: Obtaining the current GPU resource usage data of the target service namespace and the application information in the GPU resource application request, the application information including the currently applied GPU quantity, usage duration and priority parameter; According to the preset resource allocation priority rule and the current GPU resource usage data, calculating the priority score of the target service namespace; Determining whether the currently applied GPU quantity exceeds the highest GPU quota of the target service namespace, and if it exceeds, rejecting the application and returning a prompt message; Determining whether the available GPU quantity of the common namespace meets the application requirement of the target service namespace; When the application requirement of the target service namespace is not met, according to the order from low to high of the priority score, scheduling GPU resources from the idle resources of other service namespaces to the common namespace; When the GPU resource in the common namespace still cannot meet the application requirement, the application is rejected and a prompt message of insufficient resources is returned; When the available GPU quantity of the common namespace meets the application requirement, a corresponding quantity of GPU resources is allocated from the common namespace to the target service namespace.

4. The method of claim 1, wherein, After the step of allocating the GPU resource in the common namespace to the target service namespace, it further includes: Periodically obtaining resource usage rate, quota usage rate and service priority data of each service namespace; According to the resource usage rate, the quota usage rate and the service priority data, calculating the resource balance degree of each service namespace; According to the resource balance degree, it is determined whether there is an unbalanced resource allocation situation; If there is an unbalanced resource allocation, the number of migrated GPU resources between different service namespaces is calculated according to a resource balancing algorithm; The number of migrated GPU resources is migrated to the corresponding service namespace in an incremental migration manner; After migration is completed, the resource usage rate and quota usage rate of each migrated service namespace are obtained as updated resource data; According to the updated resource data, it is determined whether the resources of each service namespace reach a balanced state, and if not, migration is continued until resource balancing is achieved.

5. The method of claim 1, wherein, The step of allocating GPU resources in the common namespace to the target service namespace further comprises: According to a preset idle resource monitoring period, GPU resource usage monitoring index data of each service namespace is periodically collected; The GPU resource usage monitoring index data is compared with a preset resource recovery trigger condition to determine whether the resource recovery trigger condition is met; When the resource recovery trigger condition is met, a GPU resource dynamic expansion and contraction mechanism is started to temporarily recover idle GPU resources to the common namespace; At the same time of recovering GPU resources, the resource management strategy of the common namespace is updated, and the resource allocation strategy is adjusted according to the resource demand priority of the service namespace.

6. An apparatus for managing utilization of a cluster GPU, the apparatus comprising: Comprise: A division module is configured to determine a namespace division manner according to a preset division condition, divide namespaces based on the namespace division manner, and divide the cluster into a plurality of service namespaces and a common namespace; A setting module is configured to obtain historical resource usage data and service priority information of each service namespace, and obtain a weight coefficient of each service namespace according to the service priority information; Based on the historical resource usage data and the weight coefficient, a resource usage growth rate of each service namespace in a preset time period is predicted; according to the resource usage growth rate and the weight coefficient, an optimization algorithm is used to calculate a maximum GPU quota of each service namespace; Based on the maximum GPU quota, a minimum GPU quota of each service namespace is determined; the maximum GPU quota and the minimum GPU quota are taken as a service GPU resource quota of the corresponding service namespace; and a common GPU resource quota is calculated according to the maximum GPU quota and the minimum GPU quota. The resource allocation module is configured to receive an application request for GPU resources of the service namespace, obtain a GPU application quantity of the service namespace according to the application request, reject the application request if the GPU application quantity is greater than the maximum GPU quota, and allocate GPU resources according to the maximum GPU quota; and if the GPU application quantity is less than the minimum GPU quota, determine whether an idle public GPU resource quota in the public namespace is greater than the minimum GPU quota; and if the idle public GPU resource quota is greater than the minimum GPU quota, schedule idle public GPU resources from the public namespace to the corresponding service namespace. The collection module is configured to collect resource monitoring data of service GPU resources of each service namespace in real time. The resource recycling module is configured to determine, according to the resource monitoring data and a preset scheduling strategy, whether there is an idle service namespace, and if so, trigger a resource recycling process to recycle service GPU resources of the idle service namespace to the public namespace. The resource scheduling module is configured to receive a GPU resource application request of a target service namespace, determine whether an allocation condition is met according to application information in the GPU resource application request, current GPU resource usage data, and a preset resource allocation priority rule, and if the allocation condition is met, allocate GPU resources in the public namespace to the target service namespace.

7. A computer device, comprising: A memory and a processor are included, the memory stores computer readable instructions, and the processor implements the steps of the management method of cluster GPU utilization rate according to any one of claims 1 to 5 when executing the computer readable instructions.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the management method of cluster GPU utilization rate according to any one of claims 1 to 5.

Images