GPU management method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a GPU management method and device, electronic equipment and a computer readable storage medium, the method is applied to a cloud platform connected with a plurality of cluster nodes, each cluster node is provided with a GPU monitoring module, and the method comprises the steps that GPU change information obtained by any cluster node based on the GPU monitoring module of the cluster node is received; determining a target GPU and a change event of the target GPU according to the GPU change information; updating an original configuration file of the target GPU according to the change event to obtain a current configuration file of the target GPU; generating a configuration change notification according to the current configuration file; and distributing the configuration change notification to each GPU management and control device in the cloud platform, so that each GPU management and control device performs configuration update management on the target GPU according to the current configuration file. By applying the technical scheme, more efficient, convenient and reliable GPU adaptation can be realized in the cloud platform, and the stability of the cloud platform is effectively ensured.

Description

Technical Field

[0001] The present application relates to the field of cloud environment technology, and in particular to a GPU management method, and also to a GPU management device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In today's era of rapid technological development, cloud platforms are increasingly widely used. As a key infrastructure for promoting digital transformation in various industries, the importance of cloud platforms is self-evident. Among them, graphics processing units (GPUs) have demonstrated irreplaceable value in many fields with their excellent parallel computing capabilities. GPUs have a wide range of application scenarios, but different application scenarios have unique requirements for GPU performance indicators, which leads to the need to use different models of GPUs to meet the task requirements of specific scenarios.

[0003] However, in actual application environments, it is common to encounter situations where new models of GPUs are used on site without adaptation. At present, due to the highly personalized implementation logic of GPU hardware and its plug-ins, when faced with the adaptation requirements of new models of GPUs, traditional methods often require tedious modifications to the cloud platform code. This process is not only extremely complex to operate and maintain, requiring a lot of manpower, time and professional technical resources, but may also introduce new potential risks and unstable factors, which will have a serious impact on the stability and reliability of the cloud platform.

[0004] Therefore, how to achieve more efficient, convenient and reliable GPU adaptation in the cloud platform to effectively ensure the stability of the cloud platform is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0005] The purpose of this application is to provide a GPU management method, which can achieve more efficient, convenient and reliable GPU adaptation in a cloud platform, effectively ensuring the stability of the cloud platform; another purpose of this application is to provide a GPU management device, electronic device, computer-readable storage medium and computer program product, all of which have the above-mentioned beneficial effects.

[0006] In a first aspect, the present application provides a GPU management method, which is applied to a cloud platform, wherein the cloud platform is connected to a plurality of cluster nodes, and a GPU monitoring module is deployed on each of the cluster nodes, and the method comprises:

[0007] Receiving GPU change information obtained by any of the cluster nodes based on its own GPU monitoring module;

[0008] Determine a target GPU and a change event of the target GPU according to the GPU change information;

[0009] updating an original configuration file of the target GPU according to the change event to obtain a current configuration file of the target GPU;

[0010] generating a configuration change notification according to the current configuration file;

[0011] The configuration change notification is distributed to each GPU control device in the cloud platform, so that each GPU control device performs configuration update management on the target GPU according to the current configuration file.

[0012] Optionally, when the change event is a GPU joining event, updating the original configuration file of the target GPU according to the change event to obtain the current configuration file of the target GPU includes:

[0013] Obtaining GPU information of the target GPU according to the GPU change information; the GPU information includes device information, driver information, and plug-in information;

[0014] The data format of the GPU information is converted into a preset configuration format and saved to the original configuration file to obtain the current configuration file of the target GPU.

[0015] Optionally, when the change event is a GPU removal event, updating the original configuration file of the target GPU according to the change event to obtain the current configuration file of the target GPU includes:

[0016] Obtaining identification information of the target GPU according to the GPU change information;

[0017] Determine the original configuration file of the target GPU according to the identification information;

[0018] The file content in the original configuration file is deleted to obtain the current configuration file of the target GPU.

[0019] Optionally, distributing the configuration change notification to each GPU control device in the cloud platform so that each GPU control device performs configuration update management on the target GPU according to the current configuration file includes:

[0020] Sending the configuration change notification to the GPU device management device in the cloud platform, so that the GPU device management device performs a device management update operation on the target GPU according to the current configuration file;

[0021] Sending the configuration change notification to a GPU information reporting device in the cloud platform, so that the GPU information reporting device performs a reporting management update operation on the target GPU according to the current configuration file;

[0022] Sending the configuration change notification to the GPU quota management device in the cloud platform, so that the GPU quota management device performs a quota management update operation on the target GPU according to the current configuration file;

[0023] The configuration change notification is sent to a GPU scheduling management device in the cloud platform, so that the GPU scheduling management device performs a scheduling management update operation on the target GPU according to the current configuration file.

[0024] Optionally, sending the configuration change notification to a GPU quota management device in the cloud platform so that the GPU quota management device performs a quota management update operation on the target GPU according to the current configuration file includes:

[0025] When the change event is a GPU joining event, sending the configuration change notification to the GPU quota management device, so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file, and allocates resource quotas for the target GPU according to the application quota;

[0026] When the change event is a GPU removal event, the configuration change notification is sent to the GPU quota management device, so that the GPU quota management device obtains the current quota of the target GPU according to the current configuration file and reclaims the current quota.

[0027] Optionally, sending the configuration change notification to the GPU quota management device so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file and allocates resources to the target GPU according to the application quota, including:

[0028] The configuration change notification is sent to the GPU quota management device, so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file, and verifies the application quota using the total resource amount and the resource usage. If the verification passes, the resource quota is allocated to the target GPU according to the application quota; if the verification fails, an insufficient quota prompt is output.

[0029] Optionally, sending the configuration change notification to a GPU scheduling management device in the cloud platform so that the GPU scheduling management device performs a scheduling management update operation on the target GPU according to the current configuration file includes:

[0030] When the change event is a GPU joining event, sending the configuration change notification to the GPU scheduling management device, so that the GPU scheduling management device obtains the computing power capability of the target GPU according to the current configuration file, and performs application scheduling for the target GPU according to the computing power capability;

[0031] When the change event is a GPU removal event, the configuration change notification is sent to the GPU scheduling management device, so that the GPU scheduling management device determines the service application of the target GPU according to the current configuration file and schedules the service application to other GPUs.

[0032] In a second aspect, the present application further discloses a GPU management device, which is applied to a cloud platform, wherein the cloud platform is connected to a plurality of cluster nodes, and a GPU monitoring module is deployed on each of the cluster nodes, and the device comprises:

[0033] A receiving module, used to receive GPU change information obtained by any of the cluster nodes based on its own GPU monitoring module;

[0034] A determination module, configured to determine a target GPU and a change event of the target GPU according to the GPU change information;

[0035] An updating module, configured to update an original configuration file of the target GPU according to the change event, and obtain a current configuration file of the target GPU;

[0036] A generation module, used to generate a configuration change notification according to the current configuration file;

[0037] A distribution module is used to distribute the configuration change notification to each GPU control device in the cloud platform, so that each GPU control device performs configuration update management on the target GPU according to the current configuration file.

[0038] In a third aspect, the present application further discloses an electronic device, comprising:

[0039] Memory for storing computer programs;

[0040] A processor is used to implement the steps of any one of the GPU management methods described above when executing the computer program.

[0041] In a fourth aspect, the present application further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the GPU management methods described above are implemented.

[0042] In a fifth aspect, the present invention further discloses a computer program product, including a computer program / instruction, which implements the steps of any one of the GPU management methods described above when executed by a processor.

[0043] The present application provides a GPU management method, which is applied to a cloud platform, wherein the cloud platform is connected to multiple cluster nodes, and a GPU monitoring module is deployed on each of the cluster nodes. The method comprises: receiving GPU change information obtained by any of the cluster nodes based on its own GPU monitoring module; determining a target GPU and a change event of the target GPU according to the GPU change information; updating an original configuration file of the target GPU according to the change event to obtain a current configuration file of the target GPU; generating a configuration change notification according to the current configuration file; and distributing the configuration change notification to each GPU control device in the cloud platform, so that each of the GPU control devices performs configuration update management on the target GPU according to the current configuration file.

[0044] By applying the technical solution provided in this application, a communication connection is established between the cloud platform and each cluster node, and a GPU monitoring module is deployed on each cluster node, so that each cluster node can monitor the change events of its own GPU devices based on its own GPU monitoring module, and report it to the cloud platform, so that the cloud platform can automatically adapt and manage the GPU devices of each cluster node based on each reported information. During the adaptation management process, the target GPU that has changed and its corresponding change event can be determined according to the reported information, and the original configuration file corresponding to the target GPU can be updated to obtain the current configuration file corresponding to the target GPU. Thus, each GPU control device in the cloud platform can refer to the current configuration file of the target GPU to automatically adapt the target GPU. It can be seen that this technical solution realizes more efficient, convenient and reliable GPU automatic adaptation in the cloud platform without changing the cloud platform code, effectively ensuring the stability and reliability of the cloud platform.

[0045] In one embodiment of the present application, when the GPU quota management device in the cloud platform performs quota management update on the target GPU, if the target GPU is a newly added GPU, it will refer to the application quota of the target GPU for resource quota. In this process, the GPU quota management device will first use its own total resource amount and resource usage to verify the application quota of the target GPU. Only when the verification is passed will the resource quota be allocated to the target GPU, otherwise it will directly output a quota shortage prompt. In this way, the successful execution of the quota management update operation of the GPU quota management device in the cloud platform for the target GPU can be effectively guaranteed.

[0046] The GPU management device, electronic device, computer-readable storage medium, and computer program product provided in this application also have the above-mentioned technical effects, and this application will not repeat them here. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the prior art and the embodiments of the present application, the drawings required for use in the description of the prior art and the embodiments of the present application are briefly introduced below. Of course, the drawings described below in relation to the embodiments of the present application are only part of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work, and the obtained other drawings also belong to the protection scope of the present application.

[0048] Figure 1 A schematic diagram of the structure of a cloud system provided by this application;

[0049] Figure 2 A flowchart of a GPU management method provided in this application;

[0050] Figure 3 A schematic diagram of the structure of a cloud platform provided for this application;

[0051] Figure 4 A flowchart of an implementation of a GPU management method provided in this application;

[0052] Figure 5 A schematic diagram of the structure of a GPU management device provided in this application;

[0053] Figure 6 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0054] The core of this application is to provide a GPU management method, which can achieve more efficient, convenient and reliable GPU adaptation in the cloud platform, effectively ensuring the stability of the cloud platform; another core of this application is to provide a GPU management device, electronic device, computer-readable storage medium and computer program product, all of which have the above-mentioned beneficial effects.

[0055] In order to describe the technical solutions in the embodiments of the present application more clearly and completely, the technical solutions in the embodiments of the present application will be introduced below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0056] An embodiment of the present application provides a GPU management method.

[0057] First, please refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of a cloud system provided by the present application, which includes a cloud platform and multiple cluster nodes, and each cluster node is connected to the cloud platform, wherein a GPU monitoring module is deployed on each cluster node to realize the monitoring and management of the GPU device on its own node. The GPU management method provided by the present application is implemented based on the cloud system.

[0058] For further information, please refer to Figure 2 , Figure 2 A flow chart of a GPU management method provided in this application, which is applied to Figure 1 The cloud platform shown may include the following S101 to S105.

[0059] S101: receiving GPU change information obtained by any cluster node based on its own GPU monitoring module.

[0060] This step aims to enable the cloud platform to obtain GPU change information within the cluster node. As mentioned above, a GPU monitoring module is deployed on each cluster node in the cloud system, and the GPU monitoring module is used to implement the monitoring and management of each GPU device on the corresponding cluster node, and upload the GPU change information obtained by monitoring to the cloud platform. Among them, the GPU change information can be a newly added GPU device on the corresponding cluster node, or it can be the removal of the original GPU device on the corresponding cluster node, or it can be the change of the existing configuration information of a GPU device on the corresponding cluster node. In addition, the GPU monitoring module can be real-time operation or scheduled operation, which is not limited in this application.

[0061] S102: Determine a target GPU and a change event of the target GPU according to the GPU change information.

[0062] This step aims to determine the target GPU and its change events. The target GPU is the GPU device that has changed on the cluster node, and the change event is the actual change event that has occurred on the target GPU, such as adding a new target GPU or removing a target GPU or changing the target GPU configuration as described above. It is understandable that the cloud platform can determine the cluster node where the GPU has changed (that is, the cluster node that uploaded the GPU change information), the GPU that has changed on the cluster node, the specific change event of the GPU, etc. based on the received GPU change information, which can all be obtained by parsing the GPU change information.

[0063] S103: updating the original configuration file of the target GPU according to the change event to obtain the current configuration file of the target GPU.

[0064] This step aims to update the configuration file of the target GPU. It is understandable that when the target GPU changes, its corresponding configuration information will also change accordingly, so its configuration file can be updated to ensure the accuracy of the GPU configuration file, and then ensure the accuracy of the GPU adaptation result. In the implementation process, the original configuration file can be updated to the current configuration file by referring to the change event of the target GPU.

[0065] In one embodiment of the present application, when the change event is a GPU joining event, updating the original configuration file of the target GPU according to the change event to obtain the current configuration file of the target GPU may include:

[0066] Obtain the GPU information of the target GPU according to the GPU change information; the GPU information includes device information, driver information, and plug-in information;

[0067] The data format of the GPU information is converted into a preset configuration format and saved to the original configuration file to obtain the current configuration file of the target GPU.

[0068] An embodiment of the present application provides a method for updating a GPU configuration file in a GPU joining event scenario. Specifically, since it is a GPU joining event, the GPU monitoring module of the cluster node to which the target GPU belongs can not only sense the newly added GPU device of the host, but also further collect relevant information of the newly added GPU device, including but not limited to device information, driver information, plug-in information, etc., and generate GPU change information together and upload it to the cloud platform. Therefore, the cloud platform can parse the GPU change information to obtain various relevant information of the target GPU, that is, the GPU information of the above-mentioned target GPU. As a result, the cloud platform can format each GPU according to the data format set by itself, write it into the original configuration file, and obtain the current configuration file of the target GPU. It can be understood that since the change event of the target GPU is a GPU joining event, the original configuration file of the target GPU is a blank file.

[0069] In one embodiment of the present application, when the change event is a GPU removal event, updating the original configuration file of the target GPU according to the change event to obtain the current configuration file of the target GPU may include:

[0070] Obtain identification information of the target GPU according to the GPU change information;

[0071] Determine the original configuration file of the target GPU according to the identification information;

[0072] Delete the file content in the original configuration file and obtain the current configuration file of the target GPU.

[0073] An embodiment of the present application provides a method for updating a GPU configuration file in a GPU removal event scenario. Specifically, since it is a GPU removal event, the GPU monitoring module of the cluster node to which the target GPU belongs can not only sense that the host has removed the GPU device, but can also further collect the identification information of the removed GPU device (that is, determine which GPU device in the cluster node has been removed), such as ID information or PCI (Peripheral Component Interconnect, peripheral component interconnect standard) device number (both are unique), etc., and generate GPU change information together and upload it to the cloud platform. Therefore, the cloud platform can parse the GPU change information to obtain the identification information of the target GPU. Thus, the cloud platform can delete the file content in the original configuration file corresponding to the identification information to obtain the current configuration file of the target GPU. It can be understood that since the change event of the target GPU is a GPU removal event, the current configuration file of the target GPU is a blank file.

[0074] In addition, based on the GPU configuration file update process under the above GPU addition event and GPU removal event, the GPU configuration file update process under the GPU change event can also be obtained:

[0075] When the change event is a GPU change event, the original configuration file of the target GPU is updated according to the change event to obtain the current configuration file of the target GPU, which may include:

[0076] Obtain the GPU information of the target GPU according to the GPU change information; the GPU information includes device information, driver information, and plug-in information;

[0077] Determine the changed GPU information according to the GPU information of the target GPU and the file content in the original configuration file;

[0078] The data format of the changed GPU information is converted into a preset configuration format, and updated to the original configuration file to obtain the current configuration file of the target GPU.

[0079] S104: Generate a configuration change notification according to the current configuration file.

[0080] This step aims to generate a configuration change notification based on the current configuration file, that is, to inform each GPU control device in the cloud platform that a GPU change event has occurred in the current system, and each GPU control device needs to be updated and adapted synchronously.

[0081] S105: Distribute the configuration change notification to each GPU control device in the cloud platform, so that each GPU control device performs configuration update management on the target GPU according to the current configuration file.

[0082] This step aims to achieve overall configuration update management of the target GPU by the cloud platform by distributing configuration change notifications. Specifically, after obtaining the configuration change notification, it can be distributed to each GPU control device in the cloud platform, so that each GPU control device can read the current configuration file of the target GPU and perform configuration update management on the target GPU. Among them, each GPU control device in the cloud platform is used to implement different aspects of GPU control, such as device control, quota control, scheduling control, etc., and its number and type are mainly determined by the characteristics of the cloud platform itself, and this application does not limit this.

[0083] It can be seen that the GPU management method provided in the embodiment of the present application establishes a communication connection between the cloud platform and each cluster node, and deploys a GPU monitoring module on each cluster node, so that each cluster node can realize the change event monitoring of each GPU device based on its own GPU monitoring module, and report it to the cloud platform, so that the cloud platform can automatically adapt and manage the GPU devices of each cluster node based on each reported information. In the adaptation management process, the target GPU that has changed and its corresponding change event can be determined according to the reported information, and the original configuration file corresponding to the target GPU can be updated to obtain the current configuration file corresponding to the target GPU. Thus, each GPU control device in the cloud platform can refer to the current configuration file of the target GPU to automatically adapt the target GPU. It can be seen that the technical solution realizes more efficient, convenient and reliable GPU automatic adaptation in the cloud platform without changing the cloud platform code, which effectively ensures the stability and reliability of the cloud platform.

[0084] Based on the above embodiments:

[0085] In one embodiment of the present application, distributing a configuration change notification to each GPU control device in the cloud platform so that each GPU control device performs configuration update management on the target GPU according to the current configuration file may include:

[0086] Sending a configuration change notification to a GPU device management device in the cloud platform, so that the GPU device management device performs a device management update operation on the target GPU according to the current configuration file;

[0087] Sending a configuration change notification to a GPU information reporting device in the cloud platform, so that the GPU information reporting device performs a reporting management update operation on the target GPU according to the current configuration file;

[0088] Sending a configuration change notification to a GPU quota management device in the cloud platform, so that the GPU quota management device performs a quota management update operation on the target GPU according to the current configuration file;

[0089] A configuration change notification is sent to a GPU scheduling management device in the cloud platform, so that the GPU scheduling management device performs a scheduling management update operation on the target GPU according to the current configuration file.

[0090] The embodiments of the present application provide several types of GPU control devices, namely, a GPU device management device, a GPU information reporting device, a GPU quota management device, and a GPU scheduling management device. Among them, the GPU device management device is used to implement visual management of GPU devices, such as the removal, addition, and change of GPU devices, which can be visualized through the front-end interface. The display content may include various types of relevant information of the target GPU; the GPU information reporting device is used to implement information reporting management of GPU devices, for example, in what data format the relevant information of the target GPU is transmitted; the GPU quota management device is used to implement quota management of GPU devices, such as resource allocation and resource recovery for the target GPU; the GPU scheduling management device is used to implement scheduling management of GPU devices, for example, scheduling the target GPU to provide resource services for which application on the corresponding cluster node, scheduling the service application on the target GPU to other GPUs, etc. Therefore, after receiving the configuration change notification, each GPU control device can read the current configuration file of the target GPU and perform corresponding configuration update operations on the target GPU.

[0091] The sending of the configuration change notification to the GPU quota management device in the cloud platform so that the GPU quota management device performs a quota management update operation on the target GPU according to the current configuration file may include:

[0092] When the change event is a GPU joining event, a configuration change notification is sent to the GPU quota management device, so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file, and allocates resource quotas for the target GPU according to the application quota;

[0093] When the change event is a GPU removal event, a configuration change notification is sent to the GPU quota management device, so that the GPU quota management device obtains the current quota of the target GPU according to the current configuration file and reclaims the current quota.

[0094] The embodiment of the present application provides a method for implementing the quota management update operation of the target GPU by a GPU quota management device. For GPU joining events, since the target GPU is a newly added device, resources need to be allocated for it. In this process, the application quota of the target GPU can be determined based on its current configuration file, including the type of resources applied for (such as network resources, computing resources, storage resources, etc.) and the amount of resources, and then resources can be allocated according to the application quota. For GPU removal events, since the target GPU is a device that is about to be removed and no longer provides services for the cloud platform, its current quota can be directly reclaimed.

[0095] Furthermore, sending a configuration change notification to the GPU quota management device so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file, and allocates resource quotas for the target GPU according to the application quota, can include: sending a configuration change notification to the GPU quota management device so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file, and verifies the application quota using the total resource amount and resource usage; if the verification passes, allocates resource quotas for the target GPU according to the application quota; if the verification fails, outputs a quota insufficient prompt.

[0096] Specifically, for a newly added target GPU, the GPU quota management device in the cloud platform can first verify its application quota using its own total resource amount and resource usage when updating its quota management. Only when the verification is passed will the resource quota be allocated to the target GPU, otherwise it will directly output a prompt indicating insufficient quota. The verification process for applying for quota can specifically be to determine whether the difference between the total resource amount and resource usage of the cloud platform exceeds the application quota of the target GPU. If so, it is considered that the verification has passed, otherwise it is considered that the verification has failed.

[0097] It can be seen that the GPU quota management device in the cloud platform uses its own total resource amount and resource usage to verify the application quota of the target GPU, which can effectively ensure the successful execution of the quota management update operation of the target GPU by the GPU quota management device in the cloud platform.

[0098] The sending of the configuration change notification to the GPU scheduling management device in the cloud platform so that the GPU scheduling management device performs a scheduling management update operation on the target GPU according to the current configuration file may include:

[0099] When the change event is a GPU joining event, a configuration change notification is sent to the GPU scheduling management device, so that the GPU scheduling management device obtains the computing power capability of the target GPU according to the current configuration file, and performs application scheduling for the target GPU according to the computing power capability;

[0100] When the change event is a GPU removal event, a configuration change notification is sent to the GPU scheduling management device, so that the GPU scheduling management device determines the service application of the target GPU according to the current configuration file and schedules the service application to other GPUs.

[0101] The embodiment of the present application provides a method for implementing the scheduling management update operation of a target GPU by a GPU scheduling management device. For a GPU joining event, since the target GPU is a newly joined device and is about to provide services for the cloud platform, its computing power can be determined according to the current configuration file of the target GPU, and then application requests are allocated to the target GPU according to the computing power of the target GPU to implement application scheduling. For a GPU removal event, since the target GPU is a device that is about to be removed and no longer provides services for the cloud platform, it is necessary to schedule the application currently serving on the target GPU to other GPUs of the current cluster node.

[0102] Furthermore, sending a configuration change notification to the GPU scheduling management device so that the GPU scheduling management device determines the service application of the target GPU according to the current configuration file and schedules the service application to other GPUs may include: sending a configuration change notification to the GPU scheduling management device so that the GPU scheduling management device determines the service application of the target GPU according to the current configuration file, and determines the available GPU on the cluster node to which the target GPU belongs according to the computing power requirement of the service application, so as to schedule the service application to the available GPU with the lowest load.

[0103] Specifically, for the target GPU that is about to be removed, the GPU quota management device in the cloud platform can determine the available GPU that can provide services for the application on the cluster node to which the target GPU belongs based on the computing power requirements of the service application when scheduling and managing it. In other words, the computing power capacity of the available GPUs should at least meet the computing power requirements of the service application. Then, the available GPU with the least load pressure is selected from these available GPUs to provide services for this application, that is, the service application is scheduled to the available GPU with the least load pressure.

[0104] It can be seen that the GPU scheduling management device in the cloud platform realizes service application scheduling by combining the actual computing power and load pressure of each GPU, which can further improve the stability of the cloud system.

[0105] Based on the above embodiments, an embodiment of the present application provides another GPU management method.

[0106] Please refer to Figure 3 , Figure 3This is a schematic diagram of the structure of a cloud platform provided by this application. The cloud platform mainly includes: a GPU configuration device, a GPU information collection device, a GPU device management device, a GPU quota management device, a GPU reporting management device, and a GPU scheduling management device. The functions of each device are as follows:

[0107] (1) The GPU configuration device is used to configure and store the key information required for the cloud platform to adapt to the GPU, mainly including GPU device information, GPU driver information, and GPU plug-in information.

[0108] (2) The GPU information collection device is used to provide the GPU configuration device with relevant information about the GPU devices, drivers, and plug-ins on each host node.

[0109] (3) The GPU reporting management device is used to read and parse the GPU configuration information and transmit the GPU information in the form of tags.

[0110] (4) The GPU device management device is used to read and parse GPU configuration information, visualize GPU device information, and provide functions such as adding and removing GPU devices.

[0111] (5) The GPU quota management device is used to read and parse GPU configuration information to implement the management function of GPU resource quota.

[0112] (6) The GPU scheduling management device is used to read and parse GPU configuration information to enable applications to quickly use GPU devices.

[0113] For further information, please refer to Figure 4 , Figure 4 This is a flowchart of an implementation of a GPU management method provided in this application, and its specific implementation process is as follows:

[0114] Step 1: GPU configuration device design.

[0115] (1) Configuration format design: Plan and determine the format of GPU configuration to ensure that the configuration information is accurate, clear, and easy to process. For example, you can use configmap (an API object used to save non-confidential data in key-value pairs) to store json (a data format).

[0116] (2) Determine the type of configuration information: clearly cover device information (such as GPU model, specifications, etc.), driver information (including driver version, etc.) and plug-in information (type, function, resource restriction key, etc. of relevant plug-ins).

[0117] (a) Device information: lspci -nn |grep -i nvidia;

[0118] (b) Driver information: nvidia-smi;

[0119] (c) Plugin information: limit key.

[0120] If a GPU-related container already exists on the cluster node, the container can be used to parse the plug-in configuration; if no GPU-related container exists, an interactive interface can be provided for the user to manually fill in the plug-in information.

[0121] Please refer to Table 1, which is a GPU information table provided in this application.

[0122] Table 1 A GPU information table

[0123]

[0124] Finally, read and use the configuration information: use client-go (the official client library of Kubernetes, which provides interfaces and implementations for communicating with the Kubernetes ApiServer) to call the gpu configuration file for parsing and use.

[0125] Step 2: GPU existing device modification.

[0126] (1) GPU reporting management device transformation:

[0127] The existing GPU information reporting device (which cannot automatically obtain data information and requires manual operation) is optimized and improved, and the GPU configuration information is automatically obtained from the GPU configuration device, and the manufacturer, model and other information in the GPU configuration information are reported in the form of node labels. The node label format is as follows:

[0128] Manufacturer: xxx.com / gpu-vendor:xxx;

[0129] Model:xxx.com / gpu-pruduct:xxx.

[0130] Specifically, a GPU monitoring module is deployed on each cluster node in the cloud platform. The GPU monitoring module is equipped with a timed scanning mechanism, which can scan the GPU on the cluster node every 60 seconds. When a newly added GPU is detected, the GPU monitoring module will obtain the newly added GPU information, which includes the PCI device number (globally unique). Compare the newly obtained PCI device number with the GPU information registered in the configmap configuration. If the configmap configuration is not hit, register it. If the configmap configuration is hit, compare each configuration information and update it if there is a change. Among them, the configmap configuration is a global configuration that can be accessed by the entire cloud platform. Furthermore, the GPU reporting management device will monitor the configmap configuration in real time. When the configuration update is detected, it will automatically obtain the latest configuration, and obtain the GPU information of the PCI device number based on the latest configuration and report it to the cloud platform. As a result, the cloud platform can perceive that a new GPU has joined the cluster.

[0131] The GPU removal step is the opposite of the GPU addition step, and this application will not repeat it. When the number of GPUs is 0, the cloud platform calls the GPU configuration device and deletes the registration information of the GPU in the configmap.

[0132] (2) GPU equipment management device transformation:

[0133] The existing GPU device management device (which cannot realize automatic management of GPU devices) is optimized and improved, the GPU configuration information is automatically obtained from the GPU configuration device, and the required key GPU information is visualized.

[0134] For example, if one of the three GPU cards fails, monitoring can be detected by DCGM (Data Center GPU Manager) and an abnormal event will be generated. The GPU reporting management device can monitor the card failure, and the number of GPU cards reported to the cloud platform will change from 3 to 2, and the PCI number of the failed card will also be transmitted to the cloud platform. As a result, the cloud platform will call the GPU device management device to remove the information of the bad card.

[0135] (3) GPU quota management device modification:

[0136] The existing GPU quota management device (quota management based on manual operation) is optimized and improved, and the GPU configuration information is obtained from the GPU configuration device, thereby realizing automatic quota management of the GPU.

[0137] The logic of the GPU quota management device: For newly added GPUs or GPUs with changed configurations, obtain the GPU's application quota and allocate resources to the GPU based on its own total resources and resource usage. For removed GPUs, directly reclaim their originally allocated resources.

[0138] (4) GPU scheduling management device transformation:

[0139] The existing GPU scheduling management device (scheduling management based on manual operation) is optimized and improved, and the GPU configuration information is obtained from the GPU configuration device, thereby realizing GPU application scheduling management.

[0140] The logic of the GPU scheduling management device: For newly added GPUs or GPUs with changed configurations, obtain their computing power capabilities and perform appropriate application scheduling for them; for removed GPUs, schedule the applications they originally served to other GPUs. This process can combine data information such as remaining resources and load pressure to achieve the best matching scheduling.

[0141] Step 3: Automatic adaptation of new GPU models.

[0142] (1) Install new GPU devices on cluster nodes: Install new GPU hardware that has not yet been adapted on the host node.

[0143] (2) Sending GPU configuration creation tasks: When a cluster manages a resource pool, it automatically sends tasks to create a GPU configuration suitable for the new model.

[0144] (3) Automatic adaptation of new GPU models on the cloud platform: The cloud platform automatically completes the adaptation of new GPU models without the need for manual intervention or modification of cloud platform code.

[0145] (a) Registration of GPU information: When a GPU cluster is managed on a cloud platform, the GPU configuration device of the cloud platform will automatically detect the GPU information in each cluster node and generate the GPU information as a GPU configuration file.

[0146] (b) After the GPU configuration file is generated, the GPU reporting management device, the GPU device management device, the GPU quota management device, and the GPU scheduling management device will automatically read the GPU configuration (dynamic reading) and perform corresponding configuration update operations.

[0147] (4) Applications use new GPU models: After adaptation, various applications can successfully use the powerful computing power provided by the new GPU models, thereby improving the performance and efficiency of the applications.

[0148] It can be seen that the GPU management method provided in the embodiment of the present application establishes a communication connection between the cloud platform and each cluster node, and deploys a GPU monitoring module on each cluster node, so that each cluster node can realize the change event monitoring of each GPU device based on its own GPU monitoring module, and report it to the cloud platform, so that the cloud platform can automatically adapt and manage the GPU devices of each cluster node based on each reported information. In the adaptation management process, the target GPU that has changed and its corresponding change event can be determined according to the reported information, and the original configuration file corresponding to the target GPU can be updated to obtain the current configuration file corresponding to the target GPU. Thus, each GPU control device in the cloud platform can refer to the current configuration file of the target GPU to automatically adapt the target GPU. It can be seen that the technical solution realizes more efficient, convenient and reliable GPU automatic adaptation in the cloud platform without changing the cloud platform code, which effectively ensures the stability and reliability of the cloud platform.

[0149] An embodiment of the present application provides a GPU management device.

[0150] Please refer to Figure 5 , Figure 5 This is a structural diagram of a GPU management device provided in the present application. The GPU management device is applied to a cloud platform. The cloud platform is connected to multiple cluster nodes. A GPU monitoring module is deployed on each cluster node. The GPU management device may include:

[0151] Receiving module 1, used to receive GPU change information obtained by any cluster node based on its own GPU monitoring module;

[0152] Determining module 2, used to determine the target GPU and the change event of the target GPU according to the GPU change information;

[0153] An updating module 3 is used to update the original configuration file of the target GPU according to the change event to obtain the current configuration file of the target GPU;

[0154] Generating module 4, used to generate a configuration change notification according to the current configuration file;

[0155] The distribution module 5 is used to distribute the configuration change notification to each GPU control device in the cloud platform, so that each GPU control device can perform configuration update management on the target GPU according to the current configuration file.

[0156] It can be seen that the GPU management device provided in the embodiment of the present application establishes a communication connection between the cloud platform and each cluster node, and deploys a GPU monitoring module on each cluster node, so that each cluster node can realize the change event monitoring of each GPU device based on its own GPU monitoring module, and report it to the cloud platform, so that the cloud platform can automatically adapt and manage the GPU devices of each cluster node based on each reported information. In the adaptation management process, the target GPU that has changed and its corresponding change event can be determined according to the reported information, and the original configuration file corresponding to the target GPU can be updated to obtain the current configuration file corresponding to the target GPU. Thus, each GPU control device in the cloud platform can refer to the current configuration file of the target GPU to automatically adapt the target GPU. It can be seen that the technical solution realizes more efficient, convenient and reliable GPU automatic adaptation in the cloud platform without changing the cloud platform code, which effectively guarantees the stability and reliability of the cloud platform.

[0157] In one embodiment of the present application, when the change event is a GPU joining event, the above-mentioned update module 3 can be specifically used to obtain the GPU information of the target GPU based on the GPU change information; the GPU information includes device information, driver information, and plug-in information; the data format of the GPU information is converted into a preset configuration format, and saved to the original configuration file to obtain the current configuration file of the target GPU.

[0158] In one embodiment of the present application, when the change event is a GPU removal event, the above-mentioned update module 3 can be specifically used to obtain the identification information of the target GPU based on the GPU change information; determine the original configuration file of the target GPU based on the identification information; delete the file content in the original configuration file to obtain the current configuration file of the target GPU.

[0159] In one embodiment of the present application, the distribution module 5 may include:

[0160] A first sending unit is used to send a configuration change notification to a GPU device management device in the cloud platform, so that the GPU device management device performs a device management update operation on a target GPU according to a current configuration file;

[0161] A second sending unit is used to send a configuration change notification to a GPU information reporting device in the cloud platform, so that the GPU information reporting device performs a reporting management update operation on a target GPU according to a current configuration file;

[0162] A third sending unit is used to send a configuration change notification to a GPU quota management device in the cloud platform, so that the GPU quota management device performs a quota management update operation on a target GPU according to a current configuration file;

[0163] The fourth sending unit is used to send a configuration change notification to the GPU scheduling management device in the cloud platform, so that the GPU scheduling management device performs a scheduling management update operation on the target GPU according to the current configuration file.

[0164] In one embodiment of the present application, the third sending unit may include:

[0165] The first quota subunit is used for sending a configuration change notification to the GPU quota management device when the change event is a GPU joining event, so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file, and allocates resource quotas for the target GPU according to the application quota;

[0166] The second quota subunit is used to send a configuration change notification to the GPU quota management device when the change event is a GPU removal event, so that the GPU quota management device obtains the current quota of the target GPU according to the current configuration file and reclaims the current quota.

[0167] In one embodiment of the present application, the above-mentioned first quota sub-unit can be specifically used to send a configuration change notification to the GPU quota management device, so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file, and uses the total resource amount and resource usage to verify the application quota. If the verification passes, the resource quota is allocated to the target GPU according to the application quota. If the verification fails, an insufficient quota prompt is output.

[0168] In one embodiment of the present application, the fourth sending unit may include:

[0169] The first scheduling subunit is used for sending a configuration change notification to the GPU scheduling management device when the change event is a GPU joining event, so that the GPU scheduling management device obtains the computing power capability of the target GPU according to the current configuration file, and performs application scheduling for the target GPU according to the computing power capability;

[0170] The second scheduling subunit is used to send a configuration change notification to the GPU scheduling management device when the change event is a GPU removal event, so that the GPU scheduling management device determines the service application of the target GPU according to the current configuration file and schedules the service application to other GPUs.

[0171] For an introduction to the device provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0172] An embodiment of the present application provides an electronic device.

[0173] Please refer to Figure 6 , Figure 6This is a schematic diagram of the structure of an electronic device provided in this application, and the electronic device may include:

[0174] A memory 11, used for storing computer programs;

[0175] The processor 10 can implement the steps of any one of the above-mentioned GPU management methods when being used to execute a computer program.

[0176] like Figure 6 , which is a schematic diagram of the composition structure of an electronic device, the electronic device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 all communicate with each other through the communication bus 13.

[0177] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0178] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of the GPU management method.

[0179] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0180] Receive GPU change information obtained by any cluster node based on its own GPU monitoring module;

[0181] Determine the target GPU and the change event of the target GPU according to the GPU change information;

[0182] Update the original configuration file of the target GPU according to the change event to obtain the current configuration file of the target GPU;

[0183] Generate configuration change notifications based on the current configuration file;

[0184] Distribute configuration change notifications to each GPU control device in the cloud platform so that each GPU control device can perform configuration update management on the target GPU according to the current configuration file.

[0185] In a possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0186] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0187] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0188] Of course, it should be noted that Figure 6 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device may include Figure 6 More or fewer components than shown, or combinations of certain components.

[0189] An embodiment of the present application provides a computer-readable storage medium.

[0190] The computer-readable storage medium provided in the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned GPU management methods can be implemented.

[0191] Among them, the computer-readable storage medium can be any available medium that can be stored in a computer or a data storage device such as a server or data center that includes one or more available media integrated. For example, it can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape, etc.), an optical medium (such as a DVD) or a semiconductor medium (such as a solid-state hard disk) and other media that can store computer program codes.

[0192] For an introduction to the computer-readable storage medium provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0193] An embodiment of the present application provides a computer program product.

[0194] The computer program product provided in the embodiments of the present application includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of any one of the above-mentioned GPU management methods can be implemented.

[0195] Specifically, in the above embodiments, all or part of them can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of them can be implemented in the form of a computer program product.

[0196] Among them, the computer program product may include one or more computer programs / instructions, and when the computer program / instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application may be generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line, etc.) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0197] For an introduction to the computer program product provided in the embodiments of the present application, please refer to the above method embodiments, and the present application will not elaborate on them here.

[0198] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0199] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0200] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0201] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples herein, and the description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A GPU management method, characterized in that: Applied to a cloud platform, the cloud platform is connected to a plurality of cluster nodes, each of the cluster nodes is deployed with a GPU monitoring module, the method comprises: Receiving GPU change information obtained by any of the cluster nodes based on its own GPU monitoring module; Determine a target GPU and a change event of the target GPU according to the GPU change information; updating an original configuration file of the target GPU according to the change event to obtain a current configuration file of the target GPU; generating a configuration change notification according to the current configuration file; The configuration change notification is distributed to each GPU control device in the cloud platform, so that each GPU control device performs configuration update management on the target GPU according to the current configuration file.

2. The GPU management method according to claim 1, characterized in that: When the change event is a GPU joining event, updating the original configuration file of the target GPU according to the change event to obtain the current configuration file of the target GPU includes: Obtaining GPU information of the target GPU according to the GPU change information; the GPU information includes device information, driver information, and plug-in information; The data format of the GPU information is converted into a preset configuration format and saved to the original configuration file to obtain the current configuration file of the target GPU.

3. The GPU management method according to claim 1, characterized in that: When the change event is a GPU removal event, updating the original configuration file of the target GPU according to the change event to obtain the current configuration file of the target GPU includes: Obtaining identification information of the target GPU according to the GPU change information; Determine the original configuration file of the target GPU according to the identification information; The file content in the original configuration file is deleted to obtain the current configuration file of the target GPU.

4. The GPU management method according to claim 1, characterized in that: Distributing the configuration change notification to each GPU control device in the cloud platform so that each GPU control device performs configuration update management on the target GPU according to the current configuration file, including: Sending the configuration change notification to the GPU device management device in the cloud platform, so that the GPU device management device performs a device management update operation on the target GPU according to the current configuration file; Sending the configuration change notification to a GPU information reporting device in the cloud platform, so that the GPU information reporting device performs a reporting management update operation on the target GPU according to the current configuration file; Sending the configuration change notification to the GPU quota management device in the cloud platform, so that the GPU quota management device performs a quota management update operation on the target GPU according to the current configuration file; The configuration change notification is sent to a GPU scheduling management device in the cloud platform, so that the GPU scheduling management device performs a scheduling management update operation on the target GPU according to the current configuration file.

5. The GPU management method according to claim 4, characterized in that: Sending the configuration change notification to the GPU quota management device in the cloud platform so that the GPU quota management device performs a quota management update operation on the target GPU according to the current configuration file, including: When the change event is a GPU joining event, sending the configuration change notification to the GPU quota management device, so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file, and allocates resource quotas for the target GPU according to the application quota; When the change event is a GPU removal event, the configuration change notification is sent to the GPU quota management device, so that the GPU quota management device obtains the current quota of the target GPU according to the current configuration file and reclaims the current quota.

6. The GPU management method according to claim 5, characterized in that: Sending the configuration change notification to the GPU quota management device so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file, and allocates resource quotas for the target GPU according to the application quota, including: The configuration change notification is sent to the GPU quota management device, so that the GPU quota management device obtains the application quota of the target GPU according to the current configuration file, and verifies the application quota using the total resource amount and the resource usage. If the verification passes, the resource quota is allocated to the target GPU according to the application quota; if the verification fails, an insufficient quota prompt is output.

7. The GPU management method according to claim 4, characterized in that: Sending the configuration change notification to the GPU scheduling management device in the cloud platform so that the GPU scheduling management device performs a scheduling management update operation on the target GPU according to the current configuration file, including: When the change event is a GPU joining event, sending the configuration change notification to the GPU scheduling management device, so that the GPU scheduling management device obtains the computing power capability of the target GPU according to the current configuration file, and performs application scheduling for the target GPU according to the computing power capability; When the change event is a GPU removal event, the configuration change notification is sent to the GPU scheduling management device, so that the GPU scheduling management device determines the service application of the target GPU according to the current configuration file and schedules the service application to other GPUs.

8. A GPU management device, characterized in that: Applied to a cloud platform, the cloud platform is connected to a plurality of cluster nodes, each of the cluster nodes is deployed with a GPU monitoring module, and the device comprises: A receiving module, used to receive GPU change information obtained by any of the cluster nodes based on its own GPU monitoring module; A determination module, configured to determine a target GPU and a change event of the target GPU according to the GPU change information; An updating module, configured to update an original configuration file of the target GPU according to the change event, and obtain a current configuration file of the target GPU; A generation module, used to generate a configuration change notification according to the current configuration file; A distribution module is used to distribute the configuration change notification to each GPU control device in the cloud platform, so that each GPU control device performs configuration update management on the target GPU according to the current configuration file.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the GPU management method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the GPU management method according to any one of claims 1 to 7 are implemented.

Images