GPU performance monitoring methods, monitoring devices, and electronic equipment
By obtaining the identifier and index value of the MIG group when the GPU is in MIG mode, and then obtaining and storing its performance parameters, the problem of not being able to monitor the performance of multiple MIG groups in the prior art is solved, and real-time monitoring of GPU resources and tasks is realized.
Patent Information
- Application Number
- CN202310403108.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-14
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-04-14
AI Technical Summary
Existing technologies cannot monitor the performance of multiple MIG instances in a GPU in real time, resulting in insufficient resource utilization and difficulties in task optimization.
The target script determines whether the GPU has enabled MIG mode, obtains the identifiers and index values of each MIG group, and uses these identifiers and index values to obtain the performance parameters of each MIG group, which are then stored in the target storage unit for display.
It enables performance monitoring of multiple MIG groups in the GPU, solving the problem of real-time monitoring in existing technologies and improving resource utilization and task optimization efficiency.
Smart Images

Figure CN116560941B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of GPU performance monitoring, and more specifically, to a method, device, computer-readable storage medium, and electronic device for monitoring GPU performance. Background Technology
[0002] For AI cloud platforms, deep learning computing resources are quite valuable, and new computing power like the GA100 is even scarcer. The GA100 is an NVIDIA TESLA-A100 GPU card, a new type of computing power capable of enabling MIG (Multi-Instance GPU, a hardware-level computing power partitioning method proposed by NVIDIA) mode. NVIDIA provides the MIG function to achieve reasonable partitioning and allocation of computing power at the hardware level. MIG is a finer-grained resource partitioning for the entire GPU card, directly splitting and recombining the GPU cores and memory from a physical perspective, which can better adapt to the simultaneous execution of multiple smaller tasks. In addition, past GPU reuse used time slices to achieve concurrent execution of multiple tasks, which could not be considered as relatively independent tasks, nor could it guarantee the full utilization of resources.
[0003] MIG can generate various combinations of the GPU's seven cores and memory, allowing for flexible resource isolation and allocation to tasks. However, after enabling MIG, NVIDIA's "NVIDIA SMI" command cannot obtain detailed monitoring metrics for each MIG instance, and GPU utilization will show N / A (not used or not supported). Only data for the entire GPU and the memory usage of each MIG instance can be obtained. In the training module of the Kubernetes cloud platform, each task-generated pod (a plain old data structure) is allocated corresponding GPU resources. When MIG is enabled, the UUID (Universally Unique Identifier) of the MIG instance is automatically written to the pod's environment variables, making it impossible to directly obtain the corresponding GPU's UUID, causing discrepancies in the query monitoring data. Therefore, the optimization, training, and resource usage of multiple tasks on the GPU cannot be monitored in real time, resulting in a functional deficiency.
[0004] Therefore, there is an urgent need for a method that can monitor the performance of multiple MIG instances in a GPU. Summary of the Invention
[0005] This application provides a method, device, computer-readable storage medium, and electronic device for monitoring GPU performance, to at least address the problem in the related art that it is impossible to monitor the performance of multiple MIG groups in a GPU.
[0006] According to one embodiment of this application, a method for monitoring GPU performance is provided, comprising: determining whether the GPU has enabled MIG mode through a target script, wherein the target script is a script in the node where the GPU is located that is used to obtain and output performance parameters of multiple MIG groups in the GPU, and the MIG mode is a mode used to divide the GPU into multiple MIG groups; when the GPU has enabled the MIG mode, obtaining the identifier and index value of each MIG group in the GPU through the target script, obtaining at least a set of first performance parameters corresponding to each MIG group according to the identifier and index value of each MIG group, and storing at least the first performance parameters in a target storage unit, wherein the first performance parameters represent the performance of the computing core of each MIG group and include at least video memory parameters and dram_active parameters.
[0007] In one exemplary embodiment, before determining whether the GPU has enabled MIG mode through the target script, the method further includes: executing the NVIDIA SMI instruction in the target script; if the NVIDIA SMI instruction is successfully executed, performing the step of determining whether the GPU has enabled MIG mode; and if the execution of the NVIDIA SMI instruction fails, exiting the target script.
[0008] In one exemplary embodiment, obtaining the identifiers and index values of each MIG group in the GPU through the target script includes: obtaining the identifiers and index values of each MIG group in the GPU through the NVIDIA SMI L instruction in the target script.
[0009] In an exemplary embodiment, after obtaining the identifiers and index values of each MIG group in the GPU, the method further includes: generating a target group from the identifiers and index values of multiple MIG groups of the GPU; determining whether the group code of the target group is the same as the group code of the initial group in the orientation file; wherein the target group includes multiple target groups, each target group includes the identifier and index value of one MIG group, both the target group and the initial group have unique group codes, the orientation file is stored in the node where the GPU is located, and the initial group is generated after the MIG mode is first enabled on the GPU; if the group code of the target group is the same as the group code of the initial group, determining whether each MIG group in the target group is the same as the initial group's identifier and index value; If the storage space size of group G is the same as the storage space size of each initial MIG group in the initial group, and the storage space size of all MIG groups in the target group is the same as the storage space size of all initial MIG groups in the initial group, then the group code of the initial group is used as the group code of the target group. If the storage space size of all MIG groups in the target group is different from the storage space size of all initial MIG groups in the initial group, then the group code of the initial group is deleted, and the group code of the target group is stored in the orientation file. If the group code of the target group is different from the group code of the initial group, then the group code of the target group is stored in the orientation file.
[0010] In one exemplary embodiment, obtaining at least a set of first performance parameters corresponding to each MIG group based on the identifier and index value of each MIG group includes: obtaining a set of first performance parameters corresponding to each MIG group by using the dcgmi dmon g command of the target script, according to the identifier and index value of each MIG group in the target group corresponding to the target group code in the orientation file, wherein the orientation file is stored in the node where the GPU is located and includes the group code of the target group, the target group includes multiple target groups, and each target group includes the identifier and index value of one MIG group; obtaining a second performance parameter of the GPU by using the dcgmi dmon command of the target script, wherein the second performance parameter represents the performance of the GPU and includes at least the GPU's video memory.
[0011] In one exemplary embodiment, storing at least the first performance parameter to the target storage unit includes: the node where the GPU is located further includes a telegraf component, the telegraf component being used to execute the target script within a predetermined period, obtain the first performance parameter and the second performance parameter output by the target script, and store the first performance parameter and the second performance parameter to the target storage unit, wherein the second performance parameter is used to represent the performance of the GPU and includes at least the GPU's video memory.
[0012] In one exemplary embodiment, the method further includes: generating a MIG task, wherein the MIG task controls the GPU to enable the MIG mode to divide the GPU into multiple MIG groups, wherein the MIG task is used to control the GPU to enable the MIG mode.
[0013] In one exemplary embodiment, the method further includes: obtaining the first performance parameter and the second performance parameter stored in the target storage unit by calling an interface through a cloud platform, wherein the second performance parameter represents the performance of the GPU and includes at least the GPU's video memory; and outputting the first performance parameter and the second performance parameter to an interface for display.
[0014] According to another embodiment of this application, a GPU performance monitoring device is provided, comprising: a judgment module, configured to determine whether the GPU is in MIG mode by means of a target script, wherein the target script is a script in the node where the GPU is located that is used to obtain and output performance parameters of multiple MIG groups in the GPU, and the MIG mode is a mode used to divide the GPU into multiple MIG groups; and a storage module, configured to, when the GPU is in MIG mode, obtain the identifier and index value of each MIG group in the GPU by means of the target script, obtain at least a set of first performance parameters corresponding to each MIG group according to the identifier and index value of each MIG group, and store at least the first performance parameters in a target storage unit, wherein the first performance parameters represent the performance of the computing core of each MIG group and include at least video memory parameters and dram_active parameters.
[0015] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.
[0016] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0017] This application solves the problem of not being able to monitor the performance of multiple MIG groups in a GPU when MIG mode is enabled. By obtaining the identifier and index value of each MIG group through a target script, and then retrieving a set of performance parameters corresponding to each MIG group based on these identifiers and index values, the application stores these performance parameters in a target storage unit. Finally, it displays the performance parameters of multiple MIG groups. Compared to existing methods that can only monitor the performance parameters of the entire GPU and cannot monitor the performance of multiple MIG groups within the GPU, this application, through a target script, determines that the GPU is in MIG mode (i.e., the GPU is divided into multiple MIG groups), obtains the identifier and index value corresponding to each MIG group, and then retrieves the first performance parameters corresponding to each MIG group using these identifiers and index values. This results in multiple first performance parameters, which are then stored and displayed. Therefore, this application solves the problem of existing technologies being unable to monitor the performance of multiple MIG groups in a GPU, achieving the effect of monitoring the performance of multiple MIG groups in a GPU. Attached Figure Description
[0018] Figure 1 This is a hardware structure block diagram of a mobile terminal for a GPU performance monitoring method according to an embodiment of this application.
[0019] Figure 2 This is a flowchart of a GPU performance monitoring method according to an embodiment of this application;
[0020] Figure 3 This is a schematic diagram of the system architecture of a GPU performance monitoring method according to an embodiment of this application;
[0021] Figure 4 This is a schematic diagram of the implementation flow of the target script in a GPU performance monitoring method according to an embodiment of this application;
[0022] Figure 5 This is a structural block diagram of a GPU performance monitoring device according to an embodiment of this application.
[0023] The above figures include the following reference numerals:
[0024] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation
[0025] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.
[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0027] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a GPU performance monitoring method according to an embodiment of this application. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0028] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the GPU performance monitoring method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0029] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.
[0030] This embodiment provides a method for monitoring the GPU performance of the aforementioned mobile terminal. Figure 2 This is a flowchart of a GPU performance monitoring method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:
[0031] Step S202: Determine whether the GPU has enabled MIG mode through the target script. The target script is at least one script in the node where the GPU is located that is used to obtain and output the performance parameters of multiple MIG groups in the GPU. The MIG mode is a mode used to divide the GPU into multiple MIG groups.
[0032] Specifically, MIG (Multi-Group) is a finer-grained resource partitioning for the entire GPU. Enabling MIG mode divides the GPU into multiple MIG groups, directly splitting and recombining cores and memory from a physical perspective. This allows for greater adaptability to the simultaneous execution of multiple smaller tasks. For example, a GPU can be divided into 7 cores, each capable of executing tasks simultaneously and combining with memory individually. The GA100 is a novel computing instance capable of enabling MIG mode. This application uses a custom target script to obtain performance parameters for each MIG group within the GPU, enabling better monitoring of the fine-grained operation of each part of the GPU. Nodes are hosts located within an AI cloud platform. AI cloud platforms typically contain multiple hosts, i.e., multiple nodes. This application provides target scripts on each node requiring monitoring to monitor the performance of each MIG group within the host's GPU.
[0033] Step S204: When the GPU is in MIG mode, the identifier and index value of each MIG group in the GPU are obtained through the target script. At least one set of first performance parameters corresponding to each MIG group are obtained based on the identifier and index value of each MIG group. The first performance parameters are stored in the target storage unit. The first performance parameters represent the performance of the computing core of each MIG group and include at least the video memory parameter and the dram_active parameter.
[0034] Specifically, after the GPU is divided into multiple MIG groups, each MIG group has its own unique identifier (UUID) and index. The target script obtains the unique identifier (UUID) and index for each MIG group, and further obtains the first performance parameters corresponding to the unique identifier (UUID) and index for each MIG group. These parameters include fine-grained unit data analysis indicators such as memory parameters, dram_active parameters, fp64_active / fp32_active / fp16_active parameters, gr_engine_active parameters, sm_active parameters, sm_occupancy parameters, and tensor_active parameters. These are the first performance parameters of each MIG group and are stored in the target unit for display.
[0035] Through the above steps, when the GPU is in MIG mode, the identifier and index value of each MIG group can be obtained through the target script. Based on the identifier and index value of each MIG group, a set of performance parameters corresponding to each MIG group is obtained, and each set of performance parameters corresponding to each MIG group is stored in the target storage unit. Then, the performance parameters of multiple MIG groups are displayed. Compared with existing technologies that can only monitor the performance parameters of the entire GPU and cannot monitor the performance of multiple MIG groups within the GPU, this application, through the target script, determines that the GPU is in MIG mode, i.e., the GPU is divided into multiple MIG groups, obtains the identifier and index value corresponding to each MIG group, and then obtains the first performance parameter corresponding to each MIG group through the identifier and index value, obtaining multiple first performance parameters, storing and displaying them. Therefore, it solves the problem in existing technologies that cannot monitor the performance of multiple MIG groups within the GPU, achieving the effect of monitoring the performance of multiple MIG groups within the GPU, and enabling real-time monitoring of the GPU optimization, task training, and GPU resource usage in the task platform through the first and second performance parameters.
[0036] The entities that perform the above steps can be servers, terminals, etc., but are not limited to these.
[0037] In some optional embodiments, before determining whether the GPU has MIG mode enabled via the target script, the method further includes: executing the NVIDIA SMI instruction in the target script; if the NVIDIA SMI instruction is successfully executed, proceeding to determine whether the GPU has MIG mode enabled; if the execution of the NVIDIA SMI instruction fails, exiting the target script. This method first determines whether the GPU has MIG mode enabled, i.e., whether the GPU is divided into multiple MIG groups. This allows subsequent steps to be executed only if MIG mode is enabled, and not if it is not enabled, thereby improving the execution efficiency of subsequent steps.
[0038] Specifically, in the training module based on the Kubernetes cloud platform, the target script uses NVIDIA SMI and DCGM commands to obtain GPU performance data and MIG instance performance data. For example, the command "NVIDIA SMI --query-gpu" is used to determine if the driver and commands are available, and whether there are any GPU card drop anomalies. If normal, it can obtain the temperature, power, and memory performance data of the entire GA100 card, and whether the GA100 card has MIG mode enabled. If not, the MIG acquisition part is not executed. Corresponding commands can be "NVIDIA SMI --format=noheader,nounits,csv-query-gpu=mig.mode.current", etc.
[0039] In some optional embodiments, obtaining the identifiers and index values of each MIG group in the GPU through the target script includes: obtaining the identifiers and index values of each MIG group in the GPU through the NVIDIA SMI L instruction in the target script. This method, when MIG mode is enabled, obtains the identifiers and index values of each MIG group, thus ensuring that the first performance parameters can be further obtained using the identifiers and index values.
[0040] Specifically, in the training module of the Kubernetes cloud platform, corresponding GPU resources are allocated to each pod (a data structure) generated by each task. When MIG mode is enabled, the identifier of the MIG group will be automatically written into the environment variable of the pod. The identifier and index value of each MIG group in the GPU can be obtained through the NVIDIA SMI L instruction in the target script.
[0041] To simultaneously obtain the first performance parameters of multiple MIG groups and reduce the number of times the target script is run, in some optional embodiments, after obtaining the identifiers and index values of each MIG group in the GPU, the method further includes: generating the identifiers and index values of multiple MIG groups of the GPU into a target group; determining whether the group code of the target group is the same as the group code of the initial group in the orientation file; wherein the target group includes multiple target groups, each target group includes the identifier and index value of one MIG group, and both the target group and the initial group have unique group codes; the orientation file is stored in the node where the GPU is located; and the initial group is generated after the GPU last enabled the MIG mode; if the group code of the target group is the same as the group code of the initial group... Next, it is determined whether the storage space size of each MIG group in the target group is the same as the storage space size of each initial MIG group in the initial group. If the storage space size of all MIG groups in the target group is the same as the storage space size of all initial MIG groups in the initial group, the group code of the initial group is used as the group code of the target group. If the storage space size of all MIG groups in the target group is different from the storage space size of all initial MIG groups in the initial group, the group code of the initial group is deleted and the group code of the target group is stored in the orientation file. If the group code of the target group is different from the group code of the initial group, the group code of the target group is stored in the orientation file. This method generates identifiers and index values for target groups and stores the target group's code in a directed file located in a node. By accessing the directed file in the node, the target group's code is obtained. Based on the target group's code, the identifiers and index values of the corresponding multiple MIG groups are obtained. In this way, the first performance parameters corresponding to each MIG group can be obtained based on the identifier and index value of each MIG group. With the execution of the target script once, the first performance parameters of multiple MIG groups can be obtained, reducing the operating burden of the cloud platform.
[0042] Specifically, the obtained index values are converted to correspond with the index values required in the DCGM command, and the "dcgmi group" command is used to assign all MIG groups to a target group. The target group's code and other information are generated into a directional file, with the corresponding command being "dcgmi group-c mig-a i:0,i:1,i:2...". From the target group's code in the directional file, information such as the space occupied by each MIG group can be determined. During the execution cycle of each target script, the target group code corresponding to that cycle is generated in the targeting file. Therefore, the target group code from the previous cycle already exists in the targeting file. By comparing the space occupied by each MIG group in the previous cycle (represented by the target group code) with the space occupied by each MIG group in the current cycle (represented by the target group code), if the space occupied by each MIG group in the current cycle is the same as that in the previous cycle, it indicates that the configuration of each MIG group has not changed. Therefore, the corresponding target group can be found directly through the target group code from the previous cycle, and the target group's MIG group information can be obtained. The identifier and index value are used to further obtain the first performance parameter. When the space occupied by each MIG group in the current cycle differs from that in the previous cycle, it indicates that the configuration of each MIG group has changed. Therefore, continuing to use the target group code from the previous cycle cannot match the identifiers and index values of each MIG group within the target group, and thus the first performance parameter cannot be obtained. Therefore, the group code of the target group in the previous cycle is first deleted, and the target group code of the current cycle is stored in the orientation file, i.e., the orientation file is updated. Then, the corresponding target group is found based on the group code in the updated orientation file, and the identifiers and index values of each MIG group within the target group are obtained, thereby obtaining the first performance parameter. Of course, in the first execution cycle of the target script, there is no target group code in the orientation file. In this case, only the target group code of the first execution cycle needs to be stored in the orientation file before proceeding with the subsequent steps.
[0043] In some optional embodiments, at least one set of first performance parameters corresponding to each of the aforementioned MIG groups is obtained based on the identifier and index value of each MIG group. This includes: obtaining a set of first performance parameters corresponding to each of the aforementioned MIG groups using the `dcgmi dmon g` command of the target script, according to the identifier and index value of each of the aforementioned MIG groups in the target group corresponding to the target group code in the orientation file. The orientation file is stored in the node where the GPU resides and includes the group code of the aforementioned target group. The aforementioned target group includes multiple target groups, and each target group includes the identifier and index value of one of the aforementioned MIG groups. The second performance parameter of the GPU is obtained using the `dcgmi dmon` command of the target script. The second performance parameter represents the performance of the GPU and includes at least the GPU's video memory. This method obtains the first performance parameters of each MIG group and the second performance parameters of the GPU through the target script, thus simultaneously obtaining the performance parameters of the GPU and each MIG group to monitor resource usage, task execution, and training on the Kubernetes cloud platform.
[0044] Specifically, in the training module of the Kubernetes cloud platform, each pod generated by a task is allocated corresponding GPU resources. When MIG is enabled, the identifier of the MIG instance will be automatically written into the pod's environment variables, making it impossible to directly obtain the identifier of the corresponding GPU, which causes deviations in the query monitoring data. The corresponding commands are "dcgmi dmon-e54,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012-c 2" and "dcgmi dmon-e 1001,1002,1003,1004,1005,1006,1007,1008-g groupId-c2". Since the dcgmi command is provided by NVIDIA, but it is not yet fully optimized, experiments have shown that long-term concurrent execution of the dcgmi command will quickly accumulate memory and cause the service to freeze. Therefore, in the above steps, a targeted file is created and maintained, and the analysis index data of the GA100 card and the MIG group are obtained serially in the above steps. When the number of GA100 full cards and MIG groups reaches the threshold, when a query command is executed on the full group, the returned MIG data will be disordered due to the full card data, and therefore cannot be grouped together.
[0045] In one exemplary embodiment, storing at least the first performance parameter to the target storage unit includes: the node where the GPU resides further includes a Telegraf component, which executes the target script within a predetermined period, obtains the first and second performance parameters output by the target script, and stores the first and second performance parameters to the target storage unit. The second performance parameter represents the performance of the GPU and includes at least the GPU's video memory. This method periodically executes the target script through components in the node and stores the performance parameters output by the target script to the target storage unit, thus facilitating the invocation and execution of the target script.
[0046] Specifically, the Telegraf component is installed on each node to periodically execute the target script to collect performance data, and the target storage unit InfluxDB time-series database is set up to store the data. Data collection and display of the usage of the GA100 computing power fine-grained computing unit MIG group are carried out to strengthen and improve the monitoring of container instances' use of MIG resources for deep learning training on the Kubernetes cloud platform.
[0047] In an exemplary embodiment, to enable MIG mode for the GPU, the method further includes: generating a MIG task, in which the GPU is controlled to enable MIG mode to divide the GPU into multiple MIG groups, wherein the MIG task is used to control the GPU to enable MIG mode.
[0048] Specifically, MIG tasks are created in Kubernetes via a cloud platform, container instances for training tasks are generated, MIG resources are allocated, and identifiers are stored in environment variables.
[0049] In one exemplary embodiment, the method further includes: obtaining the first performance parameter and the second performance parameter stored in the target storage unit by calling an interface through a cloud platform, wherein the second performance parameter represents the performance of the GPU and includes at least the GPU's video memory; and outputting the first performance parameter and the second performance parameter to an interface for display.
[0050] Specifically, the NVIDIA SMI command is used to obtain the memory usage of the MIG. After the data is retrieved, the monitoring data of the MIG is filled into the corresponding index. The corresponding command is "NVIDIA SMI|grep MiB|awk-F'|''NR>gpu_count{print$1,$2,$3}"'. The collected data is then concatenated and output according to the Telegraf custom script specification.
[0051] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the GPU performance monitoring method of this application will be described in detail below with reference to specific embodiments.
[0052] This embodiment relates to a specific method for monitoring GPU performance, such as... Figure 3 and Figure 4 As shown, it includes the following steps:
[0053] Step S1: Figure 3 This is a schematic diagram of the system architecture of a GPU performance monitoring method according to an embodiment of this application, as shown below. Figure 3 As shown, based on the Kubernetes platform, two tasks, pod1 and pod2 (MIG tasks), are created to generate container instances for the training tasks, allocate MIG resources, and store MIG-UUID1 and MIG-UUID2 in the environment variable Env. Target scripts for executing NVIDIA SMI and DCGM commands are provided on all nodes within the cluster that need to be monitored. The DCGM service is started and maintained on each node. The A100 is divided into MIG0 and MIG1. The Telegraf component is installed on each node, and configurations such as script execution commands, collection intervals, and timeouts are completed. The Telegraf service is then started, periodically calling the collection script (target script) to collect monitoring data (first and second performance parameters) of the GA100 and MIG. An InfluxDB database is configured to store the monitoring data, specifically writing the first and second performance parameters collected by the Telegraf component into the InfluxDB database. Through the cloud platform AI call interface, the monitoring data (first and second performance parameters) in the InfluxDB database is queried based on the identifier and index value of the MIG used by the container instance, and then visualized on the interface (output to the display screen).
[0054] Step S2: Figure 4 This is a schematic diagram illustrating the implementation flow of the target script in a GPU performance monitoring method according to this application, as shown below. Figure 4 As shown, the NVIDIA SMI--query-gpu command is used to query the MIG (MIG) enabling status and determine whether the driver, command, etc. are available. If the above command is executed successfully, it is determined whether the GA100 card has MIG mode enabled. If the above command fails, the GPU card drop script exits.
[0055] Step S3: With MIG enabled, use the NVIDIA SMI-L command to query the correspondence between the (GA100) card and MIG, including the UUID (identifier) and index (index value). Convert the obtained index (index value) into the DCGM command format, and use the dcgmi group command to assign all MIG instances (multiple MIG groups) to one group (target group), and generate a targeting file with the group's ID (target group code) and other information.
[0056] Step S4: Determine if groupID (target group ID) is None (does not exist). If groupID (target group ID) does not exist, create group (target group) and update groupID (target group ID) storage file (store the target group ID in the target file).
[0057] Step S5: Determine if the mig configuration (the storage space occupied by the MIG group) has changed. If it has changed, delete the original groupID group (the target group code when the target script was executed in the previous cycle), create a group (target group), and update the groupID (target group code) storage file (store the target group code in the target file).
[0058] Step S6: With the mig configuration unchanged, use the dcgmi dmon-g command and the dcgmidmon command in the target script to query the whole card and the MIG group respectively, to obtain the second performance parameters of the whole card and the first performance parameters of the MIG group. Obtaining the first performance parameters of the MIG group also includes querying the MIG memory through the NVIDIA SMI|grep MiB command in the target script.
[0059] Step S7: If MIG mode is not enabled, directly execute the dcgmi dmon command to obtain the analysis index data (second performance parameters, such as Graphics Engine, SM, TC, etc.) of the GA100 card and print out the monitoring data of the GA100 card. If MIG mode is enabled, first obtain the data of the GA100 card, and then execute the dcgmi dmon command again to obtain the analysis index data (first performance parameters) of multiple MIG groups.
[0060] Step S8: Print out the GA100 card (second performance parameters) and MIG monitoring data (first performance parameters).
[0061] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0062] This embodiment also provides a GPU performance monitoring device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0063] Figure 5 This is a structural block diagram of a GPU performance monitoring device according to an embodiment of this application, such as... Figure 5 As shown, the device includes:
[0064] The judgment module 22 is used to determine whether the GPU has enabled MIG mode through the target script. The target script is at least one script in the node where the GPU is located that is used to obtain and output the performance parameters of multiple MIG groups in the GPU. The MIG mode is a mode used to divide the GPU into multiple MIG groups.
[0065] Storage module 24 is used to obtain the identifier and index value of each MIG group in the GPU through the target script when the MIG mode is enabled on the GPU, obtain at least a set of first performance parameters corresponding to each MIG group based on the identifier and index value of each MIG group, and store at least the first performance parameters in the target storage unit. The first performance parameters represent the performance of the computing core of each MIG group and include at least the video memory parameter and the dram_active parameter.
[0066] In this embodiment, when the GPU is in MIG mode, the identifier and index value of each MIG group are obtained through the target script. Based on the identifier and index value of each MIG group, a set of performance parameters corresponding to each MIG group are obtained, and each set of performance parameters corresponding to each MIG group is stored in the target storage unit. Then, the performance parameters of multiple MIG groups are displayed. Compared with existing technologies that can only monitor the performance parameters of the entire GPU and cannot monitor the performance of multiple MIG groups within the GPU, this application, through the target script, determines that the GPU is in MIG mode (i.e., the GPU is divided into multiple MIG groups), obtains the identifier and index value corresponding to each MIG group, and then obtains the first performance parameters corresponding to each MIG group using the identifier and index value, resulting in multiple first performance parameters, which are then stored and displayed. Therefore, it can solve the problem in existing technologies that cannot monitor the performance of multiple MIG groups within the GPU, achieving the effect of monitoring the performance of multiple MIG groups within the GPU.
[0067] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.
[0068] In some optional embodiments, the apparatus further includes an execution module and an exit module. The execution module executes the NVIDIA SMI instructions in the target script. If the NVIDIA SMI instructions are successfully executed, the step of determining whether the GPU has MIG mode enabled is performed. The exit module exits the target script if the execution of the NVIDIA SMI instructions fails. The apparatus first determines whether the GPU has MIG mode enabled, i.e., whether the GPU is divided into multiple MIG groups. This allows subsequent steps to be executed only if MIG mode is enabled, and not if it is not enabled, thereby improving the execution efficiency of subsequent steps.
[0069] Specifically, in the training module based on the Kubernetes cloud platform, the target script uses NVIDIA SMI and DCGM commands to obtain GPU performance data and MIG instance performance data. For example, the command "NVIDIA SMI --query-gpu" is used to determine if the driver and commands are available, and whether there are any GPU card drop anomalies. If normal, it can obtain the temperature, power, and memory performance data of the entire GA100 card, and whether the GA100 card has MIG mode enabled. If not, the MIG acquisition part is not executed. Corresponding commands can be "NVIDIA SMI --format=noheader,nounits,csv-query-gpu=mig.mode.current", etc.
[0070] In some optional embodiments, the storage module includes a first acquisition submodule, configured to acquire the identifiers and index values of each MIG group in the GPU using the NVIDIA SMI L instructions in the target script. When MIG mode is enabled, acquiring the identifiers and index values of each MIG group ensures that the first performance parameters can be further obtained using these identifiers and index values.
[0071] Specifically, in the training module of the Kubernetes cloud platform, corresponding GPU resources are allocated to each pod (a data structure) generated by each task. When MIG mode is enabled, the identifier of the MIG group will be automatically written into the environment variable of the pod. The identifier and index value of each MIG group in the GPU can be obtained through the NVIDIA SMI L instruction in the target script.
[0072] To simultaneously obtain the first performance parameters of multiple MIG groups and reduce the number of times the target script is run, in some optional embodiments, the storage module further includes a judgment submodule, a first storage submodule, and a second storage submodule. The judgment submodule is used to generate identifiers and index values of the multiple MIG groups of the GPU to a target group, and to determine whether the group code of the target group is the same as the group code of the initial group in the orientation file. The target group includes multiple target groups, each target group including the identifier and index value of one MIG group. Both the target group and the initial group have unique group codes. The orientation file is stored in the node where the GPU resides, and the initial group is generated after the GPU last enabled the MIG mode. The first storage submodule is used to determine whether the group code of the target group matches the group code of the initial group. In the case of the same situation, it is determined whether the storage space size of each MIG group in the target group is the same as the storage space size of each initial MIG group in the initial group. If the storage space size of all MIG groups in the target group is the same as the storage space size of all initial MIG groups in the initial group, the group code of the initial group is used as the group code of the target group. If the storage space size of all MIG groups in the target group is different from the storage space size of all initial MIG groups in the initial group, the group code of the initial group is deleted, and the group code of the target group is stored in the orientation file. The second storage submodule is used to store the group code of the target group in the orientation file when the group code of the target group is different from the group code of the initial group. This device generates identifiers and index values for target groups and stores the target group's code in a directed file located in the node. By accessing the directed file in the node, the target group's code is obtained. Based on the target group's code, the identifiers and index values of the corresponding multiple MIG groups are obtained. In this way, the first performance parameters corresponding to each MIG group can be obtained based on the identifier and index value of each MIG group. With the execution of the target script once, the first performance parameters of multiple MIG groups can be obtained, reducing the operating burden of the cloud platform.
[0073] Specifically, the obtained index values are converted to correspond with the index values required in the DCGM command, and the "dcgmi group" command is used to assign all MIG groups to a target group. The target group's code and other information are generated into a directional file, with the corresponding command being "dcgmi group-c mig-a i:0,i:1,i:2...". From the target group's code in the directional file, information such as the space occupied by each MIG group can be determined. During the execution cycle of each target script, the target group code corresponding to that cycle is generated in the targeting file. Therefore, the target group code from the previous cycle already exists in the targeting file. By comparing the space occupied by each MIG group in the previous cycle (represented by the target group code) with the space occupied by each MIG group in the current cycle (represented by the target group code), if the space occupied by each MIG group in the current cycle is the same as that in the previous cycle, it indicates that the configuration of each MIG group has not changed. Therefore, the corresponding target group can be found directly through the target group code from the previous cycle, and the target group's MIG group information can be obtained. The identifier and index value are used to further obtain the first performance parameter. When the space occupied by each MIG group in the current cycle differs from that in the previous cycle, it indicates that the configuration of each MIG group has changed. Therefore, continuing to use the target group code from the previous cycle cannot match the identifiers and index values of each MIG group within the target group, and thus the first performance parameter cannot be obtained. Therefore, the group code of the target group in the previous cycle is first deleted, and the target group code of the current cycle is stored in the orientation file, i.e., the orientation file is updated. Then, the corresponding target group is found based on the group code in the updated orientation file, and the identifiers and index values of each MIG group within the target group are obtained, thereby obtaining the first performance parameter. Of course, in the first execution cycle of the target script, there is no target group code in the orientation file. In this case, only the target group code of the first execution cycle needs to be stored in the orientation file before proceeding with the subsequent steps.
[0074] In some optional embodiments, the storage module further includes a second acquisition submodule and a third acquisition submodule. The second acquisition submodule is used to obtain a set of first performance parameters corresponding to each MIG group by using the `dcgmi dmon g` command of the target script, according to the identifier and index value of each MIG group in the target group corresponding to the target group code in the orientation file. The orientation file is stored in the node where the GPU resides and includes the group code of the target group. The target group includes multiple target groups, and each target group includes the identifier and index value of one MIG group. The third acquisition submodule is used to obtain the second performance parameters of the GPU by using the `dcgmi dmon` command of the target script. The second performance parameters represent the performance of the GPU and include at least the GPU's video memory. This device obtains the first performance parameters of each MIG group and the second performance parameters of the GPU through the target script, thus simultaneously obtaining the GPU's performance parameters and the performance parameters of each MIG group to monitor resource usage, task execution, and training on the Kubernetes cloud platform.
[0075] Specifically, in the training module of the Kubernetes cloud platform, each pod generated by a task is allocated corresponding GPU resources. When MIG is enabled, the identifier of the MIG instance will be automatically written into the pod's environment variables, making it impossible to directly obtain the identifier of the corresponding GPU, which causes deviations in the query monitoring data. The corresponding commands are "dcgmi dmon-e54,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012-c 2" and "dcgmi dmon-e 1001,1002,1003,1004,1005,1006,1007,1008-g groupId-c2". Since the dcgmi command is provided by NVIDIA, but it is not yet fully optimized, experiments have shown that long-term concurrent execution of the dcgmi command will quickly accumulate memory and cause the service to freeze. Therefore, in the above steps, a targeted file is created and maintained, and the analysis index data of the GA100 card and the MIG group are obtained serially in the above steps. When the number of GA100 full cards and MIG groups reaches the threshold, when a query command is executed on the full group, the returned MIG data will be disordered due to the full card data, and therefore cannot be grouped together.
[0076] In one exemplary embodiment, the storage module further includes a third storage submodule, which is used in the node where the GPU resides to include a Telegraf component. The Telegraf component is used to execute the target script within a predetermined period, obtain the first performance parameter and the second performance parameter output by the target script, and store the first performance parameter and the second performance parameter in the target storage unit. The second performance parameter represents the performance of the GPU and includes at least the GPU's video memory. This device periodically executes the target script through components in the node and stores the performance parameters output by the target script in the target storage unit, thus facilitating the invocation and execution of the target script.
[0077] Specifically, the Telegraf component is installed on each node to periodically execute the target script to collect performance data, and the target storage unit InfluxDB time-series database is set up to store the data. Data collection and display of the usage of the GA100 computing power fine-grained computing unit MIG group are carried out to strengthen and improve the monitoring of container instances' use of MIG resources for deep learning training on the Kubernetes cloud platform.
[0078] In order to enable the GPU to enable MIG mode, in an exemplary embodiment, the apparatus further includes an enabling submodule for generating a MIG task, wherein the MIG task controls the GPU to enable the MIG mode to divide the GPU into multiple MIG groups, wherein the MIG task is used to control the GPU to enable the MIG mode.
[0079] Specifically, MIG tasks are created in Kubernetes via a cloud platform, container instances for training tasks are generated, MIG resources are allocated, and identifiers are stored in environment variables.
[0080] In an exemplary embodiment, the above-mentioned device further includes a third acquisition submodule, configured to acquire the first performance parameter and the second performance parameter stored in the target storage unit by calling an interface through a cloud platform, wherein the second performance parameter represents the performance of the GPU and includes at least the GPU's video memory; and output the first performance parameter and the second performance parameter to an interface for display.
[0081] Specifically, the NVIDIA SMI command is used to obtain the memory usage of the MIG. After the data is retrieved, the monitoring data of the MIG is filled into the corresponding index. The corresponding command is "NVIDIA SMI|grep MiB|awk-F'|''NR>gpu_count{print$1,$2,$3}"'. The collected data is then concatenated and output according to the Telegraf custom script specification.
[0082] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when it is run.
[0083] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0084] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0085] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.
[0086] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.
[0087] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.
[0088] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.
Claims
1. A method for monitoring GPU performance, characterized in that, include: The target script determines whether the GPU is in MIG mode. The target script is a script in the node where the GPU is located that is used to obtain and output the performance parameters of multiple MIG groups in the GPU. The MIG mode is a mode used to divide the GPU into multiple MIG groups. When the GPU enables the MIG mode, the identifier and index value of each MIG group in the GPU are obtained through the target script. At least one set of first performance parameters corresponding to each MIG group are obtained based on the identifier and index value of each MIG group, and the first performance parameters are stored in the target storage unit. The first performance parameters represent the performance of the computing core of each MIG group and include at least the video memory parameter and the dram_active parameter. After obtaining the identifiers and index values of each MIG group in the GPU, the method further includes: generating a target group from the identifiers and index values of multiple MIG groups of the GPU; determining whether the group code of the target group is the same as the group code of the initial group in the orientation file; wherein the target group includes multiple target groups, each target group includes the identifier and index value of one MIG group, and both the target group and the initial group have unique group codes; the orientation file is stored in the node where the GPU resides; and the initial group is generated after the GPU was last enabled in MIG mode; if the group code of the target group is the same as the group code of the initial group, determining the storage location of each MIG group in the target group. If the storage space size of all MIG groups in the target group is the same as that of the initial MIG groups in the initial group, then the group code of the initial group is used as the group code of the target group. If the storage space size of all MIG groups in the target group is different from that of the initial MIG groups in the initial group, then the group code of the initial group is deleted, and the group code of the target group is stored in the orientation file. If the group code of the target group is different from that of the initial group, then the group code of the target group is stored in the orientation file.
2. The monitoring method according to claim 1, characterized in that, Before determining whether the GPU has enabled MIG mode via the target script, the following steps are also included: Execute the NVIDIA SMI instruction in the target script. If the NVIDIA SMI instruction is successfully executed, proceed to determine whether the GPU has enabled MIG mode. If the execution of the NVIDIA SMI instruction fails, exit the target script.
3. The monitoring method according to claim 1, characterized in that, The target script retrieves the identifiers and index values of each MIG group in the GPU, including: The identifiers and index values of each MIG group in the GPU are obtained through the NVIDIA SMI L instruction in the target script.
4. The monitoring method according to claim 1, characterized in that, Based on the identifier and index value of each MIG group, at least one set of first performance parameters corresponding to each MIG group is obtained, including: Using the dcgmi dmon g command in the target script, according to the identifier and index value of each MIG group in the target group corresponding to the target group code in the orientation file, a set of first performance parameters corresponding to each MIG group is obtained. The orientation file is stored in the node where the GPU is located and includes the group code of the target group. The target group includes multiple target groups, and each target group includes the identifier and index value of a MIG group. The second performance parameter of the GPU is obtained by using the dcgmi dmon command of the target script, wherein the second performance parameter represents the performance of the GPU and includes at least the GPU's video memory.
5. The monitoring method according to claim 1, characterized in that, Storing the first performance parameter to the target storage unit at least includes: The node where the GPU is located also includes a telegraf component. The telegraf component is used to execute the target script within a predetermined period, obtain the first performance parameter and the second performance parameter output by the target script, and store the first performance parameter and the second performance parameter in the target storage unit. The second performance parameter is used to represent the performance of the GPU and includes at least the GPU's video memory.
6. The monitoring method according to claim 1, characterized in that, Also includes: A MIG task is generated, in which the GPU is controlled to enable the MIG mode to divide the GPU into multiple MIG groups. The MIG task is used to control the GPU to enable the MIG mode.
7. The monitoring method according to claim 1, characterized in that, Also includes: By calling the interface through the cloud platform, the first performance parameter and the second performance parameter stored in the target storage unit are obtained, wherein the second performance parameter represents the performance of the GPU and includes at least the GPU's video memory; The first performance parameter and the second performance parameter are output to the interface for display.
8. A GPU performance monitoring device, characterized in that, include: The judgment module is used to determine whether the GPU has enabled MIG mode through a target script, wherein the target script is at least one script in the node where the GPU is located that is used to obtain and output the performance parameters of multiple MIG groups in the GPU, and the MIG mode is a mode used to divide the GPU into multiple MIG groups. The storage module is configured to, when the GPU is in MIG mode, obtain the identifier and index value of each MIG group in the GPU through the target script, obtain at least a set of first performance parameters corresponding to each MIG group based on the identifier and index value of each MIG group, and store at least the first performance parameters in the target storage unit, wherein the first performance parameters represent the performance of the computing core of each MIG group and include at least the video memory parameter and the dram_active parameter; The storage module further includes a judgment submodule, a first storage submodule, and a second storage submodule. The judgment submodule is used to generate identifiers and index values of multiple MIG groups of the GPU to a target group, and to determine whether the group code of the target group is the same as the group code of the initial group in the orientation file. The target group includes multiple target groups, each target group including an identifier and index value of a MIG group. Both the target group and the initial group have unique group codes. The orientation file is stored in the node where the GPU resides, and the initial group is generated after the MIG mode is first enabled on the GPU. The first storage submodule is used to determine, if the group code of the target group is the same as the group code of the initial group, each MIG group in the target group... The storage space size of the target group is determined by whether it is the same as the storage space size of each initial MIG group in the initial group. If the storage space size of all MIG groups in the target group is the same as the storage space size of all initial MIG groups in the initial group, the group code of the initial group is used as the group code of the target group. If the storage space size of all MIG groups in the target group is different from the storage space size of all initial MIG groups in the initial group, the group code of the initial group is deleted, and the group code of the target group is stored in the orientation file. The second storage submodule is used to store the group code of the target group in the orientation file if the group code of the target group is different from the group code of the initial group.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the monitoring method according to any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the monitoring method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Monitoring method, monitoring system and related device of GPU (Graphics Processing Unit) card
CN114860536A
