Graphics processing unit performance acquisition and analysis method, apparatus and device, and storage medium

Abstract

The invention discloses a graphic processing unit performance acquisition and analysis method, belongs to the field of data processing, and is used for quickly acquiring and analyzing performance information of each graphic processing unit during model heterogeneous mixed training. The method comprises the following steps: acquiring first model information of a plurality of first graphic processing units (GPUs) in each first computing power node, wherein the first computing power nodes are computing power nodes during model heterogeneous mixed training; calling an information acquisition interface corresponding to the first model information, and obtaining a first performance parameter of each first GPU and first event information generated during operation; determining node performance information of each first computing power node based on the first performance parameters and the first event information so as to determine overall performance information of a plurality of computing power nodes during model heterogeneous mixed training; and displaying the overall performance information in a preset interface to enable a user to perform performance analysis on each first GPU during model heterogeneous mixed training.

Description

Technical Field

[0001] This application belongs to the field of data processing, and specifically relates to a method, apparatus, device and storage medium for acquiring and analyzing the performance of a graphics processing unit. Background Technology

[0002] Performance acquisition is a crucial step in optimizing AI model performance, helping researchers and developers understand how models perform on different hardware configurations, identify bottlenecks, and perform targeted optimizations. With the rapid growth in the scale of large-scale model training, GPU computing resources from leading international manufacturers like NVIDIA have become expensive and scarce, limiting the development and application of large-scale basic models in China. Heterogeneous training technology utilizes GPUs from different manufacturers and architectures as computing resources, achieving model training capabilities comparable to a single high-performance GPU, becoming a cost-effective and efficient model training method that breaks free from the limitations of a single graphics card. However, with the increasing variety of mixed-training chips, the complex performance acquisition methods and tools for different chips increase the difficulty of learning and using them.

[0003] Therefore, a method is needed to quickly obtain and analyze the performance information of each graphics processing unit during heterogeneous mixed training of models. Summary of the Invention

[0004] This application provides a method for acquiring and analyzing the performance of graphics processing units (GPUs), enabling rapid acquisition and analysis of performance information of each GPU during heterogeneous mixed training of models.

[0005] In a first aspect, embodiments of this application provide a method for collecting and analyzing the performance of a graphics processing unit (GPU). The method includes: acquiring first model information of multiple first GPUs in each first computing power node, wherein the first computing power node is a computing power node used for heterogeneous mixed training of models; calling an information collection interface corresponding to the first model information to acquire first performance parameters and first event information generated during runtime for each first GPU; determining node performance information of each first computing power node based on the first performance parameters and the first event information to determine the overall performance information of multiple computing power nodes during heterogeneous mixed training of models; and displaying the overall performance information in a preset interface to enable users to perform performance analysis on each first GPU during heterogeneous mixed training of models.

[0006] Secondly, embodiments of this application provide a graphics processing unit performance acquisition and analysis device, which includes: a first acquisition module, configured to acquire first model information of multiple first graphics processing units (GPUs) in each first computing power node, wherein the first computing power node is a computing power node used for heterogeneous mixed training of models; a second acquisition module, configured to call an information acquisition interface corresponding to the first model information to acquire first performance parameters and first event information generated during runtime of each first GPU; a first determination module, configured to determine node performance information of each first computing power node based on the first performance parameters and the first event information, so as to determine the overall performance information of multiple computing power nodes during heterogeneous mixed training of models; and a first display module, configured to display the overall performance information in a preset interface so that users can perform performance analysis on each first GPU during heterogeneous mixed training of models.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product that, when executed by a processor, implements the steps of the method described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In this embodiment, the first model information of multiple first graphics processing units (GPUs) in each first computing power node is obtained, where the first computing power node is the computing power node used for heterogeneous mixed training of models. The information acquisition interface corresponding to the first model information is called to obtain the first performance parameters and first event information generated during the runtime of each first GPU. Based on the first performance parameters and the first event information, the node performance information of each first computing power node is determined to determine the overall performance information of multiple computing power nodes during heterogeneous mixed training of models. The overall performance information is displayed in a preset interface so that users can perform performance analysis on each first GPU during heterogeneous mixed training of models and quickly obtain the performance information of each graphics processing unit for analysis. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a method for acquiring and analyzing the performance of a graphics processing unit according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating the arrangement of operators in a second event information provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the second method for acquiring and analyzing the performance of a graphics processing unit provided in this application embodiment; Figure 4 This is a flowchart illustrating the third method for acquiring and analyzing the performance of a graphics processing unit provided in this application embodiment; Figure 5 This is a flowchart illustrating a graphics processing unit performance acquisition and analysis device provided in an embodiment of this application; Figure 6 This is a flowchart illustrating a graphics processing unit performance acquisition and analysis device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0015] The performance acquisition and analysis method for graphics processing units provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0016] Figure 1This illustration shows a method for acquiring and analyzing the performance of a graphics processing unit (GPU) according to an embodiment of the present invention. This method can be executed by an electronic device, which may include a server and / or a terminal device, such as an in-vehicle terminal or a mobile phone terminal. In other words, the method can be executed by software or hardware installed in the GPU performance acquisition and analysis device, and the method includes the following steps: Step 102: Obtain the first model information of multiple first graphics processing units (GPUs) in each first computing power node.

[0017] The first computing node is the computing node used for heterogeneous mixed training of models.

[0018] The execution entity of the graphics processing unit performance acquisition method described in this application can be a graphics processing unit performance acquisition and analysis system or graphics processing unit performance acquisition and analysis software. This embodiment takes a graphics processing unit performance acquisition and analysis system as an example, wherein the graphics processing unit is a GPU.

[0019] The GPU performance acquisition and analysis system obtains the first model information of multiple GPUs included in each first computing power node. The first computing power node is the computing power node when performing heterogeneous mixed training of models, and the first model information is the model information of each GPU when performing heterogeneous mixed training of models.

[0020] When performing heterogeneous mixed training of models, model training is required through multiple computing nodes (i.e., first computing nodes). Each computing node includes multiple GPUs of the same architecture; that is, a single computing node includes multiple GPUs of the same architecture, while different computing nodes may include GPUs of the same architecture or GPUs of different architectures. The GPU performance acquisition and analysis system obtains the model information of each first GPU included in the multiple computing nodes and uses the obtained model information of each GPU as the first model information. Step 104: Call the information acquisition interface corresponding to the first model information to obtain the first performance parameters and the first event information generated during runtime of each first GPU.

[0021] After obtaining the first model information of each first GPU, the GPU performance acquisition and analysis system calls the information acquisition interface corresponding to the first model information of each first GPU. Through the information acquisition interface, it obtains the first performance parameters of each first GPU and the first event information generated by each first GPU during runtime. The first performance parameters are the performance parameters of each first GPU, and the first event information is the event information generated by each first GPU during runtime.

[0022] In other words, the GPU performance acquisition and analysis system can use the identified GPU model information to call the corresponding official performance acquisition interface to obtain the GPU's performance parameters and event information during model training. For example, for NVIDIA graphics cards, it can directly call interfaces such as the CUPTI dynamic library to track the GPU workload, CUDA interface call status, and standard timestamps of the NVIDIA graphics card.

[0023] Specifically, the performance parameters of the first GPU can include information such as the number of CUDA cores, the number of Tensor cores, memory capacity, memory bandwidth, memory type, and power consumption. The first event information of the first GPU can include communication events, diagnostic events, scheduling events, etc. The specific event types, symbols, and descriptions included in the first event information are shown in the figure below:

[0024] Step 106: Determine the node performance information of each of the first computing power nodes based on the first performance parameters and the first event information, so as to determine the overall performance information of multiple computing power nodes during heterogeneous mixed training of the model.

[0025] After acquiring the first performance parameters and first event information of each first GPU, the GPU performance acquisition and analysis system determines the node performance information of each first computing power node based on the first performance parameters and first event information of the first GPU, and then determines the overall performance information of the overall training process (multiple first computing power nodes) when performing heterogeneous mixed training of the model based on the node information of each first computing power node.

[0026] Specifically, the GPU performance acquisition and analysis system integrates the first performance information and first event information of multiple first GPUs included in each first computing power node, and determines the integrated information as the node performance information of the first computing power node. Then, the node performance information of multiple first computing power nodes is processed as a whole, thereby determining the overall performance information based on the integrated information.

[0027] After determining the node performance information of each first computing power node and the overall performance information during heterogeneous mixed training of the model, the GPU performance acquisition and analysis system can also generate node description information and overall description information based on the determined node performance information and overall performance information. The node description information is the description information of the performance information of each node, and the overall description information is the description information of the overall performance information. For example, the node description information can be the type information of the GPU in the first computing power node, the node IP of the first computing power node, etc.

[0028] Step 108: Display the overall performance information in the preset interface so that users can perform performance analysis on each of the first GPUs during heterogeneous mixed training of models.

[0029] After determining the node performance information of each first computing power node and the overall performance information of multiple first computing power nodes, the GPU performance acquisition and analysis system can display the overall performance information in a preset interface, so that users can perform performance analysis on each first GPU during heterogeneous mixed training of models based on the displayed overall performance information.

[0030] Specifically, the GPU performance acquisition and analysis system can display overall performance information on the display page by taking the first computing power node as the unit. That is, it classifies the overall performance information according to the different computing power nodes, so that the performance information of the same computing power node (node ​​performance information) is displayed as the same type of information, and the performance information of different computing power nodes can be displayed side by side.

[0031] The graphics processing unit performance acquisition and analysis method provided in this embodiment of the invention obtains the first model information of multiple first graphics processing units (GPUs) in each first computing power node, wherein the first computing power node is the computing power node during heterogeneous mixed training of models; calls the information acquisition interface corresponding to the first model information to obtain the first performance parameters and the first event information generated during the runtime of each first GPU; determines the node performance information of each first computing power node based on the first performance parameters and the first event information to determine the overall performance information of multiple computing power nodes during heterogeneous mixed training of models; and displays the overall performance information in a preset interface so that users can perform performance analysis on each first GPU during heterogeneous mixed training of models, and can quickly obtain the performance information of each graphics processing unit for analysis during heterogeneous mixed training of models.

[0032] In one implementation, obtaining the first performance parameters and first event information generated during runtime for each of the first GPUs (step 104) can be performed via steps A1-A2: Step A1: Obtain various types of event information generated by the first GPU during runtime, and determine the first general information and the first specific information in each event information.

[0033] The first general information is the feature information common to all types of event information, and the first unique information is the feature information unique to all types of event information.

[0034] Because GPUs generate various types of event information during runtime, the identified first event information includes multiple different types of event information, and there may be multiple pieces of information for the same type of event information. After acquiring the event information generated by each first GPU during runtime, the GPU performance acquisition and analysis system can not only directly identify the event information generated by the first GPU as the first event information, but also perform processing operations on the first event information and identify the information obtained after processing as the first event information.

[0035] When processing the event information generated by each first GPU, the GPU performance acquisition and analysis system can determine the first common information and the first unique information of multiple event information in the first GPU. The first common information consists of features shared across all types of event information, while the first unique information is unique to each type of event information. In other words, the GPU performance acquisition and analysis system divides event information into common and unique components, identifying the common components as the first common information and the unique components as the first unique information.

[0036] Specifically, in order to cover all event types, this application adopts a two-level structure of "common" and "details". "Common" is used to store the common information of all events (i.e., to store the first common information), which includes: operator name (op), event type (type), timestamp (timestamp), process ID (pid), thread ID (tid), and GPU serial number (cid) within the computing node; "details" is used to store the unique information of all events (i.e., to store the first unique information), which sets different data objects or arrays according to different event types.

[0037] Step A2: Determine the first event information based on the first general information and the first specific information.

[0038] After determining the first general information and the first specific information of the first GPU, the GPU performance acquisition and analysis system can identify the first general information and the first specific information as the first event information. In other words, the GPU performance acquisition and analysis system divides the event information in the first GPU into a general part and a specific part. When multiple event information needs to be stored, only the specific part and the general part are stored, which can reduce the amount of first event information stored.

[0039] In one implementation, the first event information includes multiple basic event information. After obtaining the first performance parameters of each of the first GPUs and the first event information generated during runtime (step 104), steps B1-B4 can also be executed: Step B1: Determine the basic event information that meets the first preset condition as the second event information.

[0040] The first preset condition includes a first preset field, and the second event information includes one or more of the following: communication operator, forward propagation operator, and back propagation operator.

[0041] When performing heterogeneous training of models on a heterogeneous training platform, the speedup ratio refers to the ratio of actual computing power to theoretical computing power. The closer the value is to 1, the smaller the loss caused by mixed training compared to using a single high-performance GPU. It is an important indicator for evaluating the computing power of a mixed training platform. In the process of optimizing the speedup ratio, it is often necessary to calculate the forward or backward propagation time information of the same model when trained on different GPUs. For heterogeneous training schemes using day nodes and NVIDIA nodes, it is only necessary to calculate the forward propagation related operator time and ratio of the model on the two nodes. The model layers can then be divided into two nodes according to this ratio, thereby maximizing the computing efficiency of the two nodes.

[0042] Before determining the time information for forward and backward propagation, it is necessary to determine the forward and backward propagation operators from the first event information. Therefore, after determining the first event information, the GPU performance acquisition and analysis system can determine the basic event information that meets the first preset condition as the second event information. The first preset condition includes the first preset field, which is used to characterize the feature field information of the communication operator, forward propagation operator and backward propagation operator. The second event information is one or more of the communication operator, forward propagation operator and backward propagation operator.

[0043] In other words, the communication operator, forward propagation operator, and backpropagation operator each have corresponding feature fields. By acquiring and determining these feature fields as the first preset field, it is possible to identify the basic event information as one of the communication operator, forward propagation operator, or backpropagation operator when the basic event information includes the first preset field. The GPU performance acquisition and analysis system can determine which basic event information contains the first preset field among the multiple basic event information included in the first event information, and then identify the determined basic event information as the second event information.

[0044] Step B2: Based on the second event information, determine multiple first operators among the multiple pairs of adjacent communication operators.

[0045] Wherein, any one of the first operators includes one of the forward propagation operator and the back propagation operator, and a plurality of the first operators include the forward propagation operator and the back propagation operator.

[0046] After determining the second event information, the GPU performance acquisition and analysis system identifies multiple first operators among the multiple pairwise adjacent communication operators included in the second event information. Each first operator includes either a forward propagation operator or a backward propagation operator, and the multiple first operators include not only forward propagation operators but also backward propagation operators. For example, if there are two first operators, one may be a forward propagation operator and the other a backward propagation operator; if there are five first operators, there may be one forward propagation operator and four backward propagation operators.

[0047] The second event information is composed of forward propagation operators, back propagation operators, and communication operators arranged in a sequence. Each pair of adjacent communication operators includes either a forward propagation operator or a back propagation operator, and the number of such operators may be one or more. The GPU performance acquisition and analysis system identifies the forward propagation operator or back propagation operator included in any two adjacent communication operators as the first operator. That is, the first operator is either a forward propagation operator or a back propagation operator, and the number of such operators may be one or more. Furthermore, among the multiple first operators, at least one first operator includes an operator that is a forward propagation operator, and at least one first operator includes an operator that is a back propagation operator.

[0048] The forward propagation operator, backward propagation operator, and communication operator included in the second event information can be generated by different data communication methods (such as data parallel communication, tensor parallel communication, and pipelined parallel communication), and the arrangement rules of the forward propagation operator, backward propagation operator, and communication operator generated by different data communication methods are different.

[0049] Specifically, in practical applications, the boundaries of operators are generally NCCL communication operators or relatively long periods of GPU idle time. The length of this idle time is easily affected by various factors, making it difficult to pinpoint. To improve performance utilization, mixed training frameworks often employ DP, TP, and PP divisions. These divisions lead to certain patterns in NCCL communication, which creates conditions for automatically locating the boundaries of the forward and backward propagation processes. Therefore, filtering and recording event timestamps whose operator names (op) contain "fwd", "bwd", and "nccl" (i.e., the first preset field) is essential for subsequent calculations. Operators with the "fwd" field in their names represent operators in the forward propagation phase, operators with the "bwd" field represent operators in the backward propagation phase, and operators with the "nccl" field represent communication operators.

[0050] Because the operators for different data communication methods are arranged differently, after determining multiple first operators, the GPU performance acquisition and analysis system will also determine the communication method of the first operator based on the content information of the first operator, so as to determine the boundary information based on the communication method of the first operator.

[0051] More specifically, such as Figure 2 As shown in (a), the operators in the second event information are arranged in parallel data communication, where NCCL represents a communication process, FP represents a forward propagation process, and BP represents a backward propagation process. Figure 2 As shown in (b), the arrangement of operators in the second event information during tensor parallel communication is as follows: Figure 2 As shown in (c), the arrangement of each operator in the second event information during pipeline parallel communication is shown. FWD and BWD represent sub-processes containing forward and backward computation operators, respectively, and others represent other operator processes that together with FWD or BWD form an FP or BP.

[0052] Step B3: Obtain the boundary information of the communication operator and the number of layers divided by the first GPU.

[0053] The GPU performance acquisition and analysis system obtains the boundary information of each communication operator and the number of model layers N divided by the first GPU.

[0054] Step B4: Determine the boundary information of the forward propagation operator and the backward propagation operator based on the boundary information of the communication operator and the number of layers, so as to determine the speedup ratio during heterogeneous mixed training of the model based on the boundary information of the forward propagation operator and the backward propagation operator.

[0055] After determining the boundary information and number of layers for the first operator and communication operator, the GPU performance acquisition and analysis system determines the boundary information for the forward propagation operator and the backpropagation operator based on these boundary information and the number of layers. Based on the determined boundary information for the forward propagation operator and the backpropagation operator, the system calculates the time for a single forward or backpropagation process corresponding to the first GPU of each first computing power node, according to the timestamp. This data is then summarized and output in CSV format for speedup optimization reference.

[0056] Specifically, different methods for determining the boundaries exist for different communication methods of the first operator. When the first operator uses DP communication partitioning: there are N consecutive FP or BP processes between two NCCLs. It is only necessary to find the selected "nccl" operator, then determine whether the intermediate operator is a forward or backward process, and divide by the number of layers N of a single GPU to locate the boundaries of a single forward propagation process and a single backward propagation process. For N, if it is a uniform partition, simply divide the total number of layers of the model by the size of the PP group; for a non-uniform partition, simply read the relevant partitioning parameters from the configuration file. When the first operator uses PP communication partitioning: due to the existence of the pipeline, PP partitioning often has three cases: NCCLs sandwiched between FP and FP, FP and BP, and BP and BP. Therefore, it is only necessary to find the selected "nccl" operator, and then determine the operator category between two "nccls" to locate the boundaries of a single forward propagation process and a single backward propagation process. When the communication method of the first operator is TP communication partitioning: Unlike layer or data partitioning, tensor parallelism includes two parts: partitioning and summarizing. For the current large model based on Transformer, each model layer contains two forward communications and two backward communications to correspond to partitioning and summarizing. Therefore, firstly, it is necessary to find the "fwd" or "bwd" operator for filtering and positioning; then, find the left and right communications to locate the FWD and BWD processes. For the FWD process, it is only necessary to calculate the next "nccl" operator to determine the Others process. Combining the two can locate the boundary of a single forward propagation process; for the BWD process, it is only necessary to find the next "nccl" operator to determine the Others process. Combining the two can locate the boundary of a single backward propagation process.

[0057] Specifically, traditional forward and backward propagation calculation processes typically involve locating relevant operators using visualization tools, followed by manual statistical analysis and calculations—a tedious and technically demanding process. This application's embodiment proposes an automated calculation module that directly locates and calculates relevant operators from performance data acquisition files, effectively solving this problem.

[0058] In one implementation, determining the boundary information of the forward propagation operator and the backward propagation operator based on the boundary information of the communication operator and the number of layers (step B4) can be performed via steps C1-C3: Step C1: Determine the first data communication method included in the second event information based on multiple first operators.

[0059] The first data communication method includes one or more of the following: data parallel communication, tensor parallel communication, and pipelined parallel communication.

[0060] The GPU performance acquisition and analysis system can determine the data communication method included in the second event information, i.e., the first data communication method, by determining the data communication method of each first operator. The first data communication method includes one or more of data parallel communication, tensor parallel communication, and pipelined parallel communication. For example, when all first operators are data parallel communication, the data communication method included in the second event information is determined to be only data parallel communication; when some first operators are tensor parallel communication and some are pipelined parallel communication, the data communication method included in the second event information is determined to be both tensor parallel communication and pipelined parallel communication.

[0061] Step C2: When the first data communication method is multiple data communication methods, determine the second data communication method according to the preset priority rule.

[0062] The multiple communication methods include at least two of the data parallel communication, tensor parallel communication and pipelined parallel communication, and the second data communication method is one of the multiple data communication methods determined based on a preset priority rule.

[0063] After determining the data communication methods included in the second event information, the GPU performance acquisition and analysis system determines whether there are multiple data communication methods. If so, it determines the second data communication method according to a preset priority rule. The second data communication method is one of the multiple data communication methods included in the second event information, determined based on the preset priority rule. In other words, the GPU performance acquisition system selects the highest priority data communication method from the multiple data communication methods included in the second event information according to the preset priority rule and identifies it as the second data communication method.

[0064] In practical applications, the second event information often involves multiple communication partitions. When multiple communication methods exist within the second event information, the boundary information of the corresponding forward and backward propagation operators is not determined based on the first operator corresponding to each of the multiple communication methods. Instead, a preset priority rule is used to select the communication method with the highest priority to determine the boundary information corresponding to the forward and backward propagation operators. Furthermore, the preset priority rule can be determined by the communication partition granularity of the three data communication methods, giving higher priority to communication methods with smaller partition granularity. The partition granularity of the three communication methods, from largest to smallest, is DP > PP > TP. Therefore, the priority of the three communication methods is TP > PP > DP. For example, when both DP and PP exist, the TP communication method is selected to determine the boundary information. This reduces the amount of data processing, saves time, and makes better use of computing resources.

[0065] Step C3: Determine the boundary information of the forward propagation operator and the backward propagation operator based on the first operator corresponding to the second data communication method, the boundary information of the communication operator, and the number of layers.

[0066] After determining the second data communication method, the GPU performance acquisition and analysis system determines the boundary information of the forward propagation operator and the back propagation operator based on the first operator corresponding to the second communication method, the boundary information of the communication operator, and the number of communication layers.

[0067] Figure 3 This is a flowchart illustrating a second method for acquiring and analyzing the performance of a graphics processing unit, as provided in one embodiment of this specification. Figure 3 As shown, the schematic diagram includes: Step 302: Obtain the first model information of the multiple first graphics processing units (GPUs) in each first computing power node.

[0068] The first computing node is the computing node used for heterogeneous mixed training of models.

[0069] Step 304: Call the information acquisition interface corresponding to the first model information to obtain the first performance parameters and the first event information generated during runtime of each first GPU.

[0070] The first event information includes multiple basic event information.

[0071] Step 306: Determine the basic event information that meets the first preset condition as the second event information.

[0072] The first preset condition includes a first preset field, and the second event information includes one or more of the following: communication operator, forward propagation operator, and back propagation operator.

[0073] Step 308: Based on the second event information, determine a plurality of first operators among a plurality of pairwise adjacent communication operators.

[0074] Wherein, any one of the first operators includes one of the forward propagation operator and the back propagation operator, and a plurality of the first operators include the forward propagation operator and the back propagation operator.

[0075] Step 310: Obtain the boundary information of the communication operator and the number of layers divided by the first GPU.

[0076] Step 312: Determine the first data communication method included in the second event information based on multiple first operators.

[0077] The first data communication method includes one or more of the following: data parallel communication, tensor parallel communication, and pipelined parallel communication.

[0078] Step 314: When the first data communication method is multiple data communication methods, determine the second data communication method according to the preset priority rule.

[0079] The multiple communication methods include at least two of the data parallel communication, tensor parallel communication and pipelined parallel communication, and the second data communication method is one of the multiple data communication methods determined based on a preset priority rule.

[0080] Step 316: Determine the boundary information of the forward propagation operator and the backward propagation operator based on the first operator corresponding to the second data communication method, the boundary information of the communication operator, and the number of layers, so as to determine the speedup ratio during heterogeneous mixed training of the model based on the boundary information of the forward propagation operator and the backward propagation operator.

[0081] Step 318: Determine the node performance information of each of the first computing power nodes based on the first performance parameters and the first event information, so as to determine the overall performance information of multiple computing power nodes during heterogeneous mixed training of the model.

[0082] Step 320: Display the overall performance information in the preset interface so that users can perform performance analysis on each of the first GPUs during heterogeneous mixed training of models.

[0083] In the embodiments described in the specification, by determining the first event information that meets the first preset condition as the second event information, and then determining the second data communication method, the forward propagation and backward propagation operators in the event information can be automatically located, thereby improving the efficiency and speed of operator location.

[0084] In one implementation, determining the node performance information of each of the first computing power nodes based on the first performance parameters and the first event information to determine the overall performance information of multiple computing power nodes during heterogeneous mixed training of the model (step 106) can be performed via steps D1-D2: Step D1: Determine the first performance parameters and first event information of the multiple first GPUs included in the first computing power node as the node performance information of the first computing power node and store it in a first preset JSON file.

[0085] The GPU performance acquisition and analysis system uses the first performance parameters and first event information of all first GPUs included in the first computing power node as the node performance information of that first computing power node and stores the node performance information in a first preset JSON file. To better adapt to different GPU models and acquire GPU performance parameters and event information, this embodiment uses JSON format as the basic format for the performance acquisition file. That is, this embodiment saves the first performance parameters and first event information in JSON format, and saves the performance parameters and event information of all GPUs in each first computing power node in JSON format.

[0086] Specifically, a JSON structure is set up at the first computing node level to contain the first performance parameters and first event information of all the first GPUs of that first computing node. Its specific content can be seen in the following figure:

[0087] Step D2: Determine the node performance information of multiple first computing power nodes as the overall performance information and store it in a second preset JSON file.

[0088] After saving the node performance information of the first computing power node in JSON format, the GPU performance acquisition and analysis system will also determine the node performance information of multiple first computing power nodes as the overall performance information and store it in a second preset JSON file.

[0089] To display the performance information of multiple first-level computing nodes, it is necessary to merge the performance data collected from multiple first-level computing nodes (i.e., node performance information). This embodiment of the application utilizes an automated method to merge all specified JSON files (node ​​performance information): First, the main script checks the input parameters to ensure correctness, then initializes the basic structure of a final JSON file (for storing overall performance information) and creates a temporary directory to store the preprocessed files; second, it iterates through all JSON files under the specified path, calling the preprocessing script to preprocess each file, such as checking and ensuring that node information like hostname and GPU model has been added; finally, the jq tool is used to merge all processed JSON data and write it to the user-specified output file.

[0090] Specifically, when the GPU performance acquisition and analysis module displays overall performance information, the visualization analysis module can parse and visualize the JSON format files (node ​​performance information and overall performance information) designed in this embodiment. To reduce development costs, the visualization analysis module in this embodiment is developed based on Google's open-source Perfetto. Perfetto is a production-grade, stable, and reliable toolset that provides a complete analysis library for analyzing and tracking data, as well as a comprehensive visualization web interface. Importantly, this tool has excellent support for JSON format data. For the backend, the trace_model.ts file in the Perfetto open-source code is modified to support parsing data in the designed JSON format; secondly, the Redux call parsing logic and state management part are modified so that newly created views can obtain the parsed data. For the frontend interface, the frontend UI is modified and integrated to display node information. In this embodiment, IP and GPU architecture are used as identifiers to identify each node, and each node internally displays the model of each GPU and the corresponding CUDA process.

[0091] In one implementation, steps E1-E2 can also be performed: Step E1: Receive the second model information sent by the user.

[0092] The second model information refers to the model information of the second GPU added during heterogeneous mixed training of the model.

[0093] During heterogeneous model training, the GPU model needs to be adjusted. This adjustment requires receiving a second model information sent by the user. This second model information refers to the model of the second GPU, which is either newly added or needs adjustment during the heterogeneous training process. The second model information can be sent directly by the user or indirectly obtained through adjustment commands. These adjustment commands instruct the GPU performance acquisition and analysis system to acquire and display the performance information of the second GPU, and this instruction includes the second model information.

[0094] Step E2: Call the information acquisition interface corresponding to the second model information to obtain the second performance parameters of the second GPU and the second event information generated during runtime.

[0095] After obtaining the second model information, the GPU performance acquisition and analysis system calls the information acquisition interface corresponding to the second model information. Then, through this interface, it obtains the second performance parameters and second event information of the second GPU. The second performance parameters are the performance parameters of the second GPU, and the second event information is the event information generated during the runtime of the second GPU. In other words, when a heterogeneous training platform needs to add a GPU with a new architecture, only the corresponding interface needs to be changed and the corresponding parsing function added to the script, resulting in low expansion costs.

[0096] Figure 4 This is a flowchart illustrating a third method for acquiring and analyzing the performance of a graphics processing unit, as provided in an embodiment of this specification. Figure 4 As shown, the schematic diagram includes: Step 402: Obtain the first model information of multiple first graphics processing units (GPUs) in each first computing power node.

[0097] The first computing node is the computing node used for heterogeneous mixed training of models.

[0098] Step 404: Call the information acquisition interface corresponding to the first model information to obtain the first performance parameters and the first event information generated during runtime of each first GPU.

[0099] Step 406: Determine the first performance parameters and first event information of the multiple first GPUs included in the first computing power node as the node performance information of the first computing power node and store them in a first preset JSON file.

[0100] Step 408: Determine the node performance information of multiple first computing power nodes as the overall performance information and store it in a second preset JSON file.

[0101] Step 410: Display the overall performance information in the preset interface so that users can perform performance analysis on each of the first GPUs during heterogeneous mixed training of models.

[0102] Step 412: Receive the second model information sent by the user.

[0103] The second model information refers to the model information of the second GPU added during heterogeneous mixed training of the model.

[0104] Step 414: Call the information acquisition interface corresponding to the second model information to obtain the second performance parameters of the second GPU and the second event information generated during runtime.

[0105] In the embodiments described in the specification, by storing the second event information, node performance information, and overall performance information in JSON format, it is convenient to collect and process data, thereby improving the efficiency of information collection; by receiving the second model information sent by the user, the performance information of the second GPU can be obtained according to the corresponding interface, thereby reducing the expansion cost during heterogeneous mixed training of models.

[0106] It should be noted that the graphics processing unit (GPU) performance acquisition and analysis method provided in this application embodiment can be executed by a GPU performance acquisition and analysis device, or by a control module within that GPU performance acquisition and analysis device for executing the GPU performance acquisition and analysis method. This application embodiment uses the GPU performance acquisition and analysis device executing the GPU performance acquisition and analysis method as an example to illustrate the GPU performance acquisition and analysis device provided in this application embodiment.

[0107] Figure 5 This is a schematic diagram of the structure of a graphics processing unit performance acquisition and analysis device according to an embodiment of the present invention. Figure 5 As shown, the graphics processing unit performance acquisition and analysis device includes: a first acquisition module 502, a second acquisition module 504, a first determination module 506, and a first display module 508.

[0108] The first acquisition module 502 is used to acquire the first model information of multiple first graphics processing units (GPUs) in each first computing power node, wherein the first computing power node is the computing power node when performing heterogeneous mixed training of models. The second acquisition module 504 is used to call the information acquisition interface corresponding to the first model information to acquire the first performance parameters and the first event information generated during runtime of each first GPU. The first determining module 506 is used to determine the node performance information of each of the first computing power nodes based on the first performance parameters and the first event information, so as to determine the overall performance information of multiple computing power nodes during heterogeneous mixed training of the model. The first display module 508 is used to display the overall performance information in a preset interface so that users can perform performance analysis on each of the first GPUs during heterogeneous mixed training of models.

[0109] The graphics processing unit performance acquisition and analysis device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0110] The graphics processing unit performance acquisition and analysis device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0111] The graphics processing unit performance acquisition and analysis device provided in this application embodiment can achieve... Figures 1 to 4 The various processes implemented in the method embodiments are not described in detail here to avoid repetition.

[0112] Based on the same technical concept, embodiments of this application also provide an electronic device for executing the above-described graphics processing unit performance acquisition and analysis method. Figure 6 This is a schematic diagram of the structure of an electronic device to implement various embodiments of this application. The electronic device can vary significantly due to differences in configuration or performance, and may include a processor 602, a communications interface 604, a memory 606, and a communication bus 608. The processor 602, communications interface 604, and memory 606 communicate with each other via the communication bus 608. The processor 602 can call a computer program stored in the memory 606 and executable on the processor 602 to perform the following steps: Obtain the first model information of multiple first graphics processing units (GPUs) in each first computing power node, where the first computing power node is the computing power node used for heterogeneous mixed training of models. Call the information acquisition interface corresponding to the first model information to obtain the first performance parameters and the first event information generated during runtime of each first GPU; Based on the first performance parameters and the first event information, determine the node performance information of each of the first computing power nodes, so as to determine the overall performance information of multiple computing power nodes during heterogeneous mixed training of the model. The overall performance information is displayed in the preset interface so that users can perform performance analysis on each of the first GPUs during heterogeneous mixed training of models.

[0113] In one implementation, obtaining the first performance parameters and runtime event information of each of the first GPUs includes: The system acquires various types of event information generated during the runtime of the first GPU, and determines the first general information and the first unique information in each event information. The first general information is the common feature information in each type of event information, and the first unique information is the unique feature information in each type of event information. The first event information is determined based on the first general information and the first specific information.

[0114] In one implementation, the first event information includes multiple basic event information. After acquiring the first performance parameters of each of the first GPUs and the first event information generated during runtime, the method further includes: The basic event information that meets the first preset condition is determined as the second event information. The first preset condition includes a first preset field. The second event information includes one or more of the following: communication operator, forward propagation operator, and backward propagation operator. Based on the second event information, a plurality of first operators are determined among a plurality of pairwise adjacent communication operators, each of which includes one of the forward propagation operator and the backward propagation operator, and the plurality of first operators include both the forward propagation operator and the backward propagation operator; Obtain the boundary information of the communication operator and the number of layers divided by the first GPU; The boundary information of the forward propagation operator and the backpropagation operator is determined based on the boundary information of the communication operator and the number of layers, so as to determine the speedup ratio during heterogeneous mixed training of the model based on the boundary information of the forward propagation operator and the backpropagation operator.

[0115] In one implementation, determining the boundary information of the forward propagation operator and the backward propagation operator based on the boundary information of the communication operator and the number of layers includes: Based on multiple first operators, a first data communication method is determined in the second event information, and the first data communication method includes one or more of data parallel communication, tensor parallel communication, and pipelined parallel communication. When the first data communication method is multiple data communication methods, the second data communication method is determined according to a preset priority rule. The multiple communication methods include at least two of the data parallel communication, tensor parallel communication and pipeline parallel communication. The second data communication method is one of the multiple data communication methods determined based on the preset priority rule. The boundary information of the forward propagation operator and the backward propagation operator is determined based on the first operator corresponding to the second data communication method, the boundary information of the communication operator, and the number of layers.

[0116] In one implementation, determining the node performance information of each of the first computing power nodes based on the first performance parameter and the first event information to determine the overall performance information of multiple computing power nodes during heterogeneous mixed training of the model includes: The first performance parameters and first event information of the multiple first GPUs included in the first computing power node are determined as the node performance information of the first computing power node and stored in a first preset JSON file; The node performance information of multiple first computing power nodes is determined as the overall performance information and stored in a second preset JSON file.

[0117] In one implementation, the method further includes: Receive the second model information sent by the user. The second model information is the model information of the second GPU added during heterogeneous training of the model. Call the information acquisition interface corresponding to the second model information to obtain the second performance parameters of the second GPU and the second event information generated during runtime.

[0118] The specific execution steps can be found in the various steps of the above-described embodiment of the graphics processing unit performance acquisition and analysis method, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0119] It should be noted that the electronic devices in the embodiments of this application include: servers, terminals, or other devices besides terminals.

[0120] The above electronic device structure does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or arrange them differently. For example, an input unit may include a Graphics Processing Unit (GPU) and a microphone, and a display unit may use a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar display panels. User input units include at least one of a touch panel and other input devices. A touch panel is also called a touchscreen. Other input devices may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be elaborated further here.

[0121] Memory can be used to store software programs and various data. Memory can primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area can store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, memory can include volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0122] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and applications, while the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor.

[0123] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described graphics processing unit performance acquisition and analysis method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0124] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0125] This application also provides a computer program product. When the computer program product is executed by a processor, it implements the various processes of the above-described graphics processing unit performance acquisition and analysis method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0126] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described graphics processing unit performance acquisition and analysis method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0127] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0128] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0130] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method of graphics processing unit performance collection analysis, the method comprising: The method comprises the following steps: obtaining first model information of a plurality of first graphic processing units (GPUs) in each first computing node, the first computing node being a computing node during model heterogeneous mixed training; calling an information collection interface corresponding to the first model information to obtain first performance parameters of each first GPU and first event information generated during running; determining node performance information of each first computing node based on the first performance parameters and the first event information to determine overall performance information of a plurality of computing nodes during model heterogeneous mixed training; displaying the overall performance information in a preset interface to enable a user to perform performance analysis on each first GPU during model heterogeneous mixed training.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining a plurality of types of event information generated during running of the first GPU and determining first common information and first unique information in each event information, the first common information being characteristic information common to each type of event information, and the first unique information being characteristic information unique to each type of event information; determining the first event information based on the first common information and the first unique information.

3. The method of claim 1, wherein, The first event information comprises a plurality of basic event information, and after obtaining the first performance parameters of each first GPU and the first event information generated during running, the method further comprises the following steps: determining the basic event information satisfying a first preset condition as second event information, the first preset condition comprising a first preset field, and the second event information comprising one or more of a communication operator, a forward propagation operator and a backward propagation operator; based on the second event information, determining a plurality of first operators in a plurality of communication operators that are adjacent to each other, any first operator comprising one of the forward propagation operator and the backward propagation operator, and a plurality of first operators comprising the forward propagation operator and the backward propagation operator; obtaining boundary information of the communication operator and a number of layers divided by the first GPU; determining boundary information of the forward propagation operator and the backward propagation operator based on the boundary information of the communication operator and the number of layers to determine an acceleration ratio during model heterogeneous mixed training based on the boundary information of the forward propagation operator and the backward propagation operator.

4. The method of claim 3, wherein, The method comprises the following steps: based on a plurality of first operators, determining a first data communication mode included in the second event information, the first data communication mode comprising one or more of data parallel communication, tensor parallel communication and pipeline parallel communication; when the first data communication mode is a plurality of data communication modes, determining a second data communication mode according to a preset priority rule, the plurality of communication modes comprising at least two of the data parallel communication, the tensor parallel communication and the pipeline parallel communication, and the second data communication mode being one of the plurality of data communication modes determined based on the preset priority rule. Determine the boundary information of the forward propagation operator and the backward propagation operator based on the first operator corresponding to the second data communication mode, the boundary information of the communication operator, and the number of layers.

5. The method of claim 1, wherein, The method further comprises: Determine the node performance information of each first computing power node based on the first performance parameter and the first event information, to determine the overall performance information of multiple computing power nodes in model heterogeneous mixed training. Determine the first performance parameter and the first event information of multiple first GPUs included in the first computing power node as the node performance information of the first computing power node and store them in a first preset json file.

6. The method of claim 1, wherein, Determine the node performance information of multiple first computing power nodes as the overall performance information and store them in a second preset json file. The method further comprises: Receive second model information sent by a user, the second model information being model information of a second GPU newly added in model heterogeneous mixed training; 7. A graphics processing unit performance collection analysis apparatus, comprising: Call an information collection interface corresponding to the second model information to obtain second performance parameters of the second GPU and second event information generated during running. Comprise: A first obtaining module is configured to obtain first model information of multiple first GPUs in each first computing power node, the first computing power node being a computing power node in model heterogeneous mixed training; A second obtaining module is configured to call an information collection interface corresponding to the first model information to obtain first performance parameters of each first GPU and first event information generated during running; A first determining module is configured to determine node performance information of each first computing power node based on the first performance parameter and the first event information, to determine overall performance information of multiple computing power nodes in model heterogeneous mixed training; 8. A computer device, comprising: A first display module is configured to display the overall performance information in a preset interface, to enable a user to perform performance analysis on each first GPU in model heterogeneous mixed training. The device comprises: A processor; and 9. A storage medium, characterized by A memory arranged to store computer executable instructions configured to be executed by the processor, the executable instructions comprising steps for performing the steps of the GPU performance collection and analysis method according to any one of claims 1 to 6.

10. A computer program product, characterised in that, The storage medium is configured to store computer executable instructions, the executable instructions enabling a computer to perform the GPU performance collection and analysis method according to any one of claims 1 to 6. A computer program is included, which, when executed by a processor, implements the GPU performance collection and analysis method according to any one of claims 1 to 6.

Images