Method for obtaining accelerator card performance, accelerator card performance testing equipment and product

Abstract

The present invention discloses a method for obtaining the performance of an accelerator card, an accelerator card performance testing device and a product, and relates to the technical field of accelerator card performance testing. In this method, the performance of the accelerator card is tested through a universal access platform; secondly, the task to be executed is indirectly deployed on the accelerator card through the module corresponding to the task, thereby realizing the decoupling of the task to be executed and the accelerator card actually used. Therefore, the performance analysis of the module corresponding to the task of the universal access platform naturally does not rely on the underlying accelerator card, thereby realizing the hardware universality of the performance analysis; thirdly, the accelerator card obtains the task to be executed from the same training framework, ensuring that the training framework version is the same. It can be seen that this method, while realizing the performance testing of different accelerator cards, breaks the use barriers between the dedicated performance analysis tools of each accelerator card, and isolates the development differences of different accelerator card manufacturers at the training framework level, so that the difficulty of users selecting accelerator cards is reduced and the efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of accelerator card performance testing, and in particular to a method for obtaining accelerator card performance, accelerator card performance testing equipment, and products. Background Art

[0002] With the rapid development of artificial intelligence technology, the requirements for the computing performance of accelerator cards are getting higher and higher. In order to understand the performance of accelerator cards, it is necessary to analyze the performance of accelerator cards.

[0003] Related technologies rely on performance analysis tools provided by accelerator card manufacturers to analyze accelerator card performance. However, these tools are specific to their own accelerator cards, meaning that performance analysis tools from different manufacturers are incompatible. Furthermore, because different performance analysis tools often differ significantly in terms of test environment, usage, analysis targets, and result presentation, selecting an accelerator card becomes more difficult and inefficient for users.

[0004] It can be seen that while realizing the performance analysis of different accelerator cards, how to reduce the difficulty for users to select accelerator cards and improve the efficiency of users in selecting accelerator cards are technical problems that people in this field urgently need to solve. Summary of the Invention

[0005] The present invention provides a method for obtaining the performance of an accelerator card, an accelerator card performance testing device, and a product to at least solve the problem in the related art that performance analysis tools from different manufacturers are not compatible; and different performance analysis tools often have large differences in test environment, usage method, analysis object, result presentation, etc., which increases the difficulty and reduces the efficiency of users in selecting accelerator cards.

[0006] The present invention provides a method for obtaining accelerator card performance, comprising:

[0007] Obtain an accelerator card to be tested and a universal access platform for performing accelerator card performance testing; wherein the universal access platform is deployed with a training framework and task-specific modules; the task-specific modules include at least a runtime module and an operator module;

[0008] When it is detected that the universal access platform is deployed on the accelerator card to be tested, the system obtains the task to be executed through the training framework of the universal access platform and schedules the module corresponding to the task according to the task to be executed. The module corresponding to the task searches the task library of the task to be executed on the accelerator card to enable the accelerator card to execute the task to be executed according to the interface corresponding to the task to be executed.

[0009] The performance of the accelerator card to be tested when executing the task to be executed is obtained through the universal access platform.

[0010] The beneficial effects of the present invention are as follows: first, in a method for obtaining the performance of an accelerator card, when performing a performance test on the accelerator card to be tested, a universal access platform is deployed on the accelerator card to be tested, a task to be executed is obtained through a training framework of the universal access platform, and a module corresponding to the task to be executed is scheduled according to the task to be executed; through the module corresponding to the task, an interface corresponding to the task to be executed is searched from a task library of the task to be executed located in the accelerator card to be tested, so that the accelerator card to be tested executes the task to be executed according to the interface corresponding to the task to be executed; finally, the performance of the accelerator card to be tested when executing the task to be executed is obtained through the universal access platform. This method achieves the performance test of the accelerator card. Secondly, a task-corresponding module is deployed on the universal access platform, so that the task to be executed is indirectly deployed on the accelerator card to be tested through the task-corresponding module, rather than directly deploying the task to be executed on the accelerator card to be tested. This achieves the decoupling of the task to be executed and the accelerator card to be tested in actual use. Therefore, the performance analysis of the task-corresponding module of the universal access platform is naturally not dependent on the underlying accelerator card to be tested, thereby achieving the hardware universality of the performance analysis. Thirdly, in the method in which the performance analysis tool provided by the accelerator card manufacturer is only used for the performance test of its own accelerator card, due to the different software development progress of each accelerator card manufacturer, the test environment provided by the accelerator card manufacturer is often difficult to align in terms of the version of the training framework. When the version of the training framework cannot be aligned, it is difficult to fairly evaluate the hardware performance of different accelerator cards in executing tasks. In the method provided by the present invention, a training framework is set in the universal access platform. When the universal access platform is used to test multiple accelerator cards to be tested, the task to be executed is obtained from the same training framework, that is, the same training framework version is guaranteed. The performance of different accelerator cards to be tested is tested in the same test environment, which improves the accuracy and fairness of the performance test results. It can be seen that the method provided by the present invention not only realizes the performance testing of different accelerator cards, but also breaks down the usage barriers between the dedicated performance analysis tools of each accelerator card, and isolates the development differences of different accelerator card manufacturers at the training framework level, greatly reducing the difficulty for users to select accelerator cards and improving the efficiency of users in selecting accelerator cards.

[0011] The present invention also provides an accelerator card performance testing device, which is connected to an accelerator card to be tested; the accelerator card performance testing device is deployed with a training framework and modules corresponding to tasks; the modules corresponding to tasks include at least a runtime module and an operator module; the training framework of the accelerator card performance testing device is used to obtain tasks to be executed and schedule modules corresponding to tasks according to the tasks to be executed; the modules corresponding to the tasks are used to search for interfaces corresponding to the tasks to be executed from a task library of tasks to be executed located in the accelerator card to be tested, so that the accelerator card to be tested executes the tasks to be executed according to the interfaces corresponding to the tasks to be executed; and the performance of the accelerator card to be tested when executing the tasks to be executed is obtained.

[0012] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing any of the steps of the above-mentioned method for obtaining the performance of an accelerator card when executing the computer program.

[0013] The present invention also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for obtaining the performance of an accelerator card are implemented.

[0014] The present invention also provides a computer program product, including a computer program, which implements the steps of any of the above methods for obtaining the performance of an accelerator card when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 A flowchart of a method for obtaining accelerator card performance provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of deploying a pre-trained language model training task onto an accelerator card provided by an embodiment of the present invention;

[0018] Figure 3 A multi-level architecture diagram for model training performance analysis provided by an embodiment of the present invention;

[0019] Figure 4 A diagram of a general implementation architecture for an artificial intelligence computing power execution framework and training performance analysis provided by an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of a training time extraction tool for extracting the time consumption of a model training task provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0022] It should be noted that, in the description of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. The terms "first," "second," etc., in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence.

[0023] In order to perform performance testing on accelerator cards, related technologies use performance analysis tools provided by accelerator card manufacturers to perform performance analysis. This relies on accelerator card manufacturers providing performance analysis tools. Performance analysis tools from different accelerator card manufacturers are not compatible, and often have significant differences in test environment, usage, analysis objects, and result presentation. This greatly increases the research workload for users selecting accelerator cards for model training. Specifically:

[0024] In terms of testing environments, each accelerator card manufacturer provides its own test image. Users need to familiarize themselves with how to set up different environments, and the workload increases linearly with the type of accelerator card being researched. Secondly, due to the varying software development progress of each accelerator card manufacturer, it is often difficult to align the versions of the provided test environments for training software frameworks, such as PyTorch and Transformers. Software is also known to significantly impact training performance. Therefore, without aligned software versions, it is difficult to fairly evaluate the hardware performance of different accelerator cards on model training tasks.

[0025] The performance analysis tools provided by accelerator card manufacturers are specific to their own accelerator cards. Some manufacturers' performance analysis tools are configured via command lines, while others use a graphical interface. Different performance analysis tools also present results in different ways. These specialized designs and differences can hinder model training users in selecting training hardware, increasing the workload of evaluating the performance of individual accelerator cards and making accelerator selection more difficult and inefficient.

[0026] Therefore, the present invention provides a universal performance analysis method suitable for performance testing of various accelerator cards.

[0027] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods. Figure 1 A flowchart of a method for obtaining accelerator card performance provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0028] S10: Obtain an accelerator card to be tested and a universal access platform for performing accelerator card performance testing; wherein the universal access platform is deployed with a training framework and task-specific modules; the task-specific modules include at least a runtime module and an operator module;

[0029] S11: When it is detected that the universal access platform is deployed on the accelerator card to be tested, a task to be executed is obtained through the training framework of the universal access platform, and a module corresponding to the task is scheduled according to the task to be executed; the module corresponding to the task is used to search the task library of the task to be executed located on the accelerator card to enable the accelerator card to execute the task to be executed according to the interface corresponding to the task to be executed;

[0030] S12: Obtaining the performance of the accelerator card to be tested when executing the task to be executed through the universal access platform.

[0031] There is no limitation on the accelerator card to be tested, and it is determined according to the actual situation. Taking into account the method in which the performance analysis tool provided by the accelerator card manufacturer is only for the performance test of its own accelerator card, due to the different software development progress of each accelerator card manufacturer, the test environment provided by them is often difficult to align in terms of the version of the training framework. When the version of the training framework cannot be aligned, it is difficult to fairly evaluate the hardware performance of different accelerator cards in executing tasks. Therefore, in the method provided by the present invention, a training framework is provided in the universal access platform. The training framework includes PyTorch, Transformers, etc. It is worth noting that the type of task to be performed is different, and the training framework is different. The type of task to be performed is, for example, a training task type and an inference task type. The training framework used by the training task type is different from the training framework used by the inference task type.

[0032] To ensure that the universal access platform is suitable for testing the performance of a wide range of accelerator cards, task-specific modules are deployed on the universal access platform. These modules include at least a runtime module and an operator module. The runtime and operators are software that is very close to the underlying hardware. Different accelerator cards have different runtime and operator implementations within the universal platform.

[0033] When using the universal access platform to perform performance testing on the accelerator card to be tested, the universal access platform is deployed on the accelerator card to be tested. When the universal access platform is detected to be deployed on the accelerator card to be tested, the task to be executed is obtained through the training framework of the universal access platform. In order to achieve a refined analysis of the performance of the accelerator card to be tested, the total time consumed by model training or inference is gradually refined to the operator granularity. That is, the task to be executed is decomposed. Specifically, after obtaining the task to be executed, the universal access platform uses the module corresponding to the task to be executed to search the task library corresponding to the task to be executed, and before searching for the interface corresponding to the task to be executed, it also includes:

[0034] Decompose the tasks to be executed into operator calculation tasks and runtime call tasks;

[0035] The decomposed operator computing tasks are deployed in the operator module, and the decomposed runtime call tasks are deployed in the runtime module.

[0036] The interface corresponding to the task to be executed is searched from the task library corresponding to the task to be executed through the module corresponding to the task, including:

[0037] Use the operator module to search for the interface corresponding to the operator computing task from the operator library corresponding to the operator computing task;

[0038] The runtime module searches for the interface corresponding to the runtime calling task from the runtime library corresponding to the runtime calling task.

[0039] In order to achieve accurate division of tasks to be executed, in implementation, tasks to be executed are decomposed into operator calculation tasks and runtime call tasks, including:

[0040] Obtaining the type of the task to be executed; wherein the type of the task to be executed includes at least a training task type and an inference task type;

[0041] Determine the processing flow of the tasks to be executed according to the types of the tasks to be executed;

[0042] According to the processing flow of the tasks to be executed and the pre-set task classification rules, the tasks to be executed are decomposed level by level to obtain operator calculation tasks and runtime call tasks; among them, the tasks to be executed are the highest-level tasks among all levels of tasks; the operator calculation tasks and runtime call tasks are the lowest-level tasks among all levels of tasks.

[0043] If the task is a training task, the processing flow includes forward computation, backward computation, and gradient update. From a training perspective, model training tasks can be divided into three parts: forward computation, backward computation, and gradient update. Forward and backward computations depend on the model structure, while gradient updates are related to the optimizer. The lowest level is the runtime and operators. Figure 2 A schematic diagram of deploying a pre-trained language model training task onto an accelerator card provided by an embodiment of the present invention. Figure 2 In [1], the pre-trained language model training task is divided into forward computation, backward computation, gradient update, model structure, optimizer, operator, and runtime. The training task is deployed on accelerator card 1, accelerator card 2, ..., and accelerator card N.

[0044] There are no restrictions on pre-set task hierarchies, which are determined based on actual circumstances. For example, the network structure of a common pre-trained language model primarily consists of an embedding block, multiple repeated decoding blocks, and a normalization block. The decoding block is typically repeated dozens of times. For example, the 7-billion-parameter Llama2 model has 32 decoding layers. The decoding block can be composed of a normalization layer, a multi-head self-attention layer, and a fully connected layer. Each layer is composed of a variety of operator calculations, such as matrix multiplication and addition. Training tasks (i.e., tasks to be executed) are divided into the first level, forward computations, backward computations, and gradient updates into the second level; blocks within the network structure are divided into the third level; layers below the blocks are divided into the fourth level; and operator definitions are divided into the fifth level. Figure 3 This is a multi-level architecture diagram for model training performance analysis provided by an embodiment of the present invention. Figure 3 As shown in the figure, the first level is the pre-trained language model training task, and the second level includes forward calculation, backward calculation, and gradient update. The third level includes: embedding block, decoding block, and normalization block; taking the decoding block as an example, the fourth level includes the normalization layer, multi-head self-attention layer, normalization layer, and fully connected layer; taking the multi-head self-attention layer as an example, the fifth level includes matrix multiplication, addition, and subtraction. If the task to be executed is an inference task, the first level is the pre-trained language model inference task, and the second level includes forward calculation. The clearly granular process structure can be used for multi-level implementation of training performance analysis: from the total model training time to the operator granularity.

[0045] After the tasks to be executed are refined, in order to realize the execution of the refined tasks, corresponding functions representing the execution of the refined tasks are set in the training function.

[0046] Specifically, the training framework includes functions for characterizing the execution of tasks to be executed, functions for characterizing the execution of subtasks, functions for characterizing the execution of operator calculation tasks, and functions for characterizing the execution of runtime call tasks; among which, subtasks are the remaining tasks after decomposing the tasks to be executed level by level, except for operator calculation tasks and runtime call tasks.

[0047] Among them, taking the task to be executed as a training task as an example, the function used to characterize the execution of the task to be executed is the function used to characterize the execution of the training task; in the embodiment of the present invention, Figure 3 In the multi-level implementation architecture for model training performance analysis, the tasks in level 2, level 3, and level 4 are all called subtasks; the functions used to characterize the execution of subtasks are the functions used to characterize the execution of tasks in level 2, the functions used to characterize the execution of tasks in level 3, and the functions used to characterize the execution of tasks in level 4.

[0048] The tasks to be performed obtained through the training framework of the universal access platform include:

[0049] Receive information sent by the user representing a function for calling a task to be executed through a training framework of the universal access platform;

[0050] Decomposing tasks to be executed into operator calculation tasks and runtime call tasks includes:

[0051] The control function used to represent the call of the task to be executed calls the functions corresponding to the tasks at each level in the hierarchical order of the tasks to be executed, until the function used to represent the execution of the operator calculation task and the function used to represent the execution of the runtime call task are called, so as to decompose the task to be executed into the operator calculation task and the runtime call task.

[0052] Figure 4 This is a diagram of a general implementation architecture for an artificial intelligence computing power execution framework and training performance analysis provided by an embodiment of the present invention. Figure 4 As shown in the figure, the model training task is deployed on the universal access platform, and the universal training performance analysis module is also deployed on the universal access platform. The universal training performance analysis module includes the training framework, runtime module, and deep learning operator module. The universal access platform is deployed on accelerator cards 1 through N. The computing operator library and runtime library are deployed on each accelerator card.

[0053] The general training performance analysis module acts on the training framework, the runtime module of the accelerator universal access platform, and the deep learning operator module to obtain the model training time. The accelerator universal access platform defines a set of virtual runtime interfaces and operator interfaces, which are decoupled from the actual accelerator card used. Therefore, the performance analysis of the runtime module and operator module of the accelerator universal access platform naturally does not rely on the underlying accelerator card, thus achieving the hardware universality of training performance analysis. Based on the general training performance analysis module, Figure 2 The training performance evaluation of accelerator card 1, accelerator card 2, and more accelerator cards connected to the accelerator card universal access platform is carried out. The advantages and disadvantages of the performance of each accelerator card are clearly displayed, solving the pain point of different performance analysis tools required for different accelerator cards in related technologies.

[0054] After breaking down the tasks to be executed into operator calculation tasks and runtime call tasks, in order to achieve refined performance analysis, the general access platform obtains the performance of the accelerator card under test when executing the tasks to be executed, including:

[0055] Obtaining tasks to be timed during the execution of pending tasks; wherein the pending tasks include one or more of pending tasks, subtasks, operator calculation tasks, and runtime call tasks;

[0056] Get the duration of the task to be timed;

[0057] The performance of the accelerator card to be tested when executing the task to be executed is determined according to the time consumption of the task to be timed.

[0058] Taking the forward calculation as an example, the time consumption of the forward calculation task is obtained to serve as the performance of the accelerator card under test when executing the forward calculation task.

[0059] In order to time the task to be timed, in implementation, obtaining the time consumption of the task to be timed includes:

[0060] Receive a function for representing the start of timing that is inserted before a function for representing the execution of a task to be timed;

[0061] After detecting that the task to be timed has been completed, a prompt message indicating that the task to be timed has been completed is sent to the user, so that the user can insert a function indicating that the timing has ended after the function indicating that the task to be timed has been executed according to the prompt message;

[0062] Determining a first time point according to a function for characterizing start timing, and determining a second time point according to a function for characterizing end timing;

[0063] The time consumed by the acceleration card to be tested to execute the task to be timed is determined according to the difference between the second time point and the first time point.

[0064] Because different users have different testing requirements, you can insert functions that indicate the start of a timing task (i.e., the start timing function) and the end timing function (i.e., the end timing function). For example, to time a forward computation task, you can insert the start timing function before the forward computation task and the end timing function after the forward computation task.

[0065] In order to automatically implement time-consuming calculations, in addition to the start timing function and the end timing function, a function for printing the results (ie, the result printing function) is also provided.

[0066] Obtaining the difference between the second time point and the first time point includes:

[0067] Receive a function for printing a character result inserted after a function for representing the end timing;

[0068] The difference between the second time point and the first time point is outputted by a function for characterizing result printing.

[0069] In this method, a function for representing result printing is used to output the corresponding time consumption each time a timing task is executed.

[0070] After calculating the time consumption of tasks at each level, in order to intuitively understand the time consumption of tasks at each level, after determining the time consumption of the accelerator card to be tested to execute the task to be timed based on the difference between the second time point and the first time point, the method further includes:

[0071] The duration of the tasks to be timed is displayed in the order of the task levels.

[0072] In order to output the time consumption of tasks at all levels after the pending task is completed, after detecting that the user inserts a function for representing the end timing after the function for representing the execution of the pending task according to the prompt information, it also includes:

[0073] Sending information to the user indicating that a function for printing a result is inserted after a function for executing a task to be executed;

[0074] After detecting the existence of a function for printing representation results, the time consumption of the tasks to be timed is displayed in the order of the task hierarchy.

[0075] The bottom layer of the general multi-level training performance analysis method relies on the extraction of time-consuming data for each part of the training. The time-consuming extraction tool is implemented by the timing class. For example, in order to extract the time-consuming model training task, Figure 5 A schematic diagram of a training time extraction tool for extracting the time consumed by a model training task provided by an embodiment of the present invention. Figure 5 As shown, the timing extraction tool includes a start timing function, an end timing function, and a result printing function. These functions are collectively referred to as the timing class. The start and end timing functions are implemented using the time() function in the Python time library. The start timing function is inserted before the model training task is executed, the end timing function is inserted after the model training task is completed, and the result printing function is inserted after the end timing function. After the model training task is completed, the result printing function displays the results in order by task level.

[0076] After obtaining the time consumption of tasks at all levels, you can compare the performance of accelerator cards horizontally and identify performance bottlenecks vertically based on the time consumption, such as Figure 2 As shown. The following first describes the process of horizontal comparison. There are multiple accelerator cards to be tested. After displaying the time consumption of the tasks to be timed in the order of task levels, it also includes:

[0077] Get the time taken by the accelerator card under test to complete the timed tasks at the first level.

[0078] If a target accelerator card is detected whose task to be timed lasts longer than a time threshold, the time difference between the time of tasks to be timed on other layers of the target accelerator card and the time of tasks to be timed on the same layer of other accelerator cards is obtained; the other accelerator cards are all the accelerator cards to be tested except the target accelerator card.

[0079] The target accelerator card is adjusted according to the time-consuming difference, the target accelerator card is used as a new accelerator card to be tested, and the process returns to the step of obtaining the accelerator card to be tested and a universal access platform for performing accelerator card performance testing.

[0080] It is worth noting that the training performance analysis in related technologies acts on the runtime and operator interface of each accelerator card, such as Figure 2 As shown, there are slight differences in training time compared to the general training performance analysis statistics in this invention, namely the time introduced by the accelerator card's universal access platform. This time mainly refers to the time it takes to call from the universal platform's virtual interface to the actual interface, which is the central processing unit (CPU) time and is almost negligible compared to the pre-trained language model training time. In addition, this time is consistent for all accelerator cards connected to the universal platform, so it does not affect the horizontal comparison of the training performance of each accelerator card in the artificial intelligence computing power execution framework.

[0081] Through horizontal comparison, you can identify accelerator cards with poor performance and optimize them based on the performance test results of other accelerator cards, thereby ensuring that the performance of all accelerator cards is improved.

[0082] In practice, if the time consumption is calculated for all operators and runtimes, the amount of calculation will increase. Therefore, in order to reduce the amount of calculation, in the embodiment of the present invention, the tasks to be timed are determined to include:

[0083] Treat each subtask on the current level as a task to be timed; the current level starts from the second level;

[0084] After obtaining the time consumption of each subtask on the current level, obtain the subtask with the longest time consumption on the current level based on the time consumption of each subtask on the current level;

[0085] The subtasks on the next level of the subtask that consumes the most time on the current level are used as the subtasks on the new current level; and the step of using the subtasks on the current level as tasks to be timed is returned.

[0086] The above describes the horizontal comparison. This embodiment provides a vertical comparison method. After displaying the time consumption of the tasks to be timed in the order of the task hierarchy, it also includes:

[0087] Compare the time consumption of the tasks to be timed at the same level to determine the task to be timed that consumes the most time at the same level, and determine the target operator computing task and / or target runtime call task that consumes the most time when executing the task to be timed;

[0088] Sending information to the user that indicates the update of the operator library and / or target runtime library corresponding to the target operator computing task in the accelerator card under test;

[0089] The updated accelerator card to be tested is used as a new accelerator card to be tested, and the process returns to the step of obtaining the accelerator card to be tested and a universal access platform for performing accelerator card performance testing.

[0090] First find the most time-consuming task in the second level. If the forward time is the most consuming, then only the time-consuming tasks under the forward time will be counted. For example, continue to obtain the most time-consuming task in the third level under the forward time. Figure 4 For example, at the second level, the time consumption of forward calculation, reverse calculation and gradient update is compared. If it is found that the forward calculation takes the most time, then there is no need to calculate the time consumption of the subsequent level tasks of reverse calculation and gradient update, and only the time consumption of the subsequent level tasks of forward calculation needs to be analyzed. Continue to compare the time consumption of the embedding block, decoding block and normalization block on the third level after the forward calculation. If it is found that the decoding block takes the most time, there is no need to analyze the time consumption of the subsequent levels of the embedding block and the subsequent levels of the normalization block. Continue to compare the time consumption of the normalization layer on the fourth level after the decoding block, the time consumption of the multi-head self-attention layer, the normalization layer and the time consumption of the fully connected layer. If it is found that the multi-head self-attention layer takes the most time, there is no need to analyze the time consumption of the subsequent layers of the normalization layer and the subsequent layers of the fully connected layer. You only need to analyze the time consumption of the subsequent layers of the multi-head self-attention layer, such as the matrix multiplication time, addition time and subtraction time on the 5th level. For example, for Figure 2 Performing the 4th level performance analysis on the accelerator card in the image, it is found that the self-attention layer takes a large proportion of the time, so you can consider optimizing the self-attention implementation algorithm. Performing the 5th level performance analysis, it is found that the matrix multiplication takes a large proportion of the time, so you can consider optimizing the matrix multiplication operator implementation.

[0091] In this method, multi-level performance analysis helps users vertically identify the training performance bottleneck of a certain accelerator card in the artificial intelligence computing power execution framework and find the direction of performance optimization.

[0092] The universal, multi-level performance analysis method for model training / inference tasks on different accelerator cards described above first deploys large-model training tasks across various accelerator cards using the AI ​​Compute Execution Framework, horizontally comparing the training performance of each accelerator card. This universal performance analysis is achieved through the AI ​​Compute Execution Framework's universal accelerator access platform. Secondly, the large-model training task is broken down step by step according to the model training process, from the complete training granularity down to the operator granularity, enabling multi-level performance analysis and vertically identifying training performance bottlenecks.

[0093] In summary, this method allows users to horizontally evaluate the performance of different accelerator cards in various model training tasks with only one set of training environment, breaking the usage barriers between the dedicated performance analysis tools of various accelerator cards in related technologies, and isolating the development differences of different accelerator card manufacturers at the training framework level. The artificial intelligence computing power execution framework aims to facilitate the adaptation of more accelerator cards to large-model training through a set of general platforms and standards, and has currently completed docking with a variety of accelerator cards. The accelerator card universal access platform based on the artificial intelligence computing power execution framework provided by the present invention realizes the hardware universality of training / inference performance analysis, and realizes multi-level extraction of training / inference time-consuming based on the hierarchical process structure of large-scale model training / inference, which greatly facilitates users to horizontally compare the training / inference performance of different accelerator cards, select the accelerator card that best suits their own business, and further vertically explore the training performance bottleneck of the accelerator card to find the direction of performance optimization.

[0094] The above describes a method for obtaining accelerator card performance. This embodiment also provides an accelerator card performance testing device. The accelerator card performance testing device is connected to the accelerator card to be tested; the accelerator card performance testing device is deployed with a training framework and task-related modules; the task-related modules include at least a runtime module and an operator module;

[0095] The training framework of the accelerator card performance test equipment is used to obtain the task to be executed and schedule the module corresponding to the task according to the task to be executed; through the module corresponding to the task, the interface corresponding to the task to be executed is searched from the task library of the task to be executed located in the accelerator card to be tested, so that the accelerator card to be tested can execute the task to be executed according to the interface corresponding to the task to be executed; and the performance of the accelerator card to be tested when executing the task to be executed is obtained.

[0096] In practice, there is a need to perform performance tests on multiple accelerator cards. To meet this need, the accelerator card performance test equipment includes a main control unit and multiple independent test channels, each of which is connected to the accelerator card test interface through an isolation circuit. Before testing, the universal access platform is burned into the storage area of ​​each accelerator card. After the accelerator card performance test equipment is started, the main control unit sends a test instruction set to each channel through the time-division multiplexing bus. After the accelerator card receives the test instruction, the universal access platform deployed on the accelerator card performs a performance test on the accelerator card. The test task is identified based on the unique device identifier (ID), and the test data is transmitted back to the device's data cache array through an asynchronous interrupt mechanism. The main control unit sorts and verifies the data according to the timestamp and ID.

[0097] In this method, a universal access platform is burned into the storage area of ​​each accelerator card before testing. The main control unit sends a test instruction set to each channel through a time-division multiplexing bus, and the instruction cycle difference is used to achieve parallel loading of instructions from multiple accelerator cards. Multiple accelerator cards use their own deployed universal access platforms to perform performance tests on the accelerator cards, ultimately achieving parallel testing of multiple accelerator cards by the same device.

[0098] The accelerator card performance testing device provided in this embodiment has the same or corresponding technical features as the method for obtaining accelerator card performance described above, and has the same effect as above.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0100] An embodiment of the present invention further provides a device for obtaining accelerator card performance, comprising:

[0101] A first acquisition module is used to acquire the accelerator card to be tested and a universal access platform for performing accelerator card performance testing; wherein the universal access platform is deployed with a training framework and task-specific modules; and the task-specific modules include at least a runtime module and an operator module;

[0102] The second acquisition module is configured to, upon detecting that the universal access platform is deployed on the accelerator card under test, acquire a task to be executed through the universal access platform's training framework, schedule a module corresponding to the task according to the task to be executed, and search, using the module corresponding to the task, for an interface corresponding to the task to be executed from a task library of tasks to be executed on the accelerator card under test, so that the accelerator card under test executes the task to be executed according to the interface corresponding to the task to be executed;

[0103] The third acquisition module is used to obtain the performance of the accelerator card to be tested when executing the task to be executed through the universal access platform.

[0104] In some embodiments, the apparatus for obtaining the performance of the accelerator card further includes:

[0105] Decomposition module, used to decompose tasks to be executed into operator calculation tasks and runtime call tasks;

[0106] The deployment module is used to deploy the decomposed operator computing tasks to the operator module and the decomposed runtime call tasks to the runtime module;

[0107] The second acquisition module includes a search module for searching for an interface corresponding to the task to be executed from a task library corresponding to the task to be executed through a module corresponding to the task.

[0108] The search module specifically includes:

[0109] The first search submodule is used to search the interface corresponding to the operator computing task from the operator library corresponding to the operator computing task through the operator module;

[0110] The second search submodule is used to search the runtime library corresponding to the runtime call task through the runtime module for the interface corresponding to the runtime call task.

[0111] In some embodiments, the decomposition module includes:

[0112] A fourth acquisition module is used to obtain the type of the task to be executed; wherein the type of the task to be executed includes at least a training task type and an inference task type;

[0113] A first determining module, configured to determine a processing flow of a task to be executed according to the type of the task to be executed;

[0114] The level-by-level decomposition module is used to decompose the tasks to be executed level by level according to the processing flow of the tasks to be executed and the pre-set task classification rules to obtain operator calculation tasks and runtime call tasks; among them, the tasks to be executed are the highest-level tasks among all levels of tasks; the operator calculation tasks and runtime call tasks are the lowest-level tasks among all levels of tasks.

[0115] In some embodiments, the second acquisition module includes a first sub-acquisition module, and the first sub-acquisition module is used to acquire the task to be executed through the training framework of the universal access platform.

[0116] The first sub-acquisition module includes:

[0117] A first receiving module is configured to receive information sent by a user representing a function for calling a task to be executed through a training framework of the universal access platform;

[0118] The decomposition module is specifically used to control the function used to represent the call of the task to be executed, and call the functions corresponding to the tasks at each level in sequence according to the hierarchical order of the tasks to be executed, until the function used to represent the execution of the operator calculation task and the function used to represent the execution of the runtime call task are called, so as to decompose the task to be executed into the operator calculation task and the runtime call task.

[0119] In some embodiments, the third acquisition module includes:

[0120] A fifth acquisition module is used to acquire tasks to be timed during the execution of tasks to be executed; wherein the tasks to be timed include one or more of tasks to be executed, subtasks, operator calculation tasks, and runtime call tasks;

[0121] The sixth acquisition module is used to obtain the time consumption of the task to be timed;

[0122] The seventh acquisition module is used to determine the performance of the accelerator card to be tested when executing the task to be executed according to the time consumption of the task to be timed.

[0123] In some embodiments, the sixth acquisition module includes:

[0124] A second receiving module is used to receive a function for representing the start of timing inserted before the function for representing the execution of the task to be timed;

[0125] A first sending module is configured to send a prompt message indicating that the task to be timed has been completed to the user after detecting that the task to be timed has been completed, so that the user can insert a function indicating that the timing has ended after the function indicating that the task to be timed has been executed according to the prompt message;

[0126] a second determining module, configured to determine a first time point according to a function for representing a start timing, and to determine a second time point according to a function for representing an end timing;

[0127] A third determining module is configured to determine the time taken by the accelerator card to be tested to execute the task to be timed according to the difference between the second time point and the first time point;

[0128] The device for obtaining the performance of the accelerator card also includes:

[0129] The first display module is used to display the time consumption of the tasks to be timed in order of task levels.

[0130] In some embodiments, the apparatus for obtaining the performance of the accelerator card further includes: an eighth obtaining module, configured to obtain a difference between a second time point and a first time point.

[0131] The eighth acquisition module includes:

[0132] A third receiving module is used to receive a function for representing result printing inserted after the function for representing end timing;

[0133] The output module is used to output the difference between the second time point and the first time point through a function used to characterize result printing.

[0134] In some embodiments, the apparatus for obtaining the performance of the accelerator card further includes:

[0135] The second sending module is used to send information to the user indicating that a function for indicating result printing is inserted after the function of the task to be executed;

[0136] The second display module is used to display the time consumption of the tasks to be timed in order of task levels after detecting the existence of a function for printing the representation results.

[0137] In some embodiments, the apparatus for obtaining the performance of the accelerator card further includes:

[0138] A ninth acquisition module is used to acquire the time consumption of the task to be timed of each accelerator card to be tested on the first level;

[0139] a tenth acquisition module configured to, if a target accelerator card is detected whose task to be timed lasts longer than a time threshold, acquire a time difference between the time of tasks to be timed on other layers of the target accelerator card and the time of tasks to be timed on the same layer of other accelerator cards; wherein the other accelerator cards are all accelerator cards to be tested except the target accelerator card;

[0140] The adjustment module is used to adjust the target accelerator card according to the time-consuming difference, use the target accelerator card as a new accelerator card to be tested, and return to trigger the first acquisition module.

[0141] In some embodiments, the apparatus for obtaining the performance of the accelerator card further includes: a fourth determination module, configured to determine a task to be timed.

[0142] The fourth determination module includes:

[0143] The first is a module for treating each subtask on the current level as a task to be timed; wherein the current level starts from the second level;

[0144] The eleventh obtaining module is used to obtain the subtask with the longest time consumption on the current level according to the time consumption of each subtask on the current level after obtaining the time consumption of each subtask on the current level;

[0145] The second one is used as a module to use the subtasks of the most time-consuming subtask on the current level on the next level as the subtasks on the new current level; and returns to trigger the first one as a module.

[0146] In some embodiments, the apparatus for obtaining the performance of the accelerator card further includes:

[0147] A comparison level determination module is used to compare the time consumption of tasks to be timed on the same level to determine the task to be timed that consumes the most time on the same level, and to determine the target operator computing task and / or target runtime call task that consumes the most time when executing the task to be timed;

[0148] An update module is used to send information to the user indicating that the operator library and / or target runtime library corresponding to the target operator computing task in the accelerator card under test has been updated;

[0149] The third module is used to use the updated accelerator card to be tested as a new accelerator card to be tested and return to trigger the first acquisition module.

[0150] For descriptions of features in the embodiment corresponding to the apparatus for obtaining accelerator card performance, reference may be made to the relevant descriptions of the embodiment corresponding to the method for obtaining accelerator card performance, which will not be detailed here.

[0151] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the method for obtaining the performance of an accelerator card.

[0152] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned methods for obtaining the performance of an accelerator card when running.

[0153] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0154] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned methods for obtaining the performance of an accelerator card are implemented.

[0155] An embodiment of the present invention also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned embodiments of the method for obtaining the performance of the accelerator card.

[0156] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0157] The above is a detailed introduction to the method for obtaining the performance of an accelerator card, the accelerator card performance testing equipment, and the product provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified. These improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for obtaining accelerator card performance, characterized in that: include: Obtain an accelerator card to be tested and a universal access platform for performing accelerator card performance testing; wherein the universal access platform is deployed with a training framework and task-specific modules; the task-specific modules include at least a runtime module and an operator module; When it is detected that the universal access platform is deployed on the accelerator card to be tested, a task to be executed is obtained through the training framework of the universal access platform, and a module corresponding to the task is scheduled according to the task to be executed; and an interface corresponding to the task to be executed is searched from a task library of the task to be executed located on the accelerator card to be tested using the module corresponding to the task, so that the accelerator card to be tested executes the task to be executed according to the interface corresponding to the task to be executed; obtaining, through the universal access platform, the performance of the accelerator card to be tested when executing the task to be executed; The universal access platform obtains the performance of the accelerator card to be tested when executing the task to be executed, including: Obtaining tasks to be timed during the execution of the tasks to be executed; wherein the tasks to be timed include one or more of the tasks to be executed, subtasks, operator calculation tasks, and runtime call tasks; Obtaining the time consumption of the task to be timed; Determining the performance of the accelerator card to be tested when executing the task to be executed according to the time consumption of the task to be timed; Determining the task to be timed includes: Each subtask on the current level is used as the task to be timed; wherein the current level starts from the second level; After obtaining the time consumption of each subtask on the current level, obtaining the subtask on the current level that consumes the most time according to the time consumption of each subtask on the current level; The subtasks on the next level of the subtask that consumes the most time on the current level are used as the subtasks on the new current level; and the process returns to the step of using the subtasks on the current level as the tasks to be timed. After obtaining the task to be executed, the universal access platform searches for the interface corresponding to the task to be executed from the task library corresponding to the task to be executed by using the module corresponding to the task, and also includes: Decomposing the task to be executed into operator calculation tasks and runtime call tasks; Decomposing the task to be executed into operator calculation tasks and runtime call tasks includes: Obtain the type of the task to be executed; Determining a processing flow of the task to be executed according to the type of the task to be executed; According to the processing flow of the tasks to be executed and the pre-set task classification rules, the tasks to be executed are decomposed level by level to obtain operator calculation tasks and runtime call tasks; wherein the tasks to be executed are the highest-level tasks among all levels of tasks; the operator calculation tasks and runtime call tasks are the lowest-level tasks among all levels of tasks; The subtasks are the remaining tasks after decomposing the to-be-executed tasks level by level in all hierarchical tasks except the operator calculation tasks and runtime call tasks.

2. The method for obtaining accelerator card performance according to claim 1, characterized in that: Deploying the decomposed operator computing tasks to the operator module, and deploying the decomposed runtime call tasks to the runtime module; The searching, through the task corresponding module, for the interface corresponding to the task to be executed from the task library corresponding to the task to be executed comprises: Searching, through the operator module, for an interface corresponding to the operator computing task from the operator library corresponding to the operator computing task; The runtime module searches for an interface corresponding to the runtime calling task from a runtime library corresponding to the runtime calling task.

3. The method for obtaining accelerator card performance according to claim 2, characterized in that: The types of tasks to be executed include at least a training task type and a reasoning task type.

4. The method for obtaining accelerator card performance according to claim 3, characterized in that: The types of tasks to be executed are different, and the training frameworks are different; the training frameworks include functions for representing the execution of the tasks to be executed, functions for representing the execution of subtasks, functions for representing the execution of operator calculation tasks, and functions for representing the execution of runtime call tasks; The acquiring of tasks to be performed through the training framework of the universal access platform includes: receiving, through the training framework of the universal access platform, information sent by a user representing a function for calling the task to be executed; Decomposing the task to be executed into operator calculation tasks and runtime call tasks includes: Control the function used to characterize the call of the task to be executed, and call the functions corresponding to the tasks at each level in sequence according to the hierarchical order of the tasks to be executed, until the function used to characterize the execution of the operator calculation task and the function used to characterize the execution of the runtime call task are called, so as to decompose the task to be executed into the operator calculation task and the runtime call task.

5. The method for obtaining accelerator card performance according to claim 1, wherein: The time consumption of obtaining the task to be timed includes: receiving a function for representing the start of timing inserted before the function for representing the execution of the task to be timed; After detecting that the task to be timed has completed execution, sending a prompt message indicating that the task to be timed has completed execution to the user, so that the user can insert a function indicating that timing has ended after the function indicating that the task to be timed has completed execution according to the prompt message; Determining a first time point according to a function for characterizing start timing, and determining a second time point according to a function for characterizing end timing; Determining the time taken by the accelerator card to be tested to execute the task to be timed according to the difference between the second time point and the first time point; After determining the time taken by the accelerator card to be tested to execute the task to be timed according to the difference between the second time point and the first time point, the method further includes: The time consumption of the tasks to be timed is displayed in order of task levels.

6. The method for obtaining accelerator card performance according to claim 5, characterized in that: Obtaining the difference between the second time point and the first time point includes: receiving a function for printing a characterization result inserted after the function for representing the end timing; The difference between the second time point and the first time point is outputted by the function for characterizing result printing.

7. The method for obtaining accelerator card performance according to claim 5, characterized in that: When the task to be timed is the task to be executed, after detecting that the user inserts a function for indicating the end of timing after the function for indicating the execution of the task to be executed according to the prompt information, the method further includes: Sending to the user information indicating that a function for printing a result is inserted after the function for the task to be executed; After detecting the existence of the function for printing the characterization result, the time consumption of the tasks to be timed is displayed in sequence according to the order of the task levels.

8. The method for obtaining accelerator card performance according to claim 7, characterized in that: There are multiple accelerator cards to be tested. After displaying the time consumption of the tasks to be timed in order of task levels, the method further includes: Obtaining the time consumption of the task to be timed on the first level for each of the accelerator cards to be tested; If a target accelerator card is detected whose duration of the task to be timed is greater than a duration threshold, obtaining a time difference between the duration of the task to be timed on other levels of the target accelerator card and the duration of the task to be timed on other accelerator cards at the same level; wherein the other accelerator cards are all accelerator cards to be tested except the target accelerator card; The target accelerator card is adjusted according to the time-consuming difference, the target accelerator card is used as a new accelerator card to be tested, and the process returns to the step of obtaining the accelerator card to be tested and the universal access platform for performing accelerator card performance testing.

9. The method for obtaining accelerator card performance according to claim 5 or 6, characterized in that: After displaying the time consumption of the tasks to be timed in order of the task levels, the method further includes: Comparing the time consumption of the tasks to be timed at the same level to determine the task to be timed that consumes the most time at the same level, and determining the target operator computing task and / or target runtime calling task that consumes the most time when executing the task to be timed; Sending information to the user indicating that the operator library and / or target runtime library corresponding to the target operator computing task in the accelerator card to be tested has been updated; The updated accelerator card to be tested is used as a new accelerator card to be tested, and the process returns to the step of obtaining the accelerator card to be tested and the universal access platform for performing accelerator card performance testing.

10. An accelerator card performance test device, characterized in that: The accelerator card performance test device is connected to the accelerator card to be tested; the accelerator card performance test device is deployed with a training framework and modules corresponding to the task; the modules corresponding to the task include at least a runtime module and an operator module; The training framework of the accelerator card performance test device is used to obtain a task to be executed, schedule a module corresponding to the task according to the task to be executed, and search the task library of the task to be executed located in the accelerator card to be tested for an interface corresponding to the task to be executed from the task library of the task to be executed located in the accelerator card to be tested through the module corresponding to the task, so that the accelerator card to be tested executes the task to be executed according to the interface corresponding to the task to be executed; Obtaining the performance of the accelerator card to be tested when executing the task to be executed; Obtaining the performance of the accelerator card to be tested when executing the task to be executed includes: Obtaining tasks to be timed during the execution of the tasks to be executed; wherein the tasks to be timed include one or more of the tasks to be executed, subtasks, operator calculation tasks, and runtime call tasks; Obtaining the time consumption of the task to be timed; Determining the performance of the accelerator card to be tested when executing the task to be executed according to the time consumption of the task to be timed; Determining the task to be timed includes: Each subtask on the current level is used as the task to be timed; wherein the current level starts from the second level; After obtaining the time consumption of each subtask on the current level, obtaining the subtask on the current level that consumes the most time according to the time consumption of each subtask on the current level; The subtasks on the next level of the subtask that consumes the most time on the current level are used as the subtasks on the new current level; and the process returns to the step of using the subtasks on the current level as the tasks to be timed. After obtaining the task to be executed, and before searching the task library corresponding to the task to be executed by the module corresponding to the task, the method further includes: Decomposing the task to be executed into operator calculation tasks and runtime call tasks; Decomposing the task to be executed into operator calculation tasks and runtime call tasks includes: Obtain the type of the task to be executed; Determining a processing flow of the task to be executed according to the type of the task to be executed; According to the processing flow of the tasks to be executed and the pre-set task classification rules, the tasks to be executed are decomposed level by level to obtain operator calculation tasks and runtime call tasks; wherein the tasks to be executed are the highest-level tasks among all levels of tasks; the operator calculation tasks and runtime call tasks are the lowest-level tasks among all levels of tasks; The subtasks are the remaining tasks after decomposing the to-be-executed tasks level by level in all hierarchical tasks except the operator calculation tasks and runtime call tasks.

11. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the method for obtaining the performance of an accelerator card as claimed in any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for obtaining the performance of the accelerator card according to any one of claims 1 to 9 are implemented.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for obtaining the performance of an accelerator card as claimed in any one of claims 1 to 9 are implemented.

Images