Heterogeneous chip performance test method and device, electronic equipment and medium

By obtaining test configuration information of heterogeneous chips, generating test scripts and allocating resources, the problem of uncommon models and cumbersome configuration steps between deep learning frameworks is solved, and efficient and accurate performance testing of heterogeneous chips is achieved.

CN119025359APending Publication Date: 2024-11-26CHINA CONSTRUCTION BANK +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411070883.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the GPU performance test, the models between deep learning frameworks are not universal and the configuration steps are cumbersome, resulting in low testing efficiency and poor accuracy.

Method used

By obtaining the test configuration information of heterogeneous chips, calling the script template to generate test scripts, predicting the amount of test resources, and creating computing nodes in different chip types, allocating computing resources, executing the test process, obtaining performance comparison results, and realizing automated testing and Unified performance standards.

Benefits of technology

It improves the automated testing efficiency of heterogeneous chips in different test scenarios, ensures normal execution of the test process, improves the accuracy and efficiency of chip performance testing, and can be expanded to different platforms and hardware devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025359A_ABST
    Figure CN119025359A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of information processing. The invention discloses a heterogeneous chip performance test method and device, electronic equipment and a medium. Comprising the following steps: acquiring test configuration information of a heterogeneous chip, calling a script template corresponding to a test scene, and generating a test script based on at least two chip types, test parameters, test indexes and the script template corresponding to the test scene; predicting a test resource quantity based on the test scene and the test parameters, creating computing nodes in the chips of the at least two chip types, and allocating computing resources required by the test environment to the computing nodes based on the test resource quantity; based on the test script, controlling the computing node corresponding to each chip to execute a test process, and obtaining index data of the test index of each type of chip in the test process; and obtaining a performance comparison result of each type of chips based on the index data. Performance testing is carried out on different types of chips by constructing a consistent automatic testing script, and the applicability and the high efficiency of chip testing are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a performance testing method, device, electronic equipment and medium for heterogeneous chips. Background Art

[0002] With the vigorous development of the field of artificial intelligence, technological innovations emerge in an endless stream. In the field of GPU chips, many manufacturers have launched their own GPU chips, including training cards and inference cards. Chip types include the Hygon DCU series and Huawei Ascend NPU series.

[0003] With the emergence of more and more types of GPUs, there is an increasing need for GPU performance testing. In the field of GPU evaluation research, the existing technical solution is to use the interfaces provided by various deep learning frameworks to perform GPU deep learning performance testing. However, each deep learning framework has different deduction and deduction performance testing methods, and the models trained by each deep learning framework are not universal. The deep learning deduction performance standards are not unified and the configuration steps are cumbersome. Summary of the invention

[0004] The present invention provides a performance testing method, device, electronic device and medium for heterogeneous chips, which are used to solve the problems that models trained by various deep learning frameworks in deep learning deduction scenarios are not universal and the configuration steps are cumbersome.

[0005] According to one aspect of the present invention, a performance testing method for a heterogeneous chip is provided, comprising:

[0006] Obtaining test configuration information of heterogeneous chips, where the test configuration information includes at least two chip types, test scenarios, test parameters, and test indicators;

[0007] Calling a script template corresponding to a test scenario, and generating a test script based on at least two chip types, test parameters, test indicators, and the script template corresponding to the test scenario;

[0008] Predicting the amount of test resources based on the test scenario and the test parameters, creating computing nodes in chips of at least two chip types, and allocating computing resources required for the test environment to the computing nodes based on the amount of test resources;

[0009] Based on the test script, the computing nodes corresponding to at least two chip types are controlled to respectively execute the test process, and the indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types are obtained;

[0010] Based on the indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types, the performance comparison results of the chips of the at least two chip types are obtained.

[0011] Optionally, test configuration information of heterogeneous chips is obtained, including: displaying an interactive page, the interactive page including a test scenario selection control and a chip type selection control; determining at least two chip types in response to a trigger operation on the chip type selection control; determining a test scenario in response to a selection operation on the test scenario selection control, and displaying test parameter items and test indicator items corresponding to the test scenario on the interactive page; determining test parameters in response to a setting operation on the test parameter items, and determining test indicators based on the selection operation on the test indicator items.

[0012] Optionally, the test scenario includes one or more of the following: a model training scenario, a model or operator testing scenario, and a model network parameter sharing scenario.

[0013] Optionally, in the model training scenario, the test parameters include model type, model initialization parameters, model training batches, loss function and gradient optimizer; the test indicators include one or more of the following: training process indicators, chip operation indicators and model accuracy indicators, wherein the training process indicators include: one or more of the operator accuracy in the model, the operator calculation speed, the training time of the model and the model loss during the training process; the chip operation indicators include one or more of the video memory occupancy, memory throughput and network throughput; the model accuracy indicators include one or more of the model accuracy, the model recall rate and the model F1 value.

[0014] Optionally, based on the test script, the computing nodes corresponding to at least two chip types are controlled to execute the test process respectively, including: building an initial model based on the model type and model initialization parameters in the test parameters; sending the initial model to the computing node corresponding to each chip type; at each computing node, calling the training data set corresponding to the model type, training the initial model based on the model training batch, loss function and gradient optimizer, and obtaining a trained target model.

[0015] Optionally, indicator data corresponding to test indicators in the test process corresponding to at least two chip types are obtained, including one or more of the following: in the training process of each batch, the calculation speed and operator accuracy of each operator in the model are determined; in the training process of each batch, the training time of the model in the batch is determined, and the training time of the model is determined based on the training time of the model in each batch; the model loss is determined in the training of each batch, and / or the loss mean and the loss trend diagram are determined based on the model loss of multiple batches; in the process of executing the test process on the computing node, one or more indicator data of video memory occupancy, memory throughput and network throughput are detected; based on the verification data set, the trained target model is verified to obtain one or more indicator data of model accuracy, model recall and model F1 value.

[0016] Optionally, in a model or operator testing scenario, the test parameters include: model type; or, the test parameters include: operator type and number of operators; the test indicators include one or more of the following: model processing indicators and chip operation indicators, wherein the model processing indicators include: model calculation speed, model calculation accuracy and model calculation stability; the chip operation indicators include one or more of video memory occupancy, memory throughput, and network throughput.

[0017] Optionally, based on the test script, the computing nodes corresponding to at least two chip types are controlled to execute the test process respectively, and indicator data corresponding to the test indicators in the test process corresponding to at least two chip types are obtained, including: calling a pre-trained target model based on the model type, or calling a pre-trained target operator based on the operator type and the number of operators; processing the verification sample data based on the target model or the target operator; in the process of processing the verification sample data by the target model or the target operator, detecting the indicator data corresponding to the chip operation indicator; and / or determining the model calculation speed based on the processing time of the target model or the target operator, and / or determining one or more indicator data of model calculation accuracy and model calculation stability based on the processing result of the target model or the target operator.

[0018] Optionally, in the model network parameter sharing scenario, the test parameters include: the number of shared parameters; the test indicators include one or more of the following: sharing process indicators and chip operation indicators, wherein the sharing process indicators include: one or more of parameter sharing speed and parameter loss rate; the chip operation indicators include one or more of video memory occupancy, memory throughput, and network throughput.

[0019] Optionally, computing nodes are created in chips of at least two chip types, including: creating a computing node in each of the two chips of any type, wherein the first computing node is a source node and the second computing node is a target node; and, based on the test script, controlling the computing nodes corresponding to the at least two chip types to respectively execute the test process, and obtaining indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types, including: calling the parameters to be shared based on the number of shared parameters, and sending the parameters to be shared to the source node in each chip type; controlling the source node to synchronize the parameters to be shared to the target node; in the process of parameter sharing, determining the indicator data of the parameter sharing speed based on the duration of the parameter sharing process; and / or, comparing the parameters received by the target node with the parameters to be transmitted of the source node to determine the indicator data of the parameter loss rate; and / or, detecting the indicator data corresponding to the chip operation indicators in the process of parameter sharing.

[0020] Optionally, predicting the test resource quantity based on the test scenario and the test parameters includes: inputting the test scenario and the test parameters into a resource quantity prediction model to obtain the test resource quantity.

[0021] According to one aspect of the present invention, a performance testing device for a heterogeneous chip is provided, comprising:

[0022] A test configuration information acquisition module is used to obtain test configuration information of heterogeneous chips, where the test configuration information includes at least two chip types, test scenarios, test parameters, and test indicators;

[0023] A test script generation module, used to call a script template corresponding to a test scenario, and generate a test script based on at least two chip types, test parameters, test indicators, and the script template corresponding to the test scenario;

[0024] A computing resource determination module, configured to predict the amount of test resources based on the test scenario and the test parameters, create computing nodes in chips of at least two chip types, and allocate computing resources required for the test environment to the computing nodes based on the amount of test resources;

[0025] An indicator data determination module, used to control the computing nodes corresponding to at least two chip types to respectively execute the test process based on the test script, and obtain indicator data corresponding to the test indicators in the test process corresponding to at least two chip types;

[0026] The performance comparison result determination module is used to obtain the performance comparison results of chips of at least two chip types based on the indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types.

[0027] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0028] at least one processor; and

[0029] a memory communicatively connected to at least one processor; wherein,

[0030] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the performance testing method for heterogeneous chips according to any embodiment of the present invention.

[0031] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the performance testing method for heterogeneous chips according to any embodiment of the present invention when executed.

[0032] According to another aspect of the present invention, a computer program product is provided, characterized in that the computer program product includes a computer program, and when the computer program is executed by a processor, the performance testing method of a heterogeneous chip according to any embodiment of the present invention is implemented.

[0033] The technical solution of the embodiment of the present invention obtains test configuration information of heterogeneous chips, wherein the test configuration information includes at least two chip types, test scenarios, test parameters and test indicators; calls a script template corresponding to the test scenario, and generates a test script based on the script template corresponding to the at least two chip types, test parameters, test indicators and test scenarios; predicts the amount of test resources based on the test scenario and the test parameters, creates computing nodes in chips of at least two chip types, and allocates computing resources required for the test environment to the computing nodes based on the test resource amount; controls the computing nodes corresponding to the at least two chip types to execute the test process respectively based on the test script, and obtains indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types; obtains performance comparison results of chips of at least two chip types based on the indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types. This solution automatically generates test scripts corresponding to various test configuration information through script templates corresponding to test configuration information and test scenarios, realizes automated testing of heterogeneous chips in different scenarios, determines the computing resources required for each computing node by predicting the test resource amount and the created computing nodes, allocates resources for the test process, and ensures the normal execution of the test process. It realizes the ability to determine the test script through a test script construction method with a unified performance standard, and obtain the performance test results of the corresponding test script without repeatedly configuring the test parameter information, thereby solving the problem that models trained by different deep learning frameworks are not universal and the configuration steps are cumbersome, improves the efficiency of automated generation of test scripts for different types of chips in different test scenarios, and also improves the scalability of chip performance testing, so that it can be expanded to different platforms, systems and other hardware devices, and improves the accuracy and efficiency of performance testing of heterogeneous chips.

[0034] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 This is a flow chart of a performance testing method for a heterogeneous chip provided in the first embodiment of the present invention.

[0037] Figure 2is a flow chart of a performance testing method for a heterogeneous chip provided in Embodiment 2 of the present invention;

[0038] Figure 3 This is a flow chart of a performance testing method for a heterogeneous chip provided in Embodiment 3 of the present invention;

[0039] Figure 4 This is a flow chart of a performance testing method for a heterogeneous chip provided in Embodiment 4 of the present invention;

[0040] Figure 5 It is a schematic diagram of the structure of a performance testing device for a heterogeneous chip provided in Embodiment 5 of the present invention;

[0041] Figure 6 It is a schematic diagram of the structure of an electronic device for implementing the performance testing method of a heterogeneous chip according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first characteristic data", "second characteristic data", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] The acquisition, storage and / or processing of data in the technical solutions involved in this application shall comply with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, models, etc. may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solutions of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0045] Embodiment 1

[0046] Figure 1 This is a flow chart of a performance test method for a heterogeneous chip provided in the first embodiment of the present invention. This embodiment is applicable to the performance test of heterogeneous chips. The method can be performed by a performance test device for a heterogeneous chip. The performance test device for a heterogeneous chip can be implemented in the form of hardware and / or software. The performance test device for a heterogeneous chip can be configured in an electronic device provided in the embodiment of the present invention. Figure 1 As shown, the method includes:

[0047] S110. Obtain test configuration information of the heterogeneous chip, where the test configuration information includes at least two chip types, a test scenario, test parameters, and test indicators.

[0048] Among them, heterogeneous chips specifically refer to chips with architectures of two or more different types of microprocessors or microcontrollers. Optionally, heterogeneous chips can be chips of different GPU types, for example, GPU-type chips include but are not limited to NVIDIA GPU (Graphics Processing Unit), DCU (Deep-learning Computing Unit) and NPU (Neural-network Process Units). The test configuration information can be specifically understood as the information required to configure the chip test, and the test configuration information includes but is not limited to at least two chip types, test scenarios, test parameters and test indicators. Optionally, the test scenario includes one or more of the following: model training scenario, model or operator test scenario and model network parameter sharing scenario. Among them, the model training scenario refers to a test scenario for testing chip performance by executing a model training process, the model or operator test scenario specifically represents a test scenario for testing chip performance by executing data processing through a model or operator, and the model network parameter sharing scenario refers to a test scenario for testing chip performance by performing a model network parameter sharing process. The test parameters can be specifically understood as quantitative indicators used to measure system or device performance in a set test environment. In this embodiment, the test parameters used to measure chip performance can be set according to different test scenarios. For example, for the model training scenario, the test parameters include but are not limited to model type, model initialization parameters, model training batches, loss function, and gradient optimizer; for the model or operator test scenario, the test parameters include but are not limited to model type, operator type, and number of operators; for the model network parameter sharing scenario, the test parameters include but are not limited to the number of shared parameters. Test indicators can be specifically understood as being used to clarify and standardize the various parameters and conditions in the test process, and can be set according to different test scenarios and test parameters. For example, for model training scenarios, test indicators include but are not limited to training process indicators, chip operation indicators, and model accuracy indicators; for model or operator test scenarios, test indicators include but are not limited to model processing indicators and chip operation indicators; for model network parameter sharing scenarios, test indicators include but are not limited to sharing process indicators and chip operation indicators. The method for obtaining test configuration information can be configured through the interactive interface, or the corresponding test configuration information can be obtained by calling the configuration file.

[0049] Specifically, for the performance test of heterogeneous chips, multiple types of chips can be selected for testing. After selecting at least two types of heterogeneous chips, it is necessary to set the corresponding test configuration information, that is, it is necessary to set the corresponding test scenarios, test parameters and test indicators. The various information in the configuration test information can be set through a visual interface, or the test configuration information required for the performance test can be stored in a configuration file in advance. When a heterogeneous chip is selected, the configuration file is called to obtain the test configuration information corresponding to the heterogeneous chip.

[0050] Optionally, test configuration information of heterogeneous chips is obtained, including: displaying an interactive page, the interactive page including a test scenario selection control and a chip type selection control; determining at least two chip types in response to a trigger operation on the chip type selection control; determining a test scenario in response to a selection operation on the test scenario selection control, and displaying test parameter items and test indicator items corresponding to the test scenario on the interactive page; determining test parameters in response to a setting operation on the test parameter items, and determining test indicators based on the selection operation on the test indicator items.

[0051] Specifically, on a pre-set interactive page, by selecting at least two chip types to be tested on the chip type selection control, selecting a test scenario on the test scenario selection control, and displaying the test parameter items and test indicator items corresponding to the test scenario on the interactive page corresponding to the test scenario, selections are made according to actual needs in the test parameter items corresponding to the test scenario to obtain test parameters, and accordingly, test indicator items corresponding to the selected test parameters are selected on the test indicators corresponding to the test scenario to obtain test indicators, and the at least two chip types, test scenarios, test parameters and test indicators set through the interactive page are summarized into test configuration information for heterogeneous chips.

[0052] In this embodiment, various types of chips, test scenarios, and test parameters and test indicators corresponding to the test scenarios are configured through test configuration information, which is used for subsequent automatic generation of test scripts. There is no need to repeatedly configure the test information, which simplifies the configuration process of the test information and helps to improve the efficiency of performance testing. Parameterized testing can ensure that different data is used each time the test is run, which helps to discover performance problems of the chip under different data conditions. It is easy to maintain and can flexibly adjust the test data as needed to meet different testing requirements.

[0053] S120. Call a script template corresponding to a test scenario, and generate a test script based on at least two chip types, test parameters, test indicators, and the script template corresponding to the test scenario.

[0054] The script template may be specifically understood as a template for testing that is pre-built according to different test scenarios and may be pre-stored in the target storage space.

[0055] Specifically, when obtaining the test configuration information of the heterogeneous chip, the corresponding script template is called from the target storage space according to the test scenario, and a test script is generated according to at least two chip types, test parameters, test indicators and the script template in the test configuration information. Specifically, each item of information in the script template is compared with the test configuration information, and at least two chip types, test parameters and test indicators in the test configuration information are added to the corresponding positions in the script template, and the test script is iterated to obtain a test script that matches the test configuration information.

[0056] In this embodiment, the script template is determined by the test scenario, and the script template is updated based on the test configuration information to generate a test script corresponding to the test configuration information. This realizes the automatic generation of test scripts corresponding to different test configuration information. Based on the set indicators, the script is automatically constructed to construct a heterogeneous computing power running image based on a unified framework, algorithm, data set and evaluation. Based on different types of chips, consistent test scripts of multiple running environment versions are formed, which helps to improve the efficiency and consistency of heterogeneous chip testing.

[0057] S130 , predicting the amount of test resources based on the test scenario and the test parameters, creating computing nodes in chips of at least two chip types, and allocating computing resources required for the test environment to the computing nodes based on the amount of test resources.

[0058] The test resource amount can be specifically understood as the amount of resources required to characterize the performance test under the corresponding test scenario, and can be predicted based on the test scenario and test parameters. Exemplarily, the test resource amount corresponding to the test scenario and test parameters can be determined by calling a pre-trained resource amount prediction model, and the corresponding test resource amount can also be determined by a test resource amount calculation algorithm.

[0059] Specifically, computing nodes are created in chips of at least two chip types to execute corresponding computing tasks. Predictions are made based on different test scenarios and corresponding test parameters to obtain the amount of test resources. The heterogeneous computing service resource pool in the existing cluster can be calculated by executing the pre-created volcano script and yaml configuration file. Combined with the predicted amount of test resources, the computing resources required for the test environment are allocated to the computing nodes through automated quota calculation. The heterogeneous computing service resource pool includes multiple pods. Pod is the smallest deployment unit in the service cluster, which contains all the resources and services required to run computing tasks. Each Pod contains a set of closely related processes or computing tasks. These processes or instances share network, storage and other resources. Through Pod, developers can manage and expand computing tasks more efficiently.

[0060] On the basis of the above-mentioned embodiment, predicting the test resource quantity based on the test scenario and the test parameters includes: inputting the test scenario and the test parameters into a resource quantity prediction model to obtain the test resource quantity.

[0061] Among them, the resource quantity prediction model can be specifically understood as a machine learning model, which can be trained through historical operation data to obtain a prediction model that meets the model training requirements.

[0062] Specifically, predetermined test scenarios and test parameters are used as input parameters and input into a resource quantity prediction model. After being processed by the model, the test resource quantity corresponding to the test scenarios and test parameters is output.

[0063] In this embodiment, computing nodes are created for different types of chips respectively, and computing resources required for the test environment are allocated to each computing node according to the predicted resource quantity for executing the test task. This achieves the prediction of the test resource quantity according to the test scenario and test parameters, and then determines the computing resources required for the corresponding test environment according to the test resource quantity. This avoids the problem of allocating too many computing resources to the test environment leading to resource waste or allocating too few computing resources leading to insufficient computing resources, thereby ensuring the rational use of computing resources and the smooth execution of chip test tasks.

[0064] S140. Based on the test script, the computing nodes corresponding to at least two chip types are controlled to respectively execute the test process, and the indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types are obtained.

[0065] Specifically, by executing the test script, the computing nodes corresponding to at least two chip types are controlled to respectively execute the test process, and during the test process, the indicator data corresponding to the test indicators in the test process corresponding to each chip type are collected.

[0066] In this embodiment, the computing nodes corresponding to the various chip types are controlled to perform tests through the constructed test script, so as to obtain the indicator data corresponding to the test indicators in the test process corresponding to each chip type, thereby realizing the control of various heterogeneous chips to perform test tasks and obtain test results through the test script, improving the consistency of the test script and avoiding repeated configuration of the information required in the test process, thereby improving the test efficiency.

[0067] S150. Obtain performance comparison results of chips of at least two chip types based on indicator data corresponding to test indicators in the test process corresponding to at least two chip types.

[0068] Specifically, the indicator data corresponding to the test indicators in the test process corresponding to at least two chip types can be statistically analyzed to determine the comparison results of the indicator data of various indicators of chips of different chip types, which are used to determine the performance test results of chips of various chip types.

[0069] The technical solution of the embodiment of the present invention obtains test configuration information of heterogeneous chips, wherein the test configuration information includes at least two chip types, test scenarios, test parameters and test indicators; calls a script template corresponding to the test scenario, and generates a test script based on the script template corresponding to the at least two chip types, test parameters, test indicators and test scenarios; predicts the amount of test resources based on the test scenario and the test parameters, creates computing nodes in chips of at least two chip types, and allocates computing resources required for the test environment to the computing nodes based on the test resource amount; controls the computing nodes corresponding to the at least two chip types to execute the test process respectively based on the test script, and obtains indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types; obtains performance comparison results of chips of at least two chip types based on the indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types. This solution automatically generates test scripts corresponding to various test configuration information through script templates corresponding to test configuration information and test scenarios, realizes automated testing of heterogeneous chips in different scenarios, determines the computing resources required for each computing node by predicting the test resource amount and the created computing nodes, allocates resources for the test process, and ensures the normal execution of the test process. It realizes the ability to determine the test script through a test script construction method with a unified performance standard, and obtain the performance test results of the corresponding test script without repeatedly configuring the test parameter information, thereby solving the problem that models trained by different deep learning frameworks are not universal and the configuration steps are cumbersome, improves the efficiency of automated generation of test scripts for different types of chips in different test scenarios, and also improves the scalability of chip performance testing, so that it can be expanded to different platforms, systems and other hardware devices, and improves the accuracy and efficiency of performance testing of heterogeneous chips.

[0070] Embodiment 2

[0071] Figure 2It is a flowchart of a performance testing method for a heterogeneous chip provided in the second embodiment of the present invention. This embodiment is a further optimization of the method of the above embodiment. Optionally, when the test scenario is a model training scenario, the script template corresponding to the test scenario is called, and a test script is generated based on at least two chip types, test parameters, test indicators and the script template corresponding to the test scenario; wherein the test parameters include model type, model initialization parameters, model training batch, loss function and gradient optimizer; the test indicators include one or more of the following: training process indicators, chip operation indicators and model accuracy indicators, wherein the training process indicators include: one or more of the operator accuracy, operator calculation speed, model training time and model loss in the training process, the chip operation indicators include one or more of the video memory occupancy, memory throughput and network throughput, and the model accuracy indicators include one or more of the model accuracy, model recall and model F1 value; the initial model is constructed based on the model type and model initialization parameters in the test parameters; the initial model is sent to the computing node corresponding to each chip type; at each computing node, the training data set corresponding to the model type is called, and the initial model is trained based on the model training batch, loss function and gradient optimizer to obtain a trained target model. Figure 2 As shown, the method includes:

[0072] S210. Acquire test configuration information of the heterogeneous chip, where the test configuration information includes at least two chip types, a test scenario, test parameters, and test indicators.

[0073] S220. When the test scenario is a model training scenario, call the script template corresponding to the test scenario, and generate a test script based on at least two chip types, test parameters, test indicators and the script template corresponding to the test scenario; wherein the test parameters include model type, model initialization parameters, model training batch, loss function and gradient optimizer; the test indicators include one or more of the following: training process indicators, chip operation indicators and model accuracy indicators, wherein the training process indicators include: one or more of the operator accuracy in the model, the operator calculation speed, the training time of the model and the model loss during the training process, the chip operation indicators include one or more of the video memory occupancy, memory throughput and network throughput, and the model accuracy indicators include one or more of the model accuracy, model recall rate and model F1 value.

[0074] Wherein, when the test scenario is a model training scenario, the script template corresponding to the model training scenario is called. The test parameters include model type, model initialization parameters, model training batch, loss function and gradient optimizer, wherein the model type includes but is not limited to algorithm models such as Yolov5 and Layoutmv2; the model initialization parameters specifically represent the preset assignment operation of the model parameters before the model training begins, and the initialization parameters include but are not limited to the weights and biases of the model, etc., which are the basis for the model to learn and predict. For example, the visual model is Yolov5, and the initialized parameters are the parameters of the coco data set pre-training, and the multimodal model is Layoutlmv2, and the initialization parameters include the model structure and pre-training weights. The model training batch specifically represents the number of samples processed by the model each time the parameters are updated during the training process. It should be noted that the minimum unit of training data is a batch group, and each batch group contains several training data. Preferably, the training data of each batch group is 16 data, so that the matrix operation processing capability of the chip can be fully utilized and the data can be trained in parallel. The loss function specifically represents the function used to estimate the difference between the model prediction and the true value, including but not limited to mean square error, mean absolute error and cross entropy loss. The gradient optimizer specifically calculates the gradient of the loss function with respect to the model parameters and adjusts the model parameters according to the gradient information. The training process indicators specifically represent the quantitative indicators for measuring the model training process, including but not limited to the operator accuracy, operator calculation speed, model training time and model loss during training; the chip operation indicators specifically represent the quantitative indicators for measuring the chip operation status, including but not limited to video memory occupancy, memory throughput, network throughput; the model accuracy indicators specifically represent the quantitative indicators used to measure the performance of the model, and the model accuracy indicators include but are not limited to the model accuracy, model recall rate and model F1 value.

[0075] Specifically, when the test scenario is a model training scenario, the corresponding script template is called according to the model training scenario, and at least two chip types, test parameters, and test indicators configured by the configuration method corresponding to the test configuration information are filled into the script template corresponding to the model training scenario, and the test script is iteratively generated. The test parameters include model type, model initialization parameters, model training batch, loss function, and gradient optimizer; one or more of the training process indicators, chip operation indicators, and model accuracy indicators are selected as test indicators according to actual needs, and one or more of the test indicators are selected as test indicator items of the corresponding test indicators according to actual needs.

[0076] S230 , predicting the amount of test resources based on the test scenario and the test parameters, creating computing nodes in chips of at least two chip types, and allocating computing resources required for the test environment to the computing nodes based on the amount of test resources.

[0077] S240, constructing an initial model based on the model type and model initialization parameters in the test parameters; and sending the initial model to the computing node corresponding to each chip type.

[0078] Among them, the training models include visual perception models and multimodal semantic cognition models, and the corresponding model initialization parameters are determined according to different models. The visual perception model adopts the Yolov5 model algorithm, and by adapting domestic chips, the training code is ultimately kept consistent on different hardware platforms. The training data uses the open source standard coco128 dataset, which contains 80 categories, and the evaluation module is used to evaluate the training results. The multimodal semantic cognition model uses the Layoutlmv2 model algorithm. Layoutlmv2 is based on the Transformer component and integrates multiple modal inputs to analyze documents.

[0079] Specifically, an initial model is constructed according to the selected model type and the corresponding model initialization parameters, and the initial model is sent to the computing nodes created on the chips of each chip type, and the computing nodes perform the corresponding model training tasks.

[0080] In this embodiment, model training can be implemented for various model types. The test model can be changed by only setting parameter items in the test parameters, thereby reducing repetitive operations and improving test efficiency.

[0081] S250. At each computing node, call the training data set corresponding to the model type, train the initial model based on the model training batch, loss function and gradient optimizer, and obtain a trained target model.

[0082] It should be noted that the training data sets corresponding to different model types are not the same. The training data sets corresponding to various model types can be stored in the preset target storage space in advance, so that they can be directly called when needed. It should be noted that the training and evaluation results are constructed through automated scripts, and the training startup scripts of the two algorithm models are kept consistent, and the parameters such as model construction, training data loading, model training batches, model loss calculation method and model optimizer in the training code are kept consistent.

[0083] Specifically, at the computing nodes on chips of various chip types, the corresponding training data set is called according to the model type to train the initial model, the number of model training is controlled by the set model training batch, the loss data of each model training is calculated by the preset loss function, and the parameters of the model are adjusted by the gradient optimizer based on the loss function of the corresponding training times. When the model training batch is completed, the obtained model is the trained target model.

[0084] S260: Obtain indicator data corresponding to test indicators in a test process corresponding to at least two chip types.

[0085] Optionally, indicator data corresponding to test indicators in the test process corresponding to at least two chip types are obtained, including one or more of the following: in the training process of each batch, the calculation speed and operator accuracy of each operator in the model are determined; in the training process of each batch, the training time of the model in the batch is determined, and the training time of the model is determined based on the training time of the model in each batch; the model loss is determined in the training of each batch, and / or the loss mean and the loss trend diagram are determined based on the model loss of multiple batches; in the process of executing the test process on the computing node, one or more indicator data of video memory occupancy, memory throughput and network throughput are detected; based on the verification data set, the trained target model is verified to obtain one or more indicator data of model accuracy, model recall and model F1 value.

[0086] Among them, the loss trend diagram can be specifically understood as a schematic diagram that characterizes the loss trend, specifically referring to the changing trend of the model loss value with different batches of training. Curves can be drawn according to each batch and its corresponding model loss value to obtain the corresponding loss trend diagram.

[0087] Specifically, the index data corresponding to the test index in the test process corresponding to at least two chip types can be collected through a pre-built index data collection script. The information of the index to be collected can be set in the index data collection script according to actual needs, and the information of the index data to be collected can also be directly set in the corresponding test script. In the process of executing the test script, the index data in the test process can be obtained in the process of executing the test script. Specifically obtain one or more of the following: in the training process of each batch, the calculation speed and operator accuracy of each operator in the model are calculated by the pre-built index data collection script. Alternatively, in the training process of each batch, the training time of the corresponding batch is determined by the start time of each batch of the model, and the training time of the model is determined based on the training time of the model in each batch. Specifically, the training time corresponding to all batches can be added to obtain the training time of the model. Alternatively, in the training of each batch, the model loss of each batch is determined by the loss function, and / or the model loss of multiple batches is averaged to obtain the loss mean, and a loss trend diagram is constructed based on the model loss of multiple batches. Alternatively, during the test execution of the computing node, one or more indicator data of video memory usage, memory throughput, and network throughput are determined through the corresponding indicator data collection script, or the verification data set is used as the input data of the target model to obtain the output result of the target model, and one or more indicator data of model accuracy, model recall rate, and model F1 value are calculated based on the output result. Whether to obtain one or more of the above indicator data is set according to actual needs.

[0088] S270. Obtain performance comparison results of chips of at least two chip types based on indicator data corresponding to test indicators in the test process corresponding to at least two chip types.

[0089] The technical solution of the embodiment of the present invention obtains test configuration information of heterogeneous chips, wherein the test configuration information includes at least two chip types, test scenarios, test parameters and test indicators; calls the script template corresponding to the test scenario, and generates a test script based on the script template corresponding to at least two chip types, test parameters, test indicators and test scenarios; wherein the test scenario is a model training scenario; the test parameters include model type, model initialization parameters, model training batch, loss function and gradient optimizer; the test indicators include one or more of the following: training process indicators, chip operation indicators and model accuracy indicators, wherein the training process indicators include: one or more of the operator accuracy, operator calculation speed, model training time and model loss in the training process; the chip operation indicators include one or more of the video memory occupancy, memory throughput and network throughput; the model accuracy The accuracy index includes one or more of the model accuracy, model recall rate and model F1 value; predicting the amount of test resources based on the test scenario and test parameters, creating computing nodes in chips of at least two chip types, and allocating computing resources required for the test environment to the computing nodes based on the test resource amount; building an initial model based on the model type and model initialization parameters in the test parameters; sending the initial model to the computing node corresponding to each chip type; at each computing node, calling the training data set corresponding to the model type, training the initial model based on the model training batch, loss function and gradient optimizer to obtain a trained target model; obtaining indicator data corresponding to the test indicators in the test process corresponding to at least two chip types; obtaining the performance comparison results of chips of at least two chip types based on the indicator data corresponding to the test indicators in the test process corresponding to at least two chip types. This solution realizes the automated construction of test scripts in a model training scenario. According to the test scripts, computing nodes corresponding to at least two chip types are controlled to execute the test process respectively to obtain index data of test indexes corresponding to chips of various chip types, and then obtain performance comparison results of chips of at least two chip types. Different chip types can be performance tested through the automatically constructed test scripts, and index data of test indexes corresponding to chips of various chip types can be obtained, thereby improving the efficiency of performance testing of chips of various chip types in the model training scenario. There is no need to repeatedly construct test scripts and configure test-related information, thereby improving the accuracy and efficiency of performance testing of heterogeneous chips.

[0090] Embodiment 3

[0091] Figure 3It is a flowchart of a performance test method for a heterogeneous chip provided in the third embodiment of the present invention. This embodiment is a further optimization of the method of the above embodiment. Optionally, when the test scenario is a model or operator test scenario, a script template corresponding to the test scenario is called, and a test script is generated based on at least two chip types, test parameters, test indicators and the script template corresponding to the test scenario; wherein the test parameters include: model type; or the test parameters include: operator type and operator quantity; the test indicators include one or more of the following: model processing indicators and chip operation indicators, wherein the model processing indicators include: model calculation speed, model calculation accuracy and model calculation stability; Chip operation indicators include one or more of video memory usage, memory throughput, and network throughput; calling a pre-trained target model based on the model type, or calling a pre-trained target operator based on the operator type and the number of operators; processing the verification sample data based on the target model or the target operator; detecting the indicator data corresponding to the chip operation indicator during the processing of the verification sample data by the target model or the target operator; and / or determining the model calculation speed based on the processing time of the target model or the target operator, and / or determining one or more indicator data of the model calculation accuracy and the model calculation stability based on the processing result of the target model or the target operator. Figure 3 As shown, the method includes:

[0092] S310. Obtain test configuration information of the heterogeneous chip, where the test configuration information includes at least two chip types, a test scenario, test parameters, and test indicators.

[0093] S320. When the test scenario is a model or operator test scenario, call the script template corresponding to the test scenario, and generate a test script based on at least two chip types, test parameters, test indicators and the script template corresponding to the test scenario; wherein the test parameters include: model type; or, the test parameters include: operator type and number of operators; the test indicators include one or more of the following: model processing indicators and chip operation indicators, wherein the model processing indicators include: model calculation speed, model calculation accuracy and model calculation stability; the chip operation indicators include one or more of video memory occupancy, memory throughput, and network throughput.

[0094] Among them, operators can be specifically understood as functions or classes used to perform various mathematical operations and operations, and are the basic components of deep learning models. Operator type specifically represents the category of operators used for calculations. It can be a single operator type or a mixed operator type composed of multiple operator types. Operator types include but are not limited to pooling operators, standard convolution, deconvolution, separable convolution, and grouped convolution. The number of operators specifically refers to the number of operators required for the calculation task, which can be set according to test requirements.

[0095] Specifically, when the test scenario is a model or operator test scenario, the script template corresponding to the model or operator test scenario is called. Select the model type as the test parameter according to the test requirements, or select the operator type and the number of operators as test parameters, and select the corresponding test indicators according to the selected test parameters. The test indicators include but are not limited to model processing indicators and chip operation indicators. Among them, the model processing indicators include: model calculation speed, model calculation accuracy and model calculation stability; chip operation indicators include but are not limited to video memory occupancy, memory throughput, and network throughput. Fill the selected chip type, test parameters, and test indicators corresponding to the script template corresponding to the test scenario to generate a test script.

[0096] S330 , predicting the amount of test resources based on the test scenario and the test parameters, creating computing nodes in chips of at least two chip types, and allocating computing resources required for the test environment to the computing nodes based on the amount of test resources.

[0097] S340: Call a pre-trained target model based on the model type, or call a pre-trained target operator based on the operator type and the number of operators.

[0098] Among them, the target models corresponding to different model types can be trained in advance, and the trained target models can be stored in a preset storage space, and a mapping relationship between the model type and the target model can be constructed. When the model type is determined, the target model corresponding to the model type can be obtained according to the mapping relationship; accordingly, the target operators corresponding to different operator types and operator quantities can be stored in a preset storage space in advance, and a mapping relationship between the operator type and operator quantity and the target model can be constructed. When the operator type and operator quantity are determined, the target operator corresponding to the operator type and operator quantity can be obtained according to the mapping relationship.

[0099] Specifically, when the test scenario is a model or operator test scenario, the corresponding target model or target operator is determined according to whether the model type or the operator type and the number of operators are configured in the corresponding test configuration information. That is, if the model type is configured, the pre-trained target model is called according to the model type; if the operator type and the number of operators are configured, the pre-trained target operator is called according to the operator type and the number of operators.

[0100] S350: Process the verification sample data based on the target model or target operator.

[0101] Specifically, the corresponding verification sample data can be obtained from a preset storage space according to the target model or the target operator, and the verification sample data can be used as input data and input into the target model or the target operator for processing.

[0102] S360. During the processing of verification sample data by the target model or the target operator, indicator data corresponding to the chip operation indicator is detected; and / or, the model calculation speed is determined based on the processing time of the target model or the target operator, and / or, one or more indicator data of model calculation accuracy and model calculation stability are determined based on the processing result of the target model or the target operator.

[0103] Specifically, during the processing of the verification sample data by the target model or the target operator, the index data corresponding to the chip operation index in each test process is obtained through a pre-built index data collection script. Alternatively, the model calculation speed is calculated by collecting the start time of the processing verification sample data of the target model or the target operator. Alternatively, one or more index data of model calculation accuracy and model calculation stability are determined according to the processing results of the target model or the target operator through a pre-built index data collection script. It should be noted that the model processing indicators to be detected are set according to actual needs, that is, one or more of the chip operation indicators, model calculation speed, model calculation accuracy and model calculation stability.

[0104] S370. Obtain performance comparison results of chips of at least two chip types based on indicator data corresponding to test indicators in the test process corresponding to at least two chip types.

[0105] The technical solution of the embodiment of the present invention obtains test configuration information of heterogeneous chips, wherein the test configuration information includes at least two chip types, test scenarios, test parameters and test indicators; when the test scenario is a model or operator test scenario, the script template corresponding to the test scenario is called, and a test script is generated based on the script template corresponding to at least two chip types, test parameters, test indicators and test scenarios; wherein the test parameters include: model type; or the test parameters include: operator type and operator quantity; the test indicators include one or more of the following: model processing indicators and chip operation indicators, wherein the model processing indicators include: model calculation speed, model calculation accuracy and model calculation stability; the chip operation indicators include one or more of video memory occupancy, memory throughput and network throughput; the test resource amount is predicted based on the test scenario and the test parameters, and at least Computing nodes are created in chips of two chip types, and computing resources required for the test environment are allocated to the computing nodes based on the amount of test resources; a pre-trained target model is called based on the model type, or a pre-trained target operator is called based on the operator type and the number of operators; verification sample data is processed based on the target model or the target operator; during the processing of the verification sample data by the target model or the target operator, indicator data corresponding to the chip operation indicator is detected; and / or, the model calculation speed is determined based on the processing time of the target model or the target operator, and / or, one or more indicator data of model calculation accuracy and model calculation stability are determined based on the processing result of the target model or the target operator; performance comparison results of chips of at least two chip types are obtained based on the indicator data corresponding to the test indicators in the test process corresponding to at least two chip types. This solution realizes the automatic construction of test scripts in the test scenario of model or operator test scenario, and controls the computing nodes corresponding to at least two chip types to execute the test process respectively according to the test script to obtain the index data of the test indexes corresponding to the chips of various chip types, and then obtains the performance comparison results of the chips of at least two chip types. The performance tests can be performed on different chip types through the automatically constructed test scripts, and the index data of the test indexes corresponding to the chips of various chip types can be obtained, which improves the efficiency of performance testing of chips of various chip types in the model or operator test scenario, eliminates the need to repeatedly construct test scripts and configure test related information, and improves the accuracy and efficiency of performance testing of heterogeneous chips.

[0106] Embodiment 4

[0107] Figure 4It is a flowchart of a performance testing method for a heterogeneous chip provided in the fourth embodiment of the present invention. This embodiment is a further optimization of the method of the above embodiment. Optionally, when the test scenario is a model network parameter sharing scenario, a script template corresponding to the test scenario is called, and a test script is generated based on at least two chip types, test parameters, test indicators and the script template corresponding to the test scenario; wherein the test parameters include: the number of shared parameters; the test indicators include one or more of the following: sharing process indicators and chip operation indicators, wherein the sharing process indicators include: one or more of parameter sharing speed and parameter loss rate; the chip operation indicators include video memory occupancy, memory throughput, One or more items of network throughput; create a computing node for each of the two chips of any type, where the first computing node is the source node and the second computing node is the target node; call the parameters to be shared based on the number of shared parameters, and send the parameters to be shared to the source node in each chip type; control the source node to synchronize the parameters to be shared to the target node; during the parameter sharing process, determine the indicator data of the parameter sharing speed based on the duration of the parameter sharing process; and / or, compare the parameters received by the target node with the parameters to be transmitted by the source node to determine the indicator data of the parameter loss rate; and / or, detect the indicator data corresponding to the chip operation indicator during the parameter sharing process. Figure 4 As shown, the method includes:

[0108] S410. Acquire test configuration information of heterogeneous chips, where the test configuration information includes at least two chip types, a test scenario, test parameters, and test indicators.

[0109] S420. When the test scenario is a model network parameter sharing scenario, call the script template corresponding to the test scenario, and generate a test script based on at least two chip types, test parameters, test indicators and the script template corresponding to the test scenario; wherein the test parameters include: the number of shared parameters; the test indicators include one or more of the following: sharing process indicators and chip operation indicators, wherein the sharing process indicators include: one or more of parameter sharing speed and parameter loss rate; the chip operation indicators include one or more of video memory occupancy, memory throughput, and network throughput.

[0110] Specifically, when the test scenario is a model network parameter sharing scenario, the script template corresponding to the model network parameter sharing scenario is called. By selecting the number of shared parameters as test parameters; accordingly, select one or more of the sharing process indicators and chip operation indicators on the test indicator control corresponding to the test parameters as test indicators; select one or more of the parameter sharing speed and parameter loss rate as sharing process indicators, and select one or more of the video memory occupancy, memory throughput, and network throughput as chip operation indicators. The above test parameters and test indicators can be adjusted according to actual needs, and can be adjusted through interactive pages or by modifying configuration files. Fill the selected chip type, test parameters, and test indicators in the script template corresponding to the test scenario to generate a test script.

[0111] S430: Predict the amount of test resources based on the test scenario and test parameters.

[0112] S440: Create a computing node in each of two chips of any type, wherein the first computing node is a source node and the second computing node is a target node.

[0113] The target node may be specifically understood as a node representing the obtained shared model network parameters.

[0114] Specifically, a computing node is created in each of the two chips corresponding to any type, and one of the computing nodes of the corresponding type can be used as a source node, and the other computing node can be used as a target node.

[0115] In this embodiment, computing nodes are respectively set for two chips of the same chip type to test the parameter sharing process between the two computing nodes of the same chip type, so as to complete the test of the sharing performance between the two chips of the same chip type.

[0116] S450. Allocate computing resources required for the test environment to the computing nodes based on the test resource quantity.

[0117] S460: Call the parameters to be shared based on the number of shared parameters, and send the parameters to be shared to the source node in each chip type.

[0118] Among them, the parameters to be shared can be specifically understood as parameters that characterize the need to be shared. The corresponding parameters to be shared can be set according to the number of shared parameters. Specifically, a corresponding mapping relationship between the number of shared parameters and the parameters to be shared can be set to achieve the determination of the parameters to be shared corresponding to the number of shared parameters by matching the mapping relationship.

[0119] Specifically, the corresponding parameters to be shared are matched from the preset storage space according to the number of preset shared parameters, and the parameters to be shared are sent to the source node in each chip type.

[0120] S470 , controlling the source node to synchronize the parameters to be shared to the target node.

[0121] Specifically, in the process of sending the parameters to be shared to the source node in each chip type, the source node is controlled to synchronously transmit the received parameters to be shared to the target node of the corresponding chip type.

[0122] S480. During the parameter sharing process, determine the indicator data of the parameter sharing speed based on the duration of the parameter sharing process; and / or, compare the parameters received by the target node with the parameters to be transmitted by the source node to determine the indicator data of the parameter loss rate; and / or, detect the indicator data corresponding to the chip operation indicator during the parameter sharing process.

[0123] Specifically, during the parameter sharing process, the start time of the parameter sharing process is obtained through a pre-built indicator data collection script, the duration of the parameter sharing process is calculated according to the start time, and the parameter sharing speed is calculated according to the parameters to be shared and the duration of the parameter sharing process. Alternatively, the parameters received by the target node are traversed and compared with the parameters to be transmitted of the source node respectively, the comparison results are determined, the number of successfully matched transmission parameters and the number of unmatched parameters are obtained, and the parameter loss rate is determined according to the ratio of the number of unmatched parameters to the number of parameters to be transmitted. Alternatively, during the parameter sharing process, the indicator data corresponding to the chip operation indicator is collected through a pre-built indicator data collection script. It should be noted that during the parameter sharing process, one or more of the parameter sharing speed, parameter loss rate and chip operation indicator are set and determined according to actual needs.

[0124] S490. Obtain performance comparison results of chips of at least two chip types based on indicator data corresponding to test indicators in the test process corresponding to at least two chip types.

[0125] The technical solution of the embodiment of the present invention obtains test configuration information of heterogeneous chips, wherein the test configuration information includes at least two chip types, test scenarios, test parameters and test indicators; when the test scenario is a model network parameter sharing scenario, the script template corresponding to the test scenario is called, and a test script is generated based on the script template corresponding to at least two chip types, test parameters, test indicators and test scenarios; wherein the test parameters include: the number of shared parameters; the test indicators include one or more of the following: sharing process indicators and chip operation indicators, wherein the sharing process indicators include: one or more of parameter sharing speed and parameter loss rate; the chip operation indicators include one or more of video memory occupancy, memory throughput and network throughput; the test resource amount is predicted based on the test scenario and the test parameters; for two chips of any type, the test script is generated based on the test scenario and the test parameters; Create a computing node separately, wherein the first computing node is a source node and the second computing node is a target node; allocate computing resources required for the test environment to the computing nodes based on the amount of test resources; call the parameters to be shared based on the number of shared parameters, and send the parameters to be shared to the source node in each chip type; control the source node to synchronize the parameters to be shared to the target node; in the process of parameter sharing, determine the indicator data of the parameter sharing speed based on the duration of the parameter sharing process; and / or, compare the parameters received by the target node with the parameters to be transmitted of the source node to determine the indicator data of the parameter loss rate; and / or, detect the indicator data corresponding to the chip operation indicator during the parameter sharing process; obtain the performance comparison results of chips of at least two chip types based on the indicator data corresponding to the test indicators in the test process corresponding to at least two chip types. This solution realizes the automated construction of test scripts in a model network parameter sharing scenario. According to the test scripts, computing nodes corresponding to at least two chip types are controlled to execute the test process respectively to obtain index data of test indexes corresponding to chips of various chip types, and then obtain performance comparison results of chips of at least two chip types. Different chip types can be performance tested through the automatically constructed test scripts, and index data of test indexes corresponding to chips of various chip types can be obtained, thereby improving the efficiency of performance testing of chips of various chip types in the model network parameter sharing scenario. There is no need to repeatedly construct test scripts and configure test-related information, thereby improving the accuracy and efficiency of performance testing of heterogeneous chips.

[0126] Embodiment 5

[0127] Figure 5 Schematic diagram of the structure of a performance test device for heterogeneous chips provided in Embodiment 5 of the present invention. Figure 5 As shown, the device comprises:

[0128] A test configuration information acquisition module 510 is used to acquire test configuration information of a heterogeneous chip, wherein the test configuration information includes at least two chip types, a test scenario, test parameters, and test indicators;

[0129] A test script generation module 520, used to call a script template corresponding to a test scenario, and generate a test script based on at least two chip types, test parameters, test indicators, and the script template corresponding to the test scenario;

[0130] A computing resource determination module 530, configured to predict the amount of test resources based on the test scenario and the test parameters, create computing nodes in chips of at least two chip types, and allocate computing resources required for the test environment to the computing nodes based on the amount of test resources;

[0131] An indicator data determination module 540 is used to control the computing nodes corresponding to at least two chip types to respectively execute the test process based on the test script, and obtain indicator data corresponding to the test indicators in the test process corresponding to at least two chip types;

[0132] The performance comparison result determination module 550 is used to obtain the performance comparison results of chips of at least two chip types based on the indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types.

[0133] The technical solution of this embodiment is to obtain the test configuration information of heterogeneous chips through a test configuration information acquisition module, wherein the test configuration information includes at least two chip types, test scenarios, test parameters and test indicators; the test script generation module calls the script template corresponding to the test scenario, and generates a test script based on the script template corresponding to at least two chip types, test parameters, test indicators and test scenarios; the computing resource determination module predicts the amount of test resources based on the test scenario and test parameters, creates computing nodes in chips of at least two chip types, and allocates computing resources required for the test environment to the computing nodes based on the test resource amount; the indicator data determination module controls the computing nodes corresponding to at least two chip types to execute the test process respectively based on the test script, and obtains the indicator data corresponding to the test indicators in the test process corresponding to at least two chip types; the performance comparison result determination module obtains the performance comparison results of chips of at least two chip types based on the indicator data corresponding to the test indicators in the test process corresponding to at least two chip types. This solution automatically generates test scripts corresponding to various test configuration information through script templates corresponding to test configuration information and test scenarios, realizes automated testing of heterogeneous chips in different scenarios, determines the computing resources required for each computing node by predicting the test resource amount and the created computing nodes, allocates resources for the test process, and ensures the normal execution of the test process. It realizes the ability to determine the test script through a test script construction method with a unified performance standard, and obtain the performance test results of the corresponding test script without repeatedly configuring the test parameter information, thereby solving the problem that models trained by different deep learning frameworks are not universal and the configuration steps are cumbersome, improves the efficiency of automated generation of test scripts for different types of chips in different test scenarios, and also improves the scalability of chip performance testing, so that it can be expanded to different platforms, systems and other hardware devices, and improves the accuracy and efficiency of performance testing of heterogeneous chips.

[0134] On the basis of the above embodiment, optionally, the test configuration information acquisition module 510 is specifically used to display an interactive page, which includes a test scenario selection control and a chip type selection control; in response to a trigger operation on the chip type selection control, at least two chip types are determined; in response to a selection operation on the test scenario selection control, a test scenario is determined, and test parameter items and test indicator items corresponding to the test scenario are displayed on the interactive page; in response to a setting operation on the test parameter items, the test parameters are determined, and based on the selection operation on the test indicator items, the test indicators are determined.

[0135] Optionally, the test scenario includes one or more of the following: a model training scenario, a model or operator testing scenario, and a model network parameter sharing scenario.

[0136] In the model training scenario, the test parameters include model type, model initialization parameters, model training batch, loss function and gradient optimizer; the test indicators include one or more of the following: training process indicators, chip operation indicators and model accuracy indicators, wherein the training process indicators include: one or more of the operator accuracy in the model, the operator calculation speed, the training time of the model and the model loss during the training process, the chip operation indicators include one or more of the video memory occupancy, memory throughput and network throughput, and the model accuracy indicators include one or more of the model accuracy, model recall rate and model F1 value. Optionally, the indicator data determination module 540 is specifically used to: construct an initial model based on the model type and model initialization parameters in the test parameters; send the initial model to the computing node corresponding to each chip type; at each computing node, call the training data set corresponding to the model type, train the initial model based on the model training batch, loss function and gradient optimizer, and obtain a trained target model. During the training of each batch, the calculation speed and operator accuracy of each operator in the model are determined; during the training of each batch, the training time of the model in the batch is determined, and the training time of the model is determined based on the training time of the model in each batch; the model loss is determined in the training of each batch, and / or the loss mean and loss trend diagram are determined based on the model loss of multiple batches; during the process of executing the test process on the computing node, one or more indicator data of video memory occupancy, memory throughput and network throughput are detected; based on the verification data set, the trained target model is verified to obtain one or more indicator data of model accuracy, model recall rate and model F1 value.

[0137] Optionally, in the model or operator test scenario, the test parameters include: model type; or, the test parameters include: operator type and operator number; the test indicators include one or more of the following: model processing indicators and chip operation indicators, wherein the model processing indicators include: model calculation speed, model calculation accuracy and model calculation stability; the chip operation indicators include one or more of video memory occupancy, memory throughput and network throughput. The indicator data determination module 540 is specifically used to: call a pre-trained target model based on the model type, or call a pre-trained target operator based on the operator type and operator number; process the verification sample data based on the target model or target operator; in the process of the target model or target operator processing the verification sample data, detect the indicator data corresponding to the chip operation indicator; and / or, determine the model calculation speed based on the processing time of the target model or target operator, and / or, determine one or more indicator data of model calculation accuracy and model calculation stability based on the processing result of the target model or target operator.

[0138] Optionally, in the model network parameter sharing scenario, the test parameters include: the number of shared parameters; the test indicators include one or more of the following: sharing process indicators and chip operation indicators, wherein the sharing process indicators include: one or more of parameter sharing speed and parameter loss rate; the chip operation indicators include one or more of video memory occupancy, memory throughput, and network throughput. The computing resource determination module 530 is specifically used to create a computing node for each of the two chips of any type, wherein the first computing node is the source node and the second computing node is the target node. The indicator data determination module 540 is specifically used to: call the parameters to be shared based on the number of shared parameters, and send the parameters to be shared to the source node in each chip type; control the source node to synchronize the parameters to be shared to the target node; in the process of parameter sharing, determine the indicator data of the parameter sharing speed based on the duration of the parameter sharing process; and / or, compare the parameters received by the target node with the parameters to be transmitted of the source node to determine the indicator data of the parameter loss rate; and / or, detect the indicator data corresponding to the chip operation indicator in the process of parameter sharing.

[0139] Optionally, the computing resource determination module 530 includes a test resource quantity prediction unit, which is specifically used to input the test scenario and test parameters into the resource quantity prediction model to obtain the test resource quantity.

[0140] The performance testing device for heterogeneous chips provided in the embodiment of the present invention can execute the performance testing method for heterogeneous chips provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0141] Embodiment 6

[0142] Figure 6 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 6 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0143] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0144] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0145] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a performance test method for heterogeneous chips.

[0146] In some embodiments, the performance test method of a heterogeneous chip may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the performance test method of the heterogeneous chip described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the performance test method of the heterogeneous chip in any other appropriate manner (e.g., by means of firmware).

[0147] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0148] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0149] Embodiment 7

[0150] Embodiment 7 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a performance testing method for a heterogeneous chip, the method comprising:

[0151] Acquire test configuration information of heterogeneous chips, wherein the test configuration information includes at least two chip types, test scenarios, test parameters and test indicators; call a script template corresponding to the test scenario, and generate a test script based on the script template corresponding to the at least two chip types, test parameters, test indicators and test scenarios; predict the amount of test resources based on the test scenario and test parameters, create computing nodes in chips of at least two chip types, and allocate computing resources required for the test environment to the computing nodes based on the test resource amount; control the computing nodes corresponding to the at least two chip types to execute the test process respectively based on the test script, and obtain indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types; obtain performance comparison results of chips of at least two chip types based on the indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types.

[0152] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0155] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0156] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0157] Embodiment 8

[0158] Embodiment 8 of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a performance testing method for a heterogeneous chip according to any embodiment of the present invention.

[0159] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A performance testing method for a heterogeneous chip, characterized in that: include: Acquire test configuration information of the heterogeneous chip, wherein the test configuration information includes at least two chip types, a test scenario, test parameters, and test indicators; Calling a script template corresponding to the test scenario, and generating a test script based on the at least two chip types, the test parameters, the test indicators, and the script template corresponding to the test scenario; Predicting the amount of test resources based on the test scenario and the test parameters, creating computing nodes in the chips of the at least two chip types, and allocating computing resources required for the test environment to the computing nodes based on the amount of test resources; Based on the test script, the computing nodes corresponding to the at least two chip types are controlled to respectively execute the test process, and the indicator data corresponding to the test indicator in the test process corresponding to the at least two chip types are obtained; Based on the indicator data corresponding to the test indicator in the test process corresponding to the at least two chip types, the performance comparison results of the chips of the at least two chip types are obtained.

2. The method according to claim 1, characterized in that The obtaining of the test configuration information of the heterogeneous chip includes: Displaying an interactive page, wherein the interactive page includes a test scenario selection control and a chip type selection control; In response to a triggering operation on the chip type selection control, determining at least two chip types; In response to a selection operation on the test scenario selection control, the test scenario is determined, and test parameter items and test indicator items corresponding to the test scenario are displayed on the interactive page; In response to a setting operation on a test parameter item, the test parameter is determined, and based on a selection operation on a test indicator item, the test indicator is determined.

3. The method according to claim 1 or 2, characterized in that: The test scenario includes one or more of the following: a model training scenario, a model or operator testing scenario, and a model network parameter sharing scenario.

4. The method according to claim 3, characterized in that In the model training scenario, the test parameters include model type, model initialization parameters, model training batch, loss function and gradient optimizer; The test indicators include one or more of the following: training process indicators, chip operation indicators and model accuracy indicators, wherein the training process indicators include: one or more of the operator accuracy in the model, the operator calculation speed, the training time of the model and the model loss during the training process; the chip operation indicators include one or more of the video memory occupancy, memory throughput and network throughput; the model accuracy indicators include one or more of the model accuracy, model recall rate and model F1 value.

5. The method according to claim 4, characterized in that The controlling the computing nodes corresponding to the at least two chip types to respectively execute the test process based on the test script includes: Building an initial model based on the model type and model initialization parameters in the test parameters; sending the initial model to the computing node corresponding to each chip type; At each of the computing nodes, a training data set corresponding to the model type is called, and the initial model is trained based on the model training batch, the loss function and the gradient optimizer to obtain a trained target model.

6. The method according to claim 5, characterized in that The obtaining of the indicator data corresponding to the test indicator in the test process corresponding to the at least two chip types includes one or more of the following: During each batch of training, determine the computational speed and accuracy of each operator in the model; During the training of each batch, determining the training time of the model in the batch, and determining the training time of the model based on the training time of the model in each batch; Determine the model loss during each batch of training, and / or determine the mean loss and loss trend graph based on the model losses over multiple batches; During the process of executing the test process on the computing node, detecting one or more indicator data of the video memory occupancy, the memory throughput and the network throughput; The trained target model is verified based on the verification data set to obtain one or more indicator data of the model accuracy, the model recall rate and the model F1 value.

7. The method according to claim 3, characterized in that In the model or operator test scenario, the test parameters include: model type; Alternatively, the test parameters include: operator type and operator quantity; The test indicators include one or more of the following: model processing indicators and chip operation indicators, wherein the model processing indicators include: model calculation speed, model calculation accuracy and model calculation stability; the chip operation indicators include one or more of video memory occupancy, memory throughput, and network throughput.

8. The method according to claim 7, characterized in that The controlling the computing nodes corresponding to the at least two chip types to respectively execute the test process based on the test script, and obtaining the indicator data corresponding to the test indicator in the test process corresponding to the at least two chip types, includes: Calling a pre-trained target model based on the model type, or calling a pre-trained target operator based on the operator type and the operator quantity; Processing the verification sample data based on the target model or the target operator; During the processing of the verification sample data by the target model or the target operator, the indicator data corresponding to the chip operation indicator is detected; and / or, the model calculation speed is determined based on the processing time of the target model or the target operator, and / or, one or more indicator data of model calculation accuracy and model calculation stability are determined based on the processing result of the target model or the target operator.

9. The method according to claim 3, characterized in that: In the model network parameter sharing scenario, the test parameters include: the number of shared parameters; The test indicators include one or more of the following: sharing process indicators and chip operation indicators, wherein the sharing process indicators include one or more of parameter sharing speed and parameter loss rate; the chip operation indicators include one or more of video memory occupancy, memory throughput, and network throughput.

10. The method according to claim 9, characterized in that The creating computing nodes in the chips of the at least two chip types comprises: creating a computing node in each of the two chips of any type, wherein the first computing node is a source node and the second computing node is a target node; And, controlling the computing nodes corresponding to the at least two chip types to respectively execute the test process based on the test script, and obtaining the indicator data corresponding to the test indicator in the test process corresponding to the at least two chip types, includes: Calling parameters to be shared based on the number of shared parameters, and sending the parameters to be shared to a source node in each chip type; Controlling the source node to synchronize the parameters to be shared to the target node; During the parameter sharing process, the indicator data of the parameter sharing speed is determined based on the duration of the parameter sharing process; and / or, the parameters received by the target node and the parameters to be transmitted by the source node are compared to determine the indicator data of the parameter loss rate; and / or, the indicator data corresponding to the chip operation indicator is detected during the parameter sharing process.

11. The method according to claim 1, characterized in that The predicting the amount of test resources based on the test scenario and the test parameters includes: The test scenario and the test parameters are input into a resource quantity prediction model to obtain the test resource quantity.

12. A performance testing device for heterogeneous chips, characterized in that: include: A test configuration information acquisition module, used to acquire test configuration information of heterogeneous chips, wherein the test configuration information includes at least two chip types, test scenarios, test parameters and test indicators; A test script generation module, used to call the script template corresponding to the test scenario, and generate a test script based on the at least two chip types, the test parameters, the test indicators and the script template corresponding to the test scenario; a computing resource determination module, configured to predict the amount of test resources based on the test scenario and the test parameters, create computing nodes in the chips of the at least two chip types, and allocate computing resources required for the test environment to the computing nodes based on the amount of test resources; An indicator data determination module, used to control the computing nodes corresponding to the at least two chip types to respectively execute the test process based on the test script, and obtain the indicator data corresponding to the test indicator in the test process corresponding to the at least two chip types; The performance comparison result determination module is used to obtain the performance comparison results of the chips of the at least two chip types based on the indicator data corresponding to the test indicators in the test process corresponding to the at least two chip types.

13. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the performance testing method for a heterogeneous chip according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the performance testing method for a heterogeneous chip according to any one of claims 1 to 11 when executed.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the performance testing method for a heterogeneous chip according to any one of claims 1 to 11.

Citation Information

Cited By

  • Visual multi-chip evaluation integration method and device

    CN120045454A