Operator performance evaluation method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence chips, and provides an operator performance evaluation method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining a configuration file corresponding to a to-be-evaluated operator, enabling the configuration file to comprise a hardware configuration file and an operator configuration file, carrying out the instantiation of a global object based on the configuration file, obtaining a hardware platform instance and an operator instance; the operator instance inherits a base class in a unified manner, and the base class is a standardized interface obtained by abstracting computational logic of performance indexes of operators; the global object is a class object obtained by performing interface abstraction on a hardware platform interface and an operator interface; according to the performance index of the base class calculation operator instance when the base class calculation operator instance runs on the hardware platform instance, the evaluation result of the theoretical performance of the to-be-evaluated operator is obtained, multiplexing of the calculation logic of the operator performance index is achieved, the calculation logic of the operator performance index does not need to be rewritten for different hardware platforms, and the calculation efficiency is improved. And the evaluation efficiency of operators across hardware platforms is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence chips, and in particular to an operator performance evaluation method, device, equipment, storage medium and program product. BACKGROUND

[0002] With the development of deep learning technology, various artificial intelligence chips such as GPUs (Graphics Processing Unit) significantly improve the computing efficiency through various operators in the high-performance computing library to meet the needs of the deep learning framework. In the design optimization of artificial intelligence chips and the performance benchmarking of hardware platforms, accurately evaluating the theoretical performance of operators on hardware platforms is crucial. In the related evaluation methods of operator theoretical performance, manual configuration of hardware parameters is required, and the performance evaluation of the same operator across hardware platforms requires rewriting the performance index calculation logic for different hardware platforms, which affects the performance evaluation efficiency of operators across hardware platforms, thereby restricting the efficiency of artificial intelligence chip optimization iteration. SUMMARY

[0003] The present application provides an operator performance evaluation method, device, equipment, storage medium and program product to solve the defects in the related art that rely on manual configuration of hardware parameters and require rewriting the performance index calculation logic for different hardware platforms, which affects the performance evaluation efficiency of operators across hardware platforms.

[0004] The present application provides an operator performance evaluation method, comprising the following steps: Obtain the configuration file corresponding to the operator to be evaluated; the configuration file includes a hardware configuration file and an operator configuration file, the hardware configuration file contains hardware parameters, and the operator configuration file contains input parameters of the operator to be evaluated; Based on the configuration file, instantiate a global object to obtain a hardware platform instance corresponding to the hardware parameters and an operator instance corresponding to the input parameters; the operator instance inherits a base class, the base class is a standardized interface obtained by abstracting the calculation logic of the performance index of the operator; the global object is a class object obtained by interface abstraction of the hardware platform interface and the operator interface; According to the base class, calculate the performance index of the operator instance running on the hardware platform instance to obtain the evaluation result of the theoretical performance of the operator to be evaluated.

[0005] According to the operator performance evaluation method provided by the present application, the performance index of the operator instance running on the hardware platform instance is calculated according to the base class to obtain the evaluation result of the theoretical performance of the operator to be evaluated, which comprises: Determine the calculation scale of the operator instance according to the input parameters; determine computing resources required by the operator instance when running on the hardware platform instance based on the computing scale; the computing resources at least include computing amount and memory access amount; evaluate a performance index of the operator instance when running on the hardware platform instance according to the base class and the computing resources, to obtain an evaluation result of the theoretical performance of the operator to be evaluated; the performance index at least includes running time and computing core utilization rate.

[0006] According to the operator performance evaluation method provided by the application, the global object includes a hardware object and an operator object of each operator; the instantiation of the global object based on the configuration file obtains a hardware platform instance corresponding to the hardware parameter and an operator instance corresponding to the input parameter, including: parsing the configuration file, extracting the hardware parameter in the hardware configuration file, and extracting the input parameter of the operator to be evaluated in the operator configuration file; the hardware parameter at least includes peak computing power, bandwidth and main frequency; instantiating the operator object of the operator to be evaluated based on the input parameter, to obtain an operator instance corresponding to the input parameter; instantiating the hardware object based on the hardware parameter, to obtain a hardware platform instance corresponding to the hardware parameter; Wherein, the operator object is a class object obtained by interface abstraction of each operator, and the hardware object is a class object obtained by interface abstraction of the hardware platform; the operator object uniformly inherits the base class.

[0007] According to the operator performance evaluation method provided by the application, the hardware parameter includes multiple groups of parameters, and the instantiation of the hardware object based on the hardware parameter to obtain a hardware platform instance corresponding to the hardware parameter includes: batch instantiation of the hardware object based on the hardware parameter, to obtain multiple hardware platform instances corresponding to the hardware parameter; one hardware platform instance in the multiple hardware platform instances corresponds to one group of parameters in the hardware parameter; The evaluation result of the theoretical performance of the operator to be evaluated includes: According to the base class, the performance index of the operator instance when running on each hardware platform instance is calculated, to obtain the evaluation result of the theoretical performance of the operator to be evaluated when running on the hardware platform corresponding to each hardware platform instance.

[0008] The operator performance evaluation method provided by the application, the to-be-evaluated operators include multiple, the input parameters include at least a group of parameters of any to-be-evaluated operator;The operator object of the to-be-evaluated operator is instantiated based on the input parameters, and the operator instance corresponding to the input parameters is obtained, comprising: The operator object of each to-be-evaluated operator is batch-instantiated based on the input parameters, and multiple operator instances corresponding to the input parameters are obtained;Any operator instance in the multiple operator instances corresponds to a group of parameters in the input parameters; The performance index of the operator instance running on the hardware platform instance is calculated according to the base class, and the evaluation result of the theoretical performance of the to-be-evaluated operator is obtained, comprising: The performance index of each operator instance running on the hardware platform instance is calculated according to the base class, and the evaluation result of the theoretical performance of the to-be-evaluated operator is obtained.

[0009] The operator performance evaluation method provided by the application, the operator instance includes multiple, the hardware platform instance includes multiple;After the performance index of the operator instance running on the hardware platform instance is calculated according to the base class, and the evaluation result of the theoretical performance of the to-be-evaluated operator is obtained, further comprising: The evaluation result of the theoretical performance of the target operator instance on each hardware platform instance is obtained;The target operator instance is any one of the multiple operator instances; The evaluation results of the theoretical performance of the target operator instance on different hardware platform instances are compared, and a first comparison result is obtained; The evaluation result of the theoretical performance of each operator instance on the target hardware platform instance is obtained;The target hardware platform instance is any one of the multiple hardware platform instances; The evaluation results of the theoretical performance of different operator instances on the target hardware platform instance are compared, and a second comparison result is obtained; According to the first comparison result and / or the second comparison result, an optimization strategy for the artificial intelligence chip corresponding to the hardware platform instance is generated.

[0010] The application also provides an operator performance evaluation device, comprising the following modules: The configuration file acquisition module is used for acquiring the configuration file corresponding to the to-be-evaluated operator;The configuration file includes a hardware configuration file and an operator configuration file, the hardware configuration file contains hardware parameters, and the operator configuration file contains input parameters of the to-be-evaluated operator; An object instantiation module is configured to instantiate a global object based on the configuration file, to obtain a hardware platform instance corresponding to the hardware parameter and an operator instance corresponding to the input parameter; the operator instance inherits a base class, the base class is a standardized interface obtained by abstracting the calculation logic of the performance index of the operator; and the global object is a class object obtained by interface abstraction of the hardware platform interface and the operator interface; A performance evaluation module is configured to calculate the performance index of the operator instance when running on the hardware platform instance according to the base class, to obtain the evaluation result of the theoretical performance of the operator to be evaluated.

[0011] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the operator performance evaluation method according to any one of the above when executing the computer program.

[0012] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the operator performance evaluation method according to any one of the above.

[0013] The application further provides a computer program product, which includes a computer program, and the computer program is executable on a processor to implement the operator performance evaluation method according to any one of the above.

[0014] The operator performance evaluation method, device, equipment, storage medium and program product provided by the application, by interface abstraction of the operator and the hardware platform, obtaining the corresponding global object, then instantiating the global object obtained by abstraction through the configuration file, obtaining the operator instance and the hardware platform instance, and interface abstracting the calculation logic of the performance index required for operator evaluation, obtaining the base class uniformly inherited by the operator instance, calculating the performance index of the operator instance when running on the hardware platform instance according to the calculation logic of the performance index represented by the base class, obtaining the evaluation result of the theoretical performance of the operator to be evaluated when running on the corresponding hardware platform. By interface abstraction of the operator, the hardware platform and the performance index, the performance index is abstracted into the base class which can be inherited by the operator instance, the calculation logic of the performance index of the operator on different hardware platforms is reused, without rewriting the calculation logic of the performance index for different hardware platforms, and the evaluation efficiency of the operator across hardware platforms is improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present application or the related art, the following will briefly introduce the drawings needed in the embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0016] Figure 1 is a flowchart of an operator performance evaluation method provided by the present application.

[0017] Figure 2 is a structural diagram of an operator performance evaluation device provided by the present application.

[0018] Figure 3 is a structural diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0020] The embodiments of the present application provide an operator performance evaluation method, which is used for evaluating the theoretical performance of an operator when running on a hardware platform, so as to provide a basis for the design and optimization of the hardware platform. Specifically, Figure 1 is a flowchart of an operator performance evaluation method provided by the present application, as shown in Figure 1 the method comprises the following steps: Step 100, obtaining a configuration file corresponding to a to-be-evaluated operator; the configuration file comprises a hardware configuration file and an operator configuration file, the hardware configuration file contains hardware parameters, and the operator configuration file contains input parameters of the to-be-evaluated operator; Step 200, instantiating a global object based on the configuration file to obtain a hardware platform instance corresponding to the hardware parameters and an operator instance corresponding to the input parameters; the operator instance inherits a base class, the base class is a standardized interface obtained by abstracting the calculation logic of the performance indicators of the operator; and the global object is a class object obtained by interface abstraction of hardware platform interfaces and operator interfaces; Step 300, calculating the performance indicators of the operator instance when running on the hardware platform instance according to the base class to obtain an evaluation result of the theoretical performance of the to-be-evaluated operator.

[0021] An evaluation object is obtained, and a configuration file corresponding to the evaluation object is acquired, the configuration file including a hardware configuration file and an operator configuration file, wherein the hardware configuration file contains hardware parameters, and is used for configuring a hardware platform; the operator configuration file contains input parameters of the evaluation object, and is used for configuring the evaluation object. The hardware parameters are specifically used for configuring a hardware platform required by the evaluation object. The configuration file supports multiple formats, and the configuration file can be a table file or a CSV (Comma-Separated Values) file, and no specific limitation is made in this regard.

[0022] Optionally, the evaluation object includes one or more operators, and the operator configuration file further contains operator information of the evaluation object, the operator information including an operator name and operator description information, wherein the operator description information is used for describing operator functions and interface parameters of the operators.

[0023] Optionally, the input parameters of the evaluation object include shapes and sizes of input data and control parameters, wherein the input data is generally a multi-dimensional array, and contains features, weights, or intermediate calculation results; the control parameters further include learnable parameters, fixed parameters, and attribute parameters. Further, the learnable parameters are, for example, weights and bias values of a convolution kernel; the fixed parameters are, for example, types of an activation function and a pooling mode; and the attribute parameters are additional configurations used for controlling behaviors of the operators, such as a padding mode, an operation mode, and a data type.

[0024] Based on the configuration file corresponding to the evaluation object, a global object is instantiated to obtain a hardware platform instance corresponding to the hardware parameters and an operator instance corresponding to the input parameters of the evaluation object. Specifically, the operator object in the global object is instantiated based on the operator configuration file to obtain the operator instance, and the hardware object in the global object is instantiated based on the hardware configuration file to obtain the hardware platform instance.

[0025] The global object is a global class object obtained through interface abstraction. Optionally, the global object includes a hardware object and an operator object, and the global object is a class object obtained through interface abstraction of hardware platform interfaces and operator interfaces. Different operators correspond to different class objects, and the global object includes class objects of different operators.

[0026] Optionally, the hardware configuration file can also include configuration parameters of multiple hardware platforms. The hardware object in the global object is instantiated based on hardware parameters of different hardware platforms in the hardware configuration file to obtain hardware platform instances corresponding to different hardware parameters.

[0027] In the case where the to-be-evaluated operators include multiple operators, the operator configuration file contains operator information and input parameters of each to-be-evaluated operator, the operator information is used to distinguish different to-be-evaluated operators, the input parameters of the to-be-evaluated operator are associated with the operator information, the global object corresponding to each to-be-evaluated operator is instantiated based on the configuration file, and an operator instance of each to-be-evaluated operator is obtained.

[0028] Further, the operator instance inherits a base class, which can also be referred to as a parent class or a super class, which is a standardized interface obtained by abstracting the calculation logic of the performance indicators required for operator evaluation, and represents the calculation logic corresponding to the performance indicators of the operator. That is, all operators adopt a unified evaluation indicator system, which contains multiple performance indicators of the operator. By uniformly abstracting the calculation logic of each performance indicator through a uniform interface, a standardized interface is obtained as a base class that can be uniformly inherited by all operator instances, thereby realizing the reuse of the calculation logic of the performance indicators.

[0029] The base class, as a standardized interface obtained by abstracting the performance indicators of the operator, provides general attributes and methods for the operator instance, abstracts the commonality of the performance indicators that need to be evaluated by the operator into a general template, and reduces redundancy. On this basis, code reuse and extension are realized through the inheritance mechanism, thereby realizing the reuse of the calculation logic of the performance indicators of the operator. The operator instance, as a subclass of the base class, can directly inherit the attributes and methods of the base class, avoiding repeated coding of the same code. The operator instance, as a subclass, can override the methods of the base class to implement specific behaviors. The inheritance of the operator instance to the base class does not affect the functional implementation of the operator itself.

[0030] If the content of the base class needs to be modified, for example, a new performance indicator that needs to be evaluated by the operator is added, all inherited subclasses will automatically update the modified content of the base class, reducing maintenance costs. Moreover, the hierarchical relationship of classes is formed through inheritance, enhancing the logicality and readability of the code. The rules defined by the base class ensure that the subclasses or derived classes follow a uniform standard, improving the code standardization.

[0031] According to the base class inherited by the operator instance, the performance indicators of each to-be-evaluated operator instance running on the hardware platform instance are calculated, and the evaluation results of the theoretical performance of each to-be-evaluated operator are obtained, that is, the theoretical performance of the to-be-evaluated operator running on the hardware platform corresponding to the hardware platform instance is obtained. In the case where the to-be-evaluated operators include multiple operators, the performance indicators of the operator instance corresponding to each to-be-evaluated operator running on the hardware platform instance are calculated based on the base class, and the evaluation results of the theoretical performance of different to-be-evaluated operators are obtained.

[0032] Optionally, the hardware platform instances can also include multiple, and for each to-be-evaluated operator, the performance indicators of the operator instance corresponding to the to-be-evaluated operator running on each hardware platform instance are calculated according to the base class, to obtain the evaluation result of the theoretical performance of the to-be-evaluated operator running on different hardware platforms.

[0033] In the embodiment, the operator and the hardware platform are interface abstracted to obtain corresponding global objects, the global objects obtained by the abstraction are then instantiated through a configuration file to obtain operator instances and hardware platform instances, the performance indicators of the operators are interface abstracted to obtain a base class which is uniformly inherited by the operator instances, and the performance indicators of the operator instances running on the hardware platform instances are calculated according to the calculation logic of the performance indicators represented by the base class, to obtain the evaluation result of the theoretical performance of the to-be-evaluated operator running on the corresponding hardware platform. Through interface abstraction of the operator, the hardware platform and the performance indicators, the performance indicators are abstracted into the base class which can be inherited by the operator instances, the calculation logic of the operator performance indicators for different hardware platforms is reused, and it is not necessary to rewrite the calculation logic of the operator performance indicators for different hardware platforms, thereby improving the evaluation efficiency of the operator across hardware platforms.

[0034] In one embodiment, the global objects obtained by interface abstraction include hardware objects and operator objects of various operators, and the global objects are instantiated based on the configuration file, including instantiation of the operator objects in the global objects based on an operator configuration file, and instantiation of the hardware objects in the global objects based on a hardware configuration file. Therefore, step 200 includes: Step 201, the configuration file is parsed, hardware parameters in the hardware configuration file are extracted, and input parameters of the to-be-evaluated operator in the operator configuration file are extracted; the hardware parameters at least include peak computing power, bandwidth and main frequency; Step 202, the operator object of the to-be-evaluated operator is instantiated based on the input parameters, to obtain an operator instance corresponding to the input parameters; Step 203, the hardware object is instantiated based on the hardware parameters, to obtain a hardware platform instance corresponding to the hardware parameters; The operator object is a class object obtained by interface abstraction of various operators, and the hardware object is a class object obtained by interface abstraction of the hardware platform; the operator object uniformly inherits the base class.

[0035] When the global objects are instantiated based on the configuration files, the configuration files are parsed first, including parsing the hardware configuration file, extracting the hardware parameters from the hardware configuration file, and parsing the operator configuration file, extracting the input parameters of the to-be-evaluated operator from the operator configuration file. Then, based on the extracted input parameters of the to-be-evaluated operator, the operator object corresponding to the to-be-evaluated operator is instantiated to obtain the operator instance corresponding to the input parameters of the to-be-evaluated operator, and based on the extracted hardware parameters, the hardware object is instantiated to obtain the hardware platform instance corresponding to the hardware parameters.

[0036] The hardware parameters at least include peak computing power, bandwidth and main frequency, and the hardware parameters can further include hardware description information, which describes the bandwidth and main frequency of each level of storage unit of the hardware platform in detail, that is, the configuration parameters of each level of storage unit of the hardware platform are included in the hardware parameters, and the instantiation of the hardware object includes the parameter configuration of each level of storage unit provided in the hardware description information.

[0037] Further, the operator object is a class object obtained by interface abstraction of each operator interface, and the hardware object is a class object obtained by interface abstraction of the hardware platform interface, and the abstracted operator object uniformly inherits the base class of the computing logic of the performance indicators, so that the operator instance obtained based on the instantiation of the operator object can automatically inherit the base class and implement the logic or method in the base class, that is, the performance indicators.

[0038] Optionally, for the operator object of the same to-be-evaluated operator, different operator instances are obtained by instantiation with different input parameters, and correspondingly, for the same hardware object, different hardware platform instances can also be obtained by instantiation with different hardware parameters. In this embodiment, based on the configuration file, batch instantiation of the global objects can be realized, that is, batch instantiation of at least one of the operator object and the hardware object is performed to obtain the corresponding operator instance and hardware platform instance.

[0039] Based on this, the hardware parameters extracted from the configuration file include multiple groups of parameters, and each group of parameters can obtain a corresponding hardware platform instance by instantiating the hardware object. When calculating the performance indicators of the operator instance running on the hardware platform instance according to the base class, the performance indicators of the operator instance running on each hardware platform instance need to be calculated. Therefore, step 203 further includes: Step 213, batch instantiation of the hardware object based on the hardware parameters to obtain multiple hardware platform instances corresponding to the hardware parameters; one hardware platform instance in the multiple hardware platform instances corresponds to one group of parameters in the hardware parameters. Step 300 includes: According to the base class, performance indicators of the operator instance running on each of the hardware platform instances are calculated, to obtain an evaluation result of theoretical performance of the to-be-evaluated operator running on a hardware platform corresponding to each of the hardware platform instances.

[0040] The hardware objects are batch-instantiated based on the hardware parameters, to obtain a plurality of hardware platform instances corresponding to the hardware parameters, in which each set of parameters in the hardware parameters corresponds to one hardware platform instance, or one hardware platform instance corresponds to one set of parameters in the hardware parameters, and different hardware parameters are obtained through instantiation to obtain different hardware platform instances.

[0041] For the plurality of hardware platform instances obtained through instantiation, according to the calculation logic of the performance indicators represented by the base class, the performance indicators of the operator instance corresponding to the to-be-evaluated operator running on each of the hardware platform instances are calculated according to step 300, to obtain an evaluation result of theoretical performance of the to-be-evaluated operator running on a hardware platform corresponding to each of the hardware platform instances, and to realize performance evaluation of the to-be-evaluated operator across hardware platforms.

[0042] The performance indicators of the operator instance running on different hardware platform instances can be parallel computing or sequential computing, and no specific limitation is made.

[0043] For parallel computing, the performance indicators of the operator running on each of the hardware platform instances are calculated simultaneously according to the calculation logic of the performance indicators represented by the base class. For sequential computing, first, one of the plurality of hardware platform instances is selected as a target hardware platform instance, then the performance indicators of the operator instance corresponding to the to-be-evaluated operator running on the target hardware platform instance are calculated according to the calculation logic of the performance indicators represented by the base class, then the next target hardware platform instance is selected from the remaining hardware platform instances, and the above performance indicator calculation process is repeated until the target hardware platform instance is the last one of the plurality of hardware platform instances, so as to obtain the performance indicators of the operator instance running on each of the hardware platform instances.

[0044] Based on the performance indicators of the operator instance running on each of the hardware platform instances, the theoretical performance of the to-be-evaluated operator running on a hardware platform corresponding to each of the hardware platform instances is determined, to obtain a performance evaluation result of the to-be-evaluated operator.

[0045] In one embodiment, the to-be-evaluated operators include multiple to-be-evaluated operators, and the input parameters of the to-be-evaluated operators include at least one set of parameters of any to-be-evaluated operator, i.e., the input parameters include multiple sets. For different to-be-evaluated operators, the corresponding operator objects are different, and the operator instances obtained by instantiating the different operator objects through the input parameters are also different; for the same to-be-evaluated operator, the operator instances obtained by instantiating the same operator object based on different input parameters are also different.

[0046] Based on this, an operator instance can be obtained by instantiating the operator object through each set of input parameters, and when calculating the performance indicators of the operator instances corresponding to the to-be-evaluated operators running on the hardware platform instance according to the base class, the performance indicators of each operator instance running on the hardware platform instance need to be calculated. Therefore, step 202 further includes: instantiating the operator objects of the to-be-evaluated operators in batches based on the input parameters to obtain multiple operator instances corresponding to the input parameters; any operator instance in the multiple operator instances corresponds to a set of parameters in the input parameters; Step 300 includes: calculating the performance indicators of each operator instance running on the hardware platform instance according to the base class to obtain the evaluation result of the theoretical performance of the to-be-evaluated operators.

[0047] The operator objects of the to-be-evaluated operators are instantiated in batches based on the input parameters to obtain multiple operator instances corresponding to the input parameters, and each set of parameters in the input parameters corresponds to an operator instance, or any operator instance in the multiple operator instances corresponds to a set of parameters in the input parameters.

[0048] According to the calculation logic represented by the base class, the performance indicators of each operator instance running on the hardware platform instance are calculated according to the process shown in step 300, and the performance evaluation of different operators on the same hardware platform is implemented.

[0049] For multiple operator instances, the performance indicators of each operator instance running on the hardware platform can be calculated in parallel or sequentially, and no specific limitation is made in this regard.

[0050] For parallel computing, the performance index of each operator instance running on each hardware platform instance is calculated based on the performance index calculation logic of the base class representation.

[0051] Based on the performance index of each operator instance running on the hardware instance platform, the theoretical performance of each to-be-evaluated operator running on the hardware platform corresponding to the hardware platform instance is determined, and the performance evaluation result of each to-be-evaluated operator is obtained.

[0052] In an embodiment, based on the configuration file corresponding to the to-be-evaluated operator, the operator object and the hardware object are batch-instantiated to obtain a plurality of operator instances and a plurality of hardware platform instances, and the performance index of each operator instance running on any hardware platform instance is calculated in a parallel computing, sequential computing, or parallel and sequential cross-computing manner.

[0053] For parallel computing, the performance index of each operator instance running on each hardware platform instance is calculated based on the performance index calculation logic of the base class representation.

[0054] For sequential computing, one of the plurality of hardware platform instances is randomly selected as a target hardware platform instance, and one of the plurality of operator instances is randomly selected as a target operator instance. The performance index of the target operator instance running on the target hardware platform instance is calculated based on the performance index calculation logic of the base class representation. Further, the next one of the remaining hardware platform instances is randomly selected as the target hardware platform instance, and the performance index calculation process is repeated to calculate the performance index of the target operator running on the target hardware platform instance until the target hardware platform instance is the last one of the plurality of hardware platform instances. Then, one of the remaining operator instances is randomly selected as the target operator instance, and the performance index calculation process is repeated until the target operator instance is the last one of the plurality of operator instances.

[0055] Alternatively, any one of the plurality of hardware platform instances is selected as a target hardware platform instance, and any one of the plurality of operator instances is selected as a target operator instance. Based on the calculation logic of the performance index represented by the base class, the performance index of the target operator instance running on the target hardware platform instance is calculated. Further, any one of the remaining operator instances is selected as a target operator instance, and the above performance index calculation process is repeated until the target operator instance is the last one of the plurality of operator instances. Any next one of the remaining hardware platform instances is selected as a target hardware platform instance, and the above performance index calculation process is repeated until the target hardware platform instance is the last one of the plurality of hardware platform instances.

[0056] For parallel and sequential cross-computing, any one of the plurality of hardware platform instances is selected as a target hardware platform instance. Based on the calculation logic of the performance index represented by the base class, the performance indexes of the plurality of operator instances running on the target hardware platform instance are calculated simultaneously. Then, any next one of the remaining hardware platform instances is selected as a target hardware platform instance, and the above performance index calculation process is repeated until the target hardware platform instance is the last one of the plurality of hardware platform instances.

[0057] Alternatively, any one of the plurality of operator instances is selected as a target operator instance, and the performance indexes of the target operator instance running on each hardware platform instance are calculated based on the calculation logic of the performance index represented by the base class. Then, any one of the remaining operator instances is selected as a target operator instance, and the above performance index calculation process is repeated until the target operator is the last one of the plurality of operator instances.

[0058] It should be noted that the performance index of any one operator instance running on any hardware platform instance can be calculated in the manner shown in step 300, which will not be described again here.

[0059] In some embodiments, for the case where there are a plurality of operator instances and a plurality of hardware platform instances, since different operators are adapted to different hardware platforms, it is not necessary to calculate the performance index of each operator instance running on each hardware platform instance, but the operator instance and the hardware platform instance can be adapted to determine the hardware platform instance adapted to each operator instance. For each operator instance, the hardware platform instance adapted to the operator instance is selected from the plurality of hardware platform instances, so that the performance index of the operator instance running on the hardware platform instance adapted to the operator instance is calculated based on the calculation logic of the performance index represented by the base class.

[0060] In one embodiment, the calculation of the performance indicator of the operator instance running on the hardware platform instance is based on the evaluation of the computing resources required by the operator instance. Based on this, step 300 comprises: Step 301, determining the computing scale of the operator instance according to the input parameters; Step 302, determining the computing resources required by the operator instance running on the hardware platform instance based on the computing scale; the computing resources at least include the computing amount and the memory access amount; Step 303, evaluating the performance indicator of the operator instance running on the hardware platform instance according to the base class and the computing resources, obtaining the evaluation result of the theoretical performance of the to-be-evaluated operator; the performance indicator at least includes the running time and the computing core utilization rate.

[0061] According to the input parameters of the operator instance, the computing scale of the operator instance is determined, and the computing resources required by the operator instance running on the hardware platform instance are determined based on the computing scale; the computing resources required by the operator instance running on the hardware platform instance at least include the computing amount and the memory access amount.

[0062] According to the base class inherited by the operator instance and the computing resources required by the operator instance, the performance indicator of the operator instance running on the hardware platform is evaluated, and the evaluation result of the theoretical performance of the to-be-evaluated operator is obtained. The performance indicator of the to-be-evaluated operator at least includes the running time and the computing core utilization rate.

[0063] Optionally, for the same to-be-evaluated operator, the input parameters are different, and the operator instance instantiated is also different, and the computing resources required by the operator instance running on the hardware platform instance may also be different.

[0064] In one embodiment, the operator instance includes multiple, and the hardware platform instance also includes multiple, after obtaining the evaluation result of the theoretical performance of the to-be-evaluated operator, the theoretical performance of the operator also needs to be compared, including the longitudinal comparison of the theoretical performance of the same operator instance on different hardware platform instances, and the horizontal comparison of the theoretical performance of different operator instances on the same hardware platform instance. Based on this, after step 300, it can also include: Step 401, obtaining the evaluation result of the theoretical performance of the target operator instance on each hardware platform instance; the target operator instance is any one of the multiple operator instances; Step 402, comparing the evaluation results of the theoretical performance of the target operator instance on different hardware platform instances, obtaining a first comparison result; In step 403, evaluation results of theoretical performances of each operator instance on a target hardware platform instance are obtained; the target hardware platform instance is any one of the plurality of hardware platform instances; In step 404, the evaluation results of theoretical performances of different operator instances on the target hardware platform instance are compared to obtain a second comparison result. In step 405, an optimization strategy for the artificial intelligence chip corresponding to the hardware platform instance is generated according to the first comparison result and / or the second comparison result.

[0065] For the longitudinal comparison across hardware platforms, any operator instance is taken as a target operator instance, and evaluation results of theoretical performances of the target operator instance on each hardware platform instance are obtained. The evaluation results of theoretical performances of the target operator instance on different hardware platform instances are compared to obtain a first comparison result. For the transverse comparison of different operator instances, any hardware platform instance is taken as a target hardware platform instance, and evaluation results of theoretical performances of each operator instance on the target hardware platform instance are obtained. The evaluation results of theoretical performances of different operator instances on the target hardware platform instance are compared to obtain a second comparison result.

[0066] Finally, an optimization strategy for the artificial intelligence chip corresponding to the hardware platform instance is generated according to at least one of the first comparison result and the second comparison result. Optionally, a first optimization strategy for each hardware platform instance for the to-be-evaluated operator corresponding to the target operator instance is generated according to the first comparison result, and a second optimization strategy for the artificial intelligence chip corresponding to the target hardware platform instance is generated according to the second comparison result.

[0067] Optionally, the evaluation results of theoretical performances of the to-be-evaluated operator on the hardware platform instance are output in the form of a CSV file or a table file, so as to facilitate the comparison of different.

[0068] In an embodiment, different operator instances corresponding to a same to-be-evaluated operator are selected from the plurality of operator instances. The different operator instances of the same to-be-evaluated operator are obtained by instantiating an operator object of the same to-be-evaluated operator based on different input parameters. Evaluation results of theoretical performances of the different operator instances of the same to-be-evaluated operator on the target hardware platform instance are obtained. The evaluation results of theoretical performances of the different operator instances of the same to-be-evaluated operator on the target hardware platform instance are compared to obtain a third comparison result. A second optimization strategy for the artificial intelligence chip corresponding to the target hardware platform instance is generated according to at least one of the second comparison result and the third comparison result. Meanwhile, based on the evaluation results of theoretical performances of the different operator instances of the same to-be-evaluated operator on the same hardware platform instance, the theoretical performances of different input parameters of the same operator can be compared.

[0069] In the embodiment, by forcing all operator objects to uniformly inherit the standardized interface of the base class, it is ensured that different operator instances output the same evaluation index, and the required computing resources of the operator during running are evaluated based on the input parameters of the operator, which is used as the evaluation basis of the performance index. According to the calculation logic of the performance index represented by the base class, the theoretical performance of the operator during running on the hardware platform can be quickly and efficiently evaluated, and the evaluation efficiency of the operator performance is improved. Moreover, through the configuration file, batch configuration of the operator and the hardware platform can be realized, and the configuration efficiency of the test object is improved, so that the evaluation efficiency can be further improved.

[0070] Further, by comparing the theoretical performance of different operators on the same hardware platform, the theoretical performance of different operators or the theoretical performance of different input parameters of the same operator can be compared, and the hardware platform can be optimized in a targeted manner. By comparing the theoretical performance of the same operator instance on different hardware platforms, the performance of different hardware platforms can be benchmarked, so that the hardware platform can be optimized. By interface abstraction of the hardware platform, hardware-independent modeling is realized, and the theoretical performance of the operator on a heterogeneous hardware platform can be evaluated.

[0071] The traditional static performance model relies on manual input of hardware parameters and cannot dynamically adapt to changes in hardware configuration. In the embodiment, based on the interface abstraction of the operator, the hardware platform and the calculation logic of the performance index, when a hardware platform, an operator or a performance index is added, only the corresponding object configuration file needs to be added, without modifying the core code. The rapid access of the added hardware platform and operator is supported, which is easy to adapt to the architecture upgrade of the artificial intelligence chip, thereby improving the evaluation efficiency of the theoretical performance of the operator.

[0072] The operator performance evaluation device provided by the present application is described below. The operator performance evaluation device described below can be referred to in conjunction with the operator performance evaluation method described above.

[0073] Reference Figure 2 The operator performance evaluation device provided by the embodiment of the present application comprises: A configuration file acquisition module 10 is configured to acquire a configuration file corresponding to a to-be-evaluated operator. The configuration file comprises a hardware configuration file and an operator configuration file. The hardware configuration file contains hardware parameters, and the operator configuration file contains input parameters of the to-be-evaluated operator. An object instantiation module 20 is configured to instantiate a global object based on the configuration file to obtain a hardware platform instance corresponding to the hardware parameters and an operator instance corresponding to the input parameters. The operator instance inherits a base class, the base class is a standardized interface obtained by abstracting the calculation logic of the performance index of the operator, and the global object is a class object obtained by interface abstraction of a hardware platform interface and an operator interface. a performance evaluation module 30, configured to calculate a performance index of the operator instance running on the hardware platform instance according to the base class, to obtain an evaluation result of the theoretical performance of the operator to be evaluated.

[0074] In an embodiment, the performance evaluation module 30 is further configured to: determine a calculation scale of the operator instance according to the input parameter; determine a calculation resource required by the operator instance running on the hardware platform instance based on the calculation scale; the calculation resource at least includes a calculation amount and a memory access amount; calculate a performance index of the operator instance running on the hardware platform instance according to the base class and the calculation resource, to obtain the evaluation result of the theoretical performance of the operator to be evaluated; the performance index at least includes a running time and a calculation core utilization rate.

[0075] In an embodiment, the global objects include hardware objects and operator objects of each operator; the object instantiation module 20 is further configured to: parse the configuration file, extract a hardware parameter in the hardware configuration file, and extract an input parameter of the operator to be evaluated in the operator configuration file; the hardware parameter at least includes a peak computing power, a bandwidth and a main frequency; instantiate an operator object of the operator to be evaluated based on the input parameter, to obtain an operator instance corresponding to the input parameter; instantiate a hardware object based on the hardware parameter, to obtain a hardware platform instance corresponding to the hardware parameter; The operator object is a class object obtained by interface abstraction of each operator, and the hardware object is a class object obtained by interface abstraction of a hardware platform; the operator object uniformly inherits the base class.

[0076] In an embodiment, the hardware parameter includes multiple groups of parameters; the object instantiation module 20 is further configured to: batch-instantiate the hardware object based on the hardware parameter, to obtain multiple hardware platform instances corresponding to the hardware parameter; one hardware platform instance in the multiple hardware platform instances corresponds to one group of parameters in the hardware parameter; The performance evaluation module 30 is further configured to: calculate a performance index of the operator instance running on each hardware platform instance according to the base class, to obtain an evaluation result of the theoretical performance of the operator to be evaluated running on a hardware platform corresponding to each hardware platform instance.

[0077] In one embodiment, the to-be-evaluated operators include a plurality of to-be-evaluated operators, and the input parameters include at least one set of parameters of any of the to-be-evaluated operators; the object instantiation module 20 is further configured to: instantiating, in batches, operator objects of each of the to-be-evaluated operators based on the input parameters, to obtain a plurality of operator instances corresponding to the input parameters; any of the plurality of operator instances corresponds to a set of parameters in the input parameters; The performance evaluation module 30 is further configured to: calculating, according to the base class, a performance index of each of the operator instances when running on the hardware platform instance, to obtain an evaluation result of a theoretical performance of the to-be-evaluated operators.

[0078] In one embodiment, the operator instances include a plurality of operator instances, and the hardware platform instances include a plurality of hardware platform instances; the operator performance evaluation apparatus further includes a hardware optimization module configured to: obtaining an evaluation result of a theoretical performance of a target operator instance on each of the hardware platform instances; the target operator instance is any of the plurality of operator instances; comparing the evaluation results of the theoretical performance of the target operator instance on different hardware platform instances, to obtain a first comparison result; obtaining an evaluation result of a theoretical performance of each of the operator instances on a target hardware platform instance; the target hardware platform instance is any of the plurality of hardware platform instances; comparing the evaluation results of the theoretical performance of different operator instances on the target hardware platform instance, to obtain a second comparison result; generating an optimization strategy for an artificial intelligence chip corresponding to the hardware platform instance according to the first comparison result and / or the second comparison result.

[0079] Figure 3 An example of a schematic diagram of an entity structure of an electronic device is shown in FIG. 1. Figure 3 As shown in FIG. 1, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 can communicate with each other through the communications bus 340. The processor 310 can invoke a logical instruction in the memory 330 to execute steps of an operator performance evaluation method, for example, including: obtaining a configuration file corresponding to a to-be-evaluated operator; the configuration file includes a hardware configuration file and an operator configuration file, the hardware configuration file includes hardware parameters, and the operator configuration file includes input parameters of the to-be-evaluated operator; instantiating a global object based on the configuration file, to obtain a hardware platform instance corresponding to the hardware parameter and an operator instance corresponding to the input parameter; the operator instance inherits a base class, the base class is a standardized interface obtained by abstracting the calculation logic of the performance indicators of the operator; the global object is a class object obtained by interface abstraction of the hardware platform interface and the operator interface; calculating the performance indicators of the operator instance when running on the hardware platform instance according to the base class, to obtain the evaluation result of the theoretical performance of the operator to be evaluated.

[0080] In addition, the logical instructions in the memory 330 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0081] On the other hand, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the steps of the operator performance evaluation method provided by the above-mentioned methods, for example, including: obtaining a configuration file corresponding to the operator to be evaluated; the configuration file includes a hardware configuration file and an operator configuration file, the hardware configuration file contains hardware parameters, and the operator configuration file contains input parameters of the operator to be evaluated; instantiating a global object based on the configuration file, to obtain a hardware platform instance corresponding to the hardware parameter and an operator instance corresponding to the input parameter; the operator instance inherits a base class, the base class is a standardized interface obtained by abstracting the calculation logic of the performance indicators of the operator; the global object is a class object obtained by interface abstraction of the hardware platform interface and the operator interface; calculating the performance indicators of the operator instance when running on the hardware platform instance according to the base class, to obtain the evaluation result of the theoretical performance of the operator to be evaluated.

[0082] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the operator performance evaluation method provided by each of the above methods, for example comprising: obtaining a configuration file corresponding to the operator to be evaluated; the configuration file comprises a hardware configuration file and an operator configuration file, the hardware configuration file contains hardware parameters, and the operator configuration file contains input parameters of the operator to be evaluated; instantiating a global object based on the configuration file to obtain a hardware platform instance corresponding to the hardware parameters and an operator instance corresponding to the input parameters; the operator instance inherits a base class, the base class is a standardized interface obtained by abstracting the calculation logic of the performance indicators of the operator; and the global object is a class object obtained by interface abstraction of hardware platform interfaces and operator interfaces; calculating the performance indicators of the operator instance when running on the hardware platform instance according to the base class to obtain the evaluation result of the theoretical performance of the operator to be evaluated.

[0083] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0084] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0085] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An operator performance evaluation method, characterized in that: include: Obtain a configuration file corresponding to the operator to be evaluated; the configuration file includes a hardware configuration file and an operator configuration file, the hardware configuration file includes hardware parameters, and the operator configuration file includes input parameters of the operator to be evaluated; Instantiating the global object based on the configuration file to obtain a hardware platform instance corresponding to the hardware parameters and an operator instance corresponding to the input parameters; the operator instance inherits a base class, which is a standardized interface obtained by abstracting the calculation logic of the operator's performance indicators; The global object is a class object obtained by abstracting the hardware platform interface and the operator interface; The performance index of the operator instance when running on the hardware platform instance is calculated according to the base class to obtain an evaluation result of the theoretical performance of the operator to be evaluated.

2. The operator performance evaluation method according to claim 1, characterized in that: The step of calculating the performance index of the operator instance when running on the hardware platform instance according to the base class to obtain the evaluation result of the theoretical performance of the operator to be evaluated includes: Determine the calculation scale of the operator instance according to the input parameters; Determine, based on the computing scale, the computing resources required for the operator instance to run on the hardware platform instance; the computing resources include at least computing capacity and memory access capacity; Based on the base class and the computing resources, the performance indicators of the operator instance when running on the hardware platform instance are evaluated to obtain an evaluation result of the theoretical performance of the operator to be evaluated; the performance indicators include at least the running time and the computing core utilization rate.

3. The operator performance evaluation method according to claim 1, characterized in that: The global object includes a hardware object and operator objects of each operator; the instantiation of the global object based on the configuration file to obtain a hardware platform instance corresponding to the hardware parameters and an operator instance corresponding to the input parameters includes: Parsing the configuration file, extracting hardware parameters from the hardware configuration file, and extracting input parameters of the operator to be evaluated from the operator configuration file; the hardware parameters include at least peak computing power, bandwidth, and main frequency; Instantiating the operator object of the operator to be evaluated based on the input parameters to obtain an operator instance corresponding to the input parameters; Instantiating the hardware object based on the hardware parameters to obtain a hardware platform instance corresponding to the hardware parameters; The operator object is a class object obtained by abstracting the interface of each operator, and the hardware object is a class object obtained by abstracting the interface of the hardware platform; the operator objects uniformly inherit the base class.

4. The operator performance evaluation method according to claim 3, characterized in that: The hardware parameters include multiple groups of parameters, and the instantiating the hardware object based on the hardware parameters to obtain a hardware platform instance corresponding to the hardware parameters includes: Instantiate the hardware objects in batches based on the hardware parameters to obtain multiple hardware platform instances corresponding to the hardware parameters; one hardware platform instance in the multiple hardware platform instances corresponds to a set of parameters in the hardware parameters; The step of calculating the performance index of the operator instance when running on the hardware platform instance according to the base class to obtain the evaluation result of the theoretical performance of the operator to be evaluated includes: The performance index of the operator instance when running on each of the hardware platform instances is calculated according to the base class, and an evaluation result of the theoretical performance of the operator to be evaluated when running on the hardware platform corresponding to each of the hardware platform instances is obtained.

5. The operator performance evaluation method according to claim 3, characterized in that: The operators to be evaluated include multiple operators, the input parameters include at least one set of parameters of any of the operators to be evaluated; and the operator object of the operator to be evaluated is instantiated based on the input parameters to obtain the operator instance corresponding to the input parameters, including: Instantiating the operator objects of the operators to be evaluated in batches based on the input parameters to obtain a plurality of operator instances corresponding to the input parameters; any one of the plurality of operator instances corresponds to a set of parameters in the input parameters; The step of calculating the performance index of the operator instance when running on the hardware platform instance according to the base class to obtain the evaluation result of the theoretical performance of the operator to be evaluated includes: The performance index of each operator instance when running on the hardware platform instance is calculated according to the base class to obtain an evaluation result of the theoretical performance of the operator to be evaluated.

6. The operator performance evaluation method according to claim 1, characterized in that: The operator instances include a plurality of instances, and the hardware platform instances include a plurality of instances; after calculating the performance index of the operator instance when running on the hardware platform instance according to the base class to obtain the evaluation result of the theoretical performance of the operator to be evaluated, the method further includes: Obtaining an evaluation result of the theoretical performance of a target operator instance on each of the hardware platform instances; the target operator instance is any one of the multiple operator instances; Comparing evaluation results of theoretical performance of the target operator instance on different hardware platform instances to obtain a first comparison result; Obtaining an evaluation result of the theoretical performance of each operator instance on a target hardware platform instance; the target hardware platform instance is any one of a plurality of hardware platform instances; Comparing evaluation results of theoretical performance of different operator instances on the target hardware platform instance to obtain a second comparison result; Generate an optimization strategy for the artificial intelligence chip corresponding to the hardware platform instance based on the first comparison result and / or the second comparison result.

7. An operator performance evaluation device, characterized in that: include: A configuration file acquisition module is used to obtain a configuration file corresponding to the operator to be evaluated; the configuration file includes a hardware configuration file and an operator configuration file, the hardware configuration file includes hardware parameters, and the operator configuration file includes input parameters of the operator to be evaluated; An object instantiation module is configured to instantiate a global object based on the configuration file to obtain a hardware platform instance corresponding to the hardware parameters and an operator instance corresponding to the input parameters; the operator instance inherits a base class, which is a standardized interface obtained by abstracting the calculation logic of the operator's performance indicators; The global object is a class object obtained by abstracting the hardware platform interface and the operator interface; A performance evaluation module is used to calculate the performance index of the operator instance when it runs on the hardware platform instance according to the base class, and obtain an evaluation result of the theoretical performance of the operator to be evaluated.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the operator performance evaluation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operator performance evaluation method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the operator performance evaluation method according to any one of claims 1 to 6 is implemented.

Images