Operator performance evaluation method and device, storage medium and electronic equipment

Abstract

The invention discloses an operator performance evaluation method and device, a storage medium and electronic equipment. The method comprises the steps of obtaining a to-be-evaluated operator and an accuracy verification strategy of the to-be-evaluated operator; based on the accuracy verification strategy, performing accuracy verification on the to-be-evaluated operator, and judging whether an operator judgment criterion corresponding to the to-be-evaluated operator exists or not under the condition that a verification result meets a preset verification condition; and if the operator judgment criterion exists, generating a performance evaluation result of the to-be-evaluated operator based on the operator judgment criterion, otherwise, performing an iterative test on the to-be-evaluated operator until a preset number of iterations is reached, and generating the performance evaluation result of the to-be-evaluated operator according to the iterative test result. According to the method, the operator can be subjected to accuracy verification and performance evaluation, the high-performance operator development threshold is lowered, the manual modification process of operator development engineers is reduced, and the performance of the operator is guaranteed.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to an operator performance evaluation method and device, a storage medium, and an electronic device. Background Art

[0002] Automatic code generation technology, as an important branch in the field of software engineering, aims to transform high-level abstract descriptions into executable code through automated means, thereby improving software development efficiency and quality. In recent years, artificial intelligence technologies, especially deep learning and natural language processing, have brought new opportunities to automatic code generation. For example, a deep learning-based code generation model can generate code based on natural language descriptions or code snippets, and a natural language processing-based code generation tool can automatically generate a code framework according to user requirements.

[0003] Currently, related technologies write operators based on DSL (Domain-Specific Language) provided by compilers, such as TVM (Tensor Virtual Machine) and OpenAI Triton. Users use the languages provided by these compilers to describe the calculation rules of operators, and the compiler automatically converts the calculation process into a high-performance operator implementation for the required platform. However, this solution requires users to learn the DSL language provided by the compiler and have a certain understanding of the required platform architecture, which has a certain usage threshold. In addition, this kind of operator generation is usually limited by the DSL design and compiler implementation, and it is difficult to cover all performance optimization means. In some scenarios, the performance ceiling of the generated operators is not very high. For the automatic code generation method based on large models, engineers give operator descriptions, and the large model generates operator implementations, which improves the efficiency of operator development to a certain extent. However, most of the generated code can only meet the basic correctness requirements. Moreover, the generated operators consider insufficient boundary conditions and their performance is difficult to guarantee, making it difficult to achieve engineering implementation. To obtain good performance and comprehensively cover various input situations, it is necessary for operator developers to study the generated code, compile and run it for actual performance evaluation, and let the large model repeatedly modify and improve it. Therefore, the efficiency is still not very high, and no further methods for verifying the accuracy and evaluating the performance of operators are proposed. Summary of the Invention

[0004] This application provides an operator performance evaluation method and device, a storage medium, and an electronic device to at least solve the problems in the related technologies that operator developers need to study the generated code, compile and run it for actual performance evaluation, and let the large model repeatedly modify and improve it, resulting in low efficiency and no verification of operator accuracy and performance evaluation. The technical effect is to verify the accuracy and evaluate the performance of operators, lower the threshold for developing high-performance operators, reduce the manual modification process of operator developers, and ensure the performance of operators.

[0005] This application provides an operator performance evaluation method, including:

[0006] Obtain the operator to be evaluated and the accuracy verification strategy of the operator to be evaluated; based on the accuracy verification strategy, perform accuracy verification on the operator to be evaluated, and when the verification result meets the preset verification conditions, determine whether there is an operator evaluation criterion corresponding to the operator to be evaluated; if there is an operator evaluation criterion, generate a performance evaluation result of the operator to be evaluated based on the operator evaluation criterion, otherwise, perform iterative testing on the operator to be evaluated until the preset number of iterations is reached, and generate a performance evaluation result of the operator to be evaluated according to the iterative testing result.

[0007] This application also provides an operator performance evaluation device, including:

[0008] An acquisition module for obtaining the operator to be evaluated and the accuracy verification strategy of the operator to be evaluated;

[0009] A verification module for performing accuracy verification on the operator to be evaluated based on the accuracy verification strategy, and when the verification result meets the preset verification conditions, determining whether there is an operator evaluation criterion corresponding to the operator to be evaluated;

[0010] An evaluation module for, if there is an operator evaluation criterion, generating a performance evaluation result of the operator to be evaluated based on the operator evaluation criterion, otherwise, performing iterative testing on the operator to be evaluated until the preset number of iterations is reached, and generating a performance evaluation result of the operator to be evaluated according to the iterative testing result.

[0011] This application also provides an electronic device, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the above-mentioned operator performance evaluation method is implemented.

[0012] This application also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the above-mentioned operator performance evaluation method is implemented.

[0013] Through this application, the operator to be evaluated and the accuracy verification strategy of the operator to be evaluated are obtained. Based on the accuracy verification strategy, accuracy verification is performed on the operator to be evaluated. When the verification result meets the preset verification conditions, it is determined whether there is an operator evaluation criterion corresponding to the operator to be evaluated. If there is an operator evaluation criterion, a performance evaluation result of the operator to be evaluated is generated based on the operator evaluation criterion. Otherwise, iterative testing is performed on the operator to be evaluated until the preset number of iterations is reached, and a performance evaluation result of the operator to be evaluated is generated according to the iterative testing result. Therefore, it is possible to perform accuracy verification and performance evaluation on the operator, reduce the development threshold of high-performance operators, and reduce the process of manual modification by operator development engineers to ensure the performance of the operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To more clearly illustrate the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0015] Figure 1 FIG. is a flowchart of an operator performance evaluation method according to an embodiment of the present application;

[0016] Figure 2 FIG. is a flowchart of an operator performance evaluation method according to a specific example of the present application;

[0017] Figure 3 FIG. is a block diagram of an operator performance evaluation device according to an embodiment of the present application;

[0018] Figure 4 FIG. is a block diagram of an electronic device according to an embodiment of the present application.

[0019] Reference Numerals: 100 - operator performance evaluation device, 110 - acquisition module, 120 - verification module, 130 - evaluation module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0021] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and not to describe a specific order or sequence.

[0022] To enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0023] Automatic code generation technology, as an important branch in the field of software engineering, aims to transform high-level abstract descriptions into executable code through automated means, thereby improving software development efficiency and quality. Its development process can be roughly divided into the following stages:

[0024] 1. Birth of compiler technology: In the 1950s, with the development of computer hardware, assembly language could no longer meet the growing programming needs. The emergence of compiler technology realized the automated process of translating high-level languages into machine code, laying the foundation for automatic code generation. Rise of domain-specific languages: In the 1960s, programming languages for specific domains began to appear. These languages simplified programming tasks in specific domains by providing a higher level of abstraction, providing ideas for subsequent model-based code generation. 2. Exploration stage: Computer-aided software engineering tools were widely used during this period. These tools assisted developers in software design and code generation by providing graphical interfaces and code templates, improving development efficiency. In the late 1990s, the Object Management Group proposed MDA (Model-Driven Architecture), emphasizing the model as the core and automatically generating code through model transformation. The proposal of MDA provided a theoretical framework for automatic code generation and promoted the development of related technologies. 3. Development stage: Maturity of code generators: As software scale continued to expand and complexity increased, code generators gradually became important tools in the software development process. For example, ORM (Object-Relational Mapping) frameworks such as Hibernate can automatically generate data access code based on database models, and frameworks such as Spring Roo can generate Web application code based on domain models.

[0025] For the automatic code generation method based on large models, engineers provide operator descriptions, and the large models generate operator implementations according to the descriptions, which improves the efficiency of operator development to a certain extent. However, most of the generated code can only meet the basic correctness requirements. The generated operators consider insufficient boundary conditions and it is difficult to guarantee the performance, making it difficult to achieve engineering implementation. To obtain good performance and comprehensively cover various input situations, it is necessary for operator developers to study the generated code, compile and run it to conduct actual performance evaluations, and let the large models repeatedly modify and improve. Therefore, the efficiency is still not very high, and no further methods for verifying the accuracy and evaluating the performance of operators are proposed. For this reason, this application proposes an operator performance evaluation method, which obtains the operator to be evaluated and the accuracy verification strategy for the operator to be evaluated. Based on the accuracy verification strategy, the accuracy of the operator to be evaluated is verified. When the verification result meets the preset verification conditions, it is judged whether there is an operator evaluation criterion corresponding to the operator to be evaluated. If there is an operator evaluation criterion, the performance evaluation result of the operator to be evaluated is generated based on the operator evaluation criterion. Otherwise, the operator to be evaluated is iteratively tested until the preset number of iterations is reached, and the performance evaluation result of the operator to be evaluated is generated according to the iterative test result. Thus, the accuracy of the operator can be verified and the performance can be evaluated, reducing the threshold for developing high-performance operators and reducing the manual modification process of operator developers to ensure the performance of the operator.

[0026] An embodiment of this application provides an operator performance evaluation method. Combining the execution process of this operator performance evaluation method, the method is described in detail.

[0027] Figure 1 It is a flowchart of the operator performance evaluation method according to an embodiment of this application.

[0028] Exemplarily, as Figure 1 shown, this operator performance evaluation method includes the following steps:

[0029] S1, Obtain the operator to be evaluated and the accuracy verification strategy for the operator to be evaluated.

[0030] S2, Based on the accuracy verification strategy, verify the accuracy of the operator to be evaluated, and when the verification result meets the preset verification conditions, judge whether there is an operator evaluation criterion corresponding to the operator to be evaluated.

[0031] S3, If there is an operator evaluation criterion, generate the performance evaluation result of the operator to be evaluated based on the operator evaluation criterion. Otherwise, perform iterative testing on the operator to be evaluated until the preset number of iterations is reached, and generate the performance evaluation result of the operator to be evaluated according to the iterative test result. Among them, the preset number of iterations can be determined according to the actual situation.

[0032] Specifically, first, obtain the operator to be evaluated and the accuracy verification strategy for the operator to be evaluated. Here, the operator to be evaluated refers to the operator code that needs to be evaluated for performance and accuracy, and these operators can be generated by a certain generation model (such as a large model). In the embodiments of the present application, the operators to be evaluated may come from different sources. For example: operator codes generated by a large model, operator codes manually written by developers, or operator codes imported from other projects or libraries, etc. The accuracy verification strategy refers to a series of methods and criteria used to verify whether the operator to be evaluated is correct in function, and these strategies ensure that the generated operator can correctly execute the expected calculation tasks. For example, use a known correct reference implementation to generate a reference output, and then compare the output of the operator to be evaluated with the reference output.

[0033] After obtaining the operator to be evaluated and the accuracy verification strategy for the operator to be evaluated, the accuracy of the operator to be evaluated can be verified according to the accuracy verification strategy to ensure that it is correct in function. For example, according to the precision verification strategy, it can be judged whether the output of the operator to be evaluated meets the preset precision requirements. And when the verification result meets the preset verification conditions, it can be judged whether there is an operator evaluation criterion corresponding to the operator to be evaluated. That is to say, after the accuracy verification passes, it is necessary to judge whether there is performance benchmark data corresponding to the operator to be evaluated. The performance benchmark data is usually stored in a performance baseline database and is used to evaluate the performance of the generated operator. For example, the performance baseline database stores the performance data of operators that are known to be optimized on different hardware platforms, and the performance benchmark database can be queried according to the name of the operator to be evaluated and the target hardware platform.

[0034] In the case where there is an operator evaluation criterion, the performance evaluation result of the operator to be evaluated can be generated based on the operator evaluation criterion. That is, the operator evaluation criterion refers to the performance data of existing known optimized operators, and these data are used as the performance benchmarks for the generated operators. These benchmark data can include key performance indicators such as throughput and latency. If there is an operator evaluation criterion, the performance data of the operator to be evaluated can be directly compared with the benchmark data to generate a performance evaluation result. For example, run the operator to be evaluated, measure its throughput and latency, obtain the benchmark data of the corresponding operator and hardware platform from the performance benchmark database, compare the performance data of the operator to be evaluated with the benchmark data, calculate the performance ratio, and generate a performance evaluation report according to the comparison result.

[0035] If there is no operator evaluation criterion, the performance of the operator to be evaluated needs to be evaluated through iterative testing. The specific steps are as follows: Set a preset upper limit for the number of iterations. For example, the preset number of iterations can be 10. In each iteration, run the operator to be evaluated, measure its performance data, record the throughput and latency of each iteration, and finally generate a performance evaluation report based on the performance data of all iterations. Thus, compare the performance data of the operator to be evaluated with the benchmark data to generate a performance evaluation result, evaluate the performance of the operator to be evaluated through iterative testing, record the performance data of each iteration, and generate a performance evaluation report.

[0036] Through this method, the performance of the generated operator code can be comprehensively evaluated to ensure that it meets the requirements of actual applications.

[0037] According to an embodiment of the present application, before obtaining the operator to be evaluated, the operator performance evaluation method further includes: determining whether there is an operator generation requirement; if there is an operator generation requirement, determining the actual application scenario and target hardware characteristics based on the operator generation requirement, and determining the target operator description based on the actual application scenario and target hardware characteristics; inputting the target operator description into a pre-trained operator generation model to obtain the operator to be evaluated.

[0038] Specifically, before obtaining the operator to be evaluated, it is first necessary to determine whether there is a need for operator generation. This step is to avoid unnecessary waste of computing resources and ensure the pertinence of subsequent steps. That is, analyze whether the current project or task requires the generation of new operators. For example, in the development of deep learning models, custom convolution operations or special matrix operations may be required. That is, assume that a deep learning model is being developed and an efficient matrix multiplication operator is needed. Before starting, it is necessary to check whether the existing deep learning frameworks (such as TensorFlow or PyTorch) already provide matrix multiplication operators that meet the performance requirements. If the performance of the existing operators is insufficient, or specific optimizations are required, then there is an operator generation requirement.

[0039] In the case of an operator generation requirement, the actual application scenario and target hardware characteristics can be determined based on the operator generation requirement. This step is crucial for generating high-performance operators because different application scenarios and hardware platforms may require different optimization strategies. That is, the specific scenario where the operator will be applied can be determined. For example, whether it is used for convolution operations in a deep learning model or matrix operations in scientific computing. And the hardware platform on which the operator will run can be determined. For example, whether it runs on an NVIDIA GPU (Graphics Processing Unit) or an Intel CPU (Central Processing Unit). Different hardware platforms may support different features (such as CUDA, OpenCL, AVX instruction sets, etc.). For example, it can be assumed that a matrix multiplication operator needs to be generated for the training of a deep learning model. The actual application scenario is to train a large-scale convolutional neural network, and the target hardware is an NVIDIA A100 GPU. In this scenario, the features of the GPU, such as CUDA support and TensorCore acceleration, need to be considered.

[0040] Thus, the target operator description can be determined according to the actual application scenario and target hardware characteristics, and then the target operator description is input into a pre-trained operator generation model to obtain the operator to be evaluated. Among them, the target operator description is the key input for generating high-performance operators, aiming to help the large model better understand the user's requirements, so as to generate high-performance operator code. That is to say, a pre-trained operator generation model is selected. These models are based on deep learning technology and can generate efficient code according to the input operator description. The target operator description is input into the operator generation model, and the operator generation model generates the operator code to be evaluated according to the input description. These steps ensure that the generated operator can meet the requirements of the actual application scenario and achieve high performance on the target hardware.

[0041] According to an embodiment of the present application, the target operator description includes one or more of a mathematical formula description, a pseudocode description, and a constraint description.

[0042] Specifically, the target operator description can include one or more of mathematical formula description, pseudocode description, and constraint description. This flexible description method allows users to provide different levels of details according to actual needs, thus helping the large model generate the required operator code more accurately. That is to say, for the large model to complete the automatic generation of the operator to be evaluated, users need to provide the operator description as input. And based on fully considering the usability for operator development engineers and the model understanding efficiency, three forms of target operator description are proposed as follows. Namely, the mathematical formula description is the basis of the operator description. It uses mathematical symbols and formulas to precisely express the calculation logic of the operator. The mathematical formula is the core definition of the operator, which can avoid the ambiguity that may be brought by natural language description. The mathematical formula describes the mathematical logic of the operator, such as matrix multiplication, convolution operation, activation function, etc., and uses standard mathematical symbols and formats to ensure that the large model can accurately understand.

[0043] The pseudocode description is a description between the mathematical formula and the actual code. It uses Python-like syntax to describe the execution logic and data flow of the operator. The pseudocode can help the large model better understand details such as loop structures and data access patterns. The pseudocode describes the execution order, loop structure, and data access pattern of the operator. And the pseudocode does not need to fully conform to the syntax of a certain programming language, but should be clear enough for the large model to understand. The constraint description is a natural language description used to supplement the optimization requirements that cannot be covered by the mathematical formula and pseudocode. It provides key context information for the optimization direction, such as parallel strategies, memory access optimization, hardware characteristics, etc. The constraint description provides context information for the optimization direction, such as parallel strategies, memory access optimization, hardware characteristics, etc., and the constraint description is expressed in natural language and should be as concise and clear as possible to avoid ambiguity. For example, parallel strategy: Map the i and j dimensions to a 2D thread grid, and each thread calculates one C[i, j]. Memory characteristic: A is accessed row-continuously, B is accessed column-wise (transpose optimization is required), and D has a broadcast characteristic (broadcast in the j dimension). Hardware characteristic: The target GPU is A100, and TensorCore needs to be used.

[0044] In addition, users can selectively provide one or more of the mathematical formula description, pseudocode description, and constraint description according to their needs. This flexible description method allows users to provide different levels of details according to actual needs, thus helping the large model generate the required operator code more accurately. The specific combination methods are as follows: Provide the mathematical formula description to generate the basic version of the operator code. And on this basis, the pseudocode description can be added to optimize the loop structure and data access pattern. Finally, the constraint description can be added to generate a highly optimized operator code. Thus, users can gradually refine the operator description to help the large model generate more efficient and accurate operator code.

[0045] According to an embodiment of the present application, after inputting the target operator description into a pre-trained operator generation model, the operator performance evaluation method further includes: determining whether the operator generation model generates an information missing reminder, where the information missing reminder includes operator correction suggestions; if the operator generation model generates an information missing reminder, sending the information missing reminder to a preset user terminal, so that the user can correct the target operator description based on the operator correction suggestions.

[0046] Specifically, after inputting the target operator description into a pre-trained operator generation model, the operator generation model may find that there is information missing or needs further optimization in the target operator description provided by the user. To ensure that the generated code can meet the actual requirements, the model will generate an information missing reminder. That is, when parsing the target operator description, the operator generation model will evaluate the integrity and accuracy of the description. If the model finds that there is information missing or needs further optimization in the description, it will generate an information missing reminder. For example, when parsing, the operator generation model finds that no description about the parallel strategy is provided, so it generates an information missing reminder.

[0047] The information missing reminder is feedback information generated by the operator generation model, which points out the problems existing in the target operator description and provides correction suggestions. It clearly points out the problems existing in the target operator description, such as information missing, ambiguity or optimization requirements, and the information missing reminder includes operator correction suggestions, that is, specific correction suggestions are provided to help the user improve the target operator description. If the operator generation model generates an information missing reminder, the information missing reminder can be sent to a preset user terminal, so that the user can correct the target operator description based on the operator correction suggestions. That is, the operator generation model can send the information missing reminder to the user's user terminal (mobile terminal) through a preset communication method (such as text message, email, instant messaging tool, etc.). After receiving the reminder, the user can improve the target operator description according to the correction suggestions. Suppose the user's preset mobile terminal is a mobile phone. The operator generation model sends the information missing reminder to the user via text message: There are the following problems with your operator description: 1. No description of the parallel strategy is provided. 2. No description of memory access optimization is provided. Correction suggestions: 1. It is recommended to add a description of the parallel strategy, for example: Map i and j to a 2D thread grid, and each thread calculates a C[i, j]. 2. It is recommended to add a description of memory access optimization, for example: A is accessed continuously by row, and B is accessed by column (transpose optimization is required). Thus, the user supplements or adjusts the target operator description according to the correction suggestions, and the user can re-enter the corrected target operator description into the operator generation model to generate new operator code.

[0048] Thus, this interactive correction mechanism ensures that the generated operator code can more accurately meet the user's needs and improves the quality and performance of the generated code.

[0049] According to an embodiment of the present application, the accuracy verification strategy includes a reference output operator verification strategy, a multi-precision verification strategy, and a test case generation strategy. Based on the accuracy verification strategy, the accuracy verification of the operator to be evaluated is performed, including one of the following verification methods: when the accuracy verification strategy includes the reference output operator verification strategy, obtain the reference output operator corresponding to the operator to be evaluated, and based on the reference output operator, perform the accuracy verification of the operator to be evaluated; and / or, when the accuracy verification strategy includes the multi-precision verification strategy, determine the verification mode and error tolerance according to the application scenario of the operator to be evaluated, and perform the accuracy verification of the operator to be evaluated according to the verification mode and error tolerance; and / or, when the accuracy verification strategy includes the test case generation strategy, generate the test case corresponding to the operator to be evaluated, and perform the accuracy verification of the operator to be evaluated through the test case corresponding to the operator to be evaluated.

[0050] Specifically, the accuracy verification strategy is an important part for ensuring that the generated operator code is functionally correct and meets specific precision requirements. The accuracy verification strategy includes a reference output operator verification strategy, a multi-precision verification strategy, and a test case generation strategy. When performing the accuracy verification of the operator to be evaluated according to the accuracy verification strategy, determine the current verification strategy. When the accuracy verification strategy includes the reference output operator verification strategy, the reference output operator corresponding to the operator to be evaluated can be obtained to perform the accuracy verification of the operator to be evaluated based on the reference output operator. The reference output operator verification strategy refers to using a known correct reference implementation (such as a mature deep learning framework, such as PyTorch or TensorFlow) to generate the correct calculation result, and then comparing the output of the generated operator code with these reference results to verify its accuracy. That is, use the reference implementation (such as PyTorch) to calculate the correct result, pass the same input data to the operator code to be evaluated, calculate the output, and thus calculate the error between the output of the operator to be evaluated and the reference output to determine whether the precision requirement is met.

[0051] When the accuracy verification strategy includes the multi-precision verification strategy, the verification mode and error tolerance can be determined according to the application scenario of the operator to be evaluated, and the accuracy verification of the operator to be evaluated is performed according to the verification mode and error tolerance. That is, the multi-precision verification strategy refers to setting different precision requirements according to different scenarios to verify the correctness of the generated operator, including the verification of absolute error and relative error. The verification mode can be determined according to the application scenario of the operator to be evaluated, such as integer operation, floating-point operation, etc., and the error tolerance is set according to the verification mode. For example, integer operation may require an absolute error less than 1e-6, while floating-point operation may require a relative error less than 1e-4. Calculate the error between the output of the operator to be evaluated and the reference output to determine whether the error tolerance is met.

[0052] When the accuracy verification strategy includes a test case generation strategy, test cases corresponding to the operator to be evaluated can be generated to verify the accuracy of the operator to be evaluated through the test cases corresponding to the operator to be evaluated. That is, the test case generation strategy refers to automatically generating unit tests according to the unit test generation convention in the operator development process to verify the correctness of the operator to be evaluated. Diversified test cases can be automatically generated according to the input data type and range of the operator. For example, random matrices, all-zero matrices, all-one matrices, etc. can be generated. The generated test cases are passed to the operator code to be evaluated, the output is calculated, and the output of the operator to be evaluated is compared with the reference output to determine whether the accuracy requirement is met.

[0053] Thus, through these verification strategies, the accuracy of the generated operator code can be comprehensively verified to ensure that it meets the actual requirements.

[0054] Furthermore, according to an embodiment of the present application, the test cases corresponding to the operator to be evaluated include at least one of a base case, a boundary case, and a random sparse matrix.

[0055] Specifically, the test cases include at least one of a base case, a boundary case, and a random sparse matrix. These test cases cover different input scenarios, thus comprehensively verifying the correctness and robustness of the operator. Among them, the base case refers to a typical scenario that the operator should be able to correctly handle under normal input conditions. These cases are used to verify the basic functions of the operator under common inputs. That is, common and reasonable input data are used, such as randomly generated matrices or vectors, and it is ensured that the operator can execute correctly under normal inputs and generate the expected output. For example, for A = np.random.randn(128, 256).astype(np.float32), B = np.random.randn(256, 512).astype(np.float32), A and B are two randomly generated matrices with sizes of 128×256 and 256×512 respectively. The sizes and data distributions of these matrices are typical and are used to verify the basic functions of the matrix multiplication operator.

[0056] Boundary cases refer to scenarios where the input data is under extreme or boundary conditions. These cases are used to verify the behavior of the operator under extreme conditions and ensure that it can correctly handle boundary conditions. Input data under extreme or boundary conditions, such as all-zero matrices, all-one matrices, large-sized matrices, etc., are used to ensure that the operator can execute correctly under boundary conditions and generate the expected output. Assuming the operator to be evaluated is matrix multiplication, boundary cases can be: A = np.zeros((1024, 1024), dtype = np.float32), B = np.ones((1024, 1024), dtype = np.float32). A is an all-zero matrix and B is an all-one matrix, both of size 1024×1024. The sizes and data distributions of these matrices are extreme and are used to verify the behavior of the matrix multiplication operator under boundary conditions.

[0057] A random sparse matrix refers to a matrix in which most of the elements are zero and only a few elements are non-zero. Generate random sparse matrices where the distribution and positions of the non-zero elements are random to ensure that the operator can correctly handle sparse matrices and generate the expected output. For example, A = sparse_matrix(2048, 0.1), B = sparse_matrix(2048, 0.1). A and B are two randomly generated sparse matrices, both of size 2048×2048 and with a sparsity of 10%. The sparsity of these matrices simulates common scenarios in practical applications and is used to verify the behavior of the matrix multiplication operator when dealing with sparse matrices.

[0058] Therefore, the test case generation strategy can comprehensively use basic cases, boundary cases, and random sparse matrices to comprehensively verify the correctness and robustness of the operator. By generating diverse test cases and conducting strict verification, it can be ensured that the generated operator code can run correctly under various conditions and meet the requirements of practical applications.

[0059] According to an embodiment of the present application, the operator evaluation criteria include the target memory access bandwidth, target throughput, and target board utilization rate during the operation of the target processor. Based on the operator evaluation criteria, a performance evaluation result of the operator to be evaluated is generated, including one of the following evaluation methods: when the actual memory access bandwidth of the operator to be evaluated is greater than or equal to the target memory access bandwidth, and / or, when the actual throughput of the operator to be evaluated is greater than or equal to the target throughput, and / or, when the actual board utilization rate of the operator to be evaluated is greater than or equal to the target board utilization rate, it is determined that the operator to be evaluated meets the target performance.

[0060] Specifically, the operator evaluation criteria refer to the performance metrics that an operator needs to achieve when running on a target processor. These metrics include: the target memory access bandwidth, the target throughput, and the target board utilization rate. Among them, the target memory access bandwidth is the expected memory access bandwidth of the target processor when running the operator, the target throughput is the expected number of floating-point operations per second of the target processor when running the operator, and the target board utilization rate is the expected utilization rate of the target processor when running the operator. These criteria can be based on the performance data of existing well-optimized operators and stored in the performance benchmark database.

[0061] When generating the performance evaluation result of the operator to be evaluated according to the operator evaluation criteria, tools (such as Nsight Compute) can be used to measure the actual memory access bandwidth of the operator to be evaluated, calculate the actual throughput of the operator to be evaluated by measuring the execution time of the kernel and the number of floating-point operations executed, and use tools (such as nvidia-smi) to measure the actual board utilization rate of the operator to be evaluated. Compare the actual performance metrics of the operator to be evaluated with the operator evaluation criteria to determine whether the target performance is achieved. If the actual memory access bandwidth of the operator to be evaluated is greater than or equal to the target memory access bandwidth, or, the actual throughput of the operator to be evaluated is greater than or equal to the target throughput, or, the actual board utilization rate of the operator to be evaluated is greater than or equal to the target board utilization rate, that is, if one or more of the above conditions are met, it can be determined that the operator to be evaluated meets the target performance. Through this method, the performance of the generated operator code can be comprehensively evaluated to ensure that it meets the requirements of actual applications.

[0062] In addition, in the embodiments of the present application, based on the performance evaluation data, optimization suggestions can also be automatically generated. These optimization suggestions are aimed at improving the performance of the operator to make it closer to or exceed the performance benchmark. The optimization suggestions may include: suggestions on how to better utilize multi-threading or multi-cores to improve computing efficiency, suggestions on how to optimize the memory access pattern to reduce latency and improve bandwidth utilization, suggestions on how to utilize specific hardware features (such as NVIDIA's TensorCore) to accelerate computing, etc. Assuming that the performance evaluation result shows that the memory access bandwidth of the operator to be evaluated is lower than the target value, the following optimization suggestions can be generated: use shared memory to cache frequently accessed data to reduce the number of global memory accesses, optimize the memory access pattern to ensure continuous memory access and reduce memory fragmentation.

[0063] In addition to the automatically generated optimization suggestions, experienced developers are also allowed to manually provide optimization suggestions. These manually configured optimization suggestions can be based on the developers' professional knowledge and experience and provided to the large model as input. The manually configured optimization suggestions can be directly provided to the large model to help it generate more efficient operator code. These optimization suggestions can override the automatically generated suggestions or supplement the automatically generated suggestions to ensure that the generated operator code can meet specific performance requirements. For example, assuming that the developer believes based on experience that memory access needs to be further optimized, the following optimization suggestions can be manually provided: store matrix B after transposition to optimize the memory access pattern. By combining automatic and manual optimization suggestions, operator code that better meets the actual requirements can be generated, improving the performance and efficiency of the code. Through this method, the system can generate more efficient operator code to meet the performance requirements of actual applications.

[0064] The following combines Figure 2 to describe the method of this application.

[0065] As a specific example, the operator performance evaluation method of this application may include the following steps:

[0066] S101, in the case of an operator generation requirement, determine the actual application scenario and target hardware characteristics based on the operator generation requirement, and determine the target operator description based on the actual application scenario and target hardware characteristics.

[0067] S102, input the target operator description into a pre-trained operator generation model to obtain the operator to be evaluated.

[0068] S103, obtain the accuracy verification strategy for the operator to be evaluated.

[0069] S104, when the accuracy verification strategy includes the reference output operator verification strategy, obtain the reference output operator corresponding to the operator to be evaluated, and verify the accuracy of the operator to be evaluated based on the reference output operator; when the accuracy verification strategy includes the multi-precision verification strategy, determine the verification mode and error tolerance according to the application scenario of the operator to be evaluated, and verify the accuracy of the operator to be evaluated according to the verification mode and error tolerance; when the accuracy verification strategy includes the test case generation strategy, generate the test case corresponding to the operator to be evaluated to verify the accuracy of the operator to be evaluated through the test case corresponding to the operator to be evaluated.

[0070] S105, when the verification result meets the preset verification conditions, determine whether there is an operator evaluation criterion corresponding to the operator to be evaluated. If yes, execute step S106; if no, execute step S107.

[0071] S106, generate the performance evaluation result of the operator to be evaluated based on the operator evaluation criterion.

[0072] S107, perform iterative testing on the operator to be evaluated until the preset number of iterations is reached, and generate a performance evaluation result of the operator to be evaluated according to the iterative test results.

[0073] In summary, according to the operator performance evaluation method of the embodiments of the present application, obtain the operator to be evaluated and the accuracy verification strategy of the operator to be evaluated. Based on the accuracy verification strategy, perform accuracy verification on the operator to be evaluated, and when the verification result meets the preset verification conditions, determine whether there is an operator evaluation criterion corresponding to the operator to be evaluated. If there is an operator evaluation criterion, generate a performance evaluation result of the operator to be evaluated based on the operator evaluation criterion. Otherwise, perform iterative testing on the operator to be evaluated until the preset number of iterations is reached, and generate a performance evaluation result of the operator to be evaluated according to the iterative test results. Thus, this method can perform accuracy verification and performance evaluation on the operator, lower the development threshold of high-performance operators, reduce the process of manual modification by operator development engineers, and ensure the performance of the operator.

[0074] An embodiment of the present application also provides an operator performance evaluation device.

[0075] Figure 3 It is a block diagram of an operator performance evaluation device according to an embodiment of the present application.

[0076] As Figure 3 shown, the operator performance evaluation device 100 includes: an acquisition module 110, a verification module 120, and an evaluation module 130.

[0077] Among them, the acquisition module 110 is used to acquire the operator to be evaluated and the accuracy verification strategy of the operator to be evaluated. The verification module 120 is used to perform accuracy verification on the operator to be evaluated based on the accuracy verification strategy, and determine whether there is an operator evaluation criterion corresponding to the operator to be evaluated when the verification result meets the preset verification conditions. The evaluation module 130 is used to, if there is an operator evaluation criterion, generate a performance evaluation result of the operator to be evaluated based on the operator evaluation criterion. Otherwise, perform iterative testing on the operator to be evaluated until the preset number of iterations is reached, and generate a performance evaluation result of the operator to be evaluated according to the iterative test results.

[0078] According to an embodiment of the present application, before acquiring the operator to be evaluated, the acquisition module 110 is further used to: determine whether there is an operator generation requirement; if there is an operator generation requirement, determine the actual application scenario and target hardware characteristics based on the operator generation requirement, and determine the target operator description based on the actual application scenario and target hardware characteristics; input the target operator description into a pre-trained operator generation model to obtain the operator to be evaluated.

[0079] According to an embodiment of the present application, the target operator description includes one or more of a mathematical formula description, a pseudocode description, and a constraint description.

[0080] According to an embodiment of the present application, after the obtaining module 110 inputs the target operator description into the pre-trained operator generation model, it is further configured to: determine whether the operator generation model generates an information missing reminder, where the information missing reminder includes an operator correction suggestion; if the operator generation model generates an information missing reminder, send the information missing reminder to a preset user terminal, so that the user corrects the target operator description based on the operator correction suggestion.

[0081] According to an embodiment of the present application, the accuracy verification strategy includes a reference output operator verification strategy, a multi-precision verification strategy, and a test case generation strategy. The verification module 120 performs accuracy verification on the operator to be evaluated based on the accuracy verification strategy, specifically configured to: when the accuracy verification strategy includes the reference output operator verification strategy, obtain the reference output operator corresponding to the operator to be evaluated, and perform accuracy verification on the operator to be evaluated based on the reference output operator; and / or, when the accuracy verification strategy includes the multi-precision verification strategy, determine the verification mode and error tolerance according to the application scenario of the operator to be evaluated, and perform accuracy verification on the operator to be evaluated according to the verification mode and error tolerance; and / or, when the accuracy verification strategy includes the test case generation strategy, generate a test case corresponding to the operator to be evaluated, and perform accuracy verification on the operator to be evaluated through the test case corresponding to the operator to be evaluated.

[0082] According to an embodiment of the present application, the test case corresponding to the operator to be evaluated includes at least one of a basic case, a boundary case, and a random sparse matrix.

[0083] According to an embodiment of the present application, the operator evaluation criterion includes the target memory access bandwidth, target throughput, and target board utilization rate during the operation of the target processor. The evaluation module 130 generates a performance evaluation result of the operator to be evaluated based on the operator evaluation criterion, specifically configured to: when the actual memory access bandwidth of the operator to be evaluated is greater than or equal to the target memory access bandwidth, and / or, when the actual throughput of the operator to be evaluated is greater than or equal to the target throughput, and / or, when the actual board utilization rate of the operator to be evaluated is greater than or equal to the target board utilization rate, determine that the operator to be evaluated meets the target performance.

[0084] It should be noted that for the details not disclosed in the operator performance evaluation device of the embodiments of the present application, please refer to the details disclosed in the operator performance evaluation method of the embodiments of the present application, and will not be elaborated here specifically.

[0085] An operator performance evaluation device according to an embodiment of the present application, an acquisition module is configured to acquire an operator to be evaluated and an accuracy verification strategy for the operator to be evaluated, a verification module is configured to perform accuracy verification on the operator to be evaluated based on the accuracy verification strategy, and determine whether there is an operator evaluation criterion corresponding to the operator to be evaluated when the verification result meets a preset verification condition, and an evaluation module is configured to, if there is an operator evaluation criterion, generate a performance evaluation result of the operator to be evaluated based on the operator evaluation criterion, otherwise, perform iterative testing on the operator to be evaluated until a preset number of iterations is reached, and generate a performance evaluation result of the operator to be evaluated according to the iterative test result. Thus, the device can perform accuracy verification and performance evaluation on the operator, reduce the development threshold of high-performance operators, and reduce the manual modification process of operator development engineers to ensure the performance of the operator.

[0086] Corresponding to the above embodiment, the present application also proposes an electronic device.

[0087] As Figure 4 shown, the electronic device 200 according to an embodiment of the present application may include: a memory 210, a processor 220, and a program stored on the memory 210 and executable on the processor 220. When the processor 220 executes the program, the above-mentioned operator performance evaluation method is implemented.

[0088] The electronic device according to an embodiment of the present application can perform accuracy verification and performance evaluation on the operator by executing the above-mentioned operator performance evaluation method, reduce the development threshold of high-performance operators, and reduce the manual modification process of operator development engineers to ensure the performance of the operator.

[0089] Corresponding to the above embodiment, the present application also proposes a computer-readable storage medium.

[0090] The computer-readable storage medium according to an embodiment of the present application stores a program thereon, and when the program is executed by a processor, the above-mentioned operator performance evaluation method is implemented.

[0091] The computer-readable storage medium according to an embodiment of the present application can perform accuracy verification and performance evaluation on the operator by executing the above-mentioned operator performance evaluation method, reduce the development threshold of high-performance operators, and reduce the manual modification process of operator development engineers to ensure the performance of the operator.

[0092] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0093] The above has introduced in detail an operator performance evaluation method provided by this application. Specific examples are used herein to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. An operator performance evaluation method, characterized in that, The method includes: Obtaining an operator to be evaluated and an accuracy verification strategy for the operator to be evaluated; Based on the accuracy verification strategy, performing accuracy verification on the operator to be evaluated, and when the verification result meets a preset verification condition, determining whether there is an operator evaluation criterion corresponding to the operator to be evaluated; If the operator evaluation criterion exists, generating a performance evaluation result for the operator to be evaluated based on the operator evaluation criterion, otherwise, performing iterative testing on the operator to be evaluated until a preset number of iterations is reached, and generating a performance evaluation result for the operator to be evaluated according to the iterative test result.

2. The operator performance evaluation method according to claim 1, characterized in that Before obtaining the operator to be evaluated, it further includes: Determining whether there is an operator generation requirement; If the operator generation requirement exists, determining an actual application scenario and target hardware characteristics based on the operator generation requirement, and determining a target operator description based on the actual application scenario and the target hardware characteristics; Inputting the target operator description into a pre-trained operator generation model to obtain the operator to be evaluated.

3. The operator performance evaluation method according to claim 2, wherein The target operator description includes one or more of a mathematical formula description, a pseudocode description, and a constraint description.

4. The operator performance evaluation method according to claim 3, wherein After inputting the target operator description into the pre-trained operator generation model, it further includes: Determining whether the operator generation model generates an information missing reminder, where the information missing reminder includes an operator correction suggestion; If the operator generation model generates the information missing reminder, sending the information missing reminder to a preset user terminal so that the user can correct the target operator description based on the operator correction suggestion.

5. The operator performance evaluation method according to claim 1, characterized in that The accuracy verification strategy includes a reference output operator verification strategy, a multi-precision verification strategy, and a test case generation strategy. Based on the accuracy verification strategy, performing accuracy verification on the operator to be evaluated includes one of the following verification methods: When the accuracy verification strategy includes the reference output operator verification strategy, obtaining a reference output operator corresponding to the operator to be evaluated, and performing accuracy verification on the operator to be evaluated based on the reference output operator; And / or, when the accuracy verification strategy includes the multi-precision verification strategy, determining a verification mode and an error tolerance according to the application scenario of the operator to be evaluated, and performing accuracy verification on the operator to be evaluated according to the verification mode and the error tolerance; And / or, when the accuracy verification strategy includes the test case generation strategy, generating a test case corresponding to the operator to be evaluated to perform accuracy verification on the operator to be evaluated through the test case corresponding to the operator to be evaluated.

6. The operator performance evaluation method according to claim 5, wherein The test case corresponding to the operator to be evaluated includes at least one of a basic case, a boundary case, and a random sparse matrix.

7. The operator performance evaluation method according to claim 1, characterized in that, The operator evaluation criterion includes a target memory access bandwidth, a target throughput, and a target board utilization rate during the operation of the target processor. Generating a performance evaluation result for the operator to be evaluated based on the operator evaluation criterion includes one of the following evaluation methods: In the case where the actual memory access bandwidth of the operator to be evaluated is greater than or equal to the target memory access bandwidth, And / or, in the case where the actual throughput of the operator to be evaluated is greater than or equal to the target throughput, When the actual board utilization rate of the operator to be evaluated is greater than or equal to the target board utilization rate, it is determined that the operator to be evaluated meets the target performance.

8. An operator performance evaluation device, characterized in that, The device includes: An acquisition module, configured to acquire an operator to be evaluated and an accuracy verification strategy for the operator to be evaluated; A verification module, configured to perform accuracy verification on the operator to be evaluated based on the accuracy verification strategy, and determine whether there is an operator evaluation criterion corresponding to the operator to be evaluated when the verification result meets a preset verification condition; An evaluation module, configured to, if there is the operator evaluation criterion, generate a performance evaluation result of the operator to be evaluated based on the operator evaluation criterion; otherwise, perform iterative testing on the operator to be evaluated until a preset number of iterations is reached, and generate a performance evaluation result of the operator to be evaluated according to the iterative test result.

9. An electronic device, characterized in that, including: A memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, the operator performance evaluation method according to any one of claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that, A program is stored thereon, and when the program is executed by the processor, the operator performance evaluation method according to any one of claims 1-7 is implemented.

Images