Operator test case generation method and device, computer equipment, medium and product

Abstract

Embodiments of the present application provide an operator test case generation method and device, computer equipment, medium and product. The method comprises: after completing configuration of a generation environment, obtaining a to-be-parsed file, parsing the to-be-parsed file to obtain operator input data, operating the operator input data using the generation environment to obtain a baseline result, and writing the operator input data, the baseline result and test case associated data into a file in a preset format to generate an operator test case. Based on the method provided in the present application, the efficiency of generating test cases can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to an operator test case generation method, apparatus, computer equipment, medium, and product. Background Technology

[0002] With the improvement of computing power and the expansion of application fields of high-performance microprocessors, artificial intelligence algorithms have received increasing attention. Related algorithm models are composed of computational units, which can be called operators. The performance of operators directly affects the functionality and performance of the algorithm model. In the actual operator development process, different hardware platforms and operator implementation schemes will lead to differences in operator performance. Therefore, it is necessary to test the performance of operators to ensure the functionality and performance of the algorithm model.

[0003] In related technologies, operators are usually tested by manually writing test cases for different scenarios. However, generating test cases manually is often inefficient. Summary of the Invention

[0004] This application provides an operator test case generation method, apparatus, computer equipment, medium, and product, which can improve the efficiency of test case generation.

[0005] In a first aspect, embodiments of this application provide a method for generating operator test cases, including:

[0006] After completing the configuration of the production environment, obtain the file to be parsed;

[0007] The file to be parsed is parsed to obtain the operator input data;

[0008] Using the aforementioned generation environment, the operator input data is processed to obtain a baseline result;

[0009] The operator input data, baseline results, and test case association data are written into a file with a preset format to generate an operator test case.

[0010] In one possible implementation, parsing the file to be parsed to obtain operator input data includes:

[0011] Determine the file type of the file to be parsed;

[0012] If the file type is a feature combination file, then the feature combination file is parsed to obtain test case feature information, and the operator input data is obtained based on the test case feature information;

[0013] If the file type is a historical test case file, then the historical test case file is parsed to obtain the target test case information, and the operator input data is obtained based on the target test case information.

[0014] In one possible implementation, parsing the feature combination file to obtain test case feature information includes:

[0015] Parse the feature combination file to obtain at least one set of operator feature combinations;

[0016] One set of operator feature combinations is selected from the at least one set of operator feature combinations as the test case feature information.

[0017] In one possible implementation, the operator feature combination includes at least one feature;

[0018] Accordingly, obtaining the operator input data based on the test case feature information includes:

[0019] For any feature in the operator feature combination, a preset data randomization algorithm is used to obtain the input data corresponding to the feature;

[0020] The input data corresponding to all features in the operator feature combination are determined as the operator input data.

[0021] In one possible implementation, parsing the historical test case file to obtain target test case information includes:

[0022] The historical test case file is parsed to obtain at least one set of test case information, and a set of test case information is selected from the at least one set of test case information as the target test case information.

[0023] In one possible implementation, obtaining the operator input data based on the target test case information includes:

[0024] If the target test case information is a combination of historical operator features, then for any historical feature in the combination of historical operator features, a preset data randomization algorithm is used to obtain the input data corresponding to the historical feature, and the input data corresponding to all historical features in the combination of historical operator features are determined as the operator input data, wherein the combination of historical operator features includes at least one historical feature;

[0025] If the target test case information is true data, then the true data is determined as the operator input data.

[0026] In one possible implementation, prior to parsing the feature combination file, the method further includes:

[0027] Get the configuration file;

[0028] The configuration file is parsed to determine the target constraints;

[0029] Based on the target constraints, at least one set of operator feature combinations is obtained;

[0030] Each set of operator feature combinations is written into a file with a preset format to obtain the feature combination file.

[0031] In one possible implementation, parsing the configuration file to determine the target constraints includes:

[0032] If the configuration file corresponds to the first keyword identifier, then the target constraint is determined to be a fully randomized generation mode;

[0033] If the configuration file corresponds to the second keyword identifier, then the target constraint is determined to be a partially randomized generation mode;

[0034] If the configuration file corresponds to a third keyword identifier, then the target constraint is determined to be a fixed generation mode.

[0035] In one possible implementation, the step of using the generation environment to perform operations on the operator input data to obtain a baseline result includes:

[0036] Construct the operator comparison script corresponding to the generated environment;

[0037] The operator comparison script is used to perform calculations on the operator input data to obtain the baseline result.

[0038] In one possible implementation, the step of using a preset operator comparison script to perform operations on the operator input data to obtain the baseline result includes:

[0039] Execute the preset operator comparison script to store the operator input data to the microprocessor.

[0040] The preset operator comparison interface is invoked to perform calculations on the operator input data stored on the microprocessor side to obtain the baseline result;

[0041] The baseline results are stored on the central processing unit.

[0042] In one possible implementation, the test case association data includes at least one of benchmark performance data, hardware information, accuracy formulas, and framework information, wherein the framework information includes framework type and framework version.

[0043] In one possible implementation, the process of generating the benchmark performance data includes:

[0044] Call the acquisition tool corresponding to the generated environment;

[0045] Based on the acquisition tool, the computing performance data of the generation environment is collected;

[0046] The computational performance data of the generated environment is determined as the baseline performance data.

[0047] In one possible implementation, the configuration process of the generation environment includes:

[0048] Get configuration information;

[0049] Based on the configuration information, a generation environment is constructed for generating operator test cases.

[0050] In one possible implementation, after constructing the operator test cases, the method further includes:

[0051] The operator is tested based on the aforementioned operator test cases.

[0052] In one possible implementation, testing the operator based on the operator test cases includes:

[0053] Obtain the test results and test performance data of the operator under a preset test environment;

[0054] Based on the accuracy formula, the error between the baseline result and the test result is obtained;

[0055] If the error does not exceed the preset threshold, the operator is determined to have met the accuracy standard in the test environment; otherwise, the operator is determined to have failed to meet the accuracy standard in the test environment.

[0056] If the test performance data is higher than the baseline performance data, it is determined that the computational performance of the operator in the test environment is higher than that in the generation environment; otherwise, it is determined that the computational performance of the operator in the test environment is lower than that in the generation environment.

[0057] Secondly, this application provides an operator test case generation apparatus, comprising:

[0058] The acquisition module is used to obtain the file to be parsed after the production environment is configured.

[0059] The processing module is used to parse the file to be parsed to obtain operator input data;

[0060] The processing module is also used to perform operations on the operator input data using the generation environment to obtain a baseline result;

[0061] The processing module is also used to write the operator input data, baseline results and test case association data into a file with a preset format to generate an operator test case.

[0062] Thirdly, this application provides a computer device, including: a memory and a processor;

[0063] The memory stores computer-executed instructions;

[0064] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described above.

[0065] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described above.

[0066] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0067] The operator test case generation method, apparatus, computer equipment, medium, and product provided in this application can perform calculations on operator input data according to a configured generation environment to obtain baseline results. Furthermore, the operator input data, baseline results, and test case association data can be written into a file with a preset format to complete the generation of operator test cases. The method provided in this application eliminates the need for manually writing operator test cases; it can generate operator test cases based on a configured generation environment, thereby improving the efficiency of operator test case generation. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0069] Figure 1 This application provides a schematic diagram of the scenario.

[0070] Figure 2 A flowchart illustrating the operator test case generation method provided in this application. Figure 1 ;

[0071] Figure 3 A flowchart illustrating the operator test case generation method provided in this application. Figure 2 ;

[0072] Figure 4 A flowchart illustrating the operator test case generation method provided in this application. Figure 3 ;

[0073] Figure 5 A flowchart illustrating the operator test case generation method provided in this application. Figure 4 ;

[0074] Figure 6 A flowchart illustrating the operator test case generation method provided in this application. Figure 5 ;

[0075] Figure 7 A flowchart illustrating the operator test case generation method provided in this application. Figure 6 ;

[0076] Figure 8 A flowchart illustrating the operator test case generation method provided in this application. Figure 7 ;

[0077] Figure 9 A flowchart illustrating the operator test case generation method provided in this application. Figure 8 ;

[0078] Figure 10 A flowchart illustrating the operator test case generation method provided in this application. Figure 9 ;

[0079] Figure 11 A flowchart illustrating the operator test case generation method provided in this application. Figure 10 ;

[0080] Figure 12 A flowchart illustrating the operator test case generation method provided in this application. Figure 10 one;

[0081] Figure 13 A flowchart illustrating the operator test case generation method provided in this application. Figure 10 two;

[0082] Figure 14 A schematic diagram illustrating the overall process of generating test cases for operators;

[0083] Figure 15 A schematic diagram of the operator test case generation device provided in this application;

[0084] Figure 16 A schematic diagram of the structure of the electronic device provided in this application.

[0085] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0086] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0087] The performance of operators directly affects the functionality and performance of related algorithm models. Operator performance can be tested by constructing operator test cases. In related technologies, operator test cases are typically generated manually, but this method is often inefficient.

[0088] Figure 1 The scenario diagram provided for this application is as follows: Figure 1 As shown, this application can generate operator test cases through a computing device. The computing device can be selected as a test case generator. The test generator is a multi-functional test case generation tool, which consists of a Random module and a Generator module.

[0089] Therefore, the operator test case generation method, apparatus, computer device, medium, and product provided in this application embodiment can use a computing device to perform calculations on operator input data according to a configured generation environment to obtain baseline results. The operator input data, baseline results, and test case association data can be written into a file with a preset format to complete the generation of operator test cases. This eliminates the need for manually writing operator test cases, thus improving the efficiency of operator test case generation.

[0090] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0091] Figure 2 A flowchart illustrating the operator test case generation method provided in this application. Figure 1 ,like Figure 2 As shown, the above method includes:

[0092] S201. After completing the configuration of the production environment, obtain the file to be parsed.

[0093] Based on the scenario examples, the executing entity of this application can be Figure 1 The computing device in the example can be a test case generator.

[0094] The generation environment includes the hardware environment in which the operator runs and the software environment installed on the hardware. The test case generator obtains the file to be parsed through the preset file path in the generation environment.

[0095] S202. The file to be parsed is parsed to obtain operator input data.

[0096] Based on the scenario example, the operator input data is related to the operator type. For example, if the operator type is an addition operator, then the operator input data includes two input data.

[0097] S203. Using the aforementioned generation environment, perform calculations on the operator input data to obtain a baseline result.

[0098] Based on the scenario example, since the performance of the operator varies under different hardware and software environments, the operation on the input data of the operator can be completed in the generation environment configured in this application. The output result of the operator in this generation environment can be obtained, and the obtained output result can be determined as the benchmark for subsequent comparison. Therefore, the output result can be determined as the baseline result corresponding to the operator.

[0099] S204. Write the operator input data, baseline results, and test case association data into a file with a preset format to generate an operator test case.

[0100] Based on the scenario example, operator test cases can be stored in files of type Protocol Buffers. Protocol Buffers files have the extension prototxt or pb. Protobuf is a serialization data format and communication protocol commonly used for data exchange and serialization between different programming languages ​​or platforms. Protobuf uses a compact binary representation, offering higher efficiency and smaller file size compared to traditional text formats. It supports various data types, including integers, floating-point numbers, strings, lists, and mappings, and can handle nested data structures. Therefore, the default file format can be Protocol Buffers. After obtaining the baseline results corresponding to the operator, the operator input data, baseline results, and test case association data can be written to a Protocol Buffers file to obtain the operator test cases. The test case association data includes data related to the operator test cases, such as data related to the corresponding production environment. Based on the method provided in this example, operator test cases can be generated using a computing device, eliminating the need for manual writing and improving the efficiency of operator test case generation.

[0101] Optional, Figure 3A flowchart illustrating the operator test case generation method provided in this application. Figure 2 ,like Figure 3 As shown, S202 includes:

[0102] S301. Determine the file type of the file to be parsed.

[0103] Based on the scenario examples, the file types of the files to be parsed can be divided into two types: one is a feature combination file used to store feature combinations, and the other is a historical test case file used to store historical operator test cases.

[0104] S302. If the file type is a feature combination file, then the feature combination file is parsed to obtain test case feature information, and the operator input data is obtained based on the test case feature information.

[0105] In the example scenario, since the storage of feature combinations relies on JavaScript Object Notation (JSON) files, i.e., files with the .josn extension, if the file to be parsed has a .josn extension, it is determined to be a feature combination file. Parsing the feature combination file yields the stored feature combination information, and the operator input data is obtained based on this information.

[0106] S303. If the file type belongs to a historical test case file, then parse the historical test case file to obtain the target test case information, and obtain the operator input data based on the target test case information.

[0107] Based on the scenario example and the above content, since operator test cases can be stored in files of type Protocol Buffers, if the file to be parsed is of type Protocol Buffers (i.e., a file with the extension prototxt or pb), then the file to be parsed is determined to be a historical test case file. From the historical test cases, target test case information related to the historical operator test cases can be parsed, and the parsed test case information is determined as the operator input data.

[0108] Based on the method provided in this example, different methods can be used to obtain operator input data for different types of files to be parsed, which can improve the flexibility of obtaining operator input data.

[0109] Optional, Figure 4 A flowchart illustrating the operator test case generation method provided in this application. Figure 3 ,like Figure 4 As shown in S302, the step of parsing the feature combination file to obtain test case feature information includes:

[0110] S401. Parse the feature combination file to obtain at least one set of operator feature combinations.

[0111] Based on the scenario example, the feature combination file stores feature combination information, including at least one set of feature combinations. Therefore, after parsing the feature combination file, at least one set of feature combinations stored therein can be obtained.

[0112] S402. Select a set of operator feature combinations from the at least one set of operator feature combinations as the test case feature information.

[0113] Based on the scenario example, each set of operator feature combinations can yield corresponding operator input data, and thus, the operator test cases corresponding to that operator feature combination can be obtained from the operator input data. Therefore, the operator feature combinations parsed from the feature combination file can be sorted, and each set of operator feature combinations can be determined as test case feature information in order of their position.

[0114] Based on the method provided in this example, the operator feature combinations obtained by parsing the feature combination file are more comprehensive, which can improve the accuracy of subsequent operator test cases generated based on the operator feature combinations.

[0115] Optionally, the operator feature combination includes at least one feature;

[0116] Accordingly, Figure 5 A flowchart illustrating the operator test case generation method provided in this application. Figure 4 ,like Figure 5 As shown, in S302, the operator input data is obtained based on the test case feature information, including:

[0117] S501. For any feature in the operator feature combination, a preset data randomization algorithm is used to obtain the input data corresponding to the feature.

[0118] In the context of the scenario, the operator feature combination includes features such as tensors, scalars, and operator intrinsic parameters. Tensors refer to the input and output tensors corresponding to the operator, and include attributes such as shape, stride, and data type. Operator intrinsic parameters include attributes such as mode and preference. Data randomization algorithms include uniform distribution functions, Gaussian distributions, or binomial distributions. One of these algorithms can be used to determine the value range. For example, if the value range is [0,1], the determined randomization algorithm can be used to determine the value of each feature in the operator feature combination based on the value range, and the obtained value is then used as the input data corresponding to that feature.

[0119] S502. Determine the input data corresponding to all features in the operator feature combination as the operator input data.

[0120] Based on the scenario example, after obtaining the input data corresponding to each feature in the operator feature combination according to the above method, the input data corresponding to each feature in the operator feature combination is determined as the operator input data corresponding to that operator.

[0121] Based on the method provided in this example, a data randomization algorithm is used to obtain the input data corresponding to each feature, thereby obtaining the operator input data, which can improve the efficiency and completeness of obtaining the operator input data.

[0122] Optionally, in S303, parsing the historical test case file to obtain target test case information includes:

[0123] The historical test case file is parsed to obtain at least one set of test case information, and a set of test case information is selected from the at least one set of test case information as the target test case information.

[0124] Similarly, using the scenario example, the historical test case file stores relevant information about historical operator test cases, including at least one set of test case information related to historical operator test cases. The test case information includes two types: if the content stored in the historical test case file is a feature combination related to historical operator test cases, then the parsed test case information is the historical operator feature combination corresponding to the historical operator test case. If the content stored in the historical test case file is the operator input data corresponding to the historical operator test cases, then the parsed test case information is the operator input data.

[0125] Each set of test case information can yield corresponding operator input data, and thus the operator test case corresponding to that test case information can be obtained from the operator input data of that test case information. Therefore, the test case information parsed from the historical test case file can be sorted, and each set of test case information can be determined as the target test case information in order, and the corresponding operator input data can be obtained based on the target test case information.

[0126] Based on the method provided in this example, the test case information obtained by parsing historical test case files is more comprehensive, which can improve the accuracy of subsequently generated operator test cases.

[0127] In one possible implementation, Figure 6 A flowchart illustrating the operator test case generation method provided in this application. Figure 5 ,like Figure 6 As shown, in S303, the operator input data is obtained based on the target test case information, including:

[0128] S601. If the target test case information is a combination of historical operator features, then for any historical feature in the combination of historical operator features, a preset data randomization algorithm is used to obtain the input data corresponding to the historical feature, and the input data corresponding to all historical features in the combination of historical operator features are determined as the operator input data, wherein the combination of historical operator features includes at least one historical feature.

[0129] In a scenario example, if the parsed target test case information is a combination of historical operator features, then similarly, according to one of the data randomization algorithms—uniform distribution function, Gaussian distribution, or binomial distribution—the value of each historical feature in the historical operator feature combination is determined, and the obtained value is used as the input data corresponding to that historical feature. After obtaining the input data corresponding to each historical feature in the historical operator feature combination, the input data corresponding to each historical feature is determined as the operator input data corresponding to that operator.

[0130] S602. If the target test case information is true data, then the true data is determined as the operator input data.

[0131] Based on the scenario example, since the operator input data is the value corresponding to each feature, if the parsed target test case information is true value data, it can be determined that the parsed true value data is the historical operator input data used in the historical operator test cases. In this case, it can be directly determined as the operator input data for constructing the operator test cases this time.

[0132] Based on the method provided in this example, different processing methods can be used to obtain operator input data for the parsed historical operator feature combinations or truth data, which can improve the flexibility of obtaining operator input data.

[0133] Optional, Figure 7 A flowchart illustrating the operator test case generation method provided in this application. Figure 6 ,like Figure 7 As shown, before S302, it also includes:

[0134] S701, Obtain the configuration file.

[0135] Based on the scenario example, the configuration file can be manually constructed by relevant technical personnel. The configuration file stores the rules for generating test cases, including the constraints for generating feature combinations.

[0136] S702. Parse the configuration file to determine the target constraints.

[0137] Based on the scenario example, the constraints for generating feature combinations can be the generation conditions for feature combinations. By parsing the configuration file, information from multiple fields included in the configuration file can be obtained. Based on the information of the corresponding fields of the constraints, the target constraints configured in the configuration file can be determined.

[0138] S703. Based on the target constraints, at least one set of operator feature combinations is obtained.

[0139] Based on the scenario examples and the determined target constraints, at least one set of operator feature combinations can be randomly generated.

[0140] S704. Write each set of operator feature combinations into a file with a preset format to obtain the feature combination file.

[0141] Based on the scenario example, the preset format file can be a file with the .josn extension, such as confing.json. Each set of operator feature combinations obtained is stored in confing.json to obtain the feature combination file used to generate operator test cases.

[0142] Based on the method provided in this example, a feature combination file can be constructed, providing the necessary data for subsequent generation of operator test cases.

[0143] Optional, Figure 8 A flowchart illustrating the operator test case generation method provided in this application. Figure 7 ,like Figure 8 As shown, S702 includes:

[0144] S801. If the configuration file corresponds to the first keyword identifier, then the target constraint is determined to be a fully randomized generation mode.

[0145] Based on the scenario example, the keyword identifier field in the configuration file can be read. If the keyword identifier field is the first keyword identifier, i.e., the total-random keyword identifier, then the target constraint condition can be determined to be a fully randomized generation mode. The fully randomized generation mode means that the tensor dimensions are randomly generated within a preset dimension range, the dimension values ​​of each dimension can be randomly determined within the preset dimension value range, and a corresponding number of feature combinations are generated according to a preset quantity. The preset dimension range can be represented by the dim-range field, optionally [3, 8]. The preset dimension value range can be represented by the value-range field, optionally [1, 1024]. The number of generated feature combinations can be represented by the size field.

[0146] S802. If the configuration file corresponds to the second keyword identifier, then the target constraint is determined to be a partially randomized generation mode.

[0147] In a scenario example, if the keyword identifier field is the second keyword identifier, i.e., the `part-random` keyword identifier, then the target constraint can be determined to be a partially randomized generation mode. A partially randomized generation mode can have fixed dimensions, but the dimension values ​​for certain dimensions can be randomly determined within a preset range of dimension values. For example, the `part-size` field can represent the dimension values ​​of each dimension. If it is [1, 0, 3], it indicates that there are three dimensions, where the first dimension has a value of 1 and the third dimension has a value of 3. "0" indicates that the dimension value is random, so the dimension value corresponding to the second dimension can be randomly determined within a preset range of dimension values. The `random-value` field can represent the set of randomization methods, including methods based on value combinations and methods based on step size. When the relevant field is `mode 0`, the dimension values ​​are determined by the parameter field of `params`, and the values ​​in `params` are the dimension values. For example, if the parameter field of `params` is [1, 4], then the dimension value corresponding to the second dimension can be 1 or 4. When the relevant field is mode 1, the dimension value corresponding to the second dimension can be determined according to the random step size. In this case, params includes three parameters: the lower limit of the value, the upper limit of the value, and the step size. For example, if params includes [16,25,8], it means that the lower limit of the value is 16, the upper limit of the value is 25, and the step size is 8, so the dimension value corresponding to the second dimension can be determined to be 16 or 24.

[0148] S803. If the configuration file corresponds to a third keyword identifier, then the target constraint condition is determined to be a fixed generation mode.

[0149] Based on the scenario example, if the keyword identifier field is a third keyword identifier, i.e., the shape keyword identifier, then the target constraint condition can be determined to be a fixed generation mode. The corresponding feature combination can be generated directly according to the shape field parameter filled in the configuration file.

[0150] Based on the method provided in this example, different feature combinations can be generated by referring to different target constraints, which can improve the flexibility and comprehensiveness of generating feature combinations.

[0151] Optional, Figure 9 A flowchart illustrating the operator test case generation method provided in this application. Figure 8 ,like Figure 9 As shown, S203 includes:

[0152] S901. Construct the operator comparison script corresponding to the generated environment.

[0153] Based on the scenario example, the functions of the operator comparison script include, but are not limited to, receiving operator input data for generating operator test cases, data processing, data transfer, and calculating baseline results.

[0154] S902. Using the operator comparison script, the operator input data is processed to obtain the baseline result.

[0155] Based on scenario examples, the constructed operator comparison script is used to complete the operation on the operator input data, obtain the operation result of the operator in the generation environment, and determine the operation result of the operator in the generation environment as the baseline result of the operator.

[0156] Based on the method provided in this example, the operation on the operator input data can be completed based on the constructed operator comparison script, which can improve the efficiency of the operation on the operator input data.

[0157] Optional, Figure 10 A flowchart illustrating the operator test case generation method provided in this application. Figure 9 ,like Figure 10 As shown, S902 includes:

[0158] S1001. Execute the preset operator comparison script to store the operator input data to the microprocessor.

[0159] Based on the scenario example, since the operator comparison script's function includes data transfer, the operator input data can be transferred and stored to the microprocessor first by executing the operator comparison script.

[0160] S1002. Call the preset operator comparison interface to perform calculations on the operator input data stored on the microprocessor side to obtain the baseline result.

[0161] Based on scenario examples, the operator comparison interface can be set by relevant technical personnel. By executing the operator comparison script, the operator input data stored on the microprocessor side is called to complete the calculation and obtain the corresponding baseline result.

[0162] S1003. Store the baseline results to the central processing unit.

[0163] Based on the scenario example, the baseline results are finally transferred to the Central Processing Unit (CPU) for storage by executing the operator comparison script.

[0164] Based on the method provided in this example, the calculation processing of operator input data can be completed by executing the predetermined processing order in the operator comparison script, which can improve the efficiency and accuracy of the operation on operator input data.

[0165] Optionally, the test case association data includes at least one of benchmark performance data, hardware information, accuracy formula, and framework information, wherein the framework information includes framework type and framework version.

[0166] Based on the scenario example, the framework information includes the selected framework type and version, which are relevant to the production environment. The test case generator integrates multiple upper-level frameworks, allowing the selection of different framework types and versions to build the corresponding production environment for any operator. Therefore, when generating operator test cases, the framework information can be recorded along with the operator test cases. Hardware information is also relevant to the production environment; similarly, when generating operator test cases, hardware information can be recorded along with the operator test cases. The accuracy formula and benchmark performance data are both components of the feature combination. The accuracy formula is used for accuracy error calculation, and the benchmark performance data is relevant data on the operator's performance in the production environment. Whether the accuracy formula and benchmark performance data need to be recorded in the operator test cases depends on whether the feature combination includes these two parameters.

[0167] This example provides the content of test case association data. By adding test case association data to operator test cases, the richness of operator test cases can be improved.

[0168] Optional, Figure 11 A flowchart illustrating the operator test case generation method provided in this application. Figure 10 ,like Figure 11 As shown, the process of generating the benchmark performance data includes:

[0169] S1101. Call the acquisition tool corresponding to the generated environment.

[0170] Based on scenario examples, if the feature combination includes benchmark performance data, it can be determined that relevant performance data needs to be collected when the operator is running in the generation environment.

[0171] S1102. Based on the acquisition tool, acquire the computing performance data of the generation environment.

[0172] In conjunction with scenario examples, the data acquisition tool can be the hardware supporting tools in the production environment, and the performance data of the operator in the production environment can be the corresponding computational performance data, that is, the efficiency and accuracy data of calculating the input data of the operator.

[0173] S1103. The computational performance data of the generated environment is determined as the baseline performance data.

[0174] Based on scenario examples, the collected computational performance data of the operators in the generation environment are determined as the baseline performance data of the operators.

[0175] Based on the method provided in this example, computational performance data of the operator in the generation environment can be collected using acquisition tools to improve the accuracy and reliability of the obtained benchmark performance data of the operator.

[0176] Optional, Figure 12 A flowchart illustrating the operator test case generation method provided in this application. Figure 10 First, such as Figure 12 As shown, the configuration process of the production environment includes:

[0177] S1201, Obtain configuration information.

[0178] Based on the scenario examples, the configuration information includes hardware type, framework type, framework version, etc., all of which can be set by relevant technical personnel.

[0179] S1202. Based on the configuration information, construct a generation environment for generating operator test cases.

[0180] Based on scenario examples, after determining the hardware type, framework type, and framework version, the corresponding production environment can be built through the Artificial Intelligence (AI) computing platform.

[0181] The method provided in this example can be used to configure the production environment, thus providing a foundation for generating operator test cases in the future.

[0182] Optionally, after constructing the operator test cases, the following may also be included:

[0183] The operator is tested based on the aforementioned operator test cases.

[0184] With the example scenario, after obtaining the operator test cases, the operation of the operator in the test environment can be tested based on the obtained operator test cases to determine whether the operation of the operator is accurate in the test environment.

[0185] Optional, Figure 13 A flowchart illustrating the operator test case generation method provided in this application. Figure 10 Second, such as Figure 13 As shown, the testing of the operator based on the operator test cases includes:

[0186] S1301. Obtain the test results and test performance data of the operator under a preset test environment.

[0187] Based on the scenario example, determine the test environment to be tested. The test environment includes the corresponding hardware and software environments. Run the operator test cases in the test environment. Perform calculations on the operator input data in the operator test cases based on the test environment to obtain the calculation results in the test environment. These calculation results in the test environment are determined as the test results for that test environment. Using the hardware tools provided in the test environment, capture the runtime performance data of the operator test cases. This can be considered the computational performance data of the operator in the test environment, and it is determined as the test performance data corresponding to the operator test cases.

[0188] S1302. Based on the accuracy formula, the error between the baseline result and the test result is obtained.

[0189] With the example scenario, the accuracy formula can be used to calculate the error, so the obtained test results can be compared with the baseline results to obtain the error between the test results and the baseline results.

[0190] S1303. If the error does not exceed the preset threshold, the operator is determined to have met the accuracy standard in the test environment; otherwise, the operator is determined to have failed to meet the accuracy standard in the test environment.

[0191] Based on the scenario example, the preset threshold can be determined according to the actual situation. If the error between the test result and the baseline result does not exceed the preset threshold, it is determined that the operator's running accuracy in the test environment meets the standard; otherwise, it is determined that the operator's running accuracy in the test environment does not meet the standard.

[0192] S1304. If the test performance data is higher than the benchmark performance data, it is determined that the computational performance of the operator in the test environment is higher than that in the generation environment; otherwise, it is determined that the computational performance of the operator in the test environment is lower than that in the generation environment.

[0193] Based on scenario examples, and by comparing the test performance data with the benchmark performance data, it is determined whether the computational performance of the operator in the test environment is better than that in the production environment. Specifically, if the test performance data is higher than the benchmark performance data, it is determined that the computational performance of the operator in the test environment is better than that in the production environment; conversely, if the test performance data is lower than the benchmark performance data, it is determined that the computational performance of the operator in the test environment is lower than that in the production environment.

[0194] Based on the method provided in this example, the purpose of testing operators can be achieved based on operator test cases.

[0195] Figure 14 A schematic diagram of the overall process for generating test cases for operators, as shown below. Figure 14As shown, the test case generator first constructs a generation environment and operator comparison script for generating operator test cases based on the configuration information. Then, it determines whether the type of the additional file to be parsed is JSON. If the file is JSON, it can be parsed to obtain at least one set of operator feature combinations, denoted as Mi. If the file is not JSON, it can be parsed to obtain at least one set of test case information, denoted as Ni. Test case information can be operator feature combinations or ground truth data. For operator feature combinations, the corresponding operator input data is obtained through a data randomization algorithm; for ground truth data, it can be directly determined as the operator input data. The test case generator can determine whether to record baseline performance data based on the feature combinations, and if necessary, it can capture relevant performance data by calling the hardware tools corresponding to the generation environment. Then, the constructed operator comparison script completes the calculation of the operator input data to obtain the corresponding baseline results. Finally, the operator input data, baseline results, and test case association data are written to a Protocol Buffers file to obtain operator test cases. The test case association data includes at least one of the following: baseline performance data, hardware information, accuracy formulas, and framework information. After obtaining the operator test case data, it can be determined whether the parsing file has been processed completely, i.e., whether all operator feature combinations or test case information in the parsing file have generated relevant operator test cases. If all operator feature combinations or test case information in the parsing file have generated relevant operator test cases, processing can end. If there are still operator feature combinations or test case information in the parsing file that have not generated relevant operator test cases, the execution process of operator test cases can continue until all operator feature combinations or test case information in the parsing file have generated relevant operator test cases.

[0196] Therefore, the operator test case generation method provided in this application embodiment can use a computing device to perform calculations on operator input data according to a configured generation environment to obtain baseline results. The operator input data, baseline results, and test case association data can then be written into a file with a preset format to complete the generation of operator test cases. This eliminates the need for manually writing operator test cases, thus improving the efficiency of generating operator test cases.

[0197] Figure 15 This is a schematic diagram of the operator test case generation device provided in this application, as shown below. Figure 15 As shown, it includes:

[0198] Module 151 is used to obtain the file to be parsed after the configuration of the production environment is completed;

[0199] Processing module 152 is used to parse the file to be parsed to obtain operator input data;

[0200] The processing module 152 is further configured to use the generation environment to perform operations on the operator input data to obtain a baseline result;

[0201] The processing module 152 is also used to write the operator input data, baseline results and test case association data into a file with a preset format to generate an operator test case.

[0202] Optionally, the processing module 152 is specifically used to determine the file type of the file to be parsed;

[0203] The processing module 152 is further configured to, if the file type is a feature combination file, parse the feature combination file to obtain test case feature information, and obtain the operator input data based on the test case feature information;

[0204] The processing module 152 is further configured to, if the file type belongs to a historical test case file, parse the historical test case file to obtain target test case information, and obtain the operator input data based on the target test case information.

[0205] Optionally, the processing module 152 is further configured to parse the feature combination file to obtain at least one set of operator feature combinations;

[0206] The processing module 152 is further configured to select a set of operator feature combinations from the at least one set of operator feature combinations as the test case feature information.

[0207] Optionally, the operator feature combination includes at least one feature;

[0208] Accordingly, the processing module 152 is further configured to use a preset data randomization algorithm to obtain the input data corresponding to any feature in the operator feature combination;

[0209] The processing module 152 is further configured to determine the input data corresponding to all features in the operator feature combination as the operator input data.

[0210] Optionally, the processing module 152 is further configured to parse the historical test case file to obtain at least one set of test case information, and select a set of test case information from the at least one set of test case information as the target test case information.

[0211] Optionally, the processing module 152 is further configured to, if the target test case information is a combination of historical operator features, use a preset data randomization algorithm to obtain the input data corresponding to the historical feature for any historical feature in the combination of historical operator features, and determine the input data corresponding to all historical features in the combination of historical operator features as the operator input data, wherein the combination of historical operator features includes at least one historical feature;

[0212] The processing module 152 is further configured to determine the true value data as the operator input data if the target test case information is true value data.

[0213] Processing module 152 is also used to obtain configuration files;

[0214] Processing module 152 is also used to parse the configuration file to determine the target constraints;

[0215] The processing module 152 is further configured to obtain at least one set of operator feature combinations based on the target constraints;

[0216] The processing module 152 is also used to write each set of operator feature combinations into a file with a preset format to obtain the feature combination file.

[0217] Optionally, the processing module 152 is further configured to determine that the target constraint is a fully randomized generation mode if the configuration file corresponds to the first keyword identifier;

[0218] The processing module 152 is further configured to determine that the target constraint is a partially randomized mode if the configuration file corresponds to the second keyword identifier;

[0219] The processing module 152 is further configured to determine that the target constraint is a fixed generation mode if the configuration file corresponds to a third keyword identifier.

[0220] Optionally, the processing module 152 is further used to construct the operator comparison script corresponding to the generation environment;

[0221] The processing module 152 is further configured to use the operator comparison script to perform calculations on the operator input data to obtain the baseline result.

[0222] Optionally, the processing module 152 is further configured to execute the preset operator comparison script to store the operator input data to the microprocessor.

[0223] The processing module 152 is further used to call a preset operator comparison interface to perform calculations on the operator input data stored on the microprocessor side in order to obtain the baseline result;

[0224] The processing module 152 is further configured to store the baseline results to the central processing unit.

[0225] Optionally, the test case association data includes at least one of benchmark performance data, hardware information, accuracy formula, and framework information, wherein the framework information includes framework type and framework version.

[0226] Optionally, the processing module 152 is further used to call the acquisition tool corresponding to the generation environment;

[0227] The processing module 152 is further configured to collect computing performance data of the generation environment based on the acquisition tool.

[0228] The processing module 152 is further configured to determine the computational performance data of the generated environment as the baseline performance data.

[0229] Optionally, the processing module 152 is also used to obtain configuration information;

[0230] The processing module 152 is further configured to construct a generation environment for generating operator test cases based on the configuration information.

[0231] Optionally, the processing module 152 is also used to test the operator based on the operator test cases.

[0232] Optionally, the processing module 152 is further used to obtain the test results and test performance data of the operator under a preset test environment;

[0233] The processing module 152 is further configured to obtain the error between the baseline result and the test result based on the accuracy formula;

[0234] The processing module 152 is further configured to determine that the operation accuracy of the operator in the test environment meets the standard if the error does not exceed the preset threshold, and otherwise determine that the operation accuracy of the operator in the test environment does not meet the standard.

[0235] The processing module 152 is further configured to determine that if the test performance data is higher than the benchmark performance data, the computational performance of the operator in the test environment is higher than that in the generation environment; otherwise, the computational performance of the operator in the test environment is lower than that in the generation environment.

[0236] The operator test case generation device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0237] Figure 16A schematic diagram of the structure of the electronic device provided in this application. Figure 16 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus.

[0238] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0239] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0240] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0241] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0242] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0243] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0244] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0245] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0246] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0247] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0248] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0249] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0250] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0251] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0252] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for generating operator test cases, characterized in that, include: After completing the configuration of the production environment, obtain the file to be parsed; The file types of the files to be parsed include feature combination files and historical test case files; When the file type is a feature combination file, the feature combination file is parsed to obtain test case feature information, and operator input data is obtained based on the test case feature information; when the file type is a historical test case file, the historical test case file is parsed to obtain target test case information, and operator input data is obtained based on the target test case information. Construct the operator comparison script corresponding to the generated environment; The operator comparison script is used to perform calculations on the operator input data to obtain a baseline result; The operator input data, baseline results, and test case association data are written into a file with a preset format to generate an operator test case.

2. The method according to claim 1, characterized in that, The process of parsing the feature combination file to obtain test case feature information includes: Parse the feature combination file to obtain at least one set of operator feature combinations; One set of operator feature combinations is selected from the at least one set of operator feature combinations as the test case feature information.

3. The method according to claim 2, characterized in that, The operator feature combination includes at least one feature; Accordingly, obtaining the operator input data based on the test case feature information includes: For any feature in the operator feature combination, a preset data randomization algorithm is used to obtain the input data corresponding to the feature; The input data corresponding to all features in the operator feature combination are determined as the operator input data.

4. The method according to claim 1, characterized in that, The process of parsing the historical test case file to obtain target test case information includes: The historical test case file is parsed to obtain at least one set of test case information, and a set of test case information is selected from the at least one set of test case information as the target test case information.

5. The method according to claim 1, characterized in that, The step of obtaining the operator input data based on the target test case information includes: If the target test case information is a combination of historical operator features, then for any historical feature in the combination of historical operator features, a preset data randomization algorithm is used to obtain the input data corresponding to the historical feature, and the input data corresponding to all historical features in the combination of historical operator features are determined as the operator input data, wherein the combination of historical operator features includes at least one historical feature; If the target test case information is true data, then the true data is determined as the operator input data.

6. The method according to claim 1, characterized in that, Before parsing the feature combination file, the method further includes: Get the configuration file; The configuration file is parsed to determine the target constraints; Based on the target constraints, at least one set of operator feature combinations is obtained; Each set of operator feature combinations is written into a file with a preset format to obtain the feature combination file.

7. The method according to claim 6, characterized in that, The process of parsing the configuration file to determine the target constraints includes: If the configuration file corresponds to the first keyword identifier, then the target constraint is determined to be a fully randomized generation mode; If the configuration file corresponds to the second keyword identifier, then the target constraint is determined to be a partially randomized generation mode; If the configuration file corresponds to a third keyword identifier, then the target constraint is determined to be a fixed generation mode.

8. The method according to claim 1, characterized in that, The step of using an operator comparison script to perform operations on the operator input data to obtain the baseline result includes: Execute a preset operator comparison script to store the operator input data to the microprocessor. The preset operator comparison interface is invoked to perform calculations on the operator input data stored on the microprocessor side to obtain the baseline result; The baseline results are stored on the central processing unit.

9. The method according to claim 1, characterized in that, The test case associated data includes at least one of the following: benchmark performance data, hardware information, accuracy formula, and framework information, wherein the framework information includes framework type and framework version.

10. The method according to claim 9, characterized in that, The process of generating the benchmark performance data includes: Call the acquisition tool corresponding to the generated environment; Based on the acquisition tool, the computing performance data of the generation environment is collected; The computational performance data of the generated environment is determined as the baseline performance data.

11. The method according to claim 1, characterized in that, The configuration process of the generated environment includes: Get configuration information; Based on the configuration information, a generation environment is constructed for generating operator test cases.

12. The method according to any one of claims 1-11, characterized in that, After constructing the operator test cases, the following is also included: The operator is tested based on the aforementioned operator test cases.

13. The method according to claim 12, characterized in that, The testing of the operator based on the operator test cases includes: Obtain the test results and test performance data of the operator under a preset test environment; Based on the accuracy formula, the error between the baseline result and the test result is obtained; If the error does not exceed the preset threshold, the operator is determined to have met the accuracy standard in the test environment; otherwise, the operator is determined to have failed to meet the accuracy standard in the test environment. If the test performance data is higher than the baseline performance data, it is determined that the computational performance of the operator in the test environment is higher than that in the generation environment; otherwise, it is determined that the computational performance of the operator in the test environment is lower than that in the generation environment.

14. An operator test case generation device, characterized in that, include: The acquisition module is used to obtain the file to be parsed after the production environment is configured. The file types of the files to be parsed include feature combination files and historical test case files; The processing module is configured to, when the file type is a feature combination file, parse the feature combination file to obtain test case feature information, and obtain operator input data based on the test case feature information; when the file type is a historical test case file, parse the historical test case file to obtain target test case information, and obtain operator input data based on the target test case information. The processing module is also used to construct an operator comparison script corresponding to the generation environment; and to use the operator comparison script to perform operations on the operator input data to obtain a baseline result. The processing module is also used to write the operator input data, baseline results and test case association data into a file with a preset format to generate an operator test case.

15. A computer device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-13.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method as claimed in any one of claims 1-13.

Images