Operator precision consistency test method, device and medium

Abstract

The invention provides an operator precision consistency test method and device and a medium, and the method comprises the steps: analyzing a test configuration file, and obtaining the input configuration and output configuration of a to-be-measured operator in the test configuration file, and the test input generated based on the input configuration; based on the test input, repeatedly running the to-be-measured sub-unit for multiple times to obtain a plurality of test outputs of the to-be-measured sub-unit under the test input, the test outputs being matched with the output configuration; and performing consistency judgment on the plurality of test outputs to obtain a precision consistency test result of the to-be-measured sub-samples. According to the method, the equipment and the medium provided by the invention, the rapid configuration of the operator precision consistency test is realized through the test configuration file, so that the operator precision consistency test is realized. Compared with writing of a complete unit test script, the time and labor cost required for obtaining the test configuration file are lower, the method is more efficient and convenient, and the efficiency of operator precision consistency testing can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an operator precision consistency testing method, device and medium. Background Art

[0002] In the operator development scenario, some operators may sacrifice the consistency of calculation results in exchange for improved computing performance, which will lead to inconsistent results of multiple calculations of the operator and affect the accuracy alignment of the operator with the benchmark data (Golden). Therefore, it is necessary to perform unit testing (UT) on the accuracy consistency of the operator.

[0003] To implement unit testing, testers are usually required to write unit test scripts for each operator, and then run the unit test scripts to perform accuracy consistency tests on the operators. Writing unit test scripts for each operator increases the workload of testers and reduces testing efficiency. As the number of operators increases, the number of unit test scripts will also increase, which brings challenges to test management and maintenance. Summary of the invention

[0004] The present invention provides an operator precision consistency test method, device and medium, which are used to solve the defect in the related art that a unit test script needs to be written separately to perform operator precision consistency test, which is time-consuming and labor-intensive.

[0005] The present invention provides an operator accuracy consistency testing method, comprising: Parse the test configuration file to obtain the input configuration and output configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration; Based on the test input, repeatedly running the operator to be tested for multiple times to obtain multiple test outputs of the operator to be tested under the test input, wherein the test outputs match the output configuration; A consistency judgment is performed on the multiple test outputs to obtain an accuracy consistency test result of the operator to be tested.

[0006] According to an operator accuracy consistency test method provided by the present invention, the parsing of the test configuration file to obtain the input configuration and output configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration, includes: Parsing the test configuration file to obtain the input configuration, output configuration and test configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration and the test configuration; The test configuration includes the number of test groups, and the number of groups of the test input is consistent with the number of test groups.

[0007] According to an operator accuracy consistency test method provided by the present invention, the test configuration further includes a number of repetitions; The step of repeatedly running the operator to be tested based on the test input comprises: Based on the test input, the operator to be tested is repeatedly executed until the number of times the operator to be tested is executed reaches the number of repetitions.

[0008] According to an operator precision consistency test method provided by the present invention, the parsing of the test configuration file to obtain the input configuration, output configuration and test configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration and the test configuration, includes: Reading the test configuration from the test configuration file; Parsing the input list in the test configuration file, obtaining the input configuration of each test input in the input list, generating a test input based on the input configuration, and copying the test input to a device running the operator to be tested; Parse the output list in the test configuration file to obtain the output configuration of each test output in the output list, and control the device to initialize an empty tensor matching the output configuration for storing the test output.

[0009] According to the present invention, a method for testing operator accuracy consistency also includes: Parsing the test configuration file to obtain device information for running the operator to be tested; Based on the device information, a device running the operator to be tested is determined.

[0010] According to an operator accuracy consistency test method provided by the present invention, the consistency judgment of the multiple test outputs includes: The multiple test outputs are divided into multiple threads, and the multiple threads are controlled to perform consistency judgment on the multiple test outputs at each thread in parallel.

[0011] According to an operator precision consistency testing method provided by the present invention, the input configuration includes at least one of the type, layout and shape of the test input, and the output configuration includes at least one of the type, layout and shape of the test output.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the operator accuracy consistency testing method described in any one of the above methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described operator accuracy consistency testing methods.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned operator accuracy consistency testing methods.

[0015] The operator precision consistency test method, device and medium provided by the present invention realize the rapid configuration of the operator precision consistency test through the test configuration file, and the input configuration, output configuration and test input required for the operator precision consistency test can be conveniently obtained by parsing the test configuration file to realize the operator precision consistency test. Compared with writing a complete unit test script, the time and labor cost required to obtain the test configuration file are lower, more efficient and convenient, and can greatly improve the efficiency of the operator precision consistency test. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of the operator accuracy consistency testing method provided by the present invention.

[0018] Figure 2 This is the second flow chart of the operator accuracy consistency testing method provided by the present invention.

[0019] Figure 3 This is the third flow chart of the operator accuracy consistency testing method provided by the present invention.

[0020] Figure 4 This is the fourth flow chart of the operator accuracy consistency testing method provided by the present invention.

[0021] Figure 5 It is a schematic diagram of the multi-threaded numerical comparison process provided by the present invention.

[0022] Figure 6 It is a structural schematic diagram of the operator accuracy consistency testing device provided by the present invention.

[0023] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In the process of model adaptation, the accuracy of operator development is crucial. In order to ensure the accuracy of operator development, it is usually necessary to align the accuracy of all operators appearing in the model with the benchmark data. The accuracy alignment work generally uses the real data obtained by training the model on the benchmark hardware as the benchmark data, and then uses the input in the benchmark data as the input data. The output of the operator is obtained by calling the operator calculation in the model, and then the absolute error or relative error of the output is obtained by comparing the operator output with the output in the benchmark data. For operators with large errors, the operator can be repaired by fine-tuning parameters, so that the accuracy of the repaired operator can be consistent with the accuracy running on the benchmark hardware.

[0026] Precision alignment ensures that newly developed operators function as expected and are numerically consistent with standard results in benchmark data.

[0027] However, in actual development, in order to improve computing performance, newly developed operators sometimes make some approximate processing or optimization during the calculation process. Although these processing or optimization can significantly improve the calculation speed, they may sacrifice the consistency of the calculation results. If the results of multiple calculations of the operator are inconsistent, it will not only be detrimental to the accuracy alignment with the benchmark data, but may also have a negative impact on the accuracy convergence of the model.

[0028] In order to ensure the accuracy consistency of operators, newly developed operators generally need to undergo unit testing for accuracy consistency. Here, accuracy consistency means that when an operator performs an operation, its output results can maintain a certain accuracy and stability. The accuracy consistency of an operator is an important basis for reflecting the reliability of the operator's operation results.

[0029] To implement unit testing, testers are usually required to write a unit test script for each operator. That is, for each new operator to be tested, an additional unit test script needs to be written for the operator, and the unit test script needs to include the following five steps: (1) Create environment variables to provide the necessary configuration and parameters for the test; (2) Generate test data as the input of the operator; (3) Call the operator to generate the benchmark output Golden Output data, that is, the expected output result; (4) Create a For Loop to repeatedly test the operator and generate multiple test output Test Output data; (5) Perform a numerical comparison between the test output Test Output data and the benchmark output Golden Output data, and finally obtain the accuracy consistency result of the operator, and calculate the random error of the operator in the presence of randomness.

[0030] Writing a unit test script for each operator not only increases the workload of testers and reduces test efficiency, but also easily introduces errors. As the number of operators increases, the number of unit test scripts will also increase, which brings challenges to test management and maintenance.

[0031] In view of the above problems, an embodiment of the present invention provides an operator accuracy consistency testing method. Figure 1 This is one of the flow charts of the operator accuracy consistency test method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, parsing the test configuration file to obtain the input configuration and output configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration.

[0032] Here, the operator to be tested is the operator that needs to be tested for accuracy consistency, and the test configuration file is the file set to implement the operator accuracy consistency test. Each operator to be tested can correspond to a test configuration file, which is used to describe the configuration information of the operator to be tested when performing the operator accuracy consistency test, and specifically may include the input configuration, output configuration and other configuration information of the operator to be tested. Among them, the input configuration may be the relevant configuration information for the test input of the operator to be tested, such as the data type, data shape, layout, etc. of the test input, and the output configuration may be the relevant configuration information for the test output of the operator to be tested, such as the data type, data shape, layout, etc. of the test output.

[0033] The test configuration file may be a YAML (YAML Ain't a Markup Language) configuration file, or a JSON (JavaScript Object Notation) or other format configuration file, which is not specifically limited in the embodiment of the present invention.

[0034] The test configuration file is usually created by the tester. For example, the template of the test configuration file can be copied, and the configuration information such as the input configuration and output configuration of the operator to be tested can be filled into the template. Alternatively, the configuration information such as the input configuration and output configuration of the operator to be tested can be specified first, and then the corresponding test configuration file is generated according to the specified configuration information by a pre-set program. The embodiment of the present invention does not specifically limit this. For example, the yaml file template can be filled according to the operator function signature to obtain the test configuration file.

[0035] It is understandable that compared to writing a complete unit test script, obtaining a test configuration file requires less time and labor costs, and is more efficient and convenient. Moreover, compared to the large amount of repeated code in different unit test scripts, the configuration information in the test configuration file is specific to the operator to be tested, which greatly reduces unnecessary repetitive work and helps improve the efficiency of operator accuracy consistency testing.

[0036] For the test configuration file of the operator to be tested, the test configuration file can be parsed to obtain the configuration information contained in the test configuration file. In the process of parsing the test configuration file, the test input generated based on the configuration information for performing the operator accuracy consistency test can also be obtained.

[0037] Furthermore, in the process of parsing the test configuration file, the input configuration, output configuration and other information contained in the test configuration file can be parsed and obtained, and in the process of parsing, the test input for the operator precision consistency test can also be generated based on the input configuration contained in the test configuration file. The test input here can specifically be input data that meets the requirements of the input configuration, and the test input can be randomly generated under the data type, data shape, and numerical range defined by the known input configuration.

[0038] Step 120: Based on the test input, repeatedly run the operator to be tested for multiple times to obtain multiple test outputs of the operator to be tested under the test input, and the test outputs match the output configuration.

[0039] Specifically, after parsing the test configuration file, the accuracy consistency test can be performed on the operator to be tested. The test process here can be to apply the same test input and run the operator to be tested repeatedly for multiple times, thereby obtaining the test output obtained by each operation of the operator to be tested under the same test input. The test output obtained here is the output data that meets the requirements of the output configuration.

[0040] It can be understood that the test input applied in this process is obtained by parsing the test configuration file, and the obtained test output is obtained by running the operator to be tested. Moreover, for a set of test inputs, since the operator to be tested is run repeatedly for many times, multiple groups of test outputs can be obtained, that is, a set of test inputs corresponds to multiple groups of test outputs, and the process of performing precision consistency testing on the operator to be tested can be understood as the process of determining whether the multiple groups of test outputs corresponding to a set of test inputs are consistent. The multiple groups of test outputs here can be divided into output data as the benchmark output Golden Output and output data as the test output Test Output. That is, the benchmark output Golden Output can be one of the multiple groups of test outputs, for example, it can be the first group of the multiple groups of test outputs, or it can be any group of the multiple groups of test outputs.

[0041] Here, the device for implementing repeated multiple operations on the operator to be tested may be an artificial intelligence chip, such as a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), an NPU (Neural network Processing Unit), a DPU (Deep learning Processing Unit), an APU (Accelerated Processing Unit), and a GPGPU (General-Purpose computing on Graphics Processing Unit), which is not specifically limited in the embodiments of the present invention.

[0042] Step 130 , performing consistency judgment on the multiple test outputs to obtain the accuracy consistency test result of the operator to be tested.

[0043] Specifically, after obtaining multiple test outputs for the same test input, consistency judgment can be performed on the multiple test outputs. Here, the consistency judgment can be performed by selecting a test output from multiple test outputs as a reference output, and comparing the reference output with other test outputs one by one for consistency. Here, the consistency comparison between the reference output and any test output can be performed by comparing whether the reference output and the test output are consistent. The consistency standard here is whether the reference output is equal to the test output, or whether the difference between the reference output and the test output is within a preset difference range, that is, whether the difference between the reference output and the test output is within a tolerable range.

[0044] It can be understood that the result of consistency judgment on multiple test outputs is the precision consistency test result of the operator to be tested. For example, if at least one of the results of consistency judgment on multiple test outputs is inconsistent, the precision consistency test result of the operator to be tested is inconsistent; if the results of consistency judgment on multiple test outputs are all consistent, the precision consistency test result of the operator to be tested is consistent.

[0045] In the method provided in the embodiment of the present invention, the rapid configuration of the operator precision consistency test is realized through the test configuration file. By parsing the test configuration file, the input configuration, output configuration and test input required for the operator precision consistency test can be easily obtained to realize the operator precision consistency test. Compared with writing a complete unit test script, the time and labor cost required to obtain the test configuration file are lower, more efficient and convenient, and can greatly improve the efficiency of the operator precision consistency test.

[0046] Based on the above embodiment, in step 110, the parsing of the test configuration file to obtain the input configuration and output configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration, includes: Parsing the test configuration file to obtain the input configuration, output configuration and test configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration and the test configuration; The test configuration includes the number of test groups, and the number of groups of the test input is consistent with the number of test groups.

[0047] Specifically, the test configuration file may include not only the input configuration and output configuration of the operator to be tested, but also the test configuration of the operator to be tested. The test configuration here refers to the configuration of the accuracy consistency test of the operator to be tested, for example, it may include at least one of the number of test groups, the number of repetitions, and the test environment.

[0048] In an embodiment of the present invention, the test configuration may include the number of test groups, where the number of test groups refers to the number of test inputs to be generated, that is, when performing accuracy consistency test on the operator to be tested, several groups of test inputs are required to be tested separately.

[0049] Accordingly, in the process of parsing the test configuration file, the test configuration contained in the test configuration file can be directly parsed, and in the process of generating the test input for the accuracy consistency test, the input configuration obtained by parsing and the test configuration can also be combined to achieve it, that is, the test input thus generated can match the input configuration in terms of data type, data shape, layout, etc., and can match the number of test groups in the test configuration in terms of data volume.

[0050] In the method provided in an embodiment of the present invention, the test configuration is set in the test configuration file, so that in the process of parsing the test configuration file, the test input can be generated in combination with the input configuration and the test configuration, thereby obtaining a test input whose number of groups is consistent with the number of test groups in the test configuration, that is, only one test configuration file is needed to realize the generation of multiple groups of test inputs at one time, as well as the accuracy consistency test under multiple groups of test inputs executed for the operator to be tested, thereby greatly improving the efficiency of the accuracy consistency test.

[0051] Based on any of the above embodiments, the test configuration further includes a number of repetitions; In step 120, the repeatedly running the operator to be tested multiple times based on the test input includes: Based on the test input, the operator to be tested is repeatedly executed until the number of times the operator to be tested is executed reaches the number of repetitions.

[0052] Specifically, in the test configuration file, the test configuration may also include the number of repetitions. The number of repetitions refers to the number of times the operator to be tested is repeatedly run during the accuracy consistency test for a set of test inputs.

[0053] In the process of parsing the test configuration file, the number of repetitions in the test configuration can also be parsed.

[0054] For the number of repetitions obtained by analysis, in the process of precision consistency test for the operator to be tested, for each set of test inputs, the set of test inputs can be used as the input data of the operator to be tested, and the operator to be tested is repeatedly run, and the number of repeated runs is the number of repetitions in the test configuration. The number of groups of test outputs of the operator to be tested under a set of test inputs is also consistent with the number of repetitions.

[0055] In the method provided in the embodiment of the present invention, the number of repetitions is set in the test configuration file, so that the number of repetitions of the operator to be tested is controllable when performing the accuracy consistency test, thereby ensuring the flexibility of the accuracy consistency test.

[0056] Based on any of the above embodiments, in step 110, the parsing of the test configuration file to obtain the input configuration, output configuration and test configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration and the test configuration, includes: Reading the test configuration from the test configuration file; Parsing the input list in the test configuration file, obtaining the input configuration of each test input in the input list, generating a test input based on the input configuration, and copying the test input to a device running the operator to be tested; Parse the output list in the test configuration file to obtain the output configuration of each test output in the output list, and control the device to initialize an empty tensor matching the output configuration for storing the test output.

[0057] Specifically, the operation of parsing the test configuration file is usually implemented by the host, while the operation of the operator to be tested is implemented by a device such as an artificial intelligence chip. The host is a central processing unit or computer connected to the device, such as a CPU (Central Processing Unit), a microprocessor, etc.

[0058] During the parsing of the test configuration file, the host may read the test configuration in the test configuration file, which may specifically include the number of test groups, the number of repetitions, etc.

[0059] In addition, the host can also read the input list in the test configuration file. Here, the input list is a list of input configurations for each test input of the operator to be tested. For example, when the operator to be tested has three test inputs, namely input0, input1 and input2, the input list may include input configurations for input0, input1 and input2 respectively. After reading the input configuration, the host can also generate corresponding test inputs based on the input configuration, and considering that the test input is directly applied to the operation of the operator to be tested at the device, the host can copy the generated test input to the device running the operator to be tested, so that the device can directly read the local test input when repeatedly running the operator to be tested.

[0060] In addition, the host can also read the output list in the test configuration file. Here, the output list contains a list of output configurations for each test output of the operator to be tested. For example, when the operator to be tested has three test outputs, output0, output1, and output2, the output list can include output configurations for output0, output1, and output2 respectively. After reading the output configuration, the host can also control the device running the operator to be tested to initialize an empty tensor that matches the output configuration, so that the test output obtained after the device repeatedly runs the operator to be tested can be stored.

[0061] For example, Figure 2 This is the second flow chart of the operator accuracy consistency test method provided by the present invention, such as Figure 2 As shown, the operator accuracy consistency test can be implemented by the cooperation of CPU and GPU, wherein CPU is the host referred to in the above embodiment, and GPU is the device referred to in the above embodiment.

[0062] The operator accuracy consistency test method may include the following steps: First, read the test configuration file, which can be a yaml file, parse the various fields therein, and obtain the number of test groups random_case_num for the test configuration. random_case_num indicates that M groups of different test inputs will be generated.

[0063] In addition, you can read the input list (inputs list) in the test configuration file. Here, the input list is the list of input parameters of the operator function to be tested. The length of the input list is dynamic, and the number of input parameters of different operator functions is not fixed. Figure 2 Take inputs={input0, input1, input2} as an example.

[0064] For each input parameter input, the input configuration (tensorimfo) of the parameter can be read from the inputs list, which can include parameter name name, tensor type tensor_type (specifically op_input), data type data_type (for example, bfloat16, float32, int32, int64), layout layout (indicates the arrangement order of parameter tensor in gpu memory, for example, rowmajor or colmajor), memory_arch (indicates the way to access gpu memory, for example, uma, numa), data shape shape (indicates the shape of parameter tensor), data range data_range (indicates the data range of parameter tensor). Among them, data shape shape and data range data_range must be specified, and other field information can use the default values.

[0065] After reading the input configuration of each input parameter, the test input can be generated accordingly. Specifically, the cpu numpy data can be generated first, that is, the data in the CPU's NumPy array (ndarray object).

[0066] Here, the generation method of cpu numpy data includes: when the data type data_type=bfloat16 or float32, np.random.uniform(min_value, max_value) can be used to generate cpu numpy data, where the minimum value min_value and the maximum value max_value are determined by the data range data_range; when the data type data_type=int32 or int64, np.random.randint(min_value, max_value).astype(data_type) is used to generate cpu numpy data.

[0067] After getting the cpu numpy data, you can initialize the test input cpu torch data at the CPU. Here, cpu torch data is the test input for each input parameter.

[0068] Specifically, cpu_torch_data = torch.tensor(cpu_numpy_data, dtype=data_type).

[0069] Then, the test input cpu torch data can be copied to the memory of the gpu, that is, the test input gpu torch tensor at the GPU is obtained. The copying method here can be H2D (Host to Device) copy, which is specifically expressed as: gpu_input = torch_gpu.empty (shape, layout, dtype=data_type) gpu_input.copy_(cpu_torch_data) In addition, you can also read the output list (outputs list) in the test configuration file. Here, the output list represents the list of output parameters obtained after the operator function is calculated. The length of the outputs list is also dynamic. The number of output parameters of different operator functions is not fixed. Figure 2 Take outputs={output0, output1,output2} as an example.

[0070] For each output parameter output, you can read the output configuration tensor info of the parameter from the outputs list. In the output configuration, shape shape must be specified, and data range data_range will not appear in the output configuration.

[0071] After reading the output configuration of each output parameter, you can control the GPU to initialize the tensor for each output parameter. GPU initialization: Unlike the initialization of input parameters, for the initialization of output parameters, you only need to set it to empty tensor, that is, get a blank GPU test output gpu torch tensor, which can be expressed as: gpu_input = torch_gpu.empty (shape, layout, dtype=data_type) In addition, you can also read the comparison parameter list tensors_to_comparison, which is a subset of the output list outputs, indicating several output parameters output tensors that need to be compared in the operator test, such as {output1, output3}.

[0072] Furthermore, you can also read the number of repetitions test_times_for_each_case in the test configuration. This value indicates the number of times the operator needs to be repeated for each set of test inputs.

[0073] It is understandable that for each set of test inputs, the generation of test inputs, initialization of test outputs, and reading of the number of repetitions in the above steps need to be repeated in order to implement operator accuracy consistency testing for each set of test inputs.

[0074] For each set of test inputs, the operator is repeatedly run multiple times on the GPU, and the result of one of the runs is saved as the benchmark output golden output, and then the calculation is repeated N times to obtain N test outputs test output. Subsequently, the benchmark output golden output is compared with the N test outputs test output to obtain the accuracy consistency test result.

[0075] Based on any of the above embodiments, the method further includes: Parsing the test configuration file to obtain device information for running the operator to be tested; Based on the device information, a device running the operator to be tested is determined.

[0076] Specifically, before training a large model on a Qianka cluster, it is necessary to perform operator accuracy consistency tests on all devices in the cluster to ensure that the operating results of all devices for each operator are consistent, thus laying the foundation for long-term stable training of large models on a Qianka scale. In order to implement batch automated testing for a large number of devices, the device information for running the operator to be tested can be set in the test configuration file.

[0077] Here, the device information used to run the operator to be tested can be used to identify or locate the device running the operator to be tested. For example, the device information can be the IP address of each device node in the cluster. Therefore, by parsing the test configuration file, the device information used to run the operator to be tested can also be obtained, and the device running the operator to be tested can be directly determined through the device information. When performing the operator accuracy consistency test, the test command can be directly sent to each device in batches, thereby triggering the synchronous execution of the operator accuracy consistency test at each device.

[0078] For example, Figure 3 This is the third flow chart of the operator accuracy consistency test method provided by the present invention, such as Figure 3 As shown in the figure, the host can send test commands to each device running the operator to be tested through the SSH (Secure Shell) connection, for example Figure 3 Nodes 0 to 7 in the test can trigger consistency tests of nodes 0 to 7 for operators 1 to 3 to be tested.

[0079] Accordingly, after each device completes the consistency test, the test results can also be returned through the SSH connection. If there are inconsistencies in the test results, the host can generate an unconsistency-host_{node_name} file in the output directory to record the device node names with inconsistent test results in the consistency test.

[0080] In addition, assuming that there are M groups of test inputs, the operator is repeatedly run N+1 times for each group of test inputs, and each group of test inputs obtains a set of benchmark outputs and N groups of test outputs. When performing numerical comparison on the test outputs, a total of Number of numerical comparisons. In related technologies, numerical comparisons are performed in a serial manner. Large-scale numerical comparisons will consume a lot of time and reduce the efficiency of precision consistency testing.

[0081] Based on any of the above embodiments and in view of the above problem, in an embodiment of the present invention, in step 130, the consistency judgment of the multiple test outputs includes: The multiple test outputs are divided into multiple threads, and the multiple threads are controlled to perform consistency judgment on the multiple test outputs at each thread in parallel.

[0082] Specifically, in an embodiment of the present invention, multiple threads can be set to perform consistency judgment in parallel. The multiple threads here are multiple threads that can be executed concurrently and can realize consistency judgment for test output. The number of threads can be based on the number of computing cores of the host. For example, the number of available threads can be determined based on the number of CPU cores, thereby maximizing the utilization of the CPU to speed up the numerical comparison process.

[0083] After the operator is run, multiple test outputs can be obtained. For multiple test outputs, they can be grouped according to the consistency judgment mechanism, thereby obtaining multiple output value groups to be compared. That is, for multiple test outputs obtained under the same test output, one of them can be used as a reference output, and the reference output is grouped with other test outputs in pairs, thereby obtaining multiple output value groups.

[0084] Subsequently, threads can be allocated for multiple output value groups to be compared, that is, multiple output value groups with comparison are allocated to different threads, so that multiple threads can share the value comparison task, and multiple threads compare the values ​​of the allocated output value groups in parallel, so as to achieve the accuracy consistency judgment for the operator to be tested. The allocation method here can be an average allocation, or it can be allocated according to the processing speed of each thread, which is not specifically limited in the embodiment of the present invention.

[0085] It can be understood that, in the process of performing numerical comparison at each thread, each thread can calculate the maximum absolute error (Max Abs Error) and the maximum relative error (Max Rel Error) obtained by the numerical comparison.

[0086] Furthermore, in the process of each thread performing numerical comparison in parallel, if the numerical comparison result error of one of the threads is not 0, that is, there are two inconsistent outputs, the numerical comparison of all threads can be terminated immediately, and the accuracy consistency test result can be directly determined as inconsistent.

[0087] In addition, after the numerical comparison of each thread is completed, the numerical comparison results of all threads can be integrated to determine the accuracy consistency test result, that is, if the numerical comparison results at each thread are consistent, then the accuracy consistency test result can be determined to be consistent, otherwise it is inconsistent.

[0088] For example, Figure 4 This is the fourth flow chart of the operator accuracy consistency test method provided by the present invention. Figure 4As shown, after obtaining the test configuration file OP YAML File, the test configuration file can be parsed to obtain M groups of test inputs, namely test input 1 to test input M.

[0089] For each set of test inputs, you can run the operator once and use its output as the benchmark output GoldenOutput, and then repeat the operation N times to obtain N test outputs Test Outputs, which are combined into N output value groups Tensor Pairs in the form of (benchmark output, test output).

[0090] After all the tests of group M Test Data Group are completed, you can The (benchmark output, test output) output value group Tensor Pair is added to the thread queue Thread Queue.

[0091] Figure 5 Schematic diagram of the multi-threaded numerical comparison process provided by the present invention. Figure 5 As shown, for the thread queue Thread Queue The (benchmark output, test output) output value group Tensor Pair can be assigned to S comparison threads, namely comparison thread 1 to comparison thread S. Each thread here can calculate the maximum absolute error (Max Abs Error) and maximum relative error (Max Rel Error) obtained by the value comparison. Finally, the final accuracy consistency test result can be obtained by summarizing the value comparison results of each thread.

[0092] In the embodiment of the present invention, numerical comparison is performed in parallel through multi-threaded grouping, which greatly reduces the time required for numerical comparison during the operator precision consistency test and can significantly improve the efficiency of the operator precision consistency test.

[0093] Based on any of the above embodiments, the input configuration includes at least one of the type, layout and shape of the test input, and the output configuration includes at least one of the type, layout and shape of the test output.

[0094] Here, for input configuration, at least one of the type, layout and shape of the test input for the operator to be tested may be included, wherein the type of the test input is the data type of the input data, such as bfloat16, float32, int32, int64, etc. The layout of the test input is the arrangement order of the input data in the device memory, such as rowmajor or colmajor. The shape of the test input is the data shape of the input data, such as the size of the input data in each dimension, such as [1,4096,8192].

[0095] In addition, for the input configuration, it can also include the name of the test input, specifically the name of each input parameter in the test input; it can also include the memory architecture of the test input, specifically the way the test input accesses the memory, for example, it can be uma or numa; it can also include the data range of the test input, for example, it can be [-5,5]. The embodiment of the present invention does not specifically limit this.

[0096] For output configuration, at least one of the type, layout and shape of the test output for the operator to be tested may be included, wherein the type of the test output is the data type of the output data, such as bfloat16, float32, int32, int64, etc. The layout of the test output is the arrangement order of the input data in the device memory, such as rowmajor or colmajor. The shape of the test output is the data shape of the input and output, such as the size of the output data in each dimension, such as [1,4096,8192].

[0097] In addition, for the output configuration, it can also include the name of the test output, specifically the name of each output parameter in the test output; it can also include the memory architecture of the test output, specifically the way the test output accesses the memory, for example, it can be uma or numa, and the embodiment of the present invention does not specifically limit this.

[0098] In addition, the test configuration file can also include the name of the operator to be tested and the list of output parameters that need to be compared; it can also include test configurations such as the number of test groups and the number of repetitions; in addition, it can include relative error tolerance (rtol) and absolute error tolerance (atol) as the basis for numerical comparison to determine whether the data is consistent.

[0099] The operator accuracy consistency testing device provided by the present invention is described below. The operator accuracy consistency testing device described below and the operator accuracy consistency testing method described above can be referred to each other.

[0100] Figure 6 Schematic diagram of the structure of the operator accuracy consistency test device provided by the present invention. Figure 6 As shown, the device comprises: A parsing unit 610 is configured to parse a test configuration file to obtain an input configuration and an output configuration of an operator to be tested in the test configuration file, and a test input generated based on the input configuration. An operating unit 620 is used to repeatedly operate the operator to be tested multiple times based on the test input to obtain multiple test outputs of the operator to be tested under the test input, wherein the test outputs match the output configuration; The judgment unit 630 is used to perform consistency judgment on the multiple test outputs to obtain the accuracy consistency test result of the operator to be tested.

[0101] In the device provided by the embodiment of the present invention, the rapid configuration of the operator precision consistency test is realized through the test configuration file. By parsing the test configuration file, the input configuration, output configuration and test input required for the operator precision consistency test can be easily obtained to realize the operator precision consistency test. Compared with writing a complete unit test script, the time and labor cost required to obtain the test configuration file are lower, more efficient and convenient, and can greatly improve the efficiency of the operator precision consistency test.

[0102] Based on any of the above embodiments, the parsing unit is specifically used for: Parsing the test configuration file to obtain the input configuration, output configuration and test configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration and the test configuration; The test configuration includes the number of test groups, and the number of groups of the test input is consistent with the number of test groups.

[0103] Based on any of the above embodiments, the test configuration further includes a number of repetitions; The operation unit is specifically used for: Based on the test input, the operator to be tested is repeatedly executed until the number of times the operator to be tested is executed reaches the number of repetitions.

[0104] Based on any of the above embodiments, the parsing unit is specifically used for: Reading the test configuration from the test configuration file; Parsing the input list in the test configuration file, obtaining the input configuration of each test input in the input list, generating a test input based on the input configuration, and copying the test input to a device running the operator to be tested; Parse the output list in the test configuration file to obtain the output configuration of each test output in the output list, and control the device to initialize an empty tensor matching the output configuration for storing the test output.

[0105] Based on any of the above embodiments, the parsing unit is further used for: Parsing the test configuration file to obtain device information for running the operator to be tested; Based on the device information, a device running the operator to be tested is determined.

[0106] Based on any of the above embodiments, the judging unit is specifically configured to: The multiple test outputs are divided into multiple threads, and the multiple threads are controlled to perform consistency judgment on the multiple test outputs at each thread in parallel.

[0107] Based on any of the above embodiments, the input configuration includes at least one of the type, layout and shape of the test input, and the output configuration includes at least one of the type, layout and shape of the test output.

[0108] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the operator precision consistency test method, which includes: Parse the test configuration file to obtain the input configuration and output configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration; Based on the test input, repeatedly running the operator to be tested for multiple times to obtain multiple test outputs of the operator to be tested under the test input, wherein the test outputs match the output configuration; A consistency judgment is performed on the multiple test outputs to obtain an accuracy consistency test result of the operator to be tested.

[0109] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0110] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the operator accuracy consistency test method provided by the above methods, the method comprising: Parse the test configuration file to obtain the input configuration and output configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration; Based on the test input, repeatedly running the operator to be tested for multiple times to obtain multiple test outputs of the operator to be tested under the test input, wherein the test outputs match the output configuration; A consistency judgment is performed on the multiple test outputs to obtain an accuracy consistency test result of the operator to be tested.

[0111] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the operator accuracy consistency test method provided by the above methods, the method comprising: Parse the test configuration file to obtain the input configuration and output configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration; Based on the test input, repeatedly running the operator to be tested for multiple times to obtain multiple test outputs of the operator to be tested under the test input, wherein the test outputs match the output configuration; A consistency judgment is performed on the multiple test outputs to obtain an accuracy consistency test result of the operator to be tested.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for testing operator accuracy consistency, characterized in that: include: Parse the test configuration file to obtain the input configuration and output configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration; Based on the test input, repeatedly running the operator to be tested for multiple times to obtain multiple test outputs of the operator to be tested under the test input, wherein the test outputs match the output configuration; A consistency judgment is performed on the multiple test outputs to obtain an accuracy consistency test result of the operator to be tested.

2. The operator accuracy consistency test method according to claim 1, characterized in that: The parsing of the test configuration file to obtain the input configuration and output configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration, includes: Parsing the test configuration file to obtain the input configuration, output configuration and test configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration and the test configuration; The test configuration includes the number of test groups, and the number of groups of the test input is consistent with the number of test groups.

3. The operator accuracy consistency test method according to claim 2, characterized in that: The test configuration also includes a number of repetitions; The step of repeatedly running the operator to be tested based on the test input comprises: Based on the test input, the operator to be tested is repeatedly executed until the number of times the operator to be tested is executed reaches the number of repetitions.

4. The operator accuracy consistency test method according to claim 2, characterized in that: The parsing of the test configuration file to obtain the input configuration, output configuration and test configuration of the operator to be tested in the test configuration file, and the test input generated based on the input configuration and the test configuration, includes: Reading the test configuration from the test configuration file; Parsing the input list in the test configuration file, obtaining the input configuration of each test input in the input list, generating a test input based on the input configuration, and copying the test input to a device running the operator to be tested; Parse the output list in the test configuration file to obtain the output configuration of each test output in the output list, and control the device to initialize an empty tensor matching the output configuration for storing the test output.

5. The operator accuracy consistency test method according to claim 1, characterized in that: Also includes: Parsing the test configuration file to obtain device information for running the operator to be tested; Based on the device information, a device running the operator to be tested is determined.

6. The operator accuracy consistency test method according to claim 1, characterized in that: The performing consistency judgment on the multiple test outputs includes: The multiple test outputs are divided into multiple threads, and the multiple threads are controlled to perform consistency judgment on the multiple test outputs at each thread in parallel.

7. The operator accuracy consistency testing method according to any one of claims 1 to 6, characterized in that: The input configuration includes at least one of the type, layout, and shape of the test input, and the output configuration includes at least one of the type, layout, and shape of the test output.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the operator accuracy consistency testing method as described in any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operator accuracy consistency testing method as claimed in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the operator accuracy consistency testing method as claimed in any one of claims 1 to 7 is implemented.

Images