Operator generation test case generator, generation method and system

Abstract

The present application relates to a generator, a method and an automatic test system for generating test cases for operators in deep learning, wherein the system of the present application is integrated in an integrated circuit device, which includes a general-purpose interconnection interface and other processing devices. The system interacts with the other processing devices to jointly complete the user-specified computing operation. The integrated circuit device can also include a storage device connected to the system and the other processing devices respectively for data storage of the system and the other processing devices.

Description

TECHNICAL FIELD

[0001] The present application generally relates to the field of deep learning. More particularly, the present application relates to a generator for generating test cases for operators in deep learning, a generating method and an automatic test system. BACKGROUND

[0002] High accuracy of operator operation is the basis for high accuracy of the entire deep learning model. Therefore, the test of the deep learning operator is also critical and indispensable. At present, for operator testing, a manually written method is generally used, and a corresponding code needs to be written for each scene for testing. This method is high in cost, low in efficiency, and the test scene is often not comprehensive enough.

[0003] With the continuous development of testing technology and the continuous enrichment of application scenarios, Pairwise Independent Combinatorial Testing (PICT) and ALLPAIRS have appeared. PICT is a software test case generation tool based on orthogonal method, which requires that all level combinations of any two factors (input conditions) be covered at least once. ALLPAIRS is a test case design tool for Windows, but it is ported to multiple platforms with some minor changes to the script file. It automatically designs all experimental techniques, and through the method of this tool, a small amount of data can be selected from a large number of data combinations to generate test cases. These generation tools can combine a variety of optimal combinations. However, these tools can only meet the most basic test case scene combination construction, and cannot generate targeted test cases for operators in deep learning, nor can they generate comparison data.

[0004] As can be seen, neither of the current solutions is ideal. In order to solve the above problems, the present application proposes a scheme for generating test cases for operators in deep learning. SUMMARY

[0005] In order to at least partially solve the technical problems mentioned in the background art, the present application provides a generator, method and generating system for generating test cases for operators in deep learning.

[0006] In one aspect, the present application discloses a generator for generating test cases for operators in deep learning, comprising: a configuration file unit configured to generate a configuration file required by the test case according to data information of the operator; a scene generation unit configured to randomly combine the data information in the configuration file according to a preset combination rule to generate a plurality of test scenes; and a case generation unit configured to generate a plurality of test cases according to the plurality of test scenes.

[0007] In another aspect, the present invention discloses a method for generating test cases for operators in deep learning, the method comprising: generating a configuration file required for the test case based on the data information of the operator; randomly combining the data information in the configuration file according to preset combination rules to generate multiple test scenarios; and generating multiple test cases based on the multiple test scenarios.

[0008] In another aspect, the present invention discloses an automatic testing system, comprising: a processor configured to execute an operator in deep learning based on a test case; and a generator according to the above.

[0009] By utilizing the solution for generating test cases of the present invention, the data information required for generating a test scenario is randomly combined, and test cases are generated based on the test scenario, so that the generated multiple test cases can cover a large number of test scenarios. Furthermore, the present invention can generate test cases with wide coverage and targeting for operators in deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts.

[0011] Figure 1 is a schematic diagram of a convolutional neural network according to an embodiment of the present invention;

[0012] Figure 2 is a device diagram showing a generator of test cases according to an embodiment of the present invention;

[0013] Figure 3 is a structural diagram of an integrated circuit device according to an embodiment of the present invention;

[0014] Figure 4 is a structural diagram of a board according to an embodiment of the present invention;

[0015] Figure 5 is a flow chart illustrating a test case generation method according to an embodiment of the present invention; and

[0016] Figure 6 FIG. 4 is a flow chart illustrating another method for generating test cases according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Clearly, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all the other embodiments obtained by a person skilled in the art without creative work fall within the protection scope of the present application.

[0018] It should be understood that the terms "first", "second", "third", and "fourth" and the like in the claims, specification, and drawings of the present application are used to distinguish different objects, and are not used to describe a particular order. The terms "include" and "contain" used in the specification and claims of the present application indicate the presence of the described features, whole, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components, and / or sets thereof.

[0019] It should also be understood that the terms used in the specification of the present application herein are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and claims of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should be further understood that the term "and / or" used in the specification and claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0020] As used in the specification and claims of the present application, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context.

[0021] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0022] Deep learning is a kind of machine learning, and machine learning is a necessary path to realize artificial intelligence. The concept of deep learning is derived from the research of artificial neural networks, and the multilayer perceptron with multiple hidden layers is a kind of deep learning structure. Deep learning forms more abstract high-level representation attribute classes or features by combining low-level features to discover distributed feature representation of data. The motivation of studying deep learning is to establish a neural network that simulates the brain to analyze and learn, which simulates the mechanism of the human brain to interpret data such as images, sounds, and texts. "Deep learning" is a multi-layer network structure, similar to the cognitive structure of the human brain, which can perform computation and learning.

[0023] The convolutional neural network is a typical deep learning model, which is usually composed of the following four layer structures: an input layer, a convolution layer, a pooling layer, and a fully connected layer. Figure 1 is a schematic diagram showing the four-layer structure of the convolutional neural network 100.

[0024] The input layer 102 is configured to extract partial information from an input image and convert the partial information into a feature matrix, which contains features corresponding to the partial information.

[0025] The convolution layer 104 is configured to receive the feature matrix from the input layer 102 and perform feature extraction on the input image through a convolution operation. Although Figure 1 The convolution layer 104 only shows a single layer structure, but in actual use, multiple convolution layers can be built. The front half of the convolution layers are used to capture local and detailed information of the image, i.e., each pixel of the output image is only calculated based on a small range of values of the input image. The receptive field of the convolution layers increases layer by layer, and is used to capture more complex and abstract information of the image. After multiple convolution layers are operated, the image is finally represented in different scales. Through the convolution operation, the feature extraction of the input image is completed.

[0026] The pooling layer 106 is configured to replace a region of the image with a value, which is usually the maximum value or the average value of all values in the region. If the maximum value is used, it is called max-pooling, and if the average value is used, it is called average-pooling. Through pooling, the model size can be reduced and the calculation speed can be improved without losing too much information.

[0027] The fully connected layer 108 acts as a classifier in the entire convolutional neural network 100, which is equivalent to a feature space transformation. All useful information is extracted and integrated, and a non-linear mapping of an activation function is added. The multi-layer fully connected layer can theoretically simulate any non-linear transformation to compare information based on different classifications, so as to judge whether the input image is similar to the target of comparison.

[0028] At least one layer in the deep learning model acquires input data, which can be images, sounds, texts, etc. The input data is processed, and deep learning operators such as convolution operators, pooling operators, and activation operators are run in the processing process. The high accuracy of the operators is the basis for the high accuracy of the entire deep learning model.

[0029] A test case refers to a set of test inputs, execution conditions and expected results prepared for a specific purpose to verify whether a certain software requirement is met. In the operator testing process, different test inputs and execution conditions correspond to test cases of different test scenarios. Test scenarios include functional testing, performance testing, stress testing, bandwidth testing, I / O testing, etc.

[0030] Figure 2 An automatic test system 20 configured to test operators in deep learning is shown, which includes a generator 200 generating test cases and a tester 205. The generator 200 is configured to generate test cases for operators in a neural network, including a configuration file unit 201, a scenario generation unit 202, a case generation unit 203 and a data management unit 204. The case generation unit 203 includes a calculation unit 213 and a storage unit 223. The tester 205 is configured to test operators in a neural network based on the test cases generated by the generator 200 to determine the accuracy of the operators in the neural network, including an analysis unit 215, an execution unit 225 and a comparison unit 235.

[0031] The configuration file unit 201 is used to generate a configuration file required for generating test cases according to data information of the operator, and send the configuration file to the scenario generation unit 202, so that the scenario generation unit 202 generates multiple test scenarios.

[0032] In detail, the data information of the operator includes at least one of the number of inputs / outputs, the dimension of inputs / outputs, the data type of inputs / outputs, data distribution and parameter information. The number of inputs / outputs or the dimension of inputs / outputs is used to determine the total number of data required by the operator; the data distribution is used to determine the generation rule of the input data, including normal distribution, binomial distribution, uniform distribution, etc.; the data type of inputs / outputs is used to determine the storage type of the input / output data of the operator, so as to facilitate the participation of the input / output data in subsequent operation and memory access. In this embodiment, the configuration file uses a Json file as a carrier, and the user sets the necessary data information of the input / output number, the input / output dimension, the input / output data type, the data distribution, the parameter number, the parameter type, etc. of the operator to be tested. The Json file is a simple data exchange format, which can exchange data between servers. The Json file uses a text format completely independent of programming languages to store and represent data. The simple and clear hierarchical structure makes Json an ideal data exchange language.

[0033] The scene generation unit 202 is configured to randomly combine data information in the configuration file according to a preset combination rule to generate a plurality of test scenes. Based on the configuration file generated by the configuration file unit 201, the scene generation unit 202 parses the data information of the configuration file and randomly combines the data information according to the preset combination rule. The preset combination rule includes at least one of the number of inputs / outputs, the dimension of the inputs / outputs, the data type of the inputs / outputs, the data distribution, and the parameter information. The combination rule can realize the test cases required for traversing a large number of test scenes and edge test scenes or even all test scenes, can comprehensively cover the test scenes of the deep learning operator, and improve the accuracy of operator testing.

[0034] The use case generation unit 203 is configured to generate a plurality of test cases according to the plurality of test scenes generated by the scene generation unit 202.

[0035] The computing unit 213 is configured to run the deep learning operator based on the plurality of test scenes to obtain a plurality of computing results. Each computing result includes a true value generated by the corresponding test scene, which is the result actually obtained by the test case when executing the deep learning operator. Each test case is a set of corresponding test scenes, corresponding true values, and standard output values. In addition to running the deep learning operator, the computing unit 213 can also perform other simple tasks, such as processing intermediate results obtained during the execution of the deep learning operator, including accumulation, quantization, transposition, conversion of data types for precision improvement, and other operations.

[0036] The storage unit 223 is configured to cooperate with the computing unit 213 to save the plurality of computing results. The storage unit 223 is also configured to save the test cases according to a constraint rule. The storage unit 223 can be a buffer, a dedicated memory, a general-purpose memory, or other types of storage structures. The constraint rule includes a naming specification of the test cases and a structure specification of the test cases, wherein the structure specification includes a standard name of the data information. Optionally, the constraint rule of the test cases is set using ProtoBuf, Xml, and Json files, which are high-efficiency protocol data exchange format tool libraries. The Xml and Json files directly use field names to maintain the mapping relationship between fields and data in the serialized instance, and are generally saved in the serialized byte stream in the form of strings. In these two files, messages and message definitions are relatively independent and have good readability. The serialization methods of ProtoBuf, Xml, and Json are different, and a binary byte serialization method is used to calculate the relationship mapping before the field by using field index and field type, thereby achieving higher time efficiency and space efficiency, and being particularly suitable for occasions sensitive to data size and transmission rate.

[0037] The data management unit 204 is configured to classify the plurality of test cases according to a preset management rule. The data management unit 204 receives the test cases stored in the storage unit 223, and classifies the test cases according to different standards, so that the test cases can be uniformly managed and called after being generated, and functions such as test case uploading, test case querying, version management, and the like are supported. Further, the preset management rule includes version information of the test cases. In the development process, the saved test cases are continuously updated according to needs, and are classified according to different test case versions, so as to facilitate users to quickly query different test cases corresponding to different versions. The preset management rule further includes hardware information of the deep learning model. The hardware information of the test includes different board models, operating systems, Host system memory, CPU or GPU models, and the like. The test cases required for executing the test cases in different hardware environments can be different. The purpose of distinguishing different hardware information is to generate standard benchmark data on a reliable hardware platform, so as to compare the accuracy and performance.

[0038] The parsing unit 215 is configured to parse the test case generated by the generator 200 to obtain a test scene corresponding to the test case, a corresponding true value and a standard output value, and send the parsed result to the execution unit 225. According to the parsed result, the test is executed on the test code to obtain the test result of the test case. The comparison unit 235 compares the test result with the standard output value obtained by the parsing unit 215 to obtain the final test result. Specifically, when the test result is within the error allowable range of the standard output value, the test passes, and when the test result exceeds the error allowable threshold of the standard output value, the test fails.

[0039] Figure 3 FIG. 3 is a structural diagram illustrating an integrated circuit device 300 according to an embodiment of the present application. As shown in the figure, the integrated circuit device 300 includes a system 302, which can be a Figure 2 system 20. In addition, the integrated circuit device 300 further includes a general-purpose interconnection interface 304 and other processing devices 306.

[0040] In this embodiment, the other processing devices 306 can be one or more types of processors such as central processors, graphics processors, artificial intelligence processors, and the like general-purpose and / or special-purpose processors, the number of which is not limited but determined according to actual needs. In a specific case, the other processing devices 306 can serve as an interface between the system 302 and external data and control, performing basic controls including but not limited to data transfer, starting, stopping, and the like of the machine learning operation device.

[0041] According to the technical solution of the embodiment, the general interconnection interface 304 can be used to transmit data and control instructions between the system 302 and the other processing device 306. For example, the system 302 can obtain required input data from the other processing device 306 via the general interconnection interface 304 and write the data into a storage device on the system 302 chip. Further, the system 302 can obtain control instructions from the other processing device 306 via the general interconnection interface 304 and write the control instructions into a control cache on the system 302 chip. Alternatively or additionally, the general interconnection interface 304 can also read data in a storage module of the system 302 and transmit the data to the other processing device 306.

[0042] Optionally, the integrated circuit device 300 can further include a storage device 308, which can be connected to the system 302 and the other processing device 306 respectively. In one or more embodiments, the storage device 308 can be used to store data of the system 302 and the other processing device 306, especially for data that cannot be completely stored in the internal storage of the system 302 or the other processing device 306.

[0043] According to different application scenarios, the integrated circuit device 300 of the present application can be used as an SOC system on chip of a device such as a mobile phone, a robot, a drone, or a video acquisition device, thereby effectively reducing the core area, improving the processing speed, and reducing the overall power consumption. In this case, the general interconnection interface 304 of the integrated circuit device 300 is connected to some components of the device. The some components can be, for example, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface.

[0044] In some embodiments, the present application further discloses a chip or an integrated circuit chip, which includes the integrated circuit device 300. In other embodiments, the present application further discloses a chip packaging structure, which includes the chip.

[0045] In some embodiments, the present application further discloses a board card, which includes the chip packaging structure. Referring to Figure 4 which provides the aforementioned exemplary board card 400. In addition to the chip 402, the board card 400 can further include other supporting components, which can include but are not limited to a memory device 404, an interface device 406, and a control device 408.

[0046] The memory device 404 is connected to the chip 402 in the chip packaging structure through a bus and is used to store data. The memory device 404 can include a plurality of groups of memories 410. Each group of memories 410 is connected to the chip 402 through a bus. Each group of memories 410 can be a DDR SDRAM (Double Data Rate SDRAM).

[0047] In one embodiment, the memory device 404 can include four groups of memory 410. Each group of memory 410 can include a plurality of DDR4 chips. In one embodiment, the chip 402 can include four 72-bit DDR4 controllers inside, 64 bits of which are used for data transmission and 8 bits are used for ECC check.

[0048] In one embodiment, each group of memory 410 can include a plurality of double data rate synchronous dynamic random access memories arranged in parallel. DDR can transmit data twice in one clock cycle. A controller for controlling DDR is arranged in the chip 402 to control data transmission and data storage of each memory 410.

[0049] The interface device 406 is electrically connected with the chip 402 in the chip package structure. The interface device 406 is used to realize data transmission between the chip 402 and an external device 412 (such as a server or a computer). In one embodiment, the interface device 406 can be a standard PCIE interface. For example, data to be processed is transmitted by the server to the chip 402 through the standard PCIE interface to realize data transfer. In another embodiment, the interface device 406 can also be other interfaces, and the present application does not limit the specific forms of the above-mentioned other interfaces, as long as the interface device 406 can realize the function of switching. In addition, the calculation result of the chip 402 is still transmitted back to the external device 412 by the interface device 406.

[0050] The control device 408 is electrically connected with the chip 402 to monitor the state of the chip 402. Specifically, the chip 402 and the control device 408 can be electrically connected through an SPI interface. The control device 408 can include a single-chip microcomputer (“MCU”, MicroController Unit). The chip 402 can include a plurality of processing chips, a plurality of processing cores or a plurality of processing circuits, and can drive a plurality of loads. Thus, the chip 402 can be in different working states such as heavy load and light load. Through the control device 408, the working states of the plurality of processing chips, the plurality of processing and / or the plurality of processing circuits in the chip 402 can be regulated.

[0051] In some embodiments, the application further discloses an electronic device or apparatus comprising the above-mentioned board card 400. According to different application scenarios, the electronic device or apparatus can comprise a data processing apparatus, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a mobile phone, a vehicle record instrument, a navigation instrument, a sensor, a camera, a server, a cloud server, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a vehicle, a household appliance, and / or a medical device. The vehicle comprises an airplane, a ship and / or a vehicle; the household appliance comprises a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, an oil smoke exhauster; the medical device comprises a nuclear magnetic resonance instrument, a B-ultrasonic instrument and / or an electrocardiograph.

[0052] Figure 5 Another embodiment of the application is shown, which is a method for generating test cases for operators in deep learning, which is applied in the above-mentioned test system 20, the combination apparatus 300 or the board card 400. Figure 5 A flowchart of the method is shown.

[0053] Step 510: generating a configuration file required by the test case according to the data information of the operator. In detail, the data information of the operator comprises at least one of the number of inputs / outputs, the dimension of inputs / outputs, the data type of inputs / outputs, data distribution and parameter information. The number of inputs / outputs or the dimension of inputs / outputs is used to determine the total number of data required by the operator; the data distribution is used to determine the generation rule of the input data, including normal distribution, binomial distribution, uniform distribution, etc.; the data type of inputs / outputs is used to determine the storage type of the input / output data of the operator, so as to facilitate the participation of the input / output data in subsequent operation and memory access. In this embodiment, the configuration file uses a Json file as a carrier, and the user sets the necessary data information of the input / output number, the input / output dimension, the input / output data type, the data distribution, the parameter number, the parameter type and the like of the to-be-tested operator.

[0054] Step 520: randomly combining the data information in the configuration file according to a preset combination rule to generate a plurality of test scenes. This step further analyzes the data information of the configuration file, and randomly combines the data information according to the preset combination rule. The preset combination rule comprises at least one of the number of inputs / outputs, the dimension of inputs / outputs, the data type of inputs / outputs, the data distribution and the parameter information. The combination rule can realize the test cases required by traversing a large number of or even all test scenes, can comprehensively cover the test scenes of deep learning operators, and can improve the accuracy of operator testing.

[0055] Step 530: generating a plurality of test cases according to the plurality of test scenarios. In detail, this step can be expanded into a flowchart as shown in FIG. 5B. Figure 6

[0056] Step 610: running the deep learning operator based on the plurality of test scenarios to obtain a plurality of calculation results. Each calculation result includes a true value generated by the corresponding test scenario, which is the actual result obtained by the test case when executing the deep learning operator. Each test case is a set of corresponding test scenarios, corresponding true values and standard output values.

[0057] Step 620: saving the plurality of calculation results. The test cases are saved according to constraint rules. The constraint rules include naming specifications of the test cases and structure specifications of the test cases, wherein the structure specifications include standard names of the data information. Optionally, the constraint rules of the test cases are set using ProtoBuf, Xml and Json files, which are high-efficiency protocol data exchange format tool libraries. The Xml and Json files directly use field names to maintain the mapping relationship between fields and data in serialized instances, which are generally saved in the form of strings in serialized byte streams. In these two files, messages and message definitions are relatively independent, and are easy to read. The serialization methods of ProtoBuf, Xml and Json are different, and a binary byte serialization method is adopted to calculate the relationship mapping before the field by using field index and field type, so as to achieve higher time efficiency and space efficiency, which is particularly suitable for occasions sensitive to data size and transmission rate.

[0058] Optionally, the method of this embodiment further includes classifying the plurality of test cases according to preset management rules. In detail, the test cases are classified according to different standards, so that the test cases can be uniformly managed and called after being generated, and functions such as test case uploading, test case querying and version management are supported. Further, the preset management rules include version information of the test cases. In the research and development process, the saved test cases are continuously updated according to needs, and different versions of test cases corresponding to different versions of needs are classified for the convenience of users to quickly query. The preset management rules further include hardware information of running the deep learning model. The hardware information of the test includes different board models, operating systems, Host system memory, CPU or GPU models, and the test cases required for executing the test cases in different hardware environments may be different. The purpose of distinguishing different hardware information is to generate standard benchmark data on reliable hardware platforms for comparison of precision and performance.

[0059] ​Based on the foregoing description of each embodiment, the present application generates test cases based on a test scene generated by randomly combining data information required for generating the test scene, so that the generated multiple test cases can cover a large number of test scenes, and the present application can generate test cases with wide coverage and pertinence for operators in deep learning.

[0060] The foregoing can be better understood in light of the following clauses:

[0061] Clause A1, a generator for generating test cases for operators in a neural network, comprising: a configuration file unit configured to generate a configuration file required for the test cases according to data information of the operators; a scene generation unit configured to randomly combine the data information in the configuration file according to a preset combination rule to generate multiple test scenes; and a case generation unit configured to generate multiple test cases according to the multiple test scenes.

[0062] Clause A2, the generator of clause A1, wherein the data information comprises at least one of the number of inputs / outputs, the input / output dimension, the input / output data type, the data distribution, and the parameter information of the operators.

[0063] Clause A3, the generator of clause A2, wherein the preset combination rule comprises selecting at least one of the number of inputs / outputs, the input / output dimension, the input / output data type, the data distribution, and the parameter information.

[0064] Clause A4, the generator of clause A1, wherein the case generation unit comprises: a calculation unit configured to run the operators based on the multiple test scenes to obtain multiple calculation results; and a storage unit configured to save the multiple calculation results.

[0065] Clause A5, the generator of clause A4, wherein each calculation result comprises a true value generated by a corresponding test scene, and each test case is a set of a corresponding test scene, a corresponding true value, and a standard output value.

[0066] Clause A6, the generator of clause A5, wherein the storage unit is further configured to save the test cases according to a constraint rule.

[0067] Clause A7, the generator of clause A6, wherein the constraint rule comprises a naming specification of the test cases and a structure specification of the test cases, and the structure specification comprises a standard name of the data information.

[0068] Clause A8, the generator of clause A1, further comprising a data management unit configured to classify the multiple test cases according to a preset management rule.

[0069] Clause A9, the generator of Clause A8, the preset management rule comprises version information of the test case.

[0070] Clause A10, the generator of Clause A8, the preset management rule comprises hardware information of running the neural network.

[0071] Clause A11, a method for generating a test case for an operator in deep learning, the method comprising: generating a configuration file required by the test case according to data information of the operator; randomly combining the data information in the configuration file according to a preset combination rule to generate a plurality of test scenes; and generating a plurality of test cases according to the plurality of test scenes.

[0072] Clause A12, an automatic test system, comprising: a processor configured to execute an operator in deep learning based on a test case; and the generator of any one of Clauses A1 to A10.

[0073] The above has carried out the detailed introduction to the embodiment of the application, the principle and implementation mode of the application have been set forth in this paper by applying specific examples, the above embodiment is only for helping understanding the method of the application and its core idea; at the same time, for the general technical personnel of the field, according to the idea of the application, there will be changes in specific implementation mode and application range, and the above-mentioned, the content of the specification should not be understood as the limitation of the application.

Claims

1. A generator for generating test cases for operators in deep learning, characterized in that: include: a configuration file unit, configured to generate a configuration file required for the test case based on the data information of the operator, wherein the data information includes at least one of the number of inputs / outputs, input / output dimensions, input / output data types, data distribution, and parameter information of the operator; a scenario generating unit, configured to randomly combine the data information in the configuration file according to a preset combination rule to generate a plurality of test scenarios, wherein the preset combination rule comprises selecting at least one of the number of inputs / outputs, the input / output dimensions, the input / output data type, the data distribution, and the parameter information; as well as A use case generating unit is used to generate multiple test cases according to the multiple test scenarios.

2. The generator according to claim 1, characterized in that The use case generation unit includes: a computing unit, configured to run the operator based on the multiple test scenarios to obtain multiple computing results; and A storage unit is used to store the multiple calculation results.

3. The generator according to claim 2, characterized in that Each calculation result includes a true value generated by a corresponding test scenario, wherein each test case is a set of a corresponding test scenario, a corresponding true value, and a standard output value.

4. The generator according to claim 3, characterized in that The storage unit is further configured to store the test case according to constraint rules.

5. The generator according to claim 4, characterized in that The constraint rules include a naming specification of the test case and a structure specification of the test case, wherein the structure specification includes a standard name of the data information.

6. The generator according to claim 1, characterized in that It also includes a data management unit, which is used to classify the multiple test cases according to preset management rules.

7. The generator according to claim 6, characterized in that The preset management rules include version information of the test case.

8. The generator according to claim 6, characterized in that The preset management rules include hardware information for running the operator.

9. A method for generating test cases for operators in deep learning, characterized in that: The method comprises: Generate a configuration file required for the test case according to the data information of the operator, wherein the data information includes at least one of the number of inputs / outputs, input / output dimensions, input / output data types, data distribution, and parameter information of the operator; randomly combining the data information in the configuration file according to a preset combination rule to generate a plurality of test scenarios, wherein the preset combination rule includes selecting at least one of the number of inputs / outputs, the input / output dimensions, the input / output data type, the data distribution, and the parameter information; and A plurality of test cases are generated according to the plurality of test scenarios.

10. An automatic testing system, characterized in that: include: a processor configured to execute a deep learning operator based on a test case; as well as A generator according to any one of claims 1 to 8.

Images