Test file generation method, operator testing method, electronic device and storage medium
By flexibly configuring training steps on the artificial intelligence chip and collecting basic operator information in real time, the problems of excessive storage space and insufficient information collection are solved, and efficient test file generation and operator testing are achieved.
Patent Information
- Application Number
- CN202411223239.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-03
AI Technical Summary
Existing technologies for testing operators in AI models on AI chips suffer from problems such as excessive storage space consumption and insufficient information collection, resulting in low testing efficiency.
By flexibly configuring training steps, only the basic information of operators in the specified steps to be captured is captured, and test files are collected and generated in real time during the training process, reducing storage pressure and improving testing efficiency.
It effectively reduces local storage pressure, saves storage space, and improves the efficiency of test file generation and operator testing, enabling more efficient use of storage space and time.
Smart Images

Figure CN118760609B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to a test file generation method, an operator testing method, an electronic device, and a storage medium. Background Art
[0002] With the widespread application of deep learning technology in various fields, AI-specific chip technology continues to develop rapidly. AI frameworks (for example, TensorFlow framework, PyTorch framework, etc.) integrate algorithm encapsulation, data call, and computing resource usage, while providing developers with a development interface and an efficient execution platform. They are essential tools for AI algorithm development. AI frameworks can be used for the design, training, and verification of AI models. AI models developed based on AI frameworks can run in AI chips to perform complex computing tasks such as deep learning. Summary of the invention
[0003] At least one embodiment of the present disclosure provides a test file generation method for testing multiple operators in a model to be tested, wherein the test file generation method comprises: setting multiple training steps for the model to be tested, and specifying m training steps among the multiple training steps as m steps to be captured, wherein m is a positive integer; running the multiple training steps to train the model to be tested, wherein during the running of the multiple training steps, the basic information of each operator among the multiple operators in the m steps to be captured in real time, and the captured basic information of each operator is added to a target data file in real time; after the multiple training steps are run, a test file corresponding to each operator is generated based on the target data file.
[0004] For example, in the test file generation method provided by at least one embodiment of the present disclosure, the multiple operators include a first operator, which is called at least once during the execution of the multiple training steps, and the basic information of the first operator includes the number of times the first operator is called and data information of the tensor data of the first operator, and the real-time capture of the basic information of each of the multiple operators in the m steps to be captured includes: real-time capture of the number of times the first operator has been called in the m steps to be captured, and real-time capture of data information of the tensor data each time the first operator is called in the m steps to be captured.
[0005] For example, in the test file generation method provided by at least one embodiment of the present disclosure, the data information of the tensor data each time the first operator is called includes the data type, data shape and data range of the tensor data each time the first operator is called.
[0006] For example, in the test file generation method provided by at least one embodiment of the present disclosure, each of the multiple training steps has a corresponding identifier, and the designation of m training steps in the multiple training steps as m steps to be captured includes: selecting m identifiers from the multiple identifiers in the multiple training steps; designating the m training steps corresponding to the m identifiers as m steps to be captured, and real-time capturing of basic information of each of the multiple operators in the m steps to be captured includes: determining whether the identifier of the current training step in the multiple training steps is included in the m identifiers; in response to the identifier of the current training step being included in the m identifiers, capturing the basic information of each operator in the current training step.
[0007] For example, in the test file generation method provided by at least one embodiment of the present disclosure, the multiple operators include a first operator, the tensor data of the first operator includes at least one data shape, the basic information of the first operator includes at least one data shape of the tensor data of the first operator, and before running the multiple training steps, the test file generation method also includes: limiting the maximum number of repeated captures of each data shape of the at least one data shape of the tensor data of the first operator to n, where n is a positive integer.
[0008] For example, in the test file generation method provided by at least one embodiment of the present disclosure, the real-time capture of basic information of each of the multiple operators in the m steps to be captured includes: obtaining the maximum number of captures n for each data shape of the tensor data of the first operator; in response to the current data shape of the tensor data of the first operator to be captured, determining the number of times k that the current data shape has been repeatedly captured, where k is a positive integer; in response to k being greater than or equal to n, skipping the capture of the current data shape of the tensor data of the first operator.
[0009] For example, in the test file generation method provided by at least one embodiment of the present disclosure, the multiple operators include a first operator, and the first operator is called Y times during the execution of the multiple training steps, where Y is a positive integer. The generating a test file corresponding to each operator based on the target data file includes: parsing the target data file, splitting it to obtain basic information of each operator, wherein the basic information of the first operator includes Y entries, the i-th entry among the Y entries includes the basic information of the first operator when it is called for the i-th time, i = 1, 2,…, Y; merging the Y entries of the first operator to generate a test file corresponding to the first operator.
[0010] For example, in the test file generation method provided by at least one embodiment of the present disclosure, the i-th entry of the first operator includes data information of tensor data when the first operator is called for the i-th time, and the merging of Y entries of the first operator to generate a test file corresponding to the first operator includes: integrating the Y entries of the first operator into the test file of the first operator, wherein the test file of the first operator includes data information of the tensor data of the first operator in each of the Y entries.
[0011] For example, in the test file generation method provided by at least one embodiment of the present disclosure, the Y entries of the first operator include at least one data shape of the tensor data of the first operator, and the merging of the Y entries of the first operator to generate a test file corresponding to the first operator includes: merging the repeated data shapes in the Y entries of the first operator to obtain the merged data shape of the first operator; and recording the merged data shape of the first operator into the test file of the first operator.
[0012] For example, in the test file generation method provided by at least one embodiment of the present disclosure, the Y entries of the first operator include at least one data shape of the tensor data of the first operator, each data shape in the at least one data shape corresponds to at least one group of data ranges, and the merging of the Y entries of the first operator to generate a test file corresponding to the first operator includes: merging at least one group of data ranges corresponding to each data shape in the Y entries of the first operator to obtain the merged data range of the first operator corresponding to each data shape; and recording the merged data range of the first operator corresponding to each data shape in the test file of the first operator.
[0013] For example, in the test file generation method provided by at least one embodiment of the present disclosure, each data shape in the Y entries of the first operator corresponds to X groups of data ranges, the jth group of data ranges in the X groups of data ranges includes the jth maximum value and the jth minimum value, X is a positive integer, j = 1, 2, ..., X, and merging at least one group of data ranges corresponding to each data shape in the Y entries of the first operator to obtain the merged data range corresponding to each data shape of the first operator includes: merging the first maximum value to the Xth maximum value of the X groups of data ranges corresponding to each data shape in the Y entries of the first operator to obtain the maximum value of the merged data range corresponding to each data shape of the first operator, and merging the first minimum value to the Xth minimum value of the X groups of data ranges corresponding to each data shape in the Y entries of the first operator to obtain the minimum value of the merged data range corresponding to each data shape of the first operator.
[0014] For example, in the test file generation method provided in at least one embodiment of the present disclosure, the target data file is a json file, and the test file corresponding to each operator is a yaml file.
[0015] At least one embodiment of the present disclosure provides an operator testing method, comprising: obtaining the test file corresponding to each operator generated by the test file generation method provided by at least one embodiment of the present disclosure; based on the test file corresponding to each operator, testing multiple operators in the model to be tested in a target test platform to obtain adaptation data of each operator corresponding to the target test platform.
[0016] For example, in the operator testing method provided by at least one embodiment of the present disclosure, the multiple operators include a first operator, the test file of the first operator includes at least one data shape of tensor data of the first operator, the at least one data shape includes a first data shape, the first data shape corresponds to at least one group of data ranges, and based on the test file corresponding to each operator, the multiple operators in the model to be tested are tested in the target test platform to obtain the adaptation data of each operator corresponding to the target test platform, including: inputting the test file corresponding to the first operator into the target test platform; generating at least one group of first sample data based on at least one group of data ranges corresponding to the first data shape; running the at least one group of first sample data on the target test platform to determine the support of the target test platform for the first data shape of the first operator.
[0017] For example, in the operator testing method provided by at least one embodiment of the present disclosure, running the at least one set of first sample data on the target test platform to determine the support status of the target test platform for the first data shape of the first operator includes: in response to the failure of running the at least one set of first sample data on the target test platform, determining that the target test platform does not support the first data shape of the first operator; in response to running the at least one set of first sample data on the target test platform to obtain first output data of the first operator corresponding to the first data shape, determining the support status of the target test platform for the first data shape of the first operator based on the first output data.
[0018] For example, in the operator testing method provided by at least one embodiment of the present disclosure, based on the test file corresponding to each operator, multiple operators in the model to be tested are tested in the target test platform to obtain the adaptation data of each operator corresponding to the target test platform, and also includes: based on the support of the target test platform for each data shape of at least one data shape of the first operator, determining the support status of the target test platform for the first operator.
[0019] For example, in the operator testing method provided in at least one embodiment of the present disclosure, the multiple operators include a first operator, and in response to the target test platform supporting the first operator, the multiple operators in the model to be tested are tested in the target test platform based on the test file corresponding to each operator to obtain the adaptation data of each operator corresponding to the target test platform, including: inputting the test file corresponding to the first operator into the target test platform to generate at least one set of second sample data; running the at least one set of second sample data on the target test platform to obtain second output data of the first operator; comparing the second output data of the first operator with standard data to obtain the accuracy error between the second output data of the first operator and the standard data.
[0020] For example, in the operator testing method provided in at least one embodiment of the present disclosure, the multiple operators include a first operator, and in response to the target test platform supporting the first operator, the multiple operators in the model to be tested are tested in the target test platform based on the test file corresponding to each operator to obtain the adaptation data of each operator corresponding to the target test platform, including: inputting the test file corresponding to the first operator into the target test platform to generate at least one set of second sample data; running the at least one set of second sample data on the target test platform; after the at least one set of second sample data is run on the target test platform, calculating the running time of the first operator in the target test platform; comparing the running time of the first operator in the target test platform with the standard time to obtain the running performance of the first operator in the target test platform.
[0021] At least one embodiment of the present disclosure further provides an electronic device. The electronic device includes: a processor; a memory, including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the test file generation method provided by at least one embodiment of the present disclosure or the operator testing method provided by at least one embodiment of the present disclosure.
[0022] At least one embodiment of the present disclosure further provides a storage medium storing non-transitory computer-readable instructions, which, when executed by a computer, implements the test file generation method provided by at least one embodiment of the present disclosure or the operator testing method provided by at least one embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.
[0024] Figure 1 An exemplary flow chart of a test file generation method provided for at least one embodiment of the present disclosure;
[0025] Figure 2 A schematic diagram of an example of a test file generation method provided by at least one embodiment of the present disclosure;
[0026] Figure 3 A schematic diagram of another example of a test file generation method provided by at least one embodiment of the present disclosure;
[0027] Figure 4 A schematic diagram of an example of data information of tensor data of a first operator in a target data file provided by at least one embodiment of the present disclosure;
[0028] Figure 5 A schematic diagram of an example of data information of tensor data of a first operator in a test file of a first operator provided in at least one embodiment of the present disclosure;
[0029] Figure 6 An exemplary flow chart of an operator testing method provided for at least one embodiment of the present disclosure;
[0030] Figure 7 A schematic block diagram of a test file generating device provided for at least one embodiment of the present disclosure;
[0031] Figure 8 A schematic block diagram of an operator testing device provided for at least one embodiment of the present disclosure;
[0032] Fig. 9 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure;
[0033] Fig.10 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure; and
[0034] Fig.11 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0036] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0037] The present disclosure is described below through several specific embodiments. In order to keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of the embodiments of the present disclosure appears in more than one figure, the component is represented by the same or similar reference numeral in each figure.
[0038] The development of AI-specific chips depends not only on the maturity of hardware or software, but also on whether it can obtain broad support from deep learning frameworks. AI models developed based on deep learning frameworks are difficult to run directly on specific chips (for example, for training and reasoning), and more efficient model testing methods need to be developed to improve the adaptability of AI models in chips, thereby helping users simplify the business deployment process, reduce migration costs caused by hardware differences, and quickly obtain higher performance and energy efficiency benefits.
[0039] The artificial intelligence model includes multiple operators (Operators, Op), which correspond to the computational logic of the layers or nodes in the artificial intelligence model. For example, the convolution calculation task in the convolution layer can be an operator; the weight summation calculation task in the fully-connected layer (FC layer) can also be an operator. When implementing the artificial intelligence model, the operator can be implemented by software (for example, by computer program) or by hardware (for example, by circuit).
[0040] When an artificial intelligence chip adapts a new model, it is necessary to test the operators in the model structure. Through testing, the operator information (for example, operator name, number of calls, data type of tensor data, data shape and data range, etc.) can be obtained to efficiently test the functional support of each operator in the artificial intelligence model on the chip. In addition, before the chip adapts to the new model, it is necessary to compare the operator information obtained from the test with the standard data to obtain the accuracy and performance of the operator when it runs on the chip. For example, the standard data can be the output data and time-consuming performance obtained when the operator runs on a standard chip. Based on the above test results, the degree of adaptation of the operator in the chip can be judged, and the operators in the model can be optimized or redeveloped based on the degree of adaptation.
[0041] In order to test the functional support of each operator in the artificial intelligence model on the chip, the decorator can be registered on the operator's application programming interface (API) to obtain the operator information during model training; during the training process, the operator information can be downloaded (dumped) to the specified directory to collect the operator information used for model training in real time; downstream tasks use the collected operator information to perform tests on the chip, and perform further optimization and development work on the operator based on the test results.
[0042] In the above process of collecting operator information, the operator's tensor data (including input tensors and output tensors, etc.) will be written to the local directory in real time as training progresses, and the tensor data will occupy a large amount of local storage space. For example, model training includes multiple training steps (steps). By default, operator information will be captured from the first step of training; if the model is large or the training time is long (that is, there are many training steps), collecting the first few training steps may take up terabytes (TB) of space, greatly increasing the pressure on local storage, so that the local storage space is difficult to carry the collection of massive operator information.
[0043] In addition, due to the high pressure on local storage, if you can only start crawling from the first step of training, you may only be able to crawl data for several training steps at the beginning of training; however, the operator information that needs attention may exist in the later training steps, and the data collected in the first few training steps may not be important for testing, thus wasting local storage space.
[0044] At least one embodiment of the present disclosure provides a test file generation method for testing multiple operators in a model to be tested. The test file generation method includes: setting multiple training steps for the model to be tested, and specifying m training steps in the multiple training steps as m steps to be captured, where m is a positive integer; running the multiple training steps to train the model to be tested, wherein during the running of the multiple training steps, the basic information of each operator in the multiple operators in the m steps to be captured is captured in real time, and the captured basic information of each operator is added to the target data file in real time; after the multiple training steps are run, a test file corresponding to each operator is generated based on the target data file.
[0045] At least one embodiment of the present disclosure also provides an operator testing method, comprising: obtaining a test file corresponding to each operator generated according to the test file generation method provided by at least one embodiment of the present disclosure; based on the test file corresponding to each operator, testing multiple operators in the model to be tested in a target test platform, and obtaining adaptation data of each operator corresponding to the target test platform.
[0046] At least one embodiment of the present disclosure further provides a test file generating device, an operator testing device, an electronic device and a storage medium.
[0047] The method, apparatus, device and storage medium provided by at least one embodiment of the present disclosure can flexibly configure the steps to be captured during the model training process, and only collect the basic information of the operators required for the test in the specified steps to be captured, thereby reducing the pressure on local storage and saving storage space.
[0048] Hereinafter, at least one embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the same reference numerals in different drawings will be used to refer to the same elements described.
[0049] Figure 2 An exemplary flow chart of a test file generation method provided for at least one embodiment of the present disclosure.
[0050] For example, Figure 2 As shown, at least one embodiment of the present disclosure provides a test file generation method for testing multiple operators in a model to be tested. For example, the test file generation method includes the following steps S110 to S130.
[0051] Step S110: multiple training steps are set for the model to be tested, and m training steps among the multiple training steps are designated as m steps to be captured, where m is a positive integer.
[0052] For example, in step S110, the model to be tested can be an artificial intelligence model or other models for adapting chips, and the embodiments of the present disclosure are not limited to this. For example, the training of the model includes multiple training steps (step); the setting of multiple training steps can refer to the description in the art and will not be repeated here. For example, the training step is one of the most basic operations in model training; during the training process, the model will update the weights according to the calculated gradient, and each update can be regarded as a training step. For example, the number of training steps and the running step size of each training step can be selected based on actual needs, and the embodiments of the present disclosure are not limited to this.
[0053] For example, in step S110, by specifying m steps to be captured, only the basic information of the operators in the m steps to be captured can be collected, instead of starting to capture from the first step of training every time according to the default mode. For example, the steps to be captured can be specified based on experience or actual needs; if the operator information that needs to be paid attention to may exist in the subsequent training steps (for example, you want to download data after one day of training), the subsequent m steps that need to be paid attention to can be specified as m steps to be captured. For example, by flexibly configuring m steps to be captured, only the important operator information in the training steps that need to be paid attention to can be captured, especially when the model scale is relatively large or the training time is relatively long (that is, the number of training steps is large), which can greatly reduce the pressure on local storage and save local storage space.
[0054] Step S120: Run multiple training steps to train the model to be tested.
[0055] For example, in step S120, during the execution of multiple training steps, basic information of each operator in the multiple operators in the m steps to be captured is captured in real time, and the captured basic information of each operator is added to the target data file in real time.
[0056] For example, in step S120, when running m steps to be captured, each operator may be called once or multiple times, and it is necessary to capture the basic information of each operator each time it is called in real time. For example, the basic information of each operator may include the operator name, number of calls, parameter list, data information of the operator's tensor data, etc.; the data information of the operator's tensor data may include the type of tensor data (for example, input tensor or output tensor, etc.), data type (dtype), data shape (shape) and data range, etc. For example, the content contained in the basic information of the captured operator can be selected according to the actual test needs, and the embodiments of the present disclosure are not limited to this.
[0057] For example, in step S120, the basic information of each operator captured in real time can be added to the target data file in real time. For example, the target data file can be a json file, or other types of data files can be selected according to actual needs, which is not limited in the embodiments of the present disclosure.
[0058] Step S130: After the multiple training steps are completed, a test file corresponding to each operator is generated based on the target data file.
[0059] For example, in step S130, after the model training is completed, the target data file records the basic information of each operator captured in the m steps to be captured, and the basic information of each operator includes the information needed in the subsequent operator test. For example, further, the target data file can be parsed, split, and other operations can be performed to extract the basic information of each operator to generate a test file corresponding to each operator. For example, the test file corresponding to each operator can be a yaml file, or other types of data files can be selected according to actual needs, and the embodiments of the present disclosure are not limited to this.
[0060] In some examples, the plurality of operators includes a first operator, and the first operator is called at least once during the execution of the plurality of training steps. For example, the basic information of the first operator includes the number of times the first operator is called and data information of the tensor data of the first operator. Figure 1 In step S120, the basic information of each operator in the m to-be-captured steps is captured in real time, which may further include the following step S1201.
[0061] Step S1201: capturing in real time the number of times the first operator in the m steps to be captured has been called, and capturing in real time the data information of the tensor data each time the first operator in the m steps to be captured is called.
[0062] For example, in step S1201, since the first operator is called once or multiple times during the execution of multiple training steps, it is necessary to capture the data information of the tensor data of each operator each time it is called in real time, and record the number of times the first operator has been called in real time, so as to be added to the target data file as the basic information of the first operator.
[0063] For example, the data information of the tensor data each time the first operator is called includes the data type, data shape, and data range of the tensor data each time the first operator is called. For example, the data type of the tensor data may include float type, bf16 type, or fp32 type; the data shape of the tensor data may include two-dimensional, three-dimensional, four-dimensional, etc., for example, two-dimensional row-major order (height × width) or column-major order (width × height), four-dimensional [N, C, H, W] or [N, H, W, C] (for example, N represents the number of samples, C represents the number of channels, H and W represent the height of the data in each channel in the vertical dimension and the width in the horizontal dimension, respectively); the data range of the tensor data includes the maximum value, minimum value, average value, and matrix variance of multiple data elements in the tensor data each time the first operator is called. It should be noted that the specific forms of the data type, data shape and data range of the tensor data of the first operator can be selected according to actual needs, and the embodiments of the present disclosure do not limit this. In addition, the data information of the tensor data each time the first operator is called can also include other information such as the name and type of the tensor data, which can be selected according to actual needs, and the embodiments of the present disclosure do not limit this.
[0064] It should be noted that, in at least one embodiment of the present disclosure, the first operator may refer to any one of the multiple operators included in the model to be tested, and the selection is made according to actual needs, and the embodiments of the present disclosure do not limit this.
[0065] The test file generation method provided by at least one embodiment of the present disclosure, during the model training process, only needs to add the data type, data shape, data range and other information of the operator tensor data that will be used in subsequent tests to the target data file, without adding the actual tensor data of the operator, thereby avoiding excessive storage space occupied by the target data file and reducing the pressure on local storage.
[0066] In some examples, each training step in the plurality of training steps has a corresponding identifier (eg, stepID). Figure 1 In step S110, m training steps among the multiple training steps are designated as m steps to be captured, which may further include the following steps S1101~S1102.
[0067] Step S1101: selecting m identifiers from multiple identifiers in multiple training steps;
[0068] Step S1102: designate the m training steps corresponding to the m identifiers as m steps to be captured.
[0069] For example, in steps S1101 to S1102, m steps to be captured can be specified by specifying the identifier of the training step to be captured. For example, in the initialization stage before model training, m identifiers can be set to specify the m training steps corresponding to the m identifiers as m steps to be captured. For example, the number, selection method, and specific form of the m identifiers can be set according to actual needs, and the embodiments of the present disclosure are not limited to this.
[0070] For example, further, in Figure 1 In step S120, the basic information of each operator in the m to-be-captured steps is captured in real time, which may further include the following steps S1211~S1212.
[0071] Step S1211: determining whether the identifier of the current training step among the multiple training steps is included in the m identifiers;
[0072] Step S1212: In response to the identifier of the current training step being included in the m identifiers, basic information of each operator in the current training step is captured.
[0073] For example, in step S1211, in the current training step, it is determined whether the identifier of the current training step is included in the m identifiers of the m steps to be captured; in step S1212, if the identifier of the current training step is included in the m identifiers, the current training step is determined to be the step to be captured, and the basic information of each operator in the current training step is captured.
[0074] In some examples, the tensor data of the first operator includes at least one data shape, and the basic information of the first operator includes at least one data shape of the tensor data of the first operator. Figure 1 Before step S120, the test file generation method provided by at least one embodiment of the present disclosure further includes the following step S140.
[0075] Step S140: limiting the maximum number of times each data shape of at least one data shape of the tensor data of the first operator is repeatedly captured to n, where n is a positive integer.
[0076] For example, when the model scale is relatively large or the training time is relatively long (that is, the number of training steps is large), if the maximum number of times the same data shape of the tensor data of the first operator is captured is not limited, the same data shape of the first operator will be captured many times, resulting in a lot of duplicate data being captured, thereby wasting storage space.
[0077] For example, in step S140, by limiting the maximum number of times each data shape of the tensor data of the first operator is repeatedly captured to n, it is possible to avoid the same data shape of the first operator being captured too many times, thereby avoiding capturing too much repeated data shape information, thereby enabling more efficient use of storage space.
[0078] For example, when the maximum number of times n of grabbing the same data shape of the tensor data of the first operator is limited, Figure 1 In step S120, the basic information of each operator in the m to-be-captured steps is captured in real time, which may further include the following steps S1221~S1223.
[0079] Step S1221: Obtain the maximum number of capture times n of each data shape of the tensor data of the first operator;
[0080] Step S1222: In response to the current data shape of the tensor data of the first operator to be captured, determining the number of times k that the current data shape has been repeatedly captured, where k is a positive integer;
[0081] Step S1223: In response to k being greater than or equal to n, skipping the fetching of the current data shape of the tensor data of the first operator.
[0082] For example, in step S1221, the maximum number of captures n for each data shape of the tensor data of the first operator set in the initialization stage before model training is obtained; in step S1222, before the current data shape of the tensor data of the first operator needs to be captured, the number of times k that the current data shape has been repeatedly captured is determined; in step S1223, if k is greater than or equal to n, it means that the number of times k that the current data shape has been repeatedly captured has reached or exceeded the maximum number of captures n, then the capture of the current data shape of the tensor data of the first operator is skipped; correspondingly, if k is less than n, the current data shape of the tensor data of the first operator continues to be captured.
[0083] Figure 2 A schematic diagram of an example of a test file generation method provided by at least one embodiment of the present disclosure. For example, Figure 2 For application Figure 1 A specific example of steps S110~S120 and the above step S140.
[0084] For example, Figure 2 As shown in the figure, in the initialization phase before model training, the configuration information needed for crawling is set in the default_opt dictionary tool. For example, Figure 1 In step S110 of the program, step S1101 is executed to select 4 identifiers (here m=4) from multiple identifiers in multiple training steps, that is, dump_steps = [2, 3, 5, 10] is set in the default_opt dictionary tool; step S1102 is executed to specify the 4 training steps corresponding to identifiers 2, 3, 5, and 10 in dump_steps as 4 steps to be captured. For example, if dump_steps is not set, the capture starts from the training step with identifier 0 (i.e., the first training step) by default.
[0085] For example, Figure 2 As shown, step S140 is executed to limit the maximum number of times each data shape of the tensor data of the first operator is captured, that is, max_times_for_same_shape = 50 is set in the default_opt dictionary tool, indicating that the maximum number of times each data shape of the tensor data of the first operator is captured is limited to 50 (here n = 50). For example, taking the 24×24 data shape as an example, the maximum number of times the data shape is captured is limited to 50 times, which is sufficient to meet the needs of subsequent tests; if the maximum number of times the data shape is captured is not limited, the data shape may be captured more than 1000 times, which will affect the training speed of the model on the one hand, and on the other hand, so much repeated data shape information is not needed in the subsequent test process, resulting in a waste of storage space; therefore, by setting the maximum number of times each data shape is captured max_times_for_same_shape, it is possible to avoid capturing too much repeated data shape information, thereby reducing the pressure on local storage and saving storage space.
[0086] For example, Figure 2As shown, further, the @api_capture tool is used to register the decorator for the training process of the model to be tested (that is, the execution process of the train() function). For example, save_file = "resnet50_op_list" indicates that the target data file (for example, a json file) is named resnet50_op_list, which is related to the name of the model to be tested. For example, extra_para = default_opt is used to indicate that the default_opt dictionary tool set in the initialization phase is passed in during the training process, that is, the identifier dump_steps of the step to be captured and the maximum number of captures max_times_for_same_shape for each data shape are passed in for use in the training process of the model to be tested (that is, the execution process of the train() function).
[0087] For example, Figure 2 As shown, further, in Figure 1 In step S120, the train() function is defined and run. For example, during the running of multiple training steps (that is, during the running of the train() function), the set_training_step function (not shown in the figure) can be set in the train() function. For example, in step S1211, through the set_training_step interface, the identifier of the current training step can be determined (that is, it is determined which step is currently being trained), so as to determine whether the identifier of the current training step is included in the 4 identifiers set by dump_steps (that is, the identifier is one of 2, 3, 5, and 10). For example, in step S1212, if the identifier of the current training step is included in the 4 identifiers set by dump_steps, the basic information of each operator in the current training step is captured; if the identifier of the current training step is not included in the 4 identifiers set by dump_steps, the capture of the current training step is skipped. For example, when training to the second step, the identifier of the current training step is 3, and identifier 3 is included in the 4 identifiers (2, 3, 5, 10) set by dump_steps, then the basic information of each operator in the second step is captured; when training to the third step, the identifier of the current training step is 4, and identifier 4 is not included in the 4 identifiers (2, 3, 5, 10) set by dump_steps, then the capture of the third step is skipped; and so on in the subsequent training process.
[0088] For example, further, during the running of the train() function, for any operator (for example, operator op0), in step S1221, the maximum number of captures max_times_for_same_shape = 50 of operator op0 is obtained; in step S1222, if the current data shape of the tensor data to be captured is the current data shape of the operator op0, the number of times k that the current data shape has been repeatedly captured is determined; in step S1223, if k is greater than or equal to 50, the capture of the current data shape of the tensor data of operator op0 is skipped.
[0089] It should be noted that Figure 2 The related statements and specific setting values of the test file generation method shown are all illustrative, and the embodiments of the present disclosure do not limit the specific implementation methods of each step of the method.
[0090] The test file generation method provided by at least one embodiment of the present disclosure can flexibly configure the steps to be captured during the model training process, and only collects the basic information of the operators required for the test in the specified steps to be captured, thereby reducing the pressure on local storage and saving storage space; and by limiting the maximum number of repeated captures of each data shape of the tensor data of the same operator, it is possible to avoid the same data shape of the same operator being captured too many times, avoid capturing too much repeated data shape information, and thus enable more efficient use of storage space.
[0091] In some examples, the first operator is called Y times during the execution of multiple training steps, where Y is a positive integer. Figure 1 The step S130 may further include the following steps S131~S132.
[0092] Step S131: parsing the target data file, splitting and obtaining basic information of each operator, wherein the basic information of the first operator includes Y entries, and the i-th entry among the Y entries includes the basic information when the first operator is called for the i-th time, where i = 1, 2, ..., Y;
[0093] Step S132: Merge the Y entries of the first operator to generate a test file corresponding to the first operator.
[0094] For example, after the training of the model to be tested is completed, the final target data file will be obtained. For example, the first operator is called Y times during the running of multiple training steps, and the basic information of each time the first operator is called will be written into the target data file as a line of items. For example, in step S131, by parsing the final target data file, Y items of the first operator can be split, so that the basic information of each time the first operator is called can be obtained.
[0095] For example, in step S132, for each operator, all entries of the operator in the target data file can be merged to generate a test file corresponding to the operator. For example, the target data file can be a json file, and a yaml file corresponding to each operator can be generated based on the json file. For example, the Y entries corresponding to the first operator can be merged to generate a corresponding yaml file. For example, the json file combines the information of various operators and the information of each operator is disordered (that is, the information of the same operator is interspersed between different operators), so it is difficult to read the information of each operator, which is not conducive to testing each operator separately; compared with files in json format, files in yaml format are more concise, easy to read and easy to write, and the display form of yaml files is more conducive to testing each operator separately, thereby improving test efficiency.
[0096] Figure 3 A schematic diagram of another example of a test file generation method provided by at least one embodiment of the present disclosure. For example, Figure 3 For application Figure 1 Step S130 is a specific example.
[0097] For example, Figure 3 As shown, after the training of the model to be tested is completed, the final target data file can be obtained. For example, in step S131, the target data file is parsed and the basic information of each operator is split. For example, the basic information of each operator each time it is called will be written into the target data file as a line of entries; taking the four operators op1, op2, op3 and op4 in the target data file as an example, the basic information of op1 includes Y1 entries, the basic information of op2 includes Y2 entries, the basic information of op3 includes Y3 entries, and the basic information of op4 includes Y4 entries.
[0098] For example, Figure 3 As shown, in step S132, the 1st entry to the Y1th entry of op1, the 1st entry to the Y2th entry of op2, the 1st entry to the Y3th entry of op3, and the 1st entry to the Y4th entry of op4 are merged respectively to generate the op1 test file, the op2 test file, the op3 test file and the op4 test file respectively.
[0099] For example, Figure 3 As shown, the test files of each operator obtained by merging are input into the target test platform to test each operator. For example, taking the target test platform as the chip to be adapted as an example, based on the test files of each operator, the functional support of each operator in the model on the chip can be tested, and the operators in the model can be optimized or redeveloped based on the degree of adaptation.
[0100] It should be noted that Figure 3 The processing flow shown is only for illustration, and the embodiments of the present disclosure do not limit the number and type of operators, the number and specific form of entries, the processing method of the target data file and the operator test file, etc.
[0101] In some examples, for the merging process of the basic information of the operators, optionally, the information in the Y entries of the first operator can be integrated into a test file, or the number of calls of the first operator and related information of the tensor data can be selectively merged. The specific merging form can be selected according to actual needs, and the embodiments of the present disclosure are not limited to this.
[0102] For example, in some examples, information in Y entries of the first operator may be integrated into a test file. For example, the i-th entry of the first operator includes data information of tensor data when the first operator is called for the i-th time, and step S132 may further include the following step S1321.
[0103] Step S1321: Integrate the Y entries of the first operator into the test file of the first operator.
[0104] For example, in step S1321, the test file of the first operator includes data information of the tensor data of the first operator in each of the Y entries. For example, in this case, the data information of each call of the operator can be tested. For example, although there are some repeated data shapes in the test file, the data range corresponding to each data shape may be different; if the corresponding test sample is generated using the data range of each call, the richness of the test sample can be improved, thereby improving the accuracy of the test.
[0105] For example, in some examples, repeated data shapes in the Y entries of the first operator may be merged. For example, the Y entries of the first operator include at least one data shape of the tensor data of the first operator, and step S132 may further include the following steps S1322-S1323.
[0106] Step S1322: merging repeated data shapes in the Y entries of the first operator to obtain a merged data shape of the first operator;
[0107] Step S1323: Record the merged data shape of the first operator into the test file of the first operator.
[0108] For example, in step S1322, if the same data shape is repeated multiple times in the Y entries of the first operator, it can be simplified into a merged data shape, and then in step S1323, the merged data shape is recorded in the test file of the first operator. For example, if the Y entries of the first operator contain multiple identical data shapes [4, 16, 4096, 128], they can be merged into one [4, 16, 4096, 128] and recorded in the test file of the first operator.
[0109] For example, in some examples, the data ranges corresponding to each data shape in the Y entries of the first operator may be merged. For example, the Y entries of the first operator include at least one data shape of the tensor data of the first operator, each data shape corresponds to at least one set of data ranges, and step S132 may further include the following steps S1324-S1325.
[0110] Step S1324: merge at least one group of data ranges corresponding to each data shape in the Y entries of the first operator to obtain a merged data range corresponding to each data shape of the first operator;
[0111] Step S1325: Record the merged data range corresponding to each data shape of the first operator into the test file of the first operator.
[0112] For example, in step S1324, the maximum and minimum values of one or more groups of data ranges corresponding to the same data shape may be merged respectively, and then in step S1325, the merged data ranges corresponding to each data shape of the first operator are recorded in the test file of the first operator. For example, the merged maximum and minimum values may be the maximum and minimum values in the entire training process, rather than the maximum and minimum values of each call.
[0113] For example, specifically, each data shape in the Y entries of the first operator corresponds to X groups of data ranges, and the jth group of data ranges in the X groups of data ranges includes the jth maximum value and the jth minimum value, where X is a positive integer, j = 1, 2, ..., X. For example, step S1324 may further include: merging the 1st maximum value to the Xth maximum value of the X groups of data ranges corresponding to each data shape in the Y entries of the first operator to obtain the maximum value of the merged data range corresponding to each data shape of the first operator, and merging the 1st minimum value to the Xth minimum value of the X groups of data ranges corresponding to each data shape in the Y entries of the first operator to obtain the minimum value of the merged data range corresponding to each data shape of the first operator.
[0114] For example, by merging repeated data shapes and respectively corresponding data ranges in the Y entries of the first operator, the test file of the first operator can be simplified, thereby further reducing the storage space occupied and improving the test efficiency.
[0115] Figure 4 A schematic diagram of an example of data information of tensor data of a first operator in a target data file provided for at least one embodiment of the present disclosure.
[0116] For example, Figure 4 As shown, step S131 can be executed to parse the target data file and split to obtain the basic information of the first operator. For example, taking the output tensor outs_param of the first operator as an example, the basic information of the first operator includes 3 entries corresponding to the output tensor of the first operator; each time the first operator is called, the data information of the output tensor will be written into the target data file as one of the entries.
[0117] For example, Figure 4 As shown, the data type of the tensor data outs_param is float 32, the data shape in the first entry is [16, 512, 1024], and the corresponding data range includes the maximum value max = 573.689, the minimum value min = -985.074, the average value mean = -6.053, and the matrix variance std = 38.655 (retain 3 decimal places respectively, the same below); the data shape in the second entry is [16, 1024, 1, 512], and the corresponding data range includes the maximum value max = 8.135, the minimum value min = -8.087, the average value mean = 0.011, and the matrix variance std = 0.833; the data shape in the third entry is also [16, 1024, 1, 512], and the corresponding data range includes the maximum value max = 15.027, the minimum value min = -14.907, and the average value mean = 0.023, matrix variance std = 1.002.
[0118] For example, step S132 may be performed to merge the above three items. For example, optionally, step S1321 may be performed to integrate the above three items into the test file of the first operator, so that the corresponding test sample can be generated using the data range of each call.
[0119] For example, optionally, steps S1322 to S1325 may be performed to merge the repeated data shapes in the above three items. Figure 4As shown, the repeated data shapes in the above three entries are [16, 1024, 1, 512] in the second and third entries. The repeated data shapes in the second and third entries are merged into one [16, 1024, 1, 512], and the merged data shape is recorded in the test file of the first operator; further, different data ranges corresponding to the same data shape in the second and third entries can be merged, specifically, the maximum values 8.135 and 15.027 of the data ranges in the second and third entries are merged to obtain the maximum value 15.027 of the merged data range, and the minimum values -8.087 and -14.907 of the data ranges in the second and third entries are merged to obtain the minimum value -14.907 of the merged data range.
[0120] Figure 5 A schematic diagram of an example of data information of tensor data of a first operator in a test file of a first operator provided for at least one embodiment of the present disclosure.
[0121] For example, Figure 5 As shown, the data information of the tensor data (Tensor) of the first operator in the test file of the first operator may include the name of the tensor data (name), the type of the tensor data (tensor_type), the layout of the tensor data (layout), the format of the tensor data (format), the data shape of the tensor data (shape), and the data range (data_range) corresponding to each data shape.
[0122] For example, in Figure 5 In the example, the name of the tensor data of the first operator is self, the type is the input tensor (model_input), the layout is the activation type layout ACTIVATION, the format is NCHW, the data shape is, for example, the merged data shape, and the data range corresponding to each merged data shape is, for example, the merged data range. For example, Figure 5As shown in the figure, the merged data shapes corresponding to the self input tensor are [4, 2, 2048, 4096], [4, 2, 1024,4096], [4, 2, 2048, 2048] and [4, 2, 4096, 1024], and there is no duplication between the four data shapes; the data ranges corresponding to the four data shapes are [-4.5, 4.594, -0.003, 0.814], [-4.75, 4.844, -0.005,0.891], [-5.219, 5.344, -0.001, 0.664] and [-4.906, 5.219, -0.007, 0.779] (retain 3 decimal places respectively), and the four values in the data range are the maximum value, minimum value, mean value and matrix variance from left to right.
[0123] For example, Figure 4 The target data file in the example is in json format. Figure 5 The target data file in the example is in yaml format; compare Figure 4 and Figure 5 It can be seen that the information of each operator call can be directly recorded in the json file, that is, the json format file is conducive to the recording of data information during the model training process; the yaml format file uses a serialized data information arrangement method for each operator, so it is more concise, easy to read and easy to write, and the display form of the yaml file is more conducive to separate testing of each operator. For example, the test file generation method provided by at least one embodiment of the present disclosure can improve the efficiency of data information collection and subsequent operator testing efficiency by collaboratively using the target data file and the test file of each operator.
[0124] It should be noted that Figure 4 The example contains only part of the target data file. Figure 5 The example contains only part of the test file for the first operator, and Figure 4 and Figure 5 This is for illustration only, and the embodiments of the present disclosure do not limit the content, specific form, etc. of the target data file and the test file of the first operator.
[0125] Figure 6 An exemplary flow chart of an operator testing method provided for at least one embodiment of the present disclosure. Figure 6 As shown, at least one embodiment of the present disclosure further provides an operator testing method. For example, the operator testing method includes the following steps S210 to S220.
[0126] Step S210: obtaining a test file corresponding to each operator generated according to the test file generation method;
[0127] Step S220: Based on the test file corresponding to each operator, multiple operators in the model to be tested are tested in the target test platform to obtain adaptation data of each operator corresponding to the target test platform.
[0128] For example, in step S210, the test file generation method is the test file generation method provided by at least one embodiment of the present disclosure. The specific process of using the test file generation method to generate a test file corresponding to each operator can be referred to. Figure 1~Figure 5 The description in will not be repeated here.
[0129] For example, in step S220, based on the test file corresponding to each operator obtained in step S210, multiple operators in the model to be tested are tested in the target test platform. For example, the target test platform can be a chip to be adapted, or it can be other hardware platforms for testing, which can be selected according to actual needs, and the embodiments of the present disclosure are not limited to this.
[0130] In some examples, the adaptation data of each operator corresponding to the target test platform includes: the support of the target test platform for the first operator, the precision error between the output data of the first operator and the target data, and the running performance of the first operator in the target test platform.
[0131] For example, taking the target test platform as the chip to be adapted, based on the test file of the first operator, the functional support of the first operator on the chip can be tested; if the test result of the first operator is not supported, the first operator needs to be redeveloped to adapt the chip; if the test result of the first operator is supported, it is necessary to further test the precision error between the output data of the first operator and the target data and the operating performance of the first operator in the target test platform. If the precision error exceeds the predetermined value or the operating performance is poor, the first operator needs to be optimized to improve the adaptability of the first operator on the chip.
[0132] In some examples, the test file for the first operator includes at least one data shape of tensor data for the first operator, the at least one data shape including a first data shape, the first data shape corresponding to at least one set of data ranges. Figure 6 The step S220 may further include the following steps S2201~S2203.
[0133] Step S2201: input the test file corresponding to the first operator into the target test platform;
[0134] Step S2202: generating at least one set of first sample data based on at least one set of data range corresponding to the first data shape;
[0135] Step S2203: Run at least one set of first sample data on the target test platform to determine the support status of the target test platform for the first data shape of the first operator.
[0136] For example, in order to test whether the target test platform supports the first operator, in step S2201, the test file of the first operator of the target test platform is input, including the first data shape of the tensor data of the first operator, and the first data shape corresponds to one or more groups of data ranges; in step S2202, one or more groups of first sample data can be randomly generated through one or more groups of data ranges; in step S2203, based on the running results of at least one group of first sample data on the target test platform, the support of the target test platform for the first data shape of the first operator can be determined.
[0137] For example, step S2203 may further include: in response to at least one set of first sample data failing to run on the target test platform, determining that the target test platform does not support the first data shape of the first operator; in response to at least one set of first sample data running on the target test platform to obtain first output data of the first operator corresponding to the first data shape, determining support status of the target test platform for the first data shape of the first operator based on the first output data.
[0138] For example, in step S2203, if the first sample data fails to run on the target test platform (that is, the first output data cannot be obtained), it can be determined that the target test platform does not support the first data shape of the first operator; if the first output data of the first operator corresponding to the first data shape is obtained, but the first output data is wrong, it can also be determined that the target test platform does not support the first data shape of the first operator.
[0139] It should be noted that, in at least one embodiment of the present disclosure, the first data shape may refer to any data shape of the first operator, which is selected according to actual needs, and the embodiments of the present disclosure do not limit this.
[0140] For example, Figure 6 Step S220 may further include step S2204: determining support status of the target test platform for the first operator based on support status of the target test platform for each data shape of at least one data shape of the first operator.
[0141] For example, whether the target test platform supports the first operator is closely related to the data shape, because some operators are not universal for all data shapes and may only support some special data shapes; and the data shapes of the same operator may be different in different models to be tested. For example, in step S2204, if the target test platform supports each data shape of the first operator, it can be determined that the target test platform can support the first operator.
[0142] In some examples, if the target test platform supports the first operator, it is necessary to further test the precision error between the output data of the first operator and the target data. For example, in response to the target test platform supporting the first operator, Figure 6 The step S220 may further include the following steps S2211~S2213.
[0143] Step S2211: inputting the test file corresponding to the first operator into the target test platform to generate at least one set of second sample data;
[0144] Step S2212: running at least one set of second sample data on the target test platform to obtain second output data of the first operator;
[0145] Step S2213: Compare the second output data of the first operator with the standard data to obtain the accuracy error between the second output data of the first operator and the standard data.
[0146] For example, in step S2211, one or more groups of second sample data generated may be sample data generated based on all data shapes of the first operator (for example, may include first sample data generated based on the first data shape); in step S2212, the second output data obtained by running may be the summary output result of the first operator (for example, may be a list result containing all output values); in step S2213, the standard data may be the output data of the first operator obtained after running the model to be tested on a standard chip.
[0147] For example, by comparing the second output data with the standard data, the precision error (e.g., relative error, absolute error, etc.) between the second output data and the standard data can be obtained. For example, if the precision error obtained by comparison exceeds a predetermined value, the first operator needs to be optimized to improve the adaptability of the first operator on the target test platform.
[0148] In some examples, if the target test platform supports the first operator, it is necessary to further test the running performance of the first operator in the target test platform. Figure 6 The step S220 may further include the following steps S2221~S2224.
[0149] Step S2221: input the test file corresponding to the first operator into the target test platform to generate at least one set of second sample data;
[0150] Step S2222: running at least one set of second sample data on the target test platform;
[0151] Step S2223: after at least one set of second sample data is run on the target test platform, calculating the running time of the first operator in the target test platform;
[0152] Step S2224: Compare the running time of the first operator in the target test platform with the standard time to obtain the running performance of the first operator in the target test platform.
[0153] For example, in step S2221, one or more groups of second sample data generated may be sample data generated based on all data shapes of the first operator (for example, may include first sample data generated based on the first data shape); in step S2222, the second output data obtained by the operation may be the summary output result of the first operator (for example, a list result containing all output values); in step S2223, the running time of the first operator in the target test platform may be the time taken to run all data of the first operator on the target test platform; in step S2224, the standard time may be the running time of the first operator obtained after running the model to be tested on a standard chip.
[0154] For example, by comparing the running time of the first operator with the standard time, the time difference between the running time of the first operator and the standard time can be obtained. For example, if the time difference obtained by comparison exceeds a predetermined value, it means that the running performance of the first operator on the target test platform is poor, and the first operator needs to be optimized to improve the running performance of the first operator on the target test platform.
[0155] The operator testing method provided by at least one embodiment of the present disclosure can obtain the support degree of the target test platform for each operator and the operation accuracy and performance of each operator in the target test platform by using the test file of each operator to test the corresponding operator on the target test platform, so that the operators in the model to be tested can be redeveloped or optimized in a targeted manner, thereby improving the adaptability of the model to be tested on the target test platform.
[0156] Figure 7 A schematic block diagram of a test file generation device provided for at least one embodiment of the present disclosure. For example, the test file generation device is used to test multiple operators in a model to be tested.
[0157] For example, Figure 7 As shown, the test file generating device 200 includes a setting module 210 , a training module 220 and a generating module 230 , and the training module 220 includes a grabbing submodule 221 and an adding submodule 222 .
[0158] For example, the setting module 210 is configured to set multiple training steps for the model to be tested, and specify m training steps in the multiple training steps as m steps to be captured, where m is a positive integer. That is, the setting module 210 can be configured to perform, for example Figure 1 Step S110 is shown.
[0159] For example, the training module 220 is configured to run multiple training steps to train the model to be tested. For example, during the running of multiple training steps, the capture submodule 221 is configured to capture the basic information of each operator in the multiple operators in the m steps to be captured in real time, and the adding submodule 222 is configured to add the captured basic information of each operator to the target data file in real time. That is, the training module 220 can be configured to perform, for example, Figure 1 Step S120 is shown.
[0160] For example, the generation module 230 is configured to generate a test file corresponding to each operator based on the target data file after the multiple training steps are completed. That is, the generation module 230 can be configured to perform, for example Figure 1 Step S130 is shown.
[0161] In some examples, the target data file may be a json file, and the test file corresponding to each operator may be a yaml file.
[0162] In some examples, the multiple operators include a first operator, and the first operator is called at least once during the execution of multiple training steps; the basic information of the first operator includes the number of calls of the first operator and data information of the tensor data of the first operator. For example, the capture submodule 221 can also be configured to: capture in real time the number of times the first operator has been called in the m steps to be captured, and capture in real time the data information of the tensor data each time the first operator is called in the m steps to be captured.
[0163] For example, the data information of the tensor data each time the first operator is called includes the data type, data shape, and data range of the tensor data each time the first operator is called.
[0164] In some examples, each training step in the plurality of training steps has a corresponding identifier. For example, the setting module 210 can also be configured to: select m identifiers from the plurality of identifiers in the plurality of training steps; designate the m training steps corresponding to the m identifiers as m steps to be captured. For example, the capture submodule 221 can also be configured to: determine whether the identifier of the current training step in the plurality of training steps is included in the m identifiers; in response to the identifier of the current training step being included in the m identifiers, capture the basic information of each operator in the current training step.
[0165] In some examples, the plurality of operators include a first operator, the tensor data of the first operator includes at least one data shape, and the basic information of the first operator includes at least one data shape of the tensor data of the first operator. For example, before running a plurality of training steps, the setting module 210 may also be configured to: limit the maximum number of times each data shape of the at least one data shape of the tensor data of the first operator is repeatedly captured to n, where n is a positive integer.
[0166] For example, the capture submodule 221 can also be configured to: obtain the maximum number of captures n for each data shape of the tensor data of the first operator; in response to the current data shape of the tensor data of the first operator to be captured, determine the number of times k that the current data shape has been repeatedly captured, where k is a positive integer; in response to k being greater than or equal to n, skip the capture of the current data shape of the tensor data of the first operator.
[0167] In some examples, the multiple operators include a first operator, and the first operator is called Y times during the execution of multiple training steps, where Y is a positive integer. For example, the generation module 230 can also be configured to: parse the target data file, split it to obtain basic information of each operator, wherein the basic information of the first operator includes Y entries, and the i-th entry of the Y entries includes the basic information of the first operator when it is called for the i-th time, where i = 1, 2,…, Y; merge the Y entries of the first operator to generate a test file corresponding to the first operator.
[0168] For example, the i-th entry of the first operator includes data information of the tensor data when the first operator is called for the i-th time. For example, the generation module 230 can also be configured to integrate the Y entries of the first operator into the test file of the first operator. For example, the test file of the first operator includes data information of the tensor data of the first operator in each of the Y entries.
[0169] For example, the Y entries of the first operator include at least one data shape of the tensor data of the first operator. For example, the generation module 230 can also be configured to: merge the repeated data shapes in the Y entries of the first operator to obtain the merged data shape of the first operator; and record the merged data shape of the first operator into the test file of the first operator.
[0170] For example, the Y entries of the first operator include at least one data shape of the tensor data of the first operator, and each data shape in the at least one data shape corresponds to at least one set of data ranges. For example, the generation module 230 can also be configured to: merge at least one set of data ranges corresponding to each data shape in the Y entries of the first operator to obtain the merged data range corresponding to each data shape of the first operator; and record the merged data range corresponding to each data shape of the first operator into the test file of the first operator.
[0171] For example, each data shape in the Y entries of the first operator corresponds to X groups of data ranges, and the jth group of data ranges in the X groups of data ranges includes the jth maximum value and the jth minimum value, where X is a positive integer, j = 1, 2, ..., X. For example, the generation module 230 can also be configured to: merge the first maximum value to the Xth maximum value of the X groups of data ranges corresponding to each data shape in the Y entries of the first operator to obtain the maximum value of the merged data range corresponding to each data shape of the first operator, and merge the first minimum value to the Xth minimum value of the X groups of data ranges corresponding to each data shape in the Y entries of the first operator to obtain the minimum value of the merged data range corresponding to each data shape of the first operator.
[0172] Figure 8 A schematic block diagram of an operator testing device provided for at least one embodiment of the present disclosure.
[0173] For example, Figure 8 As shown, the operator testing device 300 includes an acquisition module 310 and a testing module 320 .
[0174] For example, the acquisition module 310 is configured to acquire a test file corresponding to each operator generated according to the test file generation method. That is, the acquisition module 310 may be configured to execute, for example Figure 6 Step S210 is shown.
[0175] For example, the test module 320 is configured to test multiple operators in the model to be tested in the target test platform based on the test file corresponding to each operator, and obtain the adaptation data of each operator corresponding to the target test platform. That is, the test module 320 can be configured to perform, for example Figure 6 Step S220 is shown.
[0176] In some examples, the plurality of operators include a first operator, the test file of the first operator includes at least one data shape of tensor data of the first operator, the at least one data shape includes a first data shape, and the first data shape corresponds to at least one set of data ranges. For example, the test module 320 may also be configured to: input the test file corresponding to the first operator into a target test platform; generate at least one set of first sample data based on at least one set of data ranges corresponding to the first data shape; and run at least one set of first sample data on the target test platform to determine the target test platform's support for the first data shape of the first operator.
[0177] For example, the test module 320 can also be configured to: in response to at least one set of first sample data failing to run on the target test platform, determine that the target test platform does not support the first data shape of the first operator; in response to at least one set of first sample data running on the target test platform to obtain first output data of the first operator corresponding to the first data shape, determine the target test platform's support for the first data shape of the first operator based on the first output data.
[0178] For example, the testing module 320 may also be configured to determine the support status of the target testing platform for the first operator based on the support status of the target testing platform for each data shape of at least one data shape of the first operator.
[0179] In some examples, multiple operators include a first operator. In response to the target test platform supporting the first operator, the test module 320 can also be configured to: input the test file corresponding to the first operator into the target test platform to generate at least one set of second sample data; run at least one set of second sample data on the target test platform to obtain second output data of the first operator; compare the second output data of the first operator with the standard data to obtain the accuracy error between the second output data of the first operator and the standard data.
[0180] In some examples, multiple operators include a first operator. In response to the target test platform supporting the first operator, the test module 320 can also be configured to: input the test file corresponding to the first operator into the target test platform to generate at least one set of second sample data; run at least one set of second sample data on the target test platform; after at least one set of second sample data is run on the target test platform, calculate the running time of the first operator in the target test platform; compare the running time of the first operator in the target test platform with the standard time to obtain the running performance of the first operator in the target test platform.
[0181] Since in the above description, for example Figure 1 The test file generation method shown and example Figure 6In the process of the operator testing method shown in FIG. 1 , the details of the operations of the test file generating device 200 and the operator testing device 300 have been introduced, so for the sake of brevity, they will not be repeated here. For relevant details, please refer to the above description of Figure 1 to Figure 6 Description.
[0182] It should be noted that in Figure 7 The test file generating device 200 and Figure 8 In the operator testing device 300 shown in the figure, the above modules can be configured as software, hardware, firmware or any combination of the above items to perform specific functions. For example, these modules can correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. Figure 7 and Figure 8 The described device may be a PC computer, a tablet device, a personal digital assistant, a smart phone, a web application, or other device capable of executing program instructions, but is not limited thereto.
[0183] In addition, although the test file generating device 200 and the operator testing device 300 are divided into modules for respectively performing corresponding processing in the above description, it is clear to those skilled in the art that the processing performed by each module can also be performed without any specific module division in the device or without clear demarcation between the modules. Figure 7 The test file generating device 200 and Figure 8 The described operator testing device 300 is not limited to include the modules described above, but some other modules (eg, a writing module, a control module, etc.) may be added as needed, or the above modules may also be combined.
[0184] At least one embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory; the memory includes one or more computer program modules; the one or more computer program modules are stored in the memory and are configured to be executed by the processor, and the one or more computer program modules include a method for implementing the test file generation method or operator testing method provided by the embodiment of the present disclosure described above.
[0185] Fig. 9 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.
[0186] For example, Fig. 9As shown, the electronic device 400 includes a processor 410 and a memory 420. For example, the memory 420 is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 410 is used to run non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are run by the processor 410, one or more steps of the test file generation method or operator test method described above can be executed. The memory 420 and the processor 410 can be interconnected through a bus system and / or other forms of connection mechanisms (not shown).
[0187] For example, the processor 410 may be a central processing unit (CPU), a digital signal processor (DSP), or other forms of processing units with data processing capabilities and / or program execution capabilities, such as a field programmable gate array (FPGA), etc.; for example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor 410 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 400 to perform desired functions.
[0188] For example, the memory 420 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 410 may run one or more computer program modules to implement various functions of the electronic device 400. Various applications and various data, as well as various data used and / or generated by the application, etc. may also be stored in the computer-readable storage medium.
[0189] It should be noted that in the embodiments of the present disclosure, the specific functions and technical effects of the electronic device 400 can refer to the above description of the test file generation method and operator testing method provided in at least one embodiment of the present disclosure, and will not be repeated here.
[0190] Fig.10 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.
[0191] For example, Fig.10 As shown, the electronic device 500 is suitable for implementing the test file generation method or operator testing method provided by the embodiment of the present disclosure. It should be noted that Fig.10The electronic device 500 shown is only an example and does not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0192] For example, Fig.10 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 51, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 52 or a program loaded from a storage device 58 to a random access memory (RAM) 53. In the RAM 53, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 51, the ROM 52, and the RAM 53 are connected to each other through a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54. Generally, the following devices can be connected to the I / O interface 55: an input device 56 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 57 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 58 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 59. The communication device 59 can allow the electronic device 500 to communicate with other electronic devices wirelessly or by wire to exchange data.
[0193] Although Fig.10 The electronic device 500 is shown to have various devices, but it should be understood that it is not required to implement or possess all the devices shown, and the electronic device 500 may alternatively implement or possess more or fewer devices.
[0194] For the detailed description and technical effects of the electronic device 500, please refer to the above description of the test file generation method or the operator testing method, which will not be repeated here.
[0195] Fig.11 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure.
[0196] For example, Fig.11 As shown, the storage medium 600 stores non-transitory computer-readable instructions 610. For example, when the non-transitory computer-readable instructions 610 are executed by a computer, one or more steps in the test file generation method or operator testing method described above are performed.
[0197] For example, the storage medium 600 can be applied to Fig. 9 For example, the storage medium 600 may be the memory 420 in the electronic device 400. For example, the description of the storage medium 600 may refer to Fig. 9 The corresponding description of the memory 420 in the electronic device 400 is shown and will not be repeated here.
[0198] There are a few points to note about this disclosure:
[0199] (1) In the drawings of the embodiments of the present disclosure, only the structures related to the embodiments of the present disclosure are involved, and other structures can refer to the general design.
[0200] (2) Unless there is a conflict, the features of the same embodiment or different embodiments of the present disclosure may be combined with each other.
[0201] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A test file generation method for testing multiple operators in a model to be tested, characterized in that: The test file generation method comprises: Setting a plurality of training steps for the model to be tested, and specifying m training steps among the plurality of training steps as m steps to be captured, wherein m is a positive integer; Running the multiple training steps to train the model to be tested, wherein during the running of the multiple training steps, basic information of each operator in the multiple operators in the m steps to be captured in real time, and adding the captured basic information of each operator to the target data file in real time; After the plurality of training steps are completed, a test file corresponding to each operator is generated based on the target data file. Each of the multiple training steps has a corresponding identifier, and the step of specifying m training steps of the multiple training steps as m steps to be captured includes: Select m identifiers from the multiple identifiers in the multiple training steps; The m training steps corresponding to the m identifiers are designated as m steps to be captured, and the real-time capturing of basic information of each operator in the multiple operators in the m steps to be captured includes: Determining whether an identifier of a current training step in the plurality of training steps is included in the m identifiers; In response to the identifier of the current training step being included in the m identifiers, capturing basic information of each operator in the current training step, The multiple operators include a first operator, the first operator is called Y times during the execution of the multiple training steps, Y is a positive integer, and the generating of a test file corresponding to each operator based on the target data file includes: Parse the target data file and split it to obtain basic information of each operator, wherein the basic information of the first operator includes Y entries, and the i-th entry of the Y entries includes the basic information of the first operator when it is called for the i-th time, i=1, 2, ..., Y; The Y entries of the first operator are merged to generate a test file corresponding to the first operator.
2. The test file generation method according to claim 1, characterized in that: The plurality of operators include a first operator, wherein the first operator is called at least once during the execution of the plurality of training steps, The basic information of the first operator includes the number of times the first operator is called and data information of the tensor data of the first operator. The real-time capturing of basic information of each operator in the m to-be-captured steps includes: Real-time capture of the number of times the first operator has been called in the m steps to be captured, and Capture in real time the data information of the tensor data each time the first operator is called in the m steps to be captured.
3. The test file generation method according to claim 2, characterized in that: The data information of the tensor data each time the first operator is called includes the data type, data shape and data range of the tensor data each time the first operator is called.
4. The test file generation method according to claim 1, characterized in that: The plurality of operators include a first operator, the tensor data of the first operator includes at least one data shape, the basic information of the first operator includes at least one data shape of the tensor data of the first operator, Before running the multiple training steps, the test file generation method further includes: The maximum number of times each data shape of at least one data shape of the tensor data of the first operator is repeatedly captured is limited to n, where n is a positive integer.
5. The test file generation method according to claim 4, characterized in that: The real-time capturing of basic information of each operator in the m to-be-captured steps includes: Obtain a maximum number of captures n for each data shape of the tensor data of the first operator; In response to the current data shape of the tensor data of the first operator to be captured, determining a number k of times the current data shape has been repeatedly captured, where k is a positive integer; In response to k being greater than or equal to n, fetching the current data shape of the tensor data of the first operator is skipped.
6. The test file generation method according to claim 1, characterized in that: The i-th entry of the first operator includes data information of the tensor data when the first operator is called for the i-th time. The merging the Y entries of the first operator to generate a test file corresponding to the first operator includes: Integrate the Y entries of the first operator into a test file of the first operator, wherein the test file of the first operator includes data information of tensor data of the first operator in each of the Y entries.
7. The test file generation method according to claim 1, characterized in that: The Y entries of the first operator include at least one data shape of tensor data of the first operator, The merging the Y entries of the first operator to generate a test file corresponding to the first operator includes: Merging repeated data shapes in the Y entries of the first operator to obtain a merged data shape of the first operator; The merged data shape of the first operator is recorded in a test file of the first operator.
8. The test file generation method according to claim 1, characterized in that: The Y entries of the first operator include at least one data shape of tensor data of the first operator, each data shape of the at least one data shape corresponding to at least one set of data ranges, The merging the Y entries of the first operator to generate a test file corresponding to the first operator includes: Merging at least one group of data ranges corresponding to each data shape in the Y entries of the first operator to obtain a merged data range corresponding to each data shape by the first operator; The merged data range corresponding to each data shape by the first operator is recorded in a test file of the first operator.
9. The test file generation method according to claim 8, characterized in that: Each data shape in the Y entries of the first operator corresponds to X groups of data ranges, the jth group of data ranges in the X groups of data ranges includes the jth maximum value and the jth minimum value, X is a positive integer, j=1, 2, ..., X, The step of merging at least one group of data ranges corresponding to each data shape in the Y entries of the first operator to obtain a merged data range corresponding to each data shape by the first operator includes: Merge the first maximum value to the Xth maximum value of the X groups of data ranges corresponding to each data shape in the Y entries of the first operator to obtain the maximum value of the merged data range corresponding to each data shape by the first operator, and The first minimum value to the Xth minimum value of the X groups of data ranges corresponding to each data shape in the Y entries of the first operator are merged to obtain the minimum value of the merged data range corresponding to each data shape of the first operator.
10. The test file generation method according to any one of claims 1 to 9, characterized in that: The target data file is a json file, and the test file corresponding to each operator is a yaml file.
11. An operator testing method, characterized in that: The operator testing method comprises: Obtain the test file corresponding to each operator generated by the test file generation method according to any one of claims 1 to 10; Based on the test file corresponding to each operator, multiple operators in the model to be tested are tested in a target test platform to obtain adaptation data of each operator corresponding to the target test platform.
12. The operator testing method according to claim 11, characterized in that: The plurality of operators include a first operator, a test file for the first operator includes at least one data shape of tensor data for the first operator, the at least one data shape includes a first data shape, the first data shape corresponds to at least one set of data ranges, The step of testing the multiple operators in the model to be tested in the target test platform based on the test file corresponding to each operator to obtain the adaptation data of each operator corresponding to the target test platform includes: Inputting a test file corresponding to the first operator into the target test platform; generating at least one set of first sample data based on at least one set of data ranges corresponding to the first data shape; The at least one set of first sample data is run on the target test platform to determine support status of the target test platform for the first data shape of the first operator.
13. The operator testing method according to claim 12, characterized in that: The running the at least one set of first sample data on the target test platform to determine the support status of the first data shape of the first operator by the target test platform includes: In response to the at least one set of first sample data failing to run on the target test platform, determining that the target test platform does not support the first data shape of the first operator; In response to the at least one set of first sample data being run on the target test platform to obtain first output data of the first operator corresponding to the first data shape, support status of the target test platform for the first data shape of the first operator is determined based on the first output data.
14. The operator testing method according to claim 12, characterized in that: The method of testing the multiple operators in the model to be tested in the target test platform based on the test file corresponding to each operator to obtain the adaptation data of each operator corresponding to the target test platform further includes: Based on the support of the target test platform for each data shape of at least one data shape of the first operator, the support of the target test platform for the first operator is determined.
15. The operator testing method according to claim 11, characterized in that: The plurality of operators include a first operator, and in response to the target test platform supporting the first operator, The step of testing the multiple operators in the model to be tested in the target test platform based on the test file corresponding to each operator to obtain the adaptation data of each operator corresponding to the target test platform includes: Inputting the test file corresponding to the first operator into the target test platform to generate at least one set of second sample data; Running the at least one set of second sample data on the target test platform to obtain second output data of the first operator; The second output data of the first operator is compared with the standard data to obtain the precision error between the second output data of the first operator and the standard data.
16. The operator testing method according to claim 11, characterized in that: The plurality of operators include a first operator, and in response to the target test platform supporting the first operator, The step of testing the multiple operators in the model to be tested in the target test platform based on the test file corresponding to each operator to obtain the adaptation data of each operator corresponding to the target test platform includes: Inputting the test file corresponding to the first operator into the target test platform to generate at least one set of second sample data; Running the at least one set of second sample data on the target test platform; After the at least one set of second sample data is run on the target test platform, calculating the running time of the first operator in the target test platform; The running time of the first operator in the target test platform is compared with the standard time to obtain the running performance of the first operator in the target test platform.
17. An electronic device, characterized in that: The electronic device comprises: processor; a memory including one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the test file generation method described in any one of claims 1 to 10 or the operator testing method described in any one of claims 11 to 16.
18. A storage medium storing non-transitory computer-readable instructions, characterized in that: When the non-transitory computer-readable instructions are executed by a computer, the test file generation method described in any one of claims 1 to 10 or the operator testing method described in any one of claims 11 to 16 is implemented.
