Operator precision verification method, system, electronic device and storage medium

Abstract

A method, system, electronic device and storage medium for verifying operator accuracy. The operator accuracy verification method comprises: obtaining a first operator to be verified; obtaining a test model based on the first operator; training the test model through reference hardware and the hardware to be tested respectively to obtain the convergence of the test model on the reference hardware and the hardware to be tested, wherein the test model is trained to convergence on at least the reference hardware; and determining whether the accuracy of the first operator on the hardware to be tested meets the standard according to the convergence of the test model on the reference hardware and the hardware to be tested. The operator accuracy verification method can accurately verify the accuracy of the operator on the hardware to be tested, and the reliability of the verification result is high.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an operator accuracy verification method, system, electronic device, and storage medium. Background Art

[0002] In recent years, with the rapid development of deep learning and artificial intelligence, new operators have emerged to adapt to increasingly complex data processing needs and diversified application scenarios. These operators play a key role in realizing more powerful and intelligent applications.

[0003] Some new hardware devices are also constantly emerging and developing, such as Graphics Processing Unit (GPU), General-Purpose computing on Graphics Processing Units (GPGPU), Tensor Processing Unit (TPU), etc. These acceleration hardware specially designed for deep learning have significant advantages in computing power, parallel processing capability and energy efficiency.

[0004] Whether it is an existing operator or a newly developed operator, when these operators are deployed on new hardware devices, it is necessary to ensure that their accuracy on the new hardware devices meets the requirements of actual applications. Therefore, before using these operators on new hardware devices, it is necessary to verify the accuracy of these operators. Only operators with accurate accuracy can be used in actual applications to ensure the accuracy of calculation results. Summary of the invention

[0005] At least one embodiment of the present disclosure provides an operator accuracy verification method, which includes: obtaining a first operator to be verified; obtaining a test model based on the first operator; training the test model through reference hardware and hardware to be tested, respectively, to obtain convergence of the test model on the reference hardware and the hardware to be tested, wherein the test model is trained to convergence on at least the reference hardware; and determining whether the accuracy of the first operator on the hardware to be tested meets the requirements based on the convergence of the test model on the reference hardware and the hardware to be tested.

[0006] For example, the operator accuracy verification method provided by at least one embodiment of the present disclosure further includes: obtaining parameter information of input data corresponding to the first operator;

[0007] The obtaining of the test model based on the first operator comprises: obtaining, through a basic model, a first sub-model for training on the reference hardware and a second sub-model for training on the hardware to be tested, based on the first operator and the parameter information;

[0008] The step of training the test model through reference hardware and the hardware to be tested respectively to obtain the convergence of the test model on the reference hardware and the hardware to be tested comprises: training the first sub-model through the reference hardware to obtain the convergence of the first sub-model on the reference hardware, and training the second sub-model through the hardware to be tested to obtain the convergence of the second sub-model on the hardware to be tested;

[0009] The determining whether the accuracy of the first operator on the hardware to be tested meets the requirements based on the convergence of the test model on the reference hardware and the hardware to be tested includes: comparing the convergence of the first sub-model on the reference hardware with the convergence of the second sub-model on the hardware to be tested, and determining whether the accuracy of the first operator on the hardware to be tested meets the requirements.

[0010] For example, in the operator accuracy verification method provided by at least one embodiment of the present disclosure, the first sub-model incorporates a first code for implementing the first operator provided by the manufacturer of the reference hardware.

[0011] For example, in the operator accuracy verification method provided by at least one embodiment of the present disclosure, the basic model includes at least one of the following operators: a shape transformation type operator for processing the input data corresponding to the first operator; a feature extraction type operator for extracting features from the input data; and a loss calculation type operator for calculating the loss value after each iteration.

[0012] For example, in the operator accuracy verification method provided by at least one embodiment of the present disclosure, the reference hardware saves the convergence information of the entire iteration stage during the process of training the test model to convergence.

[0013] For example, in the operator accuracy verification method provided by at least one embodiment of the present disclosure, in response to the reference hardware being a reference central processor located on the same device as the hardware to be tested, the reference hardware trains the test model prior to the hardware to be tested.

[0014] For example, in the operator accuracy verification method provided by at least one embodiment of the present disclosure, in response to the hardware to be tested failing to reach convergence when training the test model to the first stage, it is directly determined that the accuracy of the first operator on the hardware to be tested does not meet the standard, wherein the test model is trained to convergence by the reference hardware in the first stage.

[0015] For example, in the operator accuracy verification method provided in at least one embodiment of the present disclosure, in response to the reference hardware being a reference artificial intelligence accelerator located on a different device from the hardware to be tested, the reference hardware and the hardware to be tested train the test model in parallel.

[0016] For example, in the operator accuracy verification method provided in at least one embodiment of the present disclosure, the test model is a deep learning model obtained based on a reference model, and the scale of the test model is smaller than that of the reference model.

[0017] For example, in the operator accuracy verification method provided in at least one embodiment of the present disclosure, the test model is obtained by intercepting a portion of the reference model, or the test model is reconstructed based on the reference model.

[0018] For example, in the operator accuracy verification method provided by at least one embodiment of the present disclosure, the convergence condition includes at least one of the convergence speed and the final convergence loss value.

[0019] For example, in the operator accuracy verification method provided by at least one embodiment of the present disclosure, determining whether the accuracy of the first operator on the hardware to be tested meets the standard based on the convergence of the test model on the reference hardware and the hardware to be tested includes: in response to the final convergence loss values ​​of the test model on the reference hardware and the hardware to be tested respectively meeting a first preset condition, determining that the accuracy of the first operator on the hardware to be tested meets the standard; or, in response to the final convergence loss values ​​of the test model on the reference hardware and the hardware to be tested respectively meeting the first preset condition, and the convergence speeds of the test model on the reference hardware and the hardware to be tested respectively meeting a second preset condition, determining that the accuracy of the first operator on the hardware to be tested meets the standard.

[0020] For example, in the operator precision verification method provided by at least one embodiment of the present disclosure, the first preset condition includes: a first loss value at which the test model finally converges on the hardware to be tested is less than or equal to a second loss value at which the test model finally converges on the reference hardware; or a ratio of the first loss value at which the test model finally converges on the hardware to be tested to the second loss value at which the test model finally converges on the reference hardware is less than a preset convergence loss ratio.

[0021] For example, in the operator precision verification method provided by at least one embodiment of the present disclosure, the second preset condition includes: the convergence speed of the test model on the hardware to be tested is greater than or equal to the convergence speed of the test model on the reference hardware; or the ratio of the convergence speed of the test model on the hardware to be tested to the convergence speed of the test model on the reference hardware is greater than a preset convergence speed ratio.

[0022] For example, in the operator precision verification method provided in at least one embodiment of the present disclosure, the obtaining of the first operator to be verified currently includes: determining the first operator to be verified currently from a plurality of operators to be verified obtained in batches;

[0023] The obtaining parameter information of the input data corresponding to the first operator includes: obtaining parameter information of the input data of the first operator in multiple scenarios, wherein the input data in the multiple scenarios are different and the parameter information is the same, or the input data in the multiple scenarios are the same and the parameter information is different, or both the input data and the parameter information in the multiple scenarios are different;

[0024] The operator accuracy verification method further includes: after verifying whether the accuracy of the first operator in the multiple scenarios meets the standard, continuing to automatically verify whether the accuracy of other operators in the multiple operators to be verified meets the standard.

[0025] For example, in the operator precision verification method provided by at least one embodiment of the present disclosure, the parameter information of the input data corresponding to the first operator includes at least one of the input data quantity, input data dimension, input data shape and input data type.

[0026] For example, in the operator precision verification method provided by at least one embodiment of the present disclosure, the first operator includes a composite operator bound to multiple operators.

[0027] At least one embodiment of the present disclosure further provides an operator precision verification system, which includes: an acquisition module, a determination module and a verification module, wherein the acquisition module is configured to acquire a first operator to be verified and obtain a test model based on the first operator; the verification module is configured to train the test model through reference hardware and the hardware to be tested, respectively, to obtain the convergence of the test model on the reference hardware and the hardware to be tested, wherein the test model is trained to convergence on at least the reference hardware; and the determination module is configured to determine whether the accuracy of the first operator on the hardware to be tested meets the standard based on the convergence of the test model on the reference hardware and the hardware to be tested.

[0028] For example, in the operator accuracy verification system provided by at least one embodiment of the present disclosure, the acquisition module is also configured to obtain parameter information of the input data corresponding to the first operator, and based on the first operator and the parameter information, obtain a first sub-model for training on the reference hardware and a second sub-model for training on the hardware to be tested through a basic model; the verification module is also configured to train the first sub-model through the reference hardware to obtain the convergence of the first sub-model on the reference hardware, and train the second sub-model through the hardware to be tested to obtain the convergence of the second sub-model on the hardware to be tested; the determination module is also configured to compare the convergence of the first sub-model on the reference hardware with the convergence of the second sub-model on the hardware to be tested, to determine whether the accuracy of the first operator on the hardware to be tested meets the requirements.

[0029] At least one embodiment of the present disclosure further provides an electronic device, comprising at least one memory that non-transiently stores computer-executable instructions; and at least one processor configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, implement the operator precision verification method provided in any of the above embodiments.

[0030] At least one embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by at least one processor, the operator precision verification method provided by any of the above embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0032] Figure 1 A flowchart of an operator accuracy verification method provided by at least one embodiment of the present disclosure;

[0033] Figure 2 A flowchart of another operator accuracy verification method provided by at least one embodiment of the present disclosure;

[0034] Figure 3 A schematic diagram of reference hardware and hardware to be tested located on the same device provided for at least one embodiment of the present disclosure;

[0035] Figure 4 A schematic diagram of reference hardware and hardware to be tested located on different devices provided for at least one embodiment of the present disclosure;

[0036] Figure 5 A schematic diagram of another reference hardware and hardware to be tested located on the same device provided for at least one embodiment of the present disclosure;

[0037] Figure 6 A schematic diagram of the structure of an operator accuracy verification system provided by at least one embodiment of the present disclosure;

[0038] Figure 7 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure;

[0039] Figure 8 A schematic diagram of the structure of an electronic device provided in at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0041] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The words "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. It should be understood that the various steps recorded in the method implementation of the present disclosure can be performed in different orders and / or in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0042] The present disclosure is described below through several specific embodiments. In order to keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components (elements) may be omitted. When any component (component) of the embodiments of the present disclosure appears in more than one drawing, the component (component) is represented by the same or similar reference numeral in each drawing.

[0043] Due to different underlying hardware and algorithm ideas, the same operator will have operator accuracy deviation on different hardware. At present, for developed operators to be deployed on new hardware, the accuracy of the operator can be judged by comparing the calculation results of the operator on the new hardware with the calculation results on commercially available hardware. For newly developed operators to be deployed on new hardware, the calculation results of the operator on the new hardware can be compared with the calculation results of operators with normal accuracy and corresponding functions on commercially available hardware to judge whether the accuracy of the operator on the new hardware is up to standard. Here, commercially available hardware is usually a central processing unit (CPU), or other well-known hardware devices that can be used for comparison (such as GPUs or GPGPUs provided by some manufacturers, etc.). For these commercially available hardware, manufacturers may provide computing platforms and programming models. The accuracy of existing operators based on these platforms and programming models has usually been verified and has the commonly required accuracy. Therefore, they can be used as comparison standards. In other words, the calculation results obtained by using operators on commercially available hardware meet the actual application requirements. Operators on commercially available hardware can be used as reference operators to compare with operators deployed on new hardware, thereby verifying the accuracy of operators on new hardware.

[0044] However, this method of verifying the accuracy of an operator by comparing the computation results (or execution results) cannot accurately verify the accuracy of the operator on new hardware.

[0045] For example, one method for verifying the accuracy of an operator is to first set relative error and absolute error thresholds for the operator to be verified, then use the same input data to execute the operator on new hardware and on commercially available hardware respectively, and then compare the results obtained after executing the operator on the new hardware and on the commercially available hardware. If the relative error and absolute error of the results of the two meet the set threshold requirements, then the accuracy of the operator on the new hardware is deemed to meet the requirements; otherwise, the accuracy of the operator on the new hardware is deemed to meet the requirements.

[0046] When setting threshold requirements, we can usually only set the thresholds of relative error and absolute error through global judgment. This method has weak extrapolation ability and needs to set different threshold requirements for different types of operators. Moreover, even for the same operator, the value deviations in different scenarios vary greatly, so it is difficult to use universal relative error thresholds and absolute error thresholds to measure the accuracy of the operator.

[0047] In addition, since value deviation is not equivalent to precision deviation, misjudgment may occur, resulting in an operator with high precision on new hardware being misjudged as not meeting the precision standard because it does not meet the threshold requirement. Moreover, in many scenarios of new hardware, there will be the influence of random interference noise, which will cause the operator precision to fluctuate. In this case, it is also difficult to determine whether the operator precision meets the standard through value judgment.

[0048] Therefore, the operator accuracy verification method that makes judgments (value judgments) based on calculation results is difficult to be truly equivalent to operator accuracy verification, and has disadvantages such as difficulty in constructing thresholds and easy misjudgment.

[0049] In response to the above problems, at least one embodiment of the present disclosure provides a precision verification method, system, electronic device, and storage medium. By comparing the convergence of the test model obtained based on the current first operator to be verified on the reference hardware and the hardware to be tested, it is possible to accurately determine whether the precision of the first operator on the hardware to be tested meets the standard, and it is easy to operate and highly reliable. In addition, in at least one embodiment, it is also possible to quickly and automatically verify the precision of a large number of operators on new hardware, thereby effectively reducing time and labor costs and accelerating the development progress of operators and hardware to be tested.

[0050] Figure 1 The present invention provides a flowchart of an operator accuracy verification method according to at least one embodiment of the present invention. Figure 1 As shown, the operator accuracy verification method provided by at least one embodiment of the present disclosure includes the following steps S110~S140.

[0051] Step S110: Obtain the first operator currently to be verified.

[0052] Step S120: obtaining a test model based on the first operator.

[0053] Step S130: training the test model through the reference hardware and the hardware to be tested respectively to obtain the convergence of the test model on the reference hardware and the hardware to be tested, wherein the test model is at least trained to convergence on the reference hardware.

[0054] Step S140: Determine whether the accuracy of the first operator on the hardware to be tested meets the requirements based on the convergence of the test model on the reference hardware and the hardware to be tested.

[0055] In the embodiments of the present disclosure, the operator currently to be verified is referred to as the "first operator". An operator refers to a mathematical operation or data processing operation in the fields of mathematics, computer science, or deep learning, such as simple basic mathematical operations such as addition, multiplication, and division, or complex data processing operations such as convolution operations, pooling operations, matrix multiplication, etc.

[0056] The operator to be verified may be a computing operator, such as a mathematical operation operator (such as an addition operator, a multiplication operator, a division operator, etc.), a convolution operator (such as a two-dimensional convolution operator or a three-dimensional convolution operator, etc.), a pooling operator, a matrix operation operator, etc., or a data transformation operator or other types of operators, such as a reduce operator, a point product operator, etc. The embodiments of the present disclosure may not limit the type and specific operation of the operator to be verified. For example, the first operator may be a convolution operator in a convolutional neural network, which is used to perform a convolution operation, and can extract features in an image or data by performing a specific weighted summation on a local area of ​​the input data.

[0057] In an embodiment of the present disclosure, the test model refers to a model including a first operator or a model fused with the first operator, and in the training process of the test model, the first operator is used to implement the corresponding function. The test model may be, for example, a neural network model or other models developed based on the neural network model. Neural networks include various forms of network structures, such as multi-layer perceptrons (MLP) or deep learning networks. For example, deep learning networks include convolutional neural networks (CNN), residual networks (ResNet), recurrent neural networks (RNN), long short-term memory networks (LSTM), etc. For example, convolutional neural networks can also be subdivided into full convolutional networks, deep convolutional networks, U-nets, etc.

[0058] In step S110 and step S120, the first operator to be verified can be manually determined, and a test model for verifying accuracy can be determined based on the first operator. Then, in step S130, settings can be manually made to train the test model using reference hardware and the hardware to be tested, respectively, so as to obtain the convergence of the test model on the reference hardware and the hardware to be tested, respectively.

[0059] The hardware to be tested can be understood as new hardware developed or manufactured by the user, for example, hardware specially designed to accelerate artificial intelligence and deep learning tasks. The reference hardware can be the previous generation of hardware developed by the hardware to be tested, or it can be the aforementioned commercially available hardware. For example, the reference hardware can be a central processing unit or some widely used hardware devices, including but not limited to artificial intelligence accelerators developed by some manufacturers.

[0060] In step S130, the test model may be trained to convergence on at least the reference hardware. The following briefly describes the model training and convergence process.

[0061] Model training is a process of minimizing the loss function by continuously adjusting the model parameters. Before training begins, the network needs to be initialized and the model parameters (such as weights and biases) are given initial values. Next, the training iterative process begins. In one iteration, the training data is first input into the neural network; then forward propagation is performed so that the data starts from the input layer, passes through the hidden layer calculation, and finally reaches the output layer to obtain the prediction result; then the prediction result is compared with the true label, and the loss value is calculated using a pre-defined loss function (such as mean square error, cross entropy, etc.); then, based on the loss value, the gradient of each parameter with respect to the loss is calculated through the back propagation algorithm; finally, the optimization algorithm is used to update the parameters according to the gradient to reduce the loss.

[0062] During the training process, the model continuously adjusts parameters to gradually reduce the loss. When the loss no longer decreases significantly or reaches the predetermined stopping condition, the model is considered to have converged.

[0063] If the accuracy of the first operator is low, the test model may not be able to accurately learn the features in the data due to inaccurate calculations, resulting in increased errors in the prediction results and reduced accuracy. It is also easy to cause unstable model training and difficulty converging to a better state. Therefore, the convergence of the test model can also reflect whether the accuracy level of the first operator is up to standard. After obtaining the convergence of the test model on the reference hardware and the hardware to be tested, step S140 can be executed.

[0064] It should be noted that in step S130, the test model may not converge on the hardware to be tested. In this case, it can be directly determined that the accuracy of the first operator on the hardware to be tested does not meet the standard. That is, step S140 can be executed without waiting until the test model is trained on the hardware to be tested, so that verification can be completed in advance and additional resource waste can be reduced.

[0065] If the test model can be trained to convergence on the hardware to be tested, then by comparing the convergence of the test model on the hardware to be tested and on the reference hardware, it is possible to determine whether the accuracy of the first operator on the hardware to be tested meets the requirements. It is also possible to determine whether the accuracy of the first operator on the hardware to be tested is higher or lower than that on the reference hardware, thereby avoiding misjudgment and improving the reliability of accuracy verification.

[0066] In the embodiments of the present disclosure, the convergence situation can be judged manually or automatically through convergence information. Here, the convergence information may include the convergence speed, the final converged loss value, the stability of the convergence, the change trend of the loss value, etc. For example, the convergence speed can be understood as the number of iterations required for the loss value to drop to a certain level. A faster convergence speed means that the test model can achieve better performance in a shorter time. For example, the final converged loss value reflects the performance of the test model at the end of training. The lower the final converged loss value, the higher the performance of the test model; the stability of the convergence refers to the fluctuation of the loss value when it is close to convergence. Smaller fluctuations mean that the test model converges more stably; the change trend of the loss value can also reflect the training status of the test model. The stable and continuous decline in the loss value usually indicates that the test model is improving and learning. These convergence information can help determine whether the accuracy of the first operator meets the standard.

[0067] In the process of training the test model, it is usually necessary to go through multiple iterations to gradually optimize and stabilize the model. If only the convergence information of a single iteration or the last iteration is saved, there may be randomness or inaccuracy due to overfitting. For example, in at least one example of the embodiments of the present disclosure, the reference hardware can save the convergence information of the entire iteration stage during the process of training the test model to convergence, so that the accuracy level of the first operator can be judged more comprehensively and reliably.

[0068] For example, in at least one example of the embodiments of the present disclosure, the convergence condition includes at least one of the convergence speed and the final convergence loss value. For example, it is possible to judge whether the accuracy of the first operator on the hardware to be tested meets the standard only by the convergence speed of the test model on the reference hardware and the hardware to be tested, or it is possible to judge whether the accuracy of the first operator on the hardware to be tested meets the standard only by the final convergence loss value of the test model on the reference hardware and the hardware to be tested, or it is possible to judge whether the accuracy of the first operator on the hardware to be tested meets the standard by combining the convergence speed and the final convergence loss value. Alternatively, it is also possible to comprehensively determine the accuracy of the first operator on the hardware to be tested in combination with other convergence information such as the stability of convergence and the changing trend of the loss value, so as to more accurately and comprehensively judge whether the accuracy of the first operator on the hardware to be tested meets the standard.

[0069] For example, in at least one example of the embodiments of the present disclosure, step S140, i.e., determining whether the accuracy of the first operator on the hardware to be tested meets the standard based on the convergence of the test model on the reference hardware and the hardware to be tested, may include a specific example of: in response to the loss values ​​of the final convergence of the test model on the reference hardware and the hardware to be tested respectively meeting the first preset condition, determining that the accuracy of the first operator on the hardware to be tested meets the standard. For example, the first preset condition here may be that the first loss value of the final convergence of the test model on the hardware to be tested is less than or equal to the second loss value of the final convergence of the test model on the reference hardware, or that the ratio of the first loss value to the second loss value is less than a preset convergence loss ratio (e.g., 1 or 1.1 or other positive numbers). For example, in a specific example, the first loss value is slightly greater than the second loss value, which may also be considered to meet the first preset condition.

[0070] For example, if the final convergence loss value of the test model on the hardware to be tested is significantly larger, it can be determined that the accuracy of the first operator on the hardware to be tested does not meet the standard; if the final convergence loss value of the test model on the hardware to be tested is smaller, it means that the accuracy of the first operator on the hardware to be tested not only meets the standard, but is also higher than the accuracy of the corresponding operator on the reference hardware.

[0071] For example, in at least one example of the embodiments of the present disclosure, step S140, i.e., determining whether the accuracy of the first operator on the hardware to be tested meets the requirements based on the convergence of the test model on the reference hardware and the hardware to be tested, may include a specific example of: in response to the final convergence loss value of the test model on the reference hardware and the hardware to be tested respectively meeting a first preset condition, and the convergence speed of the test model on the reference hardware and the hardware to be tested respectively meeting a second preset condition, determining that the accuracy of the first operator on the hardware to be tested meets the requirements.

[0072] For example, the second preset condition here may be that the convergence speed of the test model on the hardware to be tested is greater than or equal to the convergence speed of the test model on the reference hardware, or that the ratio of the convergence speed of the test model on the hardware to be tested to the convergence speed of the test model on the reference hardware is greater than a preset convergence speed ratio (e.g., 0.9 or 1 or other positive numbers). For example, in a specific example, the convergence speed of the test model on the hardware to be tested is slightly slower than the convergence speed on the reference hardware, which may also be considered to meet the second preset condition.

[0073] For example, after determining that the first preset condition is met, the convergence speed of the test model on the hardware to be tested and the reference hardware can be obtained, and then it can be determined whether the second preset condition is met. If both the first preset condition and the second preset condition are met, it can be determined that the accuracy of the first operator on the hardware to be tested meets the standard. In other words, on the basis of determining that the final converged loss value of the test model on the hardware to be tested meets the first preset condition, it can be further determined whether the convergence speed of the test model on the hardware to be tested meets the second preset condition. By combining the final converged loss value and the convergence speed to comprehensively verify the accuracy of the first operator, the accuracy of the first operator can be better reflected, which is conducive to improving the reliability of verification.

[0074] Since the convergence stability and convergence speed of the test model can often better reflect the operator accuracy, and the higher the operator accuracy, the better the network convergence stability and convergence speed will usually be, therefore, by constructing a test model with the first operator added and observing the convergence of the network of the test model on the reference hardware and the hardware to be tested, it is possible to more accurately and reliably determine whether the accuracy of the first operator under the new hardware system of the hardware to be tested is higher or lower than that of the reference hardware.

[0075] Figure 2 A flowchart of another operator accuracy verification method provided by at least one embodiment of the present disclosure. Figure 1 As shown, another operator accuracy verification method provided by at least one embodiment of the present disclosure includes the following steps 210 to S250.

[0076] Step S210: Obtain the first operator currently to be verified.

[0077] Step S220: Obtain parameter information of input data corresponding to the first operator.

[0078] Step S230: Based on the first operator and the parameter information, a first sub-model for training on the reference hardware and a second sub-model for training on the hardware to be tested are respectively obtained through the basic model.

[0079] Step S240: training the first sub-model by using the reference hardware to obtain the convergence of the first sub-model on the reference hardware, and training the second sub-model by using the hardware to be tested to obtain the convergence of the second sub-model on the hardware to be tested.

[0080] Step S250: Compare the convergence of the first sub-model on the reference hardware with the convergence of the second sub-model on the hardware to be tested, and determine whether the accuracy of the first operator on the hardware to be tested meets the requirements.

[0081] For example, in step S210 and step S220, the parameter information of the first operator to be verified and the input data corresponding to the first operator can be automatically obtained through the accuracy verification framework.

[0082] For example, in at least one example of the embodiments of the present disclosure, the parameter information of the input data corresponding to the first operator includes at least one of the quantity of input data, the dimension of input data, the shape of input data, and the type of input data.

[0083] For example, in a specific example, the first operator is an addition operator, and its corresponding input data can be A+B or A+B+C, that is, the number of input data of the first operator can be 2, 3 or more.

[0084] For example, the input data dimension can be zero, one, two, three, or higher. In addition to zero dimension, one, two, or higher dimensional data also has a shape.

[0085] For example, the input data may be represented in the form of a tensor, and a specific value in the tensor may be application data, such as image data, natural language data, etc. For example, a specific value in a two-dimensional tensor may represent a grayscale value of a pixel in image data corresponding to a certain monochrome image.

[0086] For example, a zero-dimensional tensor can be understood as a scalar, such as the values ​​2, 3, 5, etc. Zero-dimensional tensors can be mainly used to represent model losses and various measurement indicators, such as accuracy, precision, etc. A one-dimensional tensor can be understood as a vector. For example, a one-dimensional tensor with a shape of 3 includes the values ​​0, 3, and 20, which can be represented as [0,3,20]. A two-dimensional tensor can be understood as a matrix. For example, a two-dimensional tensor with a shape of 2×2 includes the values ​​2, 3, 1, and 5, which can be represented as a 2-row, 2-column matrix [[2,3],[1,5]]. A two-dimensional tensor with a shape of 3×4 can be understood as a 3-row, 4-column matrix. For example, if tensor A includes the values ​​1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12, it can be represented as [ [1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11,12]].

[0087] For example, a three-dimensional tensor can be understood as a copy of multiple two-dimensional matrices. Here, copying refers to copying the shape, not the value. For example, tensor B is a three-dimensional tensor with a shape of 2×3×4. Tensor B can be understood as a combination of two matrices with the same shape and 2 rows and 3 columns. For example, these two matrices include the values ​​1, 2, 3, 4, 5, 6 and the values ​​7, 8, 9, 10, 11, 12 respectively. Tensor B can be expressed as [[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]].

[0088] Tensors can also have more dimensions, such as four-dimensional tensors, five-dimensional tensors, seven-dimensional tensors, and so on. Four-dimensional tensors can be mainly used to represent image data. For example, a four-dimensional tensor has four dimensions, and the size of each dimension is 2, 3, 4, and 4 respectively. This four-dimensional tensor with a shape of 2×3×4×4 can be understood as 2 pictures, each picture has three colors of RGB, that is, it has 3 channels, and the picture of each channel has 4 rows and 4 columns of pixels, that is, the batch, channel, height, and width of the picture can correspond to the four dimensions of the four-dimensional tensor respectively. The description of high-dimensional tensors above the four-dimensional tensor will not be repeated here one by one.

[0089] For example, the input data type includes a numeric type, a character type, or other types. Taking the numeric type as an example, the input data may be an integer or a floating point number, such as a single-precision floating point number (FP32), a half-precision floating point number (FP16), or other floating point numbers (for example, BF16, etc.).

[0090] The input data and parameter information corresponding to the first operator are different, and the verification results of its accuracy on the hardware to be tested may be different. For example, if the data type of the input data is 16 bits, and the hardware to be tested uses 20 bits to store the input data, then due to the addition of 4 bits to store the input data, the accuracy of the first operator is higher, the calculation result is relatively more accurate, and it can usually meet the actual needs, and the accuracy of the first operator will be verified as up to standard; if the input data is 32 bits, and the hardware to be tested uses 20 bits to receive the input data, then due to the low accuracy of the first operator, it may affect the convergence of the test model, and the accuracy of the first operator will be verified as not up to standard. For example, when the first operator calculates a small number of input data, its accuracy deviation may be small, but when calculating a large number of input data, its accuracy deviation may be large. For example, when the first operator calculates input data with a small dimension, its accuracy deviation may be small, but when calculating input data with a large dimension, its accuracy deviation may be large. For example, when the first operator calculates input data with a small shape, its accuracy deviation may be small, but when calculating input data with a large shape, its accuracy deviation may be large.

[0091] Therefore, when verifying the accuracy of the first operator on the hardware to be tested, it is necessary to consider the accuracy of the first operator in different scenarios. For example, in at least one example of the embodiments of the present disclosure, step S220, i.e., obtaining parameter information of the input data corresponding to the first operator, a specific example thereof may include: obtaining parameter information of the input data of the first operator in multiple scenarios, wherein the input data in multiple scenarios are different and the parameter information is the same, or the input data in multiple scenarios are the same and the parameter information is different, or both the input data and the parameter information in multiple scenarios are different. By verifying the accuracy of the first operator in multiple different scenarios, the reliability and accuracy of the verification can be improved.

[0092] For example, in at least one example of the embodiments of the present disclosure, step S210, i.e., obtaining the first operator currently to be verified, may include a specific example of: determining the first operator currently to be verified from multiple operators to be verified obtained in batches. For example, in at least one example of the embodiments of the present disclosure, after verifying whether the accuracy of the first operator in various scenarios is up to standard, it is also possible to continue to automatically verify whether the accuracy of other operators in the multiple operators to be verified is up to standard. The parameter information of multiple operators to be verified and their corresponding input data in various scenarios can be batch-inputted into the accuracy verification framework, for example, in the form of a list. The accuracy verification framework can automatically verify the accuracy of the multiple operators to be verified in the order of the multiple operators to be verified in the list, so that the hardware to be tested can continuously perform the accuracy verification work of the first operator, thereby accelerating the development progress of the operator.

[0093] For example, in step S230, the base model and the first operator may be fused to obtain a first sub-model for training on the reference hardware and a second sub-model for training on the hardware to be tested, respectively. For example, in at least one example of the embodiments of the present disclosure, the first sub-model is fused with a first code for implementing a first operator provided by a manufacturer of the reference hardware, such as a first operator included in an operator library provided by the manufacturer.

[0094] For example, the input data and parameter information corresponding to the first operator can be input into the accuracy verification framework, and the function pointer of the first operator and the function pointer corresponding to the first operator on the reference hardware can be input into the accuracy verification framework at the same time. According to the parameter information of the input data corresponding to the first operator, the basic model in the accuracy verification framework automatically generates a second sub-model that integrates the first operator and a first sub-model that integrates the first code for implementing the first operator provided by the manufacturer of the reference hardware.

[0095] Here, the first operator may be an operator that has been developed or a newly developed operator. For example, if the first operator is a newly developed operator for implementing the first operation, a second operator provided by a reference hardware manufacturer may be selected as a reference, and the second operator may implement the first operation.

[0096] For example, the first operator can be a single operator or a composite operator bound to multiple operators. For example, the first operator can be a composite operator bound to a convolution operator (conv), a batch normalization operator (batchnorm), and a rectified linear unit operator (relu), which can combine these operators to implement corresponding calculation operations. Precision verification of composite operators can reduce the number of training times, eliminate the need to verify the precision of multiple discrete operators multiple times, and improve verification speed.

[0097] In an embodiment of the present disclosure, when the accuracy of multiple operators to be verified is automatically verified through the accuracy verification framework, since the input dimension of the basic model is fixed, but the dimension of the input data of multiple operators to be verified may be different from the input dimension of the basic model, it is necessary to reserve some basic operators to ensure that the basic model can fuse the first operator to generate a second sub-model or a first sub-model that can be trained normally. For example, in one example, the basic model is a residual network, and the dimension of its input data is usually four-dimensional. If the input data is two-dimensional, three-dimensional, or five-dimensional, it is necessary to first transform the input data into four-dimensional data through a shape conversion operator before it can be input into the generated test model (including the first sub-model and the second sub-model).

[0098] For example, in at least one example of the embodiments of the present disclosure, the basic model includes at least one of the following operators: a shape transformation type operator for processing the input data corresponding to the first operator; a feature extraction type operator for extracting features from the input data; and a loss calculation type operator for calculating the loss value after each iteration.

[0099] For example, shape transformation operators include, but are not limited to, reshape operators, transpose operators, expand operators, squeeze operators, and unsqueeze operators. For example, the reshape operator can reorganize the shape of a tensor, such as reshaping a tensor with a shape of [2, 4] into a tensor with a shape of [4, 2] or [1, 8]. For example, the transpose operator can swap two specified dimensions of a tensor, such as converting a tensor with a shape of [2, 3, 4] into a tensor with a shape of [2, 4, 3]. For example, the expand operator can expand the shape of a tensor to a larger dimension, such as expanding a tensor with a shape of [3,4] to [2, 3, 4] by repeating elements, where the first dimension is repeated twice. For example, the squeeze operator can remove all dimensions of size 1 from a tensor. For example, the unsqueeze operator can add a new dimension to a tensor, such as inserting a dimension of size 1 at a specified position.

[0100] For example, the feature extraction operator is mainly used to ensure that the loss value of the test model can be normally and stably decreased and converged. For example, the loss calculation operator can be the CrossEntropyLoss operator, which is used to ensure that the loss value calculation can be performed normally.

[0101] For example, the entire architecture of the accuracy verification framework can be deployed on a central processing unit, and the central processing unit executes the operator accuracy verification method provided by the embodiment of the present disclosure.

[0102] For example, in at least one example of the embodiments of the present disclosure, the basic model may further include a normalization operator, such as a layernorm operator, an rmsnorm operator, etc. When the first operator is added to the basic model, the first operator may have a numerical impact on the basic model. In this case, it may be preset so that the first operator is located before the normalization operator, thereby solving the problem of the change in the numerical range due to the addition of the first operator through the normalization operator, and improving the stability of the generated test model.

[0103] In step S240, the first sub-model can be trained on the reference hardware until convergence. For example, if the second sub-model cannot converge on the hardware to be tested, it can be directly determined that the accuracy of the first operator on the hardware to be tested does not meet the standard, that is, step S250 can be executed without waiting for the hardware to be tested to train the second sub-model, so that verification can be completed in advance, reducing additional resource waste.

[0104] For example, if the second sub-model can be trained to convergence on the hardware to be tested, then the convergence of the second sub-model on the hardware to be tested can be compared with the convergence of the first sub-model on the reference hardware, thereby judging whether the accuracy of the first operator on the hardware to be tested meets the requirements, and also determining whether the accuracy of the first operator on the hardware to be tested is higher or lower than that on the reference hardware, thereby avoiding misjudgment and improving the reliability of accuracy verification.

[0105] For example, in at least one example of the embodiments of the present disclosure, step S250, i.e., comparing the convergence of the first sub-model on the reference hardware with the convergence of the second sub-model on the hardware to be tested to determine whether the accuracy of the first operator on the hardware to be tested meets the requirements, may include: in response to a first loss value at which the second sub-model finally converges on the hardware to be tested being less than or equal to a second loss value at which the first sub-model finally converges on the reference hardware, determining that the accuracy of the first operator on the hardware to be tested meets the requirements; or in response to a ratio of the first loss value at which the second sub-model finally converges on the hardware to be tested to the second loss value at which the first sub-model finally converges on the reference hardware being less than a preset convergence loss ratio, determining that the accuracy of the first operator on the hardware to be tested meets the requirements.

[0106] For example, other specific descriptions of step S240 and step S250 can refer to the above Figure 1 The detailed description of step S130 and step S140 in FIG. 1 is omitted here.

[0107] For example, in at least one example of the embodiments of the present disclosure, the test model is a deep learning model obtained based on the reference model, and the scale of the test model is smaller than the reference model. For example, the test model can be a small model, so as to increase the speed of verifying the accuracy of the first operator and save time cost.

[0108] For example, in at least one example of the embodiments of the present disclosure, the test model is obtained by intercepting a portion of the reference model, or the test model is reconstructed based on the reference model.

[0109] For example, a small network can be obtained by intercepting an existing ResNet network or MLP network. For example, a ResNet network includes dozens of network layers, and several of the network layers can be intercepted to construct a small deep learning model as a test model. For example, a small network can also be obtained by other means, such as constructing a small deep learning model through model compression and other technologies, and the embodiments of the present disclosure are not limited to this.

[0110] In the embodiments of the present disclosure, the hardware to be tested and the reference hardware can be located on the same device or on different devices. The two can be the same type of hardware or different types of hardware. The order of relevant steps in the operator accuracy verification method can be flexibly adjusted according to the environment in which the hardware to be tested and the reference hardware are located, thereby improving the convenience and universality of the operator accuracy verification method.

[0111] For example, in at least one example of the embodiments of the present disclosure, in response to the reference hardware being a reference central processor located on the same device as the hardware to be tested, the reference hardware trains the test model prior to the hardware to be tested.

[0112] Figure 3 A schematic diagram of reference hardware and hardware to be tested located on the same device is provided for at least one embodiment of the present disclosure. Figure 3 As shown, the reference hardware and the hardware to be tested are both located on device 1, and the reference hardware is, for example, a reference central processing unit (CPU) on device 1. For example, the test model can be first trained by the CPU until convergence is stable, and the convergence information of the entire iteration stage can be saved, and then the test model can be trained by the hardware to be tested. In response to the failure of the hardware to be tested to reach convergence when training the test model to the first stage, it is directly determined that the accuracy of the first operator on the hardware to be tested does not meet the standard. Here, the first stage refers to the stage where the test model can reach convergence on the CPU.

[0113] For example, if automatic verification is performed through the precision verification framework, the precision verification framework can first fuse the CPU operator (corresponding to the first operator) pointer into the basic model to obtain the first sub-model, and continuously iterate the training of the first sub-model until the loss value converges and stabilizes, and finally saves the convergence information of the entire iteration stage. After the CPU training is completed, the precision verification framework replaces the CPU operator with the first operator function pointer and fuses it into the basic model to obtain the second sub-model, and trains the second sub-model from scratch. If the second sub-model is still difficult to converge when it reaches the step where the CPU operator converges, it is directly determined that the accuracy of the first operator on the hardware to be tested does not meet the standard.

[0114] Since actual devices usually have a CPU, using the CPU as reference hardware can directly complete the accuracy verification of the first operator on a device, which is relatively simple to operate.

[0115] For example, in at least one example of the embodiments of the present disclosure, in response to the reference hardware being a reference artificial intelligence accelerator located on a different device from the hardware to be tested, the reference hardware trains a test model in parallel with the hardware to be tested.

[0116] Figure 4 A schematic diagram of reference hardware and hardware to be tested located on different devices provided for at least one embodiment of the present disclosure. Figure 4 As shown, the reference hardware and the hardware to be tested are located on different devices, for example, the reference hardware is located on device 2, and the hardware to be tested is located on device 3, and both device 2 and device 3 have their own CPUs. For example, the reference hardware and the hardware to be tested can both be artificial intelligence accelerators, for example, the reference hardware can be a graphics processing unit (GPU), a general-purpose computing graphics processing unit (GPGPU), a tensor processing unit (TPU), a neural network processor (Neural network Processing Unit, NPU), a deep learning processor (Deep learning Processing Unit, DPU), an accelerated processor (AcceleratedProcessing Unit, APU), etc., and correspondingly, the hardware to be tested can be a newly developed GPU, GPGPU, TPU, NPU, DPU or APU, etc. When the reference hardware and the hardware to be tested are located on different devices, both can train the test model at the same time. The processing speed of the GPU is usually faster than that of the CPU, so this method can greatly improve the verification speed and reduce the waiting time.

[0117] For example, the test model can be trained manually on the reference hardware and the hardware to be tested respectively, and then the convergence of the test model on the reference hardware and the hardware to be tested can be collected or observed respectively. Then, based on the convergence of the two, it can be determined whether the accuracy of the first operator meets the requirements.

[0118] Figure 5 A schematic diagram of another reference hardware and hardware to be tested located on the same device provided for at least one embodiment of the present disclosure. Figure 5 As shown, the reference hardware and the hardware to be tested can be located on the same device 4. Both of them are, for example, artificial intelligence accelerators. The reference hardware and the hardware to be tested can train test models in parallel, thereby increasing the verification speed and reducing waiting time.

[0119] For example, by deploying the accuracy verification framework on the CPU of device 4, the reference hardware and the hardware to be tested can automatically train the test model at the same time, and directly determine whether the accuracy of the first operator meets the standard based on the convergence of the test model on the reference hardware and the hardware to be tested.

[0120] The accuracy verification method provided by at least one embodiment of the present disclosure can accurately determine whether the accuracy of the first operator on the hardware to be tested meets the standard by comparing the convergence of the test model obtained based on the first operator to be verified on the reference hardware and the hardware to be tested. It is easy to operate and highly reliable, does not need to set relative error thresholds and absolute error thresholds for different types of operators respectively, has universality, and can also quickly and automatically verify the accuracy of a large number of operators on new hardware, thereby effectively reducing time and labor costs and accelerating the development progress of operators and hardware to be tested.

[0121] At least one embodiment of the present disclosure further provides an operator accuracy verification system. Figure 6 The structure diagram of an operator accuracy verification system provided by at least one embodiment of the present disclosure is as follows. Figure 6 As shown, an operator precision verification system 600 provided by at least one embodiment of the present disclosure includes an acquisition module 601 , a verification module 602 and a determination module 603 .

[0122] For example, the acquisition module 601 is configured to acquire the first operator currently to be verified and obtain a test model based on the first operator.

[0123] For example, the verification module 602 is configured to train the test model through the reference hardware and the hardware to be tested respectively to obtain the convergence of the test model on the reference hardware and the hardware to be tested, wherein the test model is trained to convergence at least on the reference hardware.

[0124] For example, the determination module 603 is configured to determine whether the accuracy of the first operator on the hardware to be tested meets the requirements according to the convergence of the test model on the reference hardware and the hardware to be tested.

[0125] For example, in at least one embodiment of the present disclosure, the acquisition module 601 is also configured to acquire parameter information of input data corresponding to the first operator, and based on the first operator and the parameter information, obtain a first sub-model for training on reference hardware and a second sub-model for training on the hardware to be tested through the basic model; the verification module 602 is also configured to train the first sub-model through the reference hardware to obtain the convergence of the first sub-model on the reference hardware, and train the second sub-model through the hardware to be tested to obtain the convergence of the second sub-model on the hardware to be tested; the determination module 603 is also configured to compare the convergence of the first sub-model on the reference hardware with the convergence of the second sub-model on the hardware to be tested, and determine whether the accuracy of the first operator on the hardware to be tested meets the requirements.

[0126] The acquisition module 601, the verification module 602 and the determination module 603 are respectively used to implement Figure 1 Steps S110 to S120, S130 and S140 shown in the figure can also be implemented separately Figure 2 Therefore, for the specific description of the acquisition module 601, the verification module 602 and the determination module 603, reference can be made to the relevant description in the embodiment of the above-mentioned algorithm accuracy verification method, which will not be repeated here.

[0127] The operator accuracy verification system 600 can achieve technical effects similar to those of the aforementioned algorithm accuracy verification method, which will not be described in detail here.

[0128] At least one embodiment of the present disclosure further provides an electronic device, comprising at least one memory and at least one processor, wherein the memory non-transiently stores computer executable instructions, and the processor is configured to run the computer executable instructions, wherein the computer executable instructions are executed by the processor to implement the operator precision verification method as in any of the above embodiments. The technical effect of the electronic device is the same as the technical effect of the operator precision verification method described above, and will not be repeated here.

[0129] At least one embodiment of the present disclosure also provides a non-transitory computer-readable storage medium. Figure 7 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Figure 7 As shown, one or more computer executable instructions 701 may be non-temporarily stored on a non-transitory computer readable storage medium 700. For example, when the computer executable instructions 701 are executed by a processor, one or more steps in the operator precision verification method described above may be executed. The technical effect of the non-transitory computer readable storage medium is the same as the technical effect of the operator precision verification method described above, and will not be repeated here.

[0130] For example, the non-transient computer-readable storage medium is implemented as a memory, such as a volatile memory and / or a non-volatile memory. In the above embodiment, the memory may be a volatile memory, such as a random access memory (RAM) and / or a cache, etc. Non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, a flash memory, etc. Various applications (codes, instructions) and data, as well as various data used and / or generated by the application, etc. may also be stored in the memory.

[0131] The non-transitory computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method of the present disclosure.

[0132] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0133] Some embodiments of the present disclosure further provide an electronic device, which includes the operator accuracy verification system of any of the above embodiments or can execute the operator accuracy verification method of any of the above embodiments.

[0134] Figure 8 A schematic diagram of the structure of an electronic device provided for at least one embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0135] For example, Figure 8As shown, in some examples, the electronic device 800 includes a processing device (such as a central processing unit, a graphics processing unit, etc.) 801, which can include an operator accuracy verification system of any of the above embodiments, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 to a random access memory (RAM) 803. In RAM 803, various programs and data required for the operation of the computer system are also stored. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0136] For example, the following components may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809 which may also include, for example, a network interface card such as a LAN card, a modem, etc. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data, and perform communication processing via a network such as the Internet. The drive 810 is also connected to the I / O interface 805 as needed. A removable storage medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom can be installed into the storage device 808 as needed. Although Figure 8 The electronic device 800 is shown to include various devices, but it should be understood that it is not required to implement or include all of the devices shown, and more or fewer devices may be implemented or included instead.

[0137] For example, the electronic device 800 may further include a peripheral interface (not shown in the figure), etc. The peripheral interface may be various types of interfaces, such as a USB interface, a lightning interface, etc. The communication device 809 may communicate with a network and other devices through wireless communication, such as the Internet, an intranet and / or a wireless network such as a cellular phone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN). Wireless communication may use any of a variety of communication standards, protocols, and techniques, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0138] For example, the electronic device 800 may be any device such as a mobile phone, a tablet computer, a laptop computer, an e-book, a game console, a television, a digital photo frame, a navigator, etc., or may be any combination of data processing devices and hardware, which is not limited in the embodiments of the present disclosure.

[0139] Although the disclosure has been described in detail above with general descriptions and specific implementation methods, it is obvious to those skilled in the art that some modifications or improvements may be made to the embodiments of the disclosure. Therefore, these modifications or improvements made without departing from the spirit of the disclosure are within the scope of protection claimed by the disclosure.

[0140] In addition to the above exemplary contents, the following points need to be explained in this disclosure:

[0141] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to the general design.

[0142] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present disclosure, the thickness of layers or regions is enlarged or reduced, that is, these drawings are not drawn according to the actual scale.

[0143] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0144] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. An operator accuracy verification method, comprising: Get the first operator to be verified; Obtaining a test model based on the first operator; The test model is trained by using reference hardware and the hardware to be tested respectively to obtain convergence of the test model on the reference hardware and the hardware to be tested, wherein the test model is trained to convergence on at least the reference hardware, and in the process of training the test model, the parameters of the test model are adjusted so that the loss gradually decreases, and in response to the loss no longer decreasing or reaching a predetermined stop condition, it is determined that the test model converges; According to the convergence of the test model on the reference hardware and the hardware to be tested, determine whether the accuracy of the first operator on the hardware to be tested meets the standard, wherein the convergence status is determined by convergence information, and the convergence information includes at least one of the convergence speed, the final converged loss value, the stability of the convergence, and the changing trend of the loss value.

2. The operator accuracy verification method according to claim 1, further comprising: Obtaining parameter information of input data corresponding to the first operator; The step of obtaining a test model based on the first operator includes: Based on the first operator and the parameter information, respectively obtaining a first sub-model for training on the reference hardware and a second sub-model for training on the hardware to be tested through a basic model; The step of training the test model through reference hardware and the hardware to be tested respectively to obtain convergence conditions of the test model on the reference hardware and the hardware to be tested includes: Training the first sub-model through the reference hardware to obtain a convergence condition of the first sub-model on the reference hardware, and training the second sub-model through the hardware to be tested to obtain a convergence condition of the second sub-model on the hardware to be tested; The determining, according to the convergence of the test model on the reference hardware and the hardware to be tested, whether the accuracy of the first operator on the hardware to be tested meets the requirements includes: Compare the convergence of the first sub-model on the reference hardware with the convergence of the second sub-model on the hardware to be tested, and determine whether the accuracy of the first operator on the hardware to be tested meets the requirements.

3. The operator accuracy verification method according to claim 2, wherein: The first sub-model incorporates a first code for implementing the first operator provided by a manufacturer of the reference hardware.

4. The operator accuracy verification method according to claim 2, wherein: The basic model includes at least one of the following operators: A shape conversion operator for processing the input data corresponding to the first operator; A feature extraction operator for extracting features from the input data; as well as The loss calculation operator is used to calculate the loss value after each iteration.

5. The operator accuracy verification method according to any one of claims 1 to 4, wherein: The reference hardware saves convergence information of the entire iteration phase during the process of training the test model to convergence.

6. The operator accuracy verification method according to any one of claims 1 to 4, wherein: In response to the reference hardware being a reference central processor located on the same device as the hardware to be tested, the reference hardware trains the test model prior to the hardware to be tested.

7. The operator accuracy verification method according to claim 6, wherein: In response to the hardware to be tested failing to reach convergence when training the test model to the first stage, directly determining that the accuracy of the first operator on the hardware to be tested does not meet the requirements, The test model is trained to convergence by the reference hardware in the first stage.

8. The operator accuracy verification method according to any one of claims 1 to 4, wherein: In response to the reference hardware being a reference artificial intelligence accelerator located on a different device from the hardware to be tested, the reference hardware trains the test model in parallel with the hardware to be tested.

9. The operator accuracy verification method according to any one of claims 1 to 4, wherein: The test model is a deep learning model obtained based on a reference model, and the scale of the test model is smaller than that of the reference model.

10. The operator accuracy verification method according to claim 9, wherein: The test model is obtained by intercepting a portion of the reference model, or the test model is reconstructed based on the reference model.

11. The operator accuracy verification method according to claim 1, wherein: The determining, according to the convergence of the test model on the reference hardware and the hardware to be tested, whether the accuracy of the first operator on the hardware to be tested meets the requirements includes: In response to the final converged loss values ​​of the test model on the reference hardware and the hardware to be tested respectively meeting a first preset condition, determining that the accuracy of the first operator on the hardware to be tested meets the standard; or, In response to the final converged loss values ​​of the test model on the reference hardware and the hardware to be tested respectively meeting a first preset condition, and the convergence speeds of the test model on the reference hardware and the hardware to be tested respectively meeting a second preset condition, it is determined that the accuracy of the first operator on the hardware to be tested meets the requirements.

12. The operator accuracy verification method according to claim 11, wherein: The first preset condition includes: The first loss value at which the test model finally converges on the hardware to be tested is less than or equal to the second loss value at which the test model finally converges on the reference hardware; or A ratio of a first loss value at which the test model finally converges on the hardware to be tested to a second loss value at which the test model finally converges on the reference hardware is less than a preset convergence loss ratio.

13. The operator accuracy verification method according to claim 11, wherein: The second preset condition includes: The convergence speed of the test model on the hardware to be tested is greater than or equal to the convergence speed of the test model on the reference hardware; or A ratio of a convergence speed of the test model on the hardware to be tested to a convergence speed of the test model on the reference hardware is greater than a preset convergence speed ratio.

14. The operator accuracy verification method according to claim 2, wherein: The obtaining of the first operator to be verified currently includes: Determining the first operator currently to be verified from the multiple operators to be verified acquired in batches; The obtaining parameter information of the input data corresponding to the first operator includes: Obtaining parameter information of input data of the first operator in multiple scenarios, wherein the input data in the multiple scenarios are different and the parameter information is the same, or the input data in the multiple scenarios are the same and the parameter information is different, or both the input data and the parameter information in the multiple scenarios are different; The operator accuracy verification method also includes: After verifying whether the accuracy of the first operator in the multiple scenarios meets the requirements, continue to automatically verify whether the accuracy of other operators in the multiple operators to be verified meets the requirements.

15. The operator accuracy verification method according to claim 2, wherein: The parameter information of the input data corresponding to the first operator includes at least one of the quantity of input data, the dimension of input data, the shape of input data and the type of input data.

16. The operator accuracy verification method according to any one of claims 1 to 4, wherein: The first operator includes a composite operator bound to multiple operators.

17. An operator accuracy verification system, comprising: Acquisition module, determination module and verification module, among which, The acquisition module is configured to acquire a first operator currently to be verified and obtain a test model based on the first operator; The verification module is configured to train the test model through reference hardware and hardware to be tested respectively to obtain convergence of the test model on the reference hardware and the hardware to be tested, wherein the test model is trained to convergence on at least the reference hardware, and in the process of training the test model, the parameters of the test model are adjusted so that the loss gradually decreases, and in response to the loss no longer decreasing or reaching a predetermined stop condition, it is determined that the test model converges; and The determination module is configured to determine whether the accuracy of the first operator on the hardware to be tested meets the requirements based on the convergence of the test model on the reference hardware and the hardware to be tested, wherein the convergence is determined by convergence information, and the convergence information includes at least one of the convergence speed, the final converged loss value, the stability of the convergence, and the changing trend of the loss value.

18. The operator accuracy verification system according to claim 17, wherein: The acquisition module is further configured to acquire parameter information of input data corresponding to the first operator, and based on the first operator and the parameter information, obtain a first sub-model for training on the reference hardware and a second sub-model for training on the hardware to be tested through a basic model; The verification module is further configured to train the first sub-model through the reference hardware to obtain a convergence condition of the first sub-model on the reference hardware, and train the second sub-model through the hardware to be tested to obtain a convergence condition of the second sub-model on the hardware to be tested; The determination module is further configured to compare the convergence of the first sub-model on the reference hardware with the convergence of the second sub-model on the hardware to be tested, and determine whether the accuracy of the first operator on the hardware to be tested meets the requirements.

19. An electronic device comprising: at least one memory non-transitorily storing computer executable instructions; at least one processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the operator accuracy verification method according to any one of claims 1-16 is implemented.

20. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, When the computer executable instructions are executed by at least one processor, the operator accuracy verification method according to any one of claims 1 to 16 is implemented.

Images