Data processing method and device, processor, electronic device, and storage medium

Abstract

A data processing method and device, a processor, an electronic device, and a storage medium. The present disclosure relates to the field of digital data processing. The data processing method comprises: performing inference or training operations of a network model by a hardware device, wherein the network model comprises a plurality of operators; for each operator, performing an element-wise precision verification operation on an output tensor of the operator in the inference or training operation to obtain a precision verification result of each operator, wherein the output tensor of the operator comprises a plurality of elements, and the precision verification result comprises a precision verification result of each element; and determining whether the operator meets a precision requirement based on the precision verification result of the operator. The method eliminates the precision paradox of a uniform threshold, avoids a one-size-fits-all verification method, effectively reduces the false positive rate, improves the overall verification completeness, and reduces the invalid debugging time caused by precision false positives.

Description

Technical Field

[0001] This disclosure relates to the field of electronic digital data processing, and specifically to a data processing method, processor, data processing device, electronic device, and non-transitory computer-readable storage medium. Background Technology

[0002] For operators in network models running on chips, a scientific and hierarchical verification scheme is needed to determine whether the operator accuracy is up to standard. Currently, the mainstream verification method is very crude, indiscriminately applying the same fixed error threshold to all operators such as convolution, matrix multiplication, and normalization. This uniform threshold is either too lenient (allowing problematic operators to pass) or too strict (falsely reporting normal operators). This verification method has obvious limitations. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a data processing method, comprising: performing inference or training operations on a network model by a hardware device, wherein the network model includes multiple operators; for each operator, performing an element-wise precision verification operation on the output tensor of the operator in the inference or training operation to obtain a precision verification result for each operator, wherein the output tensor of the operator includes multiple elements, and the precision verification result includes the precision verification result of each element; determining whether the operator meets the precision requirements based on the precision verification result of the operator; wherein the element-wise precision verification operation includes performing precision verification on each element in the output tensor of the operator based on an error threshold and a standard reference value corresponding to the operator, wherein the error threshold corresponding to the operator is related to the category to which the operator belongs, and operators of different categories have independent error thresholds, wherein the categories include a first category, a second category, a third category, a fourth category, a fifth category, a sixth category, a seventh category, and an eighth category, and the first category of operators belonging to the first category includes multiply-accumulate operations. The operators, belonging to the second category, are operators that include reduction and accumulation operations. The error threshold corresponding to the first category of operators is related to at least one of the data block size, accumulator precision, and accumulation execution order when the hardware device performs the inference or training operation. The error threshold corresponding to the second category of operators is related to the precision of the intermediate results generated when the hardware device performs the second category of operators. The third category of operators belonging to the third category are special function operators that use dedicated instructions provided by the hardware device. The fourth category of operators belonging to the fourth category are operators that perform stepwise basic operations. The fifth category of operators belonging to the fifth category includes operators that perform logical operations or extremum operations. The sixth category of operators belonging to the sixth category includes operators that perform shape and size transformations. The seventh category of operators belonging to the seventh category are composite operators, wherein the composite operator is formed by the fusion of multiple sub-operators. The eighth category of operators belonging to the eighth category includes operations in which multiple threads compete for the same address, and the operation relies on hardware atomic operations to resolve the competition.

[0004] For example, in the data processing method provided in at least one embodiment of this disclosure, the error threshold corresponding to the first type of operator is also positively correlated with K, where K represents the size of the accumulation dimension when the first type of operator performs the multiplication-accumulation operation, and K is a positive integer.

[0005] For example, in a data processing method provided in at least one embodiment of this disclosure, the error threshold corresponding to the first type of operator is determined based on a first base threshold and K, wherein the error threshold corresponding to the first type of operator is scaled with K, and the first base threshold is determined based on at least one of the data block size, the accumulator precision, or the accumulation execution order when the hardware device performs the inference or training operation.

[0006] For example, in the data processing method provided in at least one embodiment of this disclosure, the error thresholds corresponding to the first type of operator are distributed in a stepped manner. In response to K being less than a first threshold, the error thresholds corresponding to the first type of operator are constant values. Furthermore, when K belongs to a first range, the error thresholds corresponding to the first type of operator are less than when K belongs to a second range. Here, K is a positive integer and represents the size of the accumulation dimension when the first type of operator performs the multiplication-accumulation operation. The maximum value of the first range is less than the minimum value of the second range, and the maximum value of the second range is less than the first threshold. In response to K being greater than or equal to the first threshold, the error thresholds corresponding to the first type of operator are positively correlated with K.

[0007] For example, in the data processing method provided in at least one embodiment of this disclosure, the error threshold corresponding to the second type of operator is also positively correlated with L, wherein L represents the number of accumulations of the reduction accumulation operation included in the second type of operator, and L is a positive integer; or the error threshold corresponding to the second type of operator is also positively correlated with L and the numerical range of the input tensor of the second type of operator.

[0008] For example, in the data processing method provided in at least one embodiment of this disclosure, the error threshold corresponding to the second type of operator is determined based on a second basic threshold and a first coefficient. The second basic threshold is determined based on the accuracy of the intermediate results generated when the hardware device executes the second type of operator. The error threshold corresponding to the second type of operator is scaled with the first coefficient. The first coefficient is positively correlated with L, or the first coefficient is positively correlated with the product of L and the numerical range of the input tensor of the second type of operator.

[0009] For example, in the data processing method provided in at least one embodiment of this disclosure, the error threshold corresponding to the third type of operator is a first fixed value, and the error threshold corresponding to the fourth type of operator is a second fixed value, wherein the first fixed value is greater than the second fixed value.

[0010] For example, in the data processing method provided in at least one embodiment of this disclosure, an element-wise precision verification operation is performed on the output tensor of the operator in the inference or training operation to obtain the precision verification result of each operator, including: for any one of the first type of operator, the second type of operator, the third type of operator, or the fourth type of operator: determining the error threshold corresponding to the any one type of operator according to the category to which the any one type of operator belongs; determining the error value of the element based on the value of each element in the any one type of operator and the standard reference value corresponding to the element; comparing the error value of each element with the error threshold corresponding to the any one type of operator to obtain the precision verification result of the any one type of operator; and determining whether the operator meets the precision requirements based on the precision verification result of the operator, including: determining that the any one type of operator passes the element-wise precision verification when the error value of each element of the any one type of operator is less than or equal to the error threshold corresponding to the any one type of operator; otherwise, determining that the any one type of operator fails the element-wise precision verification.

[0011] For example, in the data processing method provided in at least one embodiment of this disclosure, performing element-wise precision verification on the output tensor of the operator in the inference or training operation to obtain the precision verification result of each operator includes: for any of the fifth type of operator or the sixth type of operator, performing element-wise bit-level verification on each element of the output tensor of the any type of operator and the standard reference value of the element to obtain the precision verification result of the any type of operator; and determining whether the operator meets the precision requirements based on the precision verification result of the operator includes: when the precision verification result of the any type of operator indicates that each element of the any type of operator is consistent with the standard reference value of the element at the bit level, determining that the any type of operator has passed the element-wise precision verification; otherwise, determining that the any type of operator has not passed the element-wise precision verification.

[0012] For example, in at least one embodiment of the data processing method provided in this disclosure, performing element-wise precision verification on the output tensor of the operator in the inference or training operation to obtain the precision verification result of each operator includes: for the seventh type of operator, performing the element-wise precision verification operation on each of the multiple sub-operators included in the seventh type of operator, and determining whether the operator meets the precision requirements based on the precision verification result of the operator, including: when each sub-operator included in the seventh type of operator passes the element-wise precision verification, determining that the seventh type of operator passes the element-wise precision verification; otherwise, determining that the seventh type of operator fails the element-wise precision verification.

[0013] For example, in the data processing method provided in at least one embodiment of this disclosure, the standard reference value corresponding to the eighth type of operator includes the first mean of the output results obtained by running the eighth type of operator N times by a reference device, where N is a positive integer greater than 1. An element-wise precision verification operation is performed on the output tensor of the operator in the inference or training operation to obtain the precision verification result of each operator, including: executing the eighth type of operator N times to obtain the second mean and variance of the output tensor of the eighth type of operator in the N runs; and determining the corresponding value of the eighth type of operator based on the second mean and the first mean. The error value is compared with the error threshold corresponding to the eighth type of operator, and the variance is compared with the preset variance threshold range to obtain the accuracy verification result of the eighth type of operator; based on the accuracy verification result of the operator, it is determined whether the operator meets the accuracy requirements, including: in response to the error value corresponding to the eighth type of operator being less than or equal to the error threshold corresponding to the eighth type of operator, and the variance being within the preset variance threshold range, it is determined that the eighth type of operator passes the element-by-element accuracy verification; otherwise, it is determined that the eighth type of operator fails the element-by-element accuracy verification.

[0014] For example, at least one embodiment of the data processing method provided in this disclosure further includes: determining a global error index for each operator based on the output tensor of each operator, wherein the global error index includes a first index for evaluating the overall deviation level of the operator, a second index for evaluating the error dispersion of the operator, and a third index for evaluating the systematic deviation direction of the operator; determining whether the operator meets the accuracy requirements based on the accuracy verification results of the operator, including: determining whether the operator meets the accuracy requirements based on the accuracy verification results of the operator and the global error index.

[0015] For example, in the data processing method provided in at least one embodiment of this disclosure, the first index of each operator is the mean of the relative errors of the multiple elements included in the operator, and the first index is determined as follows: diff1 =

[0016] =

[0017] Where, diff1 represents the first index, and T represents the total number of the plurality of elements. This represents the relative error of the i-th element among the plurality of elements, where i takes values ​​of 1, 2, ..., T. To represent a positive number indicating the first preset threshold, This represents the value of the i-th element. This represents the standard reference value corresponding to the i-th element.

[0018] For example, in the data processing method provided in at least one embodiment of this disclosure, the second index of each operator is the root mean square of the relative errors of the plurality of elements included in the operator.

[0019] For example, in the data processing method provided in at least one embodiment of this disclosure, the third index of each operator is the ratio between the sum of the error values ​​of the plurality of elements included in the operator and the first product, wherein the first product is the product of the total number of the plurality of elements and the maximum absolute value of the standard reference value among the plurality of elements; in response to the third index of the operator being within a preset range including zero values, it indicates that the operator has no systematic bias; in response to the third index of the operator being greater than the maximum value of the preset range, it indicates that the operator has a systematic positive bias; in response to the third index of the operator being less than the minimum value of the preset range, it indicates that the operator has a systematic negative bias.

[0020] For example, in a data processing method provided in at least one embodiment of this disclosure, determining whether the operator meets the accuracy requirements based on the accuracy verification result of the operator and the global error index includes: in response to the accuracy verification result of the operator indicating that the operator has passed element-by-element accuracy verification, the first index of the operator is less than or equal to a second preset threshold, the second index of the operator is less than a preset multiple of the first index, and the third index of the operator is within a preset range including zero values, determining that the operator meets the accuracy requirements; otherwise, determining that the operator does not meet the accuracy requirements.

[0021] For example, at least one embodiment of the data processing method provided in this disclosure further includes: performing performance verification on an operator that meets the accuracy requirements; and determining that the operator is qualified in response to the operator that meets the accuracy requirements passing the performance verification; wherein the performance verification includes verifying whether the computational efficiency and interface efficiency of the operator meet the performance requirements.

[0022] At least one embodiment of this disclosure provides a data processing apparatus, comprising: an operation module configured to perform inference or training operations on a network model by a hardware device, wherein the network model includes multiple operators; an element-wise precision verification module configured to perform element-wise precision verification on the output tensor of each operator in the inference or training operation, obtaining a precision verification result for each operator, wherein the output tensor of the operator includes multiple elements, and the precision verification result includes the precision verification result of each element; and a precision verification module configured to determine whether the operator meets precision requirements based on the precision verification result of the operator; wherein the element-wise precision verification operation includes performing precision verification on each element in the output tensor of the operator based on an error threshold and a standard reference value corresponding to the operator, wherein the error threshold corresponding to the operator is related to the category to which the operator belongs, and operators of different categories have independent error thresholds, the categories including a first category, a second category, a third category, a fourth category, a fifth category, a sixth category, a seventh category, and an eighth category, belonging to... The first type of operator in the first category includes operators that include multiply-accumulate operations; the second type of operator in the second category includes operators that include reduce-accumulate operations; the error threshold corresponding to the first type of operator is related to at least one of the data block size, accumulator precision, and accumulation execution order when the hardware device performs the inference or training operation; the error threshold corresponding to the second type of operator is related to the precision of the intermediate results generated when the hardware device performs the second type of operator; the third type of operator in the third category is a special function operator using dedicated instructions; the fourth type of operator in the fourth category is an operator that performs stepwise basic operations; the fifth type of operator in the fifth category includes operators that perform logical operations or extremum operations; the sixth type of operator in the sixth category includes operators that perform shape and size transformations; the seventh type of operator in the seventh category is a composite operator, wherein the composite operator is formed by the fusion of multiple sub-operators; and the eighth type of operator in the eighth category includes operations in which multiple threads compete for the same address, and the operation relies on hardware atomic operations to resolve the competition.

[0023] At least one embodiment of this disclosure provides a processor, including an instruction parsing unit and a precision verification unit, wherein the instruction parsing unit is used to receive and parse a precision verification instruction, and the precision verification unit executes a data processing method according to any embodiment of this disclosure after the instruction parsing unit parses the precision verification instruction.

[0024] At least one embodiment of this disclosure provides an electronic device, including: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to execute the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement a data processing method according to any embodiment of this disclosure.

[0025] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the data processing method according to any embodiment of this disclosure.

[0026] In the data processing method provided in at least one embodiment of this disclosure, the network model operators are divided into eight categories based on their numerical computation characteristics. Each category has an independent precision threshold, eliminating the precision paradox of a uniform threshold and avoiding a one-size-fits-all verification approach, thus reducing the false alarm rate by more than 80%. This fine-grained eight-category operator classification effectively distinguishes the inherent differences in error generation mechanisms between matrix multiplication and accumulation, hardware-specific functions, logical operations, and reduction and accumulation, effectively improving the accuracy of precision verification and enhancing the refinement of the precision verification scheme. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0028] Figure 1 This is a schematic structural diagram of a general-purpose graphics processor; Figure 2 A schematic flowchart illustrating a data processing method provided in at least one embodiment of this disclosure; Figure 3 A schematic diagram of the data processing method provided in an embodiment of this disclosure; Figure 4 A schematic block diagram of a data processing apparatus provided in at least one embodiment of this disclosure; Figure 5 A schematic structural diagram of a processor provided for at least one embodiment of this disclosure; Figure 6 A schematic block diagram of an electronic device provided in one embodiment of this disclosure; Figure 7 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0030] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0031] In a network model, an operator typically refers to the basic mathematical operations or processes used within the network. These operators are used to construct the various layers and components of the network model, enabling data transfer, transformation, and computation. They are the fundamental building blocks of the network model, defining its structure and computational flow, including inputs, outputs, and intermediate computations. In a network model, the connections between operators form a directed graph, reflecting the computational order of different operations. By combining these operators, complex and powerful network models can be constructed to handle various complex tasks and data.

[0032] When using hardware devices such as AI chips and AI accelerators to run network models for inference or training, operator precision directly affects the inference accuracy and training convergence of the network model.

[0033] Operator accuracy can be expressed as the error between the calculation results output by the operator and the standard reference value when the network model is run by the hardware device.

[0034] For example, when chips with different hardware architectures run the same network model, slight differences will occur after execution due to differences in microarchitecture, even with the same input and network model parameters. For instance, standard reference values ​​can come from the results of a reference device. For example, the reference device can employ a general-purpose hardware architecture. When the hardware device uses a self-developed architecture, the numerical deviation between the operator output of the self-developed hardware architecture running the network model and the standard reference value is verified to determine the operator accuracy.

[0035] For example, a central processing unit (CPU) can be used to run the network model, and the CPU's results can be used as a standard reference value to verify the numerical deviation between the operator output of the network model run by the graphics processing unit (GPU) or general-purpose graphics processing unit (GPU) and the standard reference value, in order to determine the operator accuracy.

[0036] Figure 1 This is a schematic diagram of the structure of a general-purpose graphics processor.

[0037] like Figure 1 As shown, a general-purpose graphics processing unit (GPGPU) is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1 The diagram shows multiple streaming processor clusters. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0038] like Figure 1 As shown, a streaming processor cluster comprises multiple compute units (CUs). Each CU performs arithmetic and logical operations, such as accumulation, reduction, and standard addition, subtraction, multiplication, and division. A CU contains multiple cores, each including an arithmetic logic unit (ALU), a floating-point unit, etc., and is used to execute specific computational tasks. Furthermore, a CU also includes registers (e.g., registers). Figure 1 The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0039] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0040] In each computing unit, the thread beam scheduling / distribution module ( Figure 1 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can be executed concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory for read and write operations, etc.

[0041] When running a network model using a parallel processor, such as a graphics processing unit (GPU) or a general-purpose GPU, various verification schemes can be employed to check the accuracy of its operators.

[0042] The most common method is the uniform threshold element-by-element comparison method, which uses a globally fixed set of relative error rtol and absolute error atol parameters to perform element-by-element numerical verification on the output of all operators. This method is simple to implement and has low integration costs, but it does not match the floating-point calculation characteristics of different operators. The numerical characteristics of different types of operators vary greatly, and the uniform threshold method is prone to problems such as missed defects or incorrect interception of normal hardware rounding deviations.

[0043] Some solutions employ a single metric, the global average relative error, relying on the mean error of all output data and a fixed threshold for verification, thus mitigating the impact of discrete outliers. However, this approach fails to reflect the dispersion of errors within the tensor, and local numerical anomalies are masked by the overall mean.

[0044] Furthermore, when the reference value is close to zero, even if the absolute error is very small, the relative error will tend to infinity. Current operator accuracy evaluation schemes generate a large number of false alarms when the operator output is close to zero. Moreover, if the operator error is systematically biased in the positive or negative direction (rather than randomly distributed), it indicates the existence of a systematic computational bias, which current operator accuracy evaluation schemes do not detect.

[0045] In summary, existing numerical verification methods for various operators have limitations at different levels, making it difficult to balance verification accuracy, false alarm rate, and defect detection capability. There is an urgent need for a systematic verification scheme that takes into account operator subdivision attributes, error statistical characteristics, near-zero compensation, and bias detection.

[0046] This disclosure provides at least one embodiment of a data processing method, processor, data processing apparatus, electronic device, and non-transitory computer-readable storage medium. In at least one embodiment, the data processing method includes: performing inference or training operations on a network model by a hardware device, wherein the network model includes multiple operators; for each operator, performing element-wise precision verification on the output tensor of the operator during the inference or training operation to obtain a precision verification result for each operator, wherein the output tensor of the operator includes multiple elements, and the precision verification result includes the precision verification result for each element; determining whether the operator meets precision requirements based on the precision verification result of the operator; wherein the element-wise precision verification operation includes performing precision verification on each element in the output tensor of the operator based on an error threshold corresponding to the operator and a standard reference value, the error threshold corresponding to the operator being related to the category to which the operator belongs, and operators of different categories having independent error thresholds, the categories including a first category, a second category, a third category, a fourth category, a fifth category, a sixth category, a seventh category, and an eighth category, the first category including multiplication and accumulation. The first category of operators includes reduction and accumulation operations. The second category includes reduction and accumulation operations. The error threshold for the first category of operators is related to at least one of the following: data block size, accumulator precision, and accumulation execution order when the hardware device performs inference or training operations. The error threshold for the second category of operators is related to the precision of the intermediate results generated when the hardware device executes the second category of operators. The third category of operators includes special function operators using dedicated instructions. The fourth category of operators includes operators that perform stepwise basic operations. The fifth category of operators includes operators that perform logical operations or extremum operations. The sixth category of operators includes operators that perform shape and size transformations. The seventh category of operators includes composite operators, which are formed by the fusion of multiple sub-operators. The eighth category of operators includes operations where multiple threads compete for the same address, and the operation relies on hardware atomic operations to resolve the competition.

[0047] In the data processing method provided in at least one embodiment of this disclosure, the network model operators are divided into eight categories based on their numerical computation characteristics. Each category has an independent precision threshold, eliminating the precision paradox of a uniform threshold and avoiding a one-size-fits-all verification approach, thus reducing the false alarm rate by more than 80%. This fine-grained eight-category operator classification effectively distinguishes the inherent differences in error generation mechanisms between matrix multiplication and accumulation, hardware-specific functions, logical operations, and reduction and accumulation, effectively improving the accuracy of precision verification and enhancing the refinement of the precision verification scheme.

[0048] The seventh type of operator (composite operator) and the eighth type of operator (nondeterministic operator) are proposed in the operator classification. Differentiated verification rules are applied to these operators to effectively reduce the false alarm rate, improve the overall verification completeness, and reduce the invalid debugging time caused by precision false alarms.

[0049] Furthermore, the error threshold corresponding to the first type of operator is related to at least one of the following: data block size, accumulator precision, and accumulation execution order when the hardware device performs inference or training operations. The error threshold corresponding to the second type of operator is related to the precision of the intermediate results generated when the hardware device executes the second type of operator. The error threshold is set in combination with the hardware device's own parameters and adjusted. The error threshold is more reasonable and accurate, which further improves the accuracy of operator precision verification when the hardware device executes the network model.

[0050] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0051] Figure 2 This is a schematic flowchart illustrating a data processing method provided in at least one embodiment of the present disclosure.

[0052] like Figure 2 As shown, at least one embodiment of the present disclosure provides a data processing method including steps S10 to S30.

[0053] For example, in step S10, the hardware device performs inference or training operations on the network model.

[0054] The network model comprises multiple operators. The network model can be any network model requiring inference or training, such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Transformer-type networks, Generative Adversarial Networks (GAN), etc.; it can also be derived from the above architectures, such as object detection networks, image segmentation networks, speech processing networks, multimodal large models, and Hybrid Expert Models (MoE). The hardware device can handle the forward inference operations of the above-mentioned network models and can also support computational tasks such as parameter training and fine-tuning of the network model; this disclosure does not impose specific limitations in this regard.

[0055] For example, the hardware device can be a device that performs inference or training operations on the network model. This hardware device could be an artificial intelligence chip, graphics processing unit (GPU), central processing unit (CPU), neural network processor (NPU), tensor processing unit (TPU), field-programmable gate array (FPGA), or application-specific integrated circuit (ASIC). It could also be a computing board integrating the above processors, an edge computing terminal, a server, an in-vehicle computing device, a mobile terminal, or a cloud computing cluster. These various hardware devices can independently or collaboratively complete forward inference operations, model parameter training, and fine-tuning operations for the network model.

[0056] For example, in step S20, for each operator, an element-wise precision verification operation is performed on the output tensor of the operator in the inference or training operation to obtain the precision verification result of each operator.

[0057] The operator's output tensor consists of multiple elements, and the precision verification result includes the precision verification result for each element.

[0058] The element-wise precision verification operation involves verifying the precision of each element in the output tensor of the operator based on the error threshold and standard reference value corresponding to the operator. The precision verification result of the operator includes the precision verification result of each element.

[0059] For example, the results of running a network model on a reference device with a different architecture than the hardware device can be used as the standard reference value for the operator, such as the results of running a central processing unit.

[0060] The error threshold corresponding to an operator is related to the category to which the operator belongs; different categories of operators have independent error thresholds.

[0061] The operators are categorized into categories one, two, three, four, five, six, seven, and eight.

[0062] The first category of operators, belonging to the first category, includes operators that include multiply-accumulate operations. For example, the first category of operators may include tensor matrix multiplication (matmul), multiply-accumulate (MAC), general matrix multiplication (GEMM), convolution (conv), deconvolution, fully connected (linear), batch matrix multiplication, and other operators that include matrix multiplication-accumulate operations.

[0063] The error threshold corresponding to the first type of operator is related to at least one of the following: the data block size, the accumulator precision, and the accumulation execution order when the hardware device performs inference or training operations.

[0064] For example, when a tensor kernel processes matrix multiplication and accumulation operations, it divides the entire tensor into multiple data blocks (blocks / tiles) for parallel processing, such as dividing it into multiple data blocks of size 8×8 or 64×64. The error threshold corresponding to the first type of operator can take into account the size of the data block.

[0065] For example, when a hardware device performs a matrix multiplication and accumulation operation, the precision of the accumulator used can be such that when the multiplication result is continuously accumulated, the data format and bit width used inside the accumulator register are usually higher than the precision of the input data, which can effectively avoid data overflow during the accumulation process.

[0066] For example, the accumulation execution order refers to the order in which the product terms are written into the accumulator for superposition and the order of traversal dimensions when the hardware device completes a matrix multiplication and accumulation operation.

[0067] The error threshold corresponding to the first type of operator can take into account the accuracy of the accumulator and the order of accumulation when the hardware device performs matrix multiplication and accumulation operations.

[0068] The second category of operators, belonging to the second category, includes operators that include reduction and accumulation operations. For example, the second category of operators may include reduction and summation operators (reduce_sum), mean reduction operators (reduce_mean), layer normalization operators (layernorm), normalized summation, various pooling summations, mean operators (mean), batch normalization operators (batchnorm), softmax operators, etc., which include reduction and accumulation operations, and the computational semantics include aggregating values ​​along a specified dimension.

[0069] The error threshold corresponding to the second type of operator is related to the precision of the intermediate results generated when the hardware device executes the second type of operator. For example, the precision of the intermediate results generated when the hardware device executes the second type of operator can refer to the intermediate parts temporarily saved and gradually accumulated during the operation, as well as the data bit width and data format used. The hardware device typically uses a higher precision than the input tensor to store the intermediate accumulation results.

[0070] The third category of operators are special function operators that use dedicated instructions provided by hardware devices. For example, third-category operators typically use dedicated instructions provided by hardware devices to implement special mathematical functions (such as transcendental functions), for example, by using lookup tables and polynomial fitting approximation. Examples of third-category operators include the power operator (pow), the exponential operator (exp), the square root operator (sqrt), the sine operator (sin), the cosine operator (cos), and the tangent operator (tan).

[0071] The fourth category of operators are those that perform element-wise basic operations, such as the addition operator (add), the subtraction operator (sub), the element-wise multiplication operator (mul), the division operator (div), and the scaling operator (scale), which are the basic arithmetic element-wise operators.

[0072] The fifth category of operators includes operators that perform logical operations or extremum operations. For example, logical operators may include equality comparison operators (equal), greater than comparison operators (greater), greater than or equal to comparison operators (greater_equal), less than comparison operators (less), less than or equal to comparison operators (less_equal), inequality comparison operators (not_equal), logical AND operators (logical_and), logical OR operators (logical_or), logical NOT operators (logical_not), etc., used to perform element-wise numerical comparisons and Boolean logic judgments. Extremum operators include, for example, maximum value operators (max), minimum value operators (min), maximum value index operators (argmax), minimum value index operators (argmin), Top-K selection operators (topk), sorting operators (sort), conditional selection operators (where), etc.

[0073] The sixth category of operators includes operators that perform shape and size transformations. For example, the sixth category of operators includes the reshape operator, the permute operator, and the cat operator. The sixth category of operators adjusts the dimensional structure of tensors, the arrangement of elements, or merges multiple tensors. They do not perform arithmetic operations on tensor elements and there is no numerical calculation process.

[0074] The seventh category of operators, belonging to the seventh class, are composite operators. Composite operators are formed by fusing multiple sub-operators, which are the basic operators. For example, they can be operators from the first, second, third, fourth, fifth, sixth, or eighth class. For instance, seventh-class operators may include Flash-Attention operators, Fused-Multi-Layer Perceptron operators, etc.

[0075] The eighth category of operators includes operations where multiple threads compete for the same memory address, and these operations rely on hardware atomic operations to resolve the competition. For example, the eighth category of operators are nondeterministic operators, meaning their results are not deterministic, and multiple executions may yield different results. This is because, in the eighth category of operators, multiple threads writing to the same memory address in parallel can cause conflicts, which are resolved by hardware atomic instructions. Hardware atomic operations themselves do not guarantee the execution order of threads; therefore, in floating-point scenarios, the results are nondeterministic. Examples of the eighth category of operators include the scatter_add operator and the atomicadd operator.

[0076] In step S30, based on the accuracy verification results of the operator, it is determined whether the operator meets the accuracy requirements.

[0077] The data processing method provided in at least one embodiment of this disclosure proposes for the first time an eight-category operator precision classification, avoiding a one-size-fits-all verification approach and reducing the false alarm rate by more than 80%. The fine-grained eight-category operator classification effectively distinguishes the inherent differences in error generation mechanisms between matrix multiplication and accumulation, hardware-specific functions, logical operations, and reduction and accumulation, effectively improving the accuracy of precision verification, enhancing the refinement of the precision verification scheme, improving the overall verification completeness, and reducing the ineffective debugging time caused by precision false alarms.

[0078] Currently, a fixed threshold is used for Type I operators, including matrix multiplication and accumulation, lacking dynamic scaling of the accumulation dimension. However, based on the numerical computational characteristics of Type I operators, the accumulation error of matrix multiplication increases linearly with the accumulation dimension K. Therefore, different K values ​​using the same threshold should be differentiated. For example, suppose the multiplication operation in matrix multiplication and accumulation is represented as A[M,K]×B[K,N]=C[M,N], where M, N, and K are positive integers, and K is the accumulation dimension.

[0079] For example, in some embodiments, the error threshold corresponding to the first type of operator is also positively correlated with K. K represents the size of the accumulation dimension when the first type of operator performs a multiplication-accumulation operation, and K is a positive integer. For example, the larger K is, the larger the error threshold corresponding to the first type of operator is, and the smaller K is, the smaller the error threshold corresponding to the first type of operator is.

[0080] For example, the error threshold corresponding to the first type of operator is determined based on a first base threshold and K. The error threshold for the first type of operator scales with K, and the first base threshold is determined based on at least one of the data block size, accumulator precision, or accumulation execution order when the hardware device performs inference or training operations. In other words, the rounding error of floating-point accumulation increases with the number of accumulations K; the larger K is, the greater the allowable error.

[0081] For example, the error threshold corresponding to the first type of operator can be expressed by the following formula 1: Threshold_A (K) = K / 256×(atol1, rtol1) (Formula 1) Where Threshold_A(K) represents the error threshold corresponding to the first type of operator, (atol1, rtol1) represents the first basic threshold, where atol1 is the absolute error and rtol1 is the relative error, and (atol1, rtol1) is determined based on at least one of the data block size, accumulator precision, or accumulation execution order when the hardware device performs inference or training operations, and K represents the size of the accumulation dimension when the first type of operator performs multiply-accumulate operations.

[0082] For example, in some other embodiments, the error thresholds corresponding to the first type of operator exhibit a stepped distribution, that is, a dynamic stepped distribution.

[0083] The cumulative error of matrix multiplication increases linearly with K. When K is large, the error threshold corresponding to the first type of operator is more significantly affected by K. To reduce the computational overhead of using continuous functions, a stepped distribution is adopted. In the stepped distribution, the error threshold corresponding to the first type of operator is not fixed but scales with K. The error threshold corresponding to the first type of operator has different values ​​in different ranges.

[0084] For example, in response to K being less than a first threshold, the error threshold corresponding to the first type of operator is a constant value, and the error threshold corresponding to the first type of operator when K belongs to a first range is less than the error threshold corresponding to the first type of operator when K belongs to a second range, where K is a positive integer and represents the size of the accumulation dimension when the first type of operator performs a multiplication-accumulation operation, the maximum value of the first range is less than the minimum value of the second range, and the maximum value of the second range is less than the first threshold; in response to K being greater than or equal to the first threshold, the error threshold corresponding to the first type of operator is positively correlated with K.

[0085] For example, when K is less than the first threshold, the error threshold corresponding to the first type of operator is a fixed value, that is, it does not scale with K. For example, in one example, K=8192, but of course, this disclosure is not limited to this. When K is greater than or equal to the first threshold, the error threshold corresponding to the first type of operator is positively correlated with K.

[0086] For example, when K is less than the first threshold, it can be further subdivided into different ranges based on the different values ​​of K. In different ranges, the specific values ​​of the error threshold (fixed value) corresponding to the first type of operator are also different, but overall, the trend is still that the error threshold is larger when K is larger.

[0087] For example, in one example with K=8192, when K is less than 1024, the error threshold corresponding to the first type of operator is (atol2, rtol2). When K is greater than or equal to 1024 and less than 8192, the error threshold corresponding to the first type of operator is (atol3, rtol3), where atol2 is less than atol3 and rtol2 is less than rtol3. atol2, atol3, rtol2, and rtol3 can be set empirically, with atol2 and atol3 representing absolute errors and rtol2 and rtol3 representing relative errors. When K is less than 8192, the error threshold corresponding to the first type of operator is a fixed value, but the error threshold when K is in the range [1024, 8192) is greater than the error threshold when K is in the range (0, 1024). When K is greater than or equal to 8192, the error threshold corresponding to the first type of operator can be determined using Formula 1 as described above, which will not be elaborated further here.

[0088] For example, in some embodiments, different error thresholds can be set for the first type of operator depending on the precision of the floating-point numbers used by the hardware device when executing the network model. For example, assuming the hardware device uses BF16 floating-point type to perform inference or training operations on the network model, the error threshold corresponding to the first type of operator can be the first basic threshold; assuming the hardware device uses FP32 floating-point type to perform inference or training operations on the network model, the error threshold corresponding to the first type of operator can adopt the stepped distribution as described above, or be positively correlated with K.

[0089] The data processing method provided in at least one embodiment of this disclosure has an error threshold corresponding to the first type of operator that is related to the accumulation dimension K. This can accurately reflect the impact of the accumulation dimension on the accuracy, provide an accurate and reasonable error threshold for the first type of operator, avoid a one-size-fits-all verification method, and effectively reduce the false alarm rate. The mathematical derivation is based on the floating-point accumulation error theory and has theoretical rigor.

[0090] For example, in some embodiments, the error threshold corresponding to the second type of operator is also positively correlated with L, where L represents the number of accumulations in the reduction-accumulation operation included in the second type of operator, and L is a positive integer. The number of accumulations L is the total number of elements participating in the addition along the reduction dimension when performing a reduction-accumulation operation.

[0091] For example, the error threshold of the second type of operator dynamically scales with the accumulation quantity L. For example, the error threshold corresponding to the second type of operator is determined based on a second basic threshold and a first coefficient, wherein the error threshold corresponding to the second type of operator scales with the first coefficient, the second basic threshold is determined based on the accuracy of the intermediate results generated when the hardware device executes the second type of operator, and the first coefficient is positively correlated with L.

[0092] For example, the error threshold of the second type of operator can be determined using the following formula 2: Threshold_B (L) = a×log2(L+1)×(atol4, rtol4) (Formula 2) Where Threshold_B (L) represents the error threshold corresponding to the second type of operator, a represents the type coefficient, different second type of operators have corresponding default values, (atol4, rtol4) represents the second basic threshold, where atol4 is the absolute error and rtol4 is the relative error, (atol4, rtol4) is determined based on the accuracy of the intermediate results generated when the hardware device executes the second type of operator, and L represents the number of accumulations included in the reduction accumulation operation.

[0093] For example, in other embodiments, the error threshold corresponding to the second type of operator is also positively correlated with L and the numerical range of the input tensor of the second type of operator. For example, the numerical range is the range of each element in the predefined input tensor. For example, in one example, assuming that the numerical range of each element in the predefined input tensor is [-1, 1], then the numerical range of the input tensor of the second type of operator is 2.

[0094] The error threshold for the second type of operator dynamically scales with the accumulation quantity L and the numerical range of the input tensor of the second type of operator. For example, the error threshold corresponding to the second type of operator is determined based on a second basic threshold and a first coefficient, wherein the first coefficient is positively correlated with the product of L and the numerical range of the input tensor of the second type of operator.

[0095] For example, the error threshold of the second type of operator can be determined using the following formula 3: Threshold_B (L) = a×log2(L×accum_count+1)×(atol4, rtol4) (Formula 3) Where Threshold_B (L) represents the error threshold corresponding to the second type of operator, a represents the type coefficient, (atol4, rtol4) represents the second basic threshold, L represents the number of accumulations included in the reduction accumulation operation, and accum_count represents the numerical range of the input tensor of the second type of operator.

[0096] For the second type of operator, in order to reduce computational overhead, a dynamic step distribution can be adopted, similar to the first type of operator. In the step distribution, the error threshold corresponding to the second type of operator is not fixed and scales with L. In different value ranges, the error threshold corresponding to the second type of operator has different values. For example, when L is small, a fixed value is taken, and when L is large, it is dynamically scaled with the first coefficient. For details, please refer to the relevant description of the first type of operator, which will not be repeated here.

[0097] For example, in some embodiments, different error thresholds can be set for the second type of operator depending on the precision of the floating-point numbers used by the hardware device when executing the network model. For example, assuming the hardware device uses BF16 floating-point type to perform inference or training operations on the network model, the error threshold corresponding to the second type of operator can be the second basic threshold; assuming the hardware device uses FP32 floating-point type to perform inference or training operations on the network model, the error threshold corresponding to the second type of operator can adopt the stepped distribution as described above, or be positively correlated with L (or the numerical range of L and the input tensor).

[0098] The data processing method provided in at least one embodiment of this disclosure has an error threshold corresponding to the second type of operator that is related to the accumulation quantity L (or the accumulation quantity L and the numerical range of the input tensor). This can accurately reflect the impact of the accumulation quantity and the numerical range of the input tensor on the accuracy, provide an accurate and reasonable error threshold for the second type of operator, avoid a one-size-fits-all verification method, and effectively reduce the false alarm rate.

[0099] The error threshold corresponding to the third type of operator is the first fixed value, and the error threshold corresponding to the fourth type of operator is the second fixed value. The first fixed value is greater than the second fixed value.

[0100] The fourth type of operator can be precisely implemented using standard floating-point multipliers and adders. Theoretically, it only suffers from standard floating-point rounding errors, which are stable and extremely small. Although the division operator (div) is an arithmetic operation, its hardware implementation often uses iterative approximation methods. The third type of operator cannot be precisely analyzed by a finite number of multiplications and additions. Internally, the hardware generally uses dedicated arithmetic instructions, employing methods such as lookup tables and polynomial approximations to approximate the true values. This is an approximate calculation. Therefore, in addition to floating-point rounding errors, the third type of operator also suffers from additional errors introduced by numerical approximation. Under the same data precision conditions, the deviation between the calculated result and the theoretical true value is usually higher than that of the fourth type of operator.

[0101] Considering the inherent system bias in the third type of operator, its error standard is configured more leniently than that of the fourth type of operator. The third type of operator uses a fixed medium threshold (first fixed value), while the fourth type of operator uses a fixed strict threshold (second fixed value), employing a smaller error threshold. When the error threshold includes both relative and absolute error thresholds, "smaller" here means that at least one of the relative and absolute error thresholds is smaller. This provides accurate and reasonable error thresholds for both the third and fourth types of operators, avoiding a one-size-fits-all verification approach and effectively reducing the false alarm rate.

[0102] For example, the error threshold can also take into account the floating-point precision used by the hardware device when executing the network model. For some high-precision (e.g., using FP32 floating-point type) inference and training scenarios, the same error threshold can be set for the third type of operator and the fourth type of operator. For some relatively low-precision (e.g., using BF16 floating-point type) inference and training scenarios, different error thresholds can be set for the third type of operator and the fourth type of operator. This disclosure does not impose specific restrictions in this regard.

[0103] For any of the first, second, third, or fourth type of operators, step S20 may include: determining the error threshold corresponding to any type of operator based on the category to which the operator belongs; determining the error value of an element based on the value of each element in the operator and the standard reference value corresponding to the element; and comparing the error value of each element with the error threshold corresponding to the operator to obtain the accuracy verification result of the operator.

[0104] For the aforementioned first, second, third, or fourth type of operators, a specific error threshold corresponding to that type of operator can be determined based on the aforementioned content. Then, for a given operator belonging to the first, second, third, or fourth type, the error value of each element is determined. For example, the error value of each element can include both the absolute error and the relative error of that element. Next, the absolute and relative errors of each element are compared with the error threshold determined based on the operator's category to obtain the accuracy verification result of that operator. For example, the accuracy verification result can include the comparison result of the error value of each element with the error threshold.

[0105] For any one of the first, second, third, or fourth type of operators, step S30 may include: if the error value of each element of any one type of operator is less than or equal to the error threshold corresponding to that one type of operator, determine that the one type of operator has passed the element-by-element precision verification; otherwise, determine that the one type of operator has not passed the element-by-element precision verification.

[0106] For example, for an operator belonging to the first, second, third, or fourth category of operators, if the error value of any element of the operator is greater than the error threshold, the operator is considered to have failed the element-by-element accuracy verification and its accuracy does not meet the requirements. If the error values ​​of all elements of the operator are less than or equal to the error threshold corresponding to the operator, the operator is considered to have passed the element-by-element accuracy verification. In some embodiments, the operator can be considered to meet the accuracy requirements. In other embodiments, the global error index can be further combined for subsequent accuracy verification.

[0107] For example, for the fifth and sixth type operators, which include logical or extreme value calculations, this involves lossless data transformation. Theoretically, there is no floating-point rounding involved. Therefore, ideally, the output tensor should be completely consistent with the standard reference value (bit-level consistency). The verification threshold is set most stringently, requiring zero tolerance. In this case, for the fifth and sixth type operators, the error threshold is not numerical; accuracy verification requires that the values ​​of each element of the operator's output tensor be bit-accurate with their corresponding standard reference values.

[0108] For any of the fifth or sixth type of operators, step S20 may include: performing element-by-element bit-level verification on each element and the standard reference value of the element in the output tensor of any type of operator to obtain the accuracy verification result of any type of operator.

[0109] In other words, for operators belonging to category 5 or 6, in the element-wise precision verification operation of the operator, each element in the output tensor of the operator is verified bit-by-bit against the standard reference value corresponding to the operator to obtain the precision verification result of the operator. For example, each element can be verified bit-by-bit against the standard reference value corresponding to that element to obtain the precision verification result of the operator.

[0110] For any of the fifth or sixth type of operators, step S30 may include: if the precision verification result of any type of operator indicates that each element of the operator is consistent with the standard reference value of the element at the bit level, determine that the operator has passed the element-by-element precision verification; otherwise, determine that the operator has not passed the element-by-element precision verification.

[0111] For example, if there is one element in the operator that is not consistent with the corresponding standard reference value at the bit level, the operator is considered to have failed the element-by-element precision verification and its precision does not meet the requirements. If all elements of the operator are consistent with the corresponding standard reference value at the bit level, the operator is considered to have passed the element-by-element precision verification. In some embodiments, the operator can be considered to meet the precision requirements. In other embodiments, the global error index can be further combined for subsequent precision verification.

[0112] For example, for the seventh type of operator, which is a composite fusion operator composed of multiple sub-operators, step S20 may include performing element-wise precision verification on each of the multiple sub-operators included in the seventh type of operator.

[0113] In other words, for the seventh type of operator, it is decomposed or split into multiple sub-operators, and then element-by-element precision verification is performed on each of the multiple sub-operators according to the aforementioned process.

[0114] For example, based on the type of each sub-operator, an error threshold corresponding to that sub-operator is determined. Based on the error threshold and a standard reference value, the accuracy of each element in the output tensor of that sub-operator is verified. The specific accuracy verification process can be referred to the description of relevant embodiments in this disclosure, depending on the type of sub-operator, and will not be repeated here.

[0115] For the seventh type of operator, step S30 may include: if each sub-operator included in the seventh type of operator passes the element-wise precision verification, determine that the seventh type of operator has passed the element-wise precision verification; otherwise, determine that the seventh type of operator has not passed the element-wise precision verification.

[0116] For example, taking the fusion multilayer perceptron operator as an example, this operator can be decomposed into three fully connected operators (belonging to the first type of operator), one element-wise multiplication operator (belonging to the fourth type of operator), and one activation function operator (belonging to the third type of operator). Therefore, for the fusion multilayer perceptron operator, it can be decomposed into five sub-operators, and then the five sub-operators are respectively subjected to element-wise precision verification operations according to the aforementioned content.

[0117] For example, for fully connected operators, element-wise accuracy verification is performed with reference to the description of the first type of operators mentioned above; for element-wise multiplication operators, element-wise accuracy verification is performed with reference to the description of the fourth type of operators mentioned above; for activation function operators, element-wise accuracy verification is performed with reference to the description of the third type of operators mentioned above. For example, if all five sub-operators pass the element-wise accuracy verification, the fused multilayer perceptron operator is considered to have passed the element-wise accuracy verification. In some embodiments, the operator can be considered to meet the accuracy requirements. In other embodiments, further accuracy verification can be performed by combining the global error index. If any sub-operator fails the element-wise accuracy verification, the fused multilayer perceptron operator is considered to have failed the element-wise accuracy verification and does not meet the accuracy requirements.

[0118] For example, when the eighth type of operator is executed, multiple threads write to the same memory address in parallel, causing a conflict. The competition is resolved by hardware atomic instructions. Therefore, the eighth type of operator is a non-deterministic operator, meaning that the results of multiple executions may be different. If a general error judgment standard is directly adopted, it is very easy to identify the normal result fluctuations caused by the hardware parallel mechanism as operator defects, resulting in continuous false alarms.

[0119] In at least one embodiment of this disclosure, a method for verifying the accuracy of an eighth-type operator is provided. Specifically, for the eighth-type operator, the corresponding standard reference value is the first average of the output results obtained by running the eighth-type operator N times on a reference device, where N is a positive integer greater than 1. For example, N=3, etc. The reference device can be a general-purpose architecture device, such as a central processing unit.

[0120] For the eighth type of operator, step S20 may include: executing the eighth type of operator N times to obtain the second mean and variance of the output tensor of the eighth type of operator in the N runs; determining the error value corresponding to the eighth type of operator based on the second mean and the first mean; comparing the error value with the error threshold corresponding to the eighth type of operator, and comparing the variance with the preset variance threshold range to obtain the accuracy verification result of the eighth type of operator.

[0121] For the eighth type of operator, step S30 may include: in response to the error value corresponding to the eighth type of operator being less than the error threshold corresponding to the eighth type of operator and the variance being within the range of a preset variance threshold, determining that the eighth type of operator has passed the element-by-element accuracy verification; otherwise, determining that the eighth type of operator has not passed the element-by-element accuracy verification.

[0122] Considering that the results of multiple executions of the eighth type operator may differ, both the standard reference value and the output tensor are determined by averaging the results of multiple runs. For example, after executing the eighth type operator N times, the mean and variance of the N results are calculated. The error between the mean of the results and the standard reference value is compared with an error threshold, and the variance is compared with a preset variance threshold range. If the error value is less than or equal to the error threshold corresponding to the eighth type operator, and the variance is within the preset variance threshold range, the eighth type operator is considered to meet the accuracy requirements. In some embodiments, a global error index can be further combined for subsequent accuracy verification. If the error value is greater than the error threshold corresponding to the eighth type operator, or the variance is not within the preset variance threshold range, the eighth type operator is considered to have failed the element-by-element accuracy verification and does not meet the accuracy requirements.

[0123] In the data processing method provided in at least one embodiment of this disclosure, operators are classified into eight levels with fine granularity. Each type of operator has an independent error threshold, and detailed accuracy verification methods are provided for each category. This eliminates the accuracy paradox of uniform thresholds, avoids a one-size-fits-all verification approach, reduces the false alarm rate by more than 80%, effectively improves the accuracy of accuracy verification, and enhances the refinement of the accuracy verification scheme. In particular, a seventh type of operator (composite operator) and an eighth type of operator (non-deterministic operator) are proposed in the operator classification. For the characteristics of these operators, detailed and reliable accuracy verification methods are provided, effectively reducing the false alarm rate, improving the overall verification completeness, and reducing the invalid debugging time caused by accuracy false alarms.

[0124] The data processing method provided in at least one embodiment of this disclosure also provides a multi-dimensional fusion accuracy verification method.

[0125] In at least one embodiment, the data processing method further includes: determining a global error index for each operator based on the output tensor of each operator, wherein the global error index includes a first index for evaluating the overall deviation level of the operator, a second index for evaluating the error dispersion of the operator, and a third index for evaluating the systematic deviation direction of the operator; and determining whether the operator meets the accuracy requirements based on the accuracy verification results of the operator, including: determining whether the operator meets the accuracy requirements based on the accuracy verification results of the operator and the global error index.

[0126] In other words, for each operator, in addition to performing element-by-element accuracy verification as described above, it is also necessary to determine the global error index of each operator and combine the global error index to determine whether the operator meets the accuracy requirements.

[0127] For example, the first index diff1 of each operator is the mean of the relative errors of the multiple elements included in the operator. The first index diff1 of each operator is used to evaluate the overall deviation level of the operator, reflect the relative deviation of the overall output tensor of the operator, and evaluate whether the overall normal floating-point noise is normal.

[0128] Current relative error calculations, when the standard reference value is close to zero, will cause the relative error to tend to infinity even if the absolute error is very small. This leads to a large number of false alarms when the standard reference value is in the near-zero region, significantly reducing the accuracy of precision verification. The data processing method provided in at least one embodiment of this disclosure considers the negative impact of the standard reference value being in the near-zero region when determining the relative error, thus avoiding the relative error explosion problem caused by the standard reference value being close to zero.

[0129] The first index, diff1, of each operator can be determined with reference to the following formula 4: diff1 =

[0130] = (Formula 4) Where diff1 represents the first index, and T represents the total number of elements. This represents the relative error of the i-th element among multiple elements, where i takes values ​​of 1, 2, ..., T, and threshold is a positive number representing the first preset threshold. This represents the value of the i-th element. This represents the standard reference value corresponding to the i-th element.

[0131] The threshold can be set as needed, for example, it can be set to a suitable positive number close to 0, such as 0.1, to avoid the problem of relative error explosion caused by dividing by a very small value.

[0132] Therefore, in at least one embodiment of this disclosure, the impact of relative error explosion caused by the near-zero value region is considered when determining the first index. When the absolute value of the standard reference value is large, the relative error is determined by conventional methods. When the standard reference value is very small, the threshold is used as the denominator for relative error calculation, which solves the false alarm problem caused by the standard reference value being in the near-zero value region and effectively improves the accuracy of precision verification.

[0133] For example, the second index for each operator is the root mean square of the relative errors of the multiple elements included in the operator. The calculation of the relative error is still similar to that of the first index, taking into account the influence of the near-zero region. The second index can measure the dispersion and volatility of the errors of each element.

[0134] For example, the second index diff2 of each operator can be determined with reference to the following formula 5: diff2 =

[0135] = (Formula 5) Where diff2 represents the second index, and T represents the total number of elements. This represents the error parameter of the i-th element out of T elements, where i takes values ​​of 1, 2, ..., T. To represent a positive number indicating the first preset threshold, This represents the value of the i-th element. This represents the standard reference value corresponding to the i-th element.

[0136] If the operator's error is systematically biased towards the positive or negative direction (rather than being randomly distributed), it indicates a systematic computational bias. Systematic computational bias often indicates a logical flaw in the operator. In training scenarios, for example, if the gradient is systematically shifted, it can lead to poorer convergence, increased loss, or non-convergence. In inference scenarios, it can cause output distribution drift and overall probability distribution drift, resulting in a decrease in the quality of the output results.

[0137] The data processing method provided in at least one embodiment of this disclosure can detect the systematic bias of an operator using a third index. For example, the third index of each operator is the ratio between the sum of the error values ​​of the multiple elements included in the operator and a first product, wherein the first product is the product of the total number of multiple elements and the maximum absolute value of the standard reference values ​​among the multiple elements; in response to the third index of the operator being within a preset range including zero values, it indicates that the operator has no systematic bias; in response to the third index of the operator being greater than the maximum value of the preset range, it indicates that the operator has a systematic positive bias; in response to the third index of the operator being less than the minimum value of the preset range, it indicates that the operator has a systematic negative bias.

[0138] For example, the third index diff3 of each operator can be determined with reference to the following formula 6: diff3 = (Formula 6) Where diff3 represents the third index, and T represents the total number of elements. This represents the value of the i-th element. This represents the standard reference value corresponding to the i-th element, where i takes values ​​of 1, 2, ..., T. MAX( ) represents the maximum absolute value among the standard reference values ​​corresponding to multiple elements.

[0139] The preset range can be determined based on practical experience. The preset range needs to include zero values, for example, it can be set to [-0.1, 0.1], etc. This disclosure does not impose specific restrictions on this.

[0140] If the third index of the operator is within the preset range, it indicates that the operator is randomly distributed and there is no systematic bias; if the third index of the operator is greater than the maximum value of the preset range, it indicates that the operator error is systematically biased in the positive direction; if the third index of the operator is less than the minimum value of the preset range, it indicates that the operator error is systematically biased in the negative direction.

[0141] The data processing method provided in at least one embodiment of this disclosure can detect the systematic bias of operators through a third indicator, discover systematic calculation deviations in advance, avoid accuracy problems caused after the network model is officially launched, identify possible operator logic problems in advance, identify errors that will accumulate layer by layer in advance, intercept hidden problems where a single operator indicator seems qualified but the business fails after stacking, supplement the blind spots of the existing indicator system, and complete the indicator dimensions.

[0142] For example, when further determining the global error index, step S30 may include: determining whether the operator meets the accuracy requirements based on the operator's accuracy verification results and the global error index.

[0143] For example, determining whether an operator meets the accuracy requirements based on the operator's accuracy verification results and global error index can include: in response to the operator's accuracy verification results indicating that the operator has passed element-by-element accuracy verification, the operator's first index is less than a second preset threshold, the operator's second index is less than a preset multiple of the first index, and the operator's third index is within a preset range including zero values, the operator is determined to meet the accuracy requirements; otherwise, the operator is determined not to meet the accuracy requirements.

[0144] For example, in the above embodiments, if the operator passes the element-by-element accuracy verification after being performed with reference to the embodiments described above, and the first indicator diff1 is less than or equal to the second preset threshold, the second indicator is less than a preset multiple of the first indicator (e.g., 5 times), and the third indicator is within a preset range including zero values ​​(indicating random error distribution), then the operator is considered to meet the accuracy requirements. If the operator fails the element-by-element accuracy verification, or the first indicator is greater than the second preset threshold, or the second indicator is much greater than the first indicator (e.g., greater than or equal to a preset multiple of the first indicator), or the third indicator indicates the presence of systematic positive or negative bias, then the operator is considered to not meet the accuracy requirements.

[0145] In the above embodiments, this disclosure integrates global error metrics and element-by-element accuracy verification into a unified verification framework, providing a more comprehensive systematic accuracy verification solution. Global metrics can be used to determine whether the overall accuracy of the operator is up to standard, while element-by-element accuracy verification can be used for detailed verification analysis, effectively improving the accuracy of accuracy verification and reducing false alarms.

[0146] Currently, accuracy verification and performance verification are separate processes, lacking a unified quality assessment framework. An operator may have acceptable accuracy but extremely poor performance, or excellent performance but unacceptable accuracy.

[0147] The data processing method provided in at least one embodiment of this disclosure further includes: performing performance verification on an operator that meets the accuracy requirements; determining that the operator is qualified in response to the operator that meets the accuracy requirements passing the performance verification; wherein, the performance verification includes verifying whether the computational efficiency and interface efficiency of the operator meet the performance requirements.

[0148] For example, performance verification can determine performance metrics in the following ways: compute_efficiency = FP1 / FP2 × 100% io_efficiency = (Theoretical IO volume) / (Actual IO time × Peak bandwidth) × 100% Among them, compute_efficiency represents computational efficiency, characterizing the utilization level of the hardware computing unit; FP1 represents actual FLOPS (Floating-Point Operations Per Second), representing the actual floating-point operation performance achieved by the hardware; FP2 represents theoretical peak FLOPS, which refers to the maximum number of floating-point operations that the hardware computing unit can complete per second under ideal full-load conditions, determined by hardware specifications.

[0149] io_efficiency represents interface efficiency, used to characterize the utilization efficiency of hardware storage bandwidth resources; theoretical IO quantity refers to the total number of bytes of data that theoretically need to be read from and written to video memory during the execution of the operator; actual IO time refers to the measured time consumed by the operator to complete all data read and write operations during actual operation; peak bandwidth refers to the theoretical maximum memory bandwidth given by the hardware specifications.

[0150] For example, if the computational efficiency and I / O efficiency of an operator that meets the accuracy requirements are not lower than a preset efficiency threshold (e.g., 60%), it means that the operator has passed the performance verification and is a qualified operator. If either of them is lower than the preset efficiency threshold, the operator is considered not to meet the performance requirements.

[0151] For example, in at least one embodiment of this disclosure, operators that have passed the accuracy test also need to undergo performance gating. If the performance test is passed, the operator is deemed to be qualified, thereby unifying the accuracy test and performance acceptance into the same quality assessment system and simplifying the operator delivery process.

[0152] Figure 3 This is a schematic diagram of the data processing method provided in one embodiment of the present disclosure.

[0153] like Figure 3 As shown, the inputs to this data processing method include standard reference values, operator implementation, and operator metadata.

[0154] Operator metadata includes the operator's category, the operator's shape and size, and the operator's parameter type.

[0155] Standard reference values ​​are used to determine error values. For example, the output tensors of each operator when a reference device (e.g., a central processing unit) executes a network model can be used as standard reference values.

[0156] The operator implementation is the operator implementation when the network model is executed by the hardware device, such as including the output tensor of the operator, which is the actual output value used for accuracy verification. Specific examples of hardware devices can be found in the relevant description of step S10 above, and will not be repeated here.

[0157] refer to Figure 3 In the element-wise precision verification section, operator metadata is used for operator classification, i.e., determining which category an operator belongs to. Then, the error threshold corresponding to that operator can be calculated or queried based on its category. For example, assuming an operator belongs to category one, its error threshold can be dynamically determined by the calculation engine according to Formula 1; assuming an operator belongs to category two, its error threshold can be dynamically determined by the calculation engine according to Formula 2 or Formula 3; assuming an operator belongs to category three or four, its error threshold can be obtained by looking up a table, and so on.

[0158] Subsequently, based on the operator's output tensor, standard reference value, and error threshold determined according to the operator's category, element-by-element accuracy verification of each operator is performed with reference to the relevant description of step S20 above, which will not be repeated here.

[0159] refer to Figure 3 The global error index part is determined based on the output tensor of the operator and the standard reference value, referring to the aforementioned embodiments to determine the first index, second index and third index of each operator. The specific process will not be repeated here.

[0160] Subsequently, at the comprehensive evaluation level, accuracy is determined based on the accuracy verification results and global error indicators.

[0161] For example, taking the first type of operator as an example, assuming that the error values ​​of all elements of the operator are less than the corresponding error threshold, the first index of the operator is less than or equal to the second preset threshold, the second index is less than a preset multiple of the first index, and the third index is within a preset range including zero values, then the operator is considered to meet the accuracy requirements; otherwise, the operator is considered not to meet the accuracy requirements. The same logic applies to the second, third, and fourth types of operators, which will not be elaborated here.

[0162] For example, taking the fifth type of operator as an example, assuming that all elements of the operator are consistent with the bit level of the standard reference value corresponding to the operator, the first index of the operator is less than or equal to the second preset threshold, the second index is less than a preset multiple of the first index, and the third index is within a preset range including zero values, then the operator is considered to meet the precision requirements; otherwise, the operator is considered not to meet the precision requirements. The same applies to the sixth type of operator, which will not be elaborated here.

[0163] For example, taking the seventh type of operator as an example, assuming that all sub-operators of this operator have passed element-wise precision verification, the first index of the operator is less than or equal to the second preset threshold, the second index is less than a preset multiple of the first index, and the third index is within a preset range including zero values, then the operator is considered to meet the precision requirements; otherwise, the operator is considered not to meet the precision requirements.

[0164] For example, taking the eighth type of operator as an example, if the error between the mean of the execution results of the operator running multiple times and the standard reference value is less than the error threshold, and the variance of the execution results of the operator running multiple times is within the preset variance threshold range, the first index of the operator is less than or equal to the second preset threshold, the second index is less than the preset multiple of the first index, and the third index is within the preset range including zero values, then the operator is considered to meet the accuracy requirements; otherwise, the operator is considered not to meet the accuracy requirements.

[0165] refer to Figure 3 The performance verification section. Figure 3 In this process, performance verification is performed simultaneously on all operators. Of course, performance verification can also be performed only on operators that meet the accuracy requirements to reduce the computational burden caused by performance verification.

[0166] For example, the computational efficiency and interface efficiency are determined based on the operator implementation when the hardware device executes the network model. If the computational efficiency and interface efficiency are not lower than the preset efficiency threshold, it means that the operator has passed the performance verification.

[0167] Subsequently, a quality report is output based on the results of performance and accuracy assessments. The quality report provides conclusions on whether the performance and accuracy of each operator in the network model meet the performance requirements and accuracy requirements, thus unifying accuracy verification and performance acceptance into the same quality assessment system and simplifying the operator delivery process.

[0168] At least one embodiment of this disclosure also provides a data processing apparatus. Figure 4 This is a schematic block diagram of a data processing apparatus provided for at least one embodiment of the present disclosure.

[0169] like Figure 4 As shown, the data processing device 100 includes an operation module 101, an element-by-element accuracy verification module 102, and an accuracy inspection module 103.

[0170] For example, operation module 101 is configured to perform inference or training operations on a network model by a hardware device, wherein the network model includes multiple operators.

[0171] For example, the element-wise precision verification module 102 is configured to perform element-wise precision verification on the output tensor of each operator in the inference or training operation, and obtain the precision verification result of each operator. The output tensor of the operator includes multiple elements, and the precision verification result includes the precision verification result of each element.

[0172] The accuracy verification module 103 is configured to determine whether the operator meets the accuracy requirements based on the accuracy verification results of the operator.

[0173] The element-wise precision verification operation involves verifying the precision of each element of the output tensor of the operator based on the error threshold corresponding to the operator. The error threshold corresponding to the operator is related to the category to which the operator belongs, and operators of different categories have independent error thresholds.

[0174] The categories include Category 1, Category 2, Category 3, Category 4, Category 5, Category 6, Category 7, and Category 8.

[0175] The first type of operator belonging to the first category is an operator that includes multiply-accumulate operations, and the second type of operator belonging to the second category is an operator that includes reduce-accumulate operations. The error threshold corresponding to the first type of operator is related to at least one of the data block size, accumulator precision, and accumulation execution order when the hardware device performs inference or training operations. The error threshold corresponding to the second type of operator is related to the precision of the intermediate results generated when the hardware device executes the second type of operator.

[0176] The third category of operators, which belong to the third category, are special function operators that use dedicated instructions. The fourth category of operators, which belong to the fourth category, are operators that perform stepwise basic operations.

[0177] The fifth category of operators includes operators that perform logical operations or extremum operations, while the sixth category of operators includes operators that perform shape and size transformations.

[0178] The seventh category of operators, which belongs to the seventh category, are composite operators, which are formed by the fusion of multiple sub-operators. The eighth category of operators, which belongs to the eighth category, includes operations in which multiple threads compete for the same address, and the competition is resolved by hardware atomic operations.

[0179] For example, the operation module 101, the element-by-element accuracy verification module 102, and the accuracy inspection module 103 include code and programs stored in memory. These modules are implemented, for example, as a central processing unit (CPU) or other processing units with data processing and / or instruction execution capabilities. The processing unit can be a general-purpose processor, or a microcontroller, digital signal processor, dedicated image processing chip, or field-programmable logic array, etc. The operation module 101, element-by-element accuracy verification module 102, and accuracy inspection module 103 execute the code and program to achieve some or all of the functions described above. For example, the operation module 101, element-by-element accuracy verification module 102, and accuracy inspection module 103 can be a single circuit board or a combination of multiple circuit boards to implement the functions described above. In the embodiments of this application, the one or more circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) processor-executable firmware stored in the memory.

[0180] It should be noted that the operation module 101 can be used to implement Figure 2 The element-by-element precision verification module 102 shown in step S10 can be used to implement... Figure 2 The accuracy verification module 103 can be used to implement step S20 as shown. Figure 2 The step S30 is shown. Therefore, for a detailed description of the functions that the operation module 101, the element-by-element accuracy verification module 102, and the accuracy inspection module 103 can be achieved, please refer to the relevant descriptions of steps S10 to S30 in the embodiments of the above data processing method; repeated details will not be repeated here. Furthermore, the data processing device 100 can achieve similar technical effects to the aforementioned data processing method, which will not be described further here.

[0181] It should be noted that in at least one embodiment of this disclosure, the data processing device 100 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0182] For example, in some embodiments, the data processing device 100 may be a data processing operator, which is an operation or function that changes the precision of a value during data processing or calculation. The data processing operator can be used to process a value from one precision to another to meet different computational or display requirements. For example, the data processing device 100 may be implemented in hardware, software, or a combination of both; this disclosure does not impose specific limitations in this regard.

[0183] Figure 5 This is a schematic structural diagram of a processor provided for at least one embodiment of the present disclosure. For example... Figure 5 As shown, the processor 200 includes an instruction parsing unit 201 and a precision verification unit 202.

[0184] For example, instruction parsing unit 201 is used to receive and parse precision verification instructions.

[0185] For example, after the instruction parsing unit parses the accuracy verification instruction, the accuracy verification unit 202 executes the data processing method according to any embodiment of the present disclosure.

[0186] Specifically, when processor-based upper-layer software (such as AI applications, HPC applications, and scientific computing applications) sends precision verification instructions for computation processing to the processor (such as CPU or GPU) through a unified encapsulated function library, the precision verification instructions can carry information such as standard reference values ​​and storage addresses of operator metadata, or access addresses of operator implementation-related information. When the processor receives the precision verification instructions, the instruction parsing unit 201 parses the precision verification instructions to obtain the input parameters, and the processor schedules the precision verification unit to execute the data processing task.

[0187] For the specific process of executing the accuracy verification instruction using the accuracy verification unit 202, please refer to steps S10-S30 in the data processing method described above. Repeated steps will not be repeated here.

[0188] The data processing method, data processing apparatus, or data processing approach provided in at least one embodiment of this disclosure can be applied to different systems or devices, such as those used in... Figure 6The illustrated electronic device 300 can be a terminal, such as a mobile phone, tablet computer, laptop computer, AR device, VR device, vehicle terminal, etc., or it can be a server. The data processing method provided in at least one embodiment of this disclosure can be applied to scenarios involving data processing in the electronic device 300, such as those related to CPU, High Performance Computing (HPC), and Artificial Intelligence (AI), for example, scalar computation units, vector computation units, matrix computation units, and tensor computation units. Of course, this disclosure is not limited to this; any scenario, device, or apparatus involving accuracy verification can employ the data processing method or data processing apparatus provided in at least one embodiment of this disclosure.

[0189] In some embodiments, the data processing apparatus provided in at least one embodiment of this disclosure can be a chip, such as a system-on-a-chip (SoC). A SoC includes a processor, which can be a single-core or multi-core processor, memory, and I / O interfaces, etc. The processor can load data and applications from memory and then process the data, such as performing computational processing involving data processing.

[0190] For example, refer to Figure 6 When the electronic device 300 shown implements the data processing method provided in at least one embodiment of this disclosure, the processing device 301 executes various appropriate actions and processes according to non-transitory computer-readable instructions stored in the memory to implement the data processing method. For example, input parameters (such as operator metadata, standard reference values, etc.) are stored in the memory, such as registers, caches, or main memory. When data processing is required, the input parameters are read from the memory, and the processing device 301 performs accuracy verification on the input parameters according to steps S10-S30 of the data processing method described in at least one embodiment of this disclosure to obtain an accuracy verification result.

[0191] Furthermore, it should be noted that the data processing method provided in at least one embodiment of this disclosure is not limited to the specific structure of the network model, and the specific physical meaning of the relevant parameters of the processed operators (e.g., input tensors and output tensors) may differ depending on the application field. For example, the data processing method provided in at least one embodiment of this disclosure can be applied to fields involving operator accuracy verification, such as natural language processing, text processing, computer vision, speech processing, and multimodal information fusion processing.

[0192] For example, the data processing method provided in at least one embodiment of this disclosure can be adapted to various large language models and the reasoning or training processes of traditional text neural networks in the field of natural language processing. It can be widely used in specific tasks such as text generation, multilingual machine translation, code generation and code completion models, intelligent question answering and dialogue systems, long text summarization and information extraction, sentiment analysis and text sentiment discrimination, keyword extraction, text correction and polishing, intent recognition, named entity recognition, document reading comprehension, logical reasoning and mathematical problem solving, copywriting, knowledge base question answering, and text classification.

[0193] For example, the data processing method provided in at least one embodiment of this disclosure can be adapted to the entire process of structured and unstructured text processing in the field of text processing, and is suitable for various text parsing, processing and optimization tasks. Examples include applications such as optical character recognition (OCR), document text parsing, content extraction, key information filtering, and document classification and archiving, and it is also suitable for refined text processing scenarios such as long text splitting, text polishing, sentence optimization, and privacy text desensitization.

[0194] For example, the data processing method provided in at least one embodiment of this disclosure can be adapted to the operator accuracy verification of various visual networks in the field of computer vision, such as convolutional neural networks and Transformer visual models, covering intelligent processing scenarios across all dimensions of images and videos. For example, it can be applied to classic visual tasks such as image classification, object detection, image segmentation, image super-resolution reconstruction, image denoising, image enhancement, feature extraction, face recognition, and behavior recognition; it can also support scenarios such as video object tracking, video frame interpolation, video quality restoration, video behavior analysis, and real-time video monitoring.

[0195] For example, the data processing method provided in at least one embodiment of this disclosure can be adapted to the operator accuracy verification of various network models related to speech signal processing and intelligent speech parsing in the field of speech processing, covering intelligent applications throughout the entire speech process. Application scenarios may include real-time speech recognition, offline speech-to-text conversion, multilingual speech translation, voice wake-up, voiceprint recognition, identity verification, and other speech perception tasks. Simultaneously, it can support speech generation and optimization tasks such as speech synthesis, timbre conversion, speech noise reduction, speech enhancement, audio emotion analysis, and speech segmentation optimization.

[0196] For example, the data processing method provided in at least one embodiment of this disclosure can be adapted to the accuracy verification of fusion inference models for multiple types of information such as text, images, voice, and video in the field of multimodal information fusion processing, supporting cross-modal intelligent processing in various scenarios. For example, it can be applied to basic fusion tasks such as image-text matching, image-text generation, image caption generation, video content understanding, cross-modal retrieval, and multimodal content classification, and can also support advanced application scenarios such as intelligent question-answering multimodal interaction, image-text content review, video content summarization generation, and cross-modal content conversion.

[0197] Of course, this disclosure is not limited to this. For other application scenarios or fields, as long as it is necessary to verify the operator accuracy of the network model, the data processing method described in at least one embodiment of this disclosure can be applied, and will not be elaborated here.

[0198] Figure 6 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. For example... Figure 6 As shown, the electronic device 300 is, for example, suitable for implementing the data processing method or data processing technique provided in the embodiments of this disclosure. It should be noted that... Figure 6 The components of the electronic device 300 shown are merely exemplary and not limiting. The electronic device 300 may have other components as needed for the actual application.

[0199] like Figure 6 As shown, the electronic device 300 may include a processing device 301 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in memory to achieve various functions.

[0200] For example, when the computer-readable instruction processing device 301 executes, it can perform one or more steps of the data processing method according to any of the above embodiments. It should be noted that a detailed description of the processing procedure of the data processing method can be found in the relevant descriptions in the embodiments of the above data processing method.

[0201] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory, etc., whereby computer-readable instructions can be loaded from storage device 308 into RAM 303 to execute. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. Various applications and various data, such as various data used and / or generated by applications, may also be stored in the computer-readable storage medium.

[0202] For example, the processing device 301, ROM 302, and RAM 303 are interconnected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0203] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, flash memory, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 6 An electronic device 300 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 300 may alternatively implement or possess more or fewer devices. For example, a processing device 301 can control other components in the electronic device 300 to perform desired functions. The processing device 301 may be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a tensor processor (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) may be an x86, ARM, RISC-V architecture, etc. The GPU may be directly integrated into the SOC, directly integrated onto the motherboard, or built into the northbridge chip of the motherboard.

[0204] Figure 7 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0205] For example, such as Figure 7 As shown, storage medium 400 can be a non-transitory computer-readable storage medium on which one or more computer-readable instructions 401 can be stored non-transitory. For example, when the computer-readable instructions 401 are executed by a processor, one or more steps in the data processing method described above can be performed.

[0206] For example, the storage medium 400 can be used in an electronic device 300, such as the storage medium 400 including the storage device 308 in the electronic device 300.

[0207] For example, a storage device may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and a processor may execute these instructions to perform various functions of the processor. Various application programs and various data may also be stored in the storage medium.

[0208] For example, the storage medium may include a memory card for a smartphone, a cache component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0210] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0211] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0212] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0213] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0214] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0215] The following points should be noted regarding this disclosure: (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0216] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0217] The above description is only a specific embodiment of this disclosure, but the protection scope of this disclosure is not limited thereto. The protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A data processing method, characterized in that, include: The inference or training operation of the network model is performed by a hardware device, wherein the network model includes multiple operators; For each operator, an element-wise precision verification operation is performed on the output tensor of the operator in the inference or training operation to obtain the precision verification result of each operator. The output tensor of the operator includes multiple elements, and the precision verification result of the operator includes the precision verification result of each of the multiple elements. Based on the accuracy verification results of the operator, determine whether the operator meets the accuracy requirements; The element-by-element accuracy verification operation includes verifying the accuracy of each element in the output tensor of the operator based on the error threshold and standard reference value corresponding to the operator. The error threshold corresponding to the operator is related to the category to which the operator belongs; different categories of operators have independent error thresholds. The categories include Category 1, Category 2, Category 3, Category 4, Category 5, Category 6, Category 7, and Category 8. The first type of operator belonging to the first category is an operator that includes multiply-accumulate operations, and the second type of operator belonging to the second category is an operator that includes reduce-accumulate operations. The error threshold corresponding to the first type of operator is related to at least one of the data block size, accumulator precision, and accumulation execution order when the hardware device performs the inference or training operation. The error threshold corresponding to the second type of operator is related to the precision of the intermediate results generated when the hardware device performs the second type of operator. The third category of operators, belonging to the third category, consists of special function operators that use dedicated instructions provided by the hardware device; the fourth category of operators, belonging to the fourth category, consists of operators that perform stepwise basic operations. The fifth category of operators includes operators that perform logical operations or extremum operations, while the sixth category of operators includes operators that perform shape and size transformations. The seventh type of operator belonging to the seventh category is a composite operator, wherein the composite operator is formed by the fusion of multiple sub-operators. The eighth type of operator, belonging to the eighth category, includes operations where multiple threads compete for the same address, and the competition is resolved by hardware atomic operations.

2. The data processing method according to claim 1, characterized in that, The error threshold corresponding to the first type of operator is also positively correlated with K, where K represents the size of the accumulation dimension when the first type of operator performs the multiplication-accumulation operation, and K is a positive integer.

3. The data processing method according to claim 2, characterized in that, The error threshold corresponding to the first type of operator is determined based on the first basic threshold and K. Wherein, the error threshold corresponding to the first type of operator is scaled with K, and the first basic threshold is determined based on at least one of the data block size, the accumulator precision, or the accumulation execution order when the hardware device performs the inference or training operation.

4. The data processing method according to claim 1, characterized in that, The error thresholds corresponding to the first type of operator exhibit a stepped distribution. In response to K being less than a first threshold, the error threshold corresponding to the first type of operator is a fixed value, and the error threshold corresponding to the first type of operator when K belongs to a first range is less than the error threshold corresponding to the first type of operator when K belongs to a second range, wherein K is a positive integer and represents the size of the accumulation dimension when the first type of operator performs the multiplication-accumulation operation, the maximum value of the first range is less than the minimum value of the second range, and the maximum value of the second range is less than the first threshold; In response to K being greater than or equal to the first threshold, the error threshold corresponding to the first type of operator is positively correlated with K.

5. The data processing method according to claim 1, characterized in that, The error threshold corresponding to the second type of operator is also positively correlated with L, where L represents the number of accumulations in the reduction-accumulation operation included in the second type of operator, and L is a positive integer; or The error threshold corresponding to the second type of operator is also positively correlated with the numerical range of L and the input tensor of the second type of operator.

6. The data processing method according to claim 5, characterized in that, The error threshold corresponding to the second type of operator is determined based on the second basic threshold and the first coefficient. The second basic threshold is determined based on the accuracy of the intermediate results generated when the hardware device executes the second type of operator. The error threshold corresponding to the second type of operator is scaled by the first coefficient. The first coefficient is positively correlated with L, or the first coefficient is positively correlated with the product of the numerical range of L and the input tensor of the second type of operator.

7. The data processing method according to claim 1, characterized in that, The error threshold corresponding to the third type of operator is a first fixed value, and the error threshold corresponding to the fourth type of operator is a second fixed value, wherein the first fixed value is greater than the second fixed value.

8. The data processing method according to claim 1, characterized in that, An element-wise precision verification operation is performed on the output tensor of the operator in the inference or training operation to obtain the precision verification result of each operator, including: For any one of the first type of operator, the second type of operator, the third type of operator, or the fourth type of operator: Based on the category to which any of the operators belong, determine the error threshold corresponding to any of the operators; Based on the value of each element in any type of operator and the standard reference value corresponding to the element, the error value of the element is determined; The error value of each element is compared with the error threshold corresponding to any type of operator to obtain the accuracy verification result of any type of operator; Based on the accuracy verification results of the operator, determine whether the operator meets the accuracy requirements, including: If the error value of each element of any type of operator is less than or equal to the error threshold corresponding to any type of operator, then any type of operator is determined to have passed the element-by-element precision verification; otherwise, it is determined that any type of operator has failed the element-by-element precision verification.

9. The data processing method according to claim 1, characterized in that, An element-wise precision verification operation is performed on the output tensor of the operator in the inference or training operation to obtain the precision verification result of each operator, including: For any of the fifth or sixth type of operators, perform bit-level verification on each element of the output tensor of any type of operator and the standard reference value of the element to obtain the accuracy verification result of any type of operator. Based on the accuracy verification results of the operator, determine whether the operator meets the accuracy requirements, including: If the accuracy verification result of any type of operator indicates that each element of the operator is consistent with the standard reference value of the element at the bit level, it is determined that the operator has passed the element-by-element accuracy verification; otherwise, it is determined that the operator has failed the element-by-element accuracy verification.

10. The data processing method according to claim 1, characterized in that, An element-wise precision verification operation is performed on the output tensor of the operator in the inference or training operation to obtain the precision verification result of each operator, including: For the seventh type of operator, the element-wise precision verification operation is performed on each of the multiple sub-operators included in the seventh type of operator. Based on the accuracy verification results of the operator, determine whether the operator meets the accuracy requirements, including: When each sub-operator included in the seventh type of operator passes the element-wise precision verification, it is determined that the seventh type of operator has passed the element-wise precision verification; otherwise, it is determined that the seventh type of operator has not passed the element-wise precision verification.

11. The data processing method according to claim 1, characterized in that, The standard reference value corresponding to the eighth type of operator includes the first mean of the output results obtained by running the eighth type of operator N times by the reference device, where N is a positive integer greater than 1. An element-wise precision verification operation is performed on the output tensor of the operator in the inference or training operation to obtain the precision verification result of each operator, including: By executing the eighth type of operator N times, the second mean and variance of the output tensor of the eighth type of operator are obtained in N runs; Based on the second mean and the first mean, the error value corresponding to the eighth type of operator is determined; The error value corresponding to the eighth type of operator is compared with the error threshold corresponding to the eighth type of operator, and the variance is compared with the preset variance threshold range to obtain the accuracy verification result of the eighth type of operator; Based on the accuracy verification results of the operator, determine whether the operator meets the accuracy requirements, including: If the error value corresponding to the eighth type of operator is less than or equal to the error threshold corresponding to the eighth type of operator, and the variance is within the preset variance threshold range, it is determined that the eighth type of operator has passed the element-by-element accuracy verification; otherwise, it is determined that the eighth type of operator has not passed the element-by-element accuracy verification.

12. The data processing method according to any one of claims 1-11, characterized in that, Also includes: Based on the output tensor of each operator, a global error index for the operator is determined, wherein the global error index includes a first index for evaluating the overall deviation level of the operator, a second index for evaluating the error dispersion of the operator, and a third index for evaluating the systematic deviation direction of the operator. Based on the accuracy verification results of the operator, determine whether the operator meets the accuracy requirements, including: Based on the accuracy verification results of the operator and the global error index, it is determined whether the operator meets the accuracy requirements.

13. The data processing method according to claim 12, characterized in that, The first index of each operator is the mean of the relative errors of the multiple elements included in the operator. The first index is determined as follows: diff1 = = Where, diff1 represents the first index, and T represents the total number of the plurality of elements. This represents the relative error of the i-th element among the plurality of elements, where i takes values ​​of 1, 2, ..., T. To represent a positive number indicating the first preset threshold, This represents the value of the i-th element. This represents the standard reference value corresponding to the i-th element.

14. The data processing method according to claim 12, characterized in that, The second index of each operator is the root mean square of the relative errors of the multiple elements included in the operator.

15. The data processing method according to claim 12, characterized in that, The third index of each operator is the ratio between the sum of the error values ​​of the multiple elements included in the operator and the first product, wherein the first product is the product of the total number of the multiple elements and the maximum absolute value of the standard reference value among the multiple elements; If the third index of the operator is within a preset range including zero values, it indicates that the operator does not have systematic bias. If the third index of the operator is greater than the maximum value of the preset range, it indicates that the operator has a systematic positive bias. If the third index of the operator is less than the minimum value of the preset range, it indicates that the operator has a systematic negative bias.

16. The data processing method according to claim 12, characterized in that, Based on the accuracy verification results of the operator and the global error index, determine whether the operator meets the accuracy requirements, including: In response to the accuracy verification result of the operator indicating that the operator has passed the element-by-element accuracy verification, if the first index of the operator is less than or equal to a second preset threshold, the second index of the operator is less than a preset multiple of the first index, and the third index of the operator is within a preset range including zero values, it is determined that the operator meets the accuracy requirements; otherwise, it is determined that the operator does not meet the accuracy requirements.

17. The data processing method according to any one of claims 1-11, characterized in that, Also includes: Performance verification is performed on operators that meet the accuracy requirements; In response to the operator meeting the accuracy requirements passing the performance verification, the operator is determined to be qualified; The performance verification includes verifying whether the computational efficiency and interface efficiency of the operator meet the performance requirements.

18. A data processing apparatus, characterized in that, include: An operation module is configured to perform inference or training operations on a network model by a hardware device, wherein the network model includes multiple operators; The element-wise precision verification module is configured to perform an element-wise precision verification operation on the output tensor of the operator in the inference or training operation for each operator, and obtain the precision verification result of each operator. The output tensor of the operator includes multiple elements, and the precision verification result of the operator includes the precision verification result of each of the multiple elements. The accuracy verification module is configured to determine whether the operator meets the accuracy requirements based on the accuracy verification results of the operator. The element-by-element accuracy verification operation includes verifying the accuracy of each element in the output tensor of the operator based on the error threshold and standard reference value corresponding to the operator. The error threshold corresponding to the operator is related to the category to which the operator belongs; different categories of operators have independent error thresholds. The categories include Category 1, Category 2, Category 3, Category 4, Category 5, Category 6, Category 7, and Category 8. The first type of operator belonging to the first category is an operator that includes multiply-accumulate operations, and the second type of operator belonging to the second category is an operator that includes reduce-accumulate operations. The error threshold corresponding to the first type of operator is related to at least one of the data block size, accumulator precision, and accumulation execution order when the hardware device performs the inference or training operation. The error threshold corresponding to the second type of operator is related to the precision of the intermediate results generated when the hardware device performs the second type of operator. The third category of operators, belonging to the third category, consists of special function operators that use dedicated instructions; the fourth category of operators, belonging to the fourth category, consists of operators that perform stepwise basic operations. The fifth category of operators includes operators that perform logical operations or extremum operations, while the sixth category of operators includes operators that perform shape and size transformations. The seventh type of operator belonging to the seventh category is a composite operator, wherein the composite operator is formed by the fusion of multiple sub-operators. The eighth type of operator, belonging to the eighth category, includes operations where multiple threads compete for the same address, and the competition is resolved by hardware atomic operations.

19. A processor, characterized in that, It includes an instruction parsing unit and a precision verification unit, wherein, The instruction parsing unit is used to receive and parse precision verification instructions. After the instruction parsing unit parses the accuracy verification instruction, the accuracy verification unit executes the data processing method according to any one of claims 1-17.

20. An electronic device, characterized in that, include: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the data processing method according to any one of claims 1-17.

21. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they implement the data processing method according to any one of claims 1-17.

Images