Quantization method, computing device, and computer-readable storage medium

Abstract

The present disclosure discloses a quantization method of a neural network, a computing device and a computer readable storage medium. The computing device can be included in a combined processing device, which can further include an interface device and other processing devices. The computing device interacts with the other processing devices to jointly complete a user-specified computing operation. The combined processing device can further include a storage device connected to the computing device and the other processing devices respectively for storing data of the computing device and the other processing devices. The scheme of the present disclosure can greatly reduce the operation time required for computing quantization parameters while maintaining the required network accuracy.

Description

Technical Field

[0001] This disclosure generally relates to the field of artificial intelligence. More specifically, this disclosure relates to a method for quantizing neural networks, a computing device, a neural network chip, a circuit board, and a computer-readable storage medium. Background Technology

[0002] Deep learning has proven highly effective in tasks such as image classification, object detection, natural language processing, and artificial intelligence. With the advancement of deep learning research, increasingly deeper and larger networks have emerged. These larger and deeper networks offer better accuracy or performance. However, the better the network's performance, the greater the computational and memory requirements. Therefore, reducing the computational and storage overhead of neural networks is crucial to addressing this issue.

[0003] The industry typically uses quantization to solve this problem. Quantization involves replacing high-precision numbers (e.g., 32-bit floating-point numbers (FP32)) with lower-precision numbers (e.g., 8-bit fixed-point numbers (INT8)) for calculations. Through specific quantization methods, power consumption and computation time can be significantly reduced while maintaining the original performance.

[0004] Different types of neural networks, such as convolutional neural networks and fully connected neural networks, often employ different quantization methods. Traditional online quantization for recurrent neural networks (RNNs) involves calculating quantization parameters for the data of each RNN layer at every time point. This significantly increases the computational load for RNNs, leading to increased power consumption and longer computation time. Offline quantization for RNNs, on the other hand, uses the same set of quantization parameters for each RNN layer regardless of the time point, resulting in a decrease in the accuracy of the quantized network. Summary of the Invention

[0005] To address at least one or more of the technical problems mentioned above, this disclosure proposes a scheme for quantizing neural networks in several aspects. This scheme significantly reduces the computation time required to calculate the quantization parameters while maintaining a certain level of accuracy by sharing the same quantization parameters across all layers of the recurrent neural network at one or more time points. The neural network quantization scheme provided in this disclosure can be applied to various fields, such as image processing, video processing, speech / audio processing, text processing, etc. These processes may include, but are not limited to, object recognition, detection, and classification.

[0006] In a first aspect, this disclosure provides a computing device comprising: a memory storing program instructions and time nodes of a recurrent neural network and associated quantization parameters thereon, wherein one or more consecutive time nodes are associated with the same quantization parameters; and a processor coupled to the memory and configured to load the program instructions to perform: determining the current time node of data to be quantized in the recurrent neural network; retrieving the quantization parameters associated with the time node from the memory; and performing quantization on the data to be quantized using the quantization parameters.

[0007] In a second aspect, this disclosure provides a neural network chip, characterized in that the neural network chip includes the computing device described in any embodiment of the first aspect of this disclosure.

[0008] In a third aspect, this disclosure provides a board, characterized in that the board includes the neural network chip described in any embodiment of the second aspect of this disclosure.

[0009] In a fourth aspect, this disclosure provides a method for quantizing data of a recurrent neural network executed by a processor, the method comprising: determining the current time point of the data to be quantized in the recurrent neural network; obtaining predetermined quantization parameters associated with the time point; and performing quantization on the data to be quantized using the quantization parameters; wherein one or more consecutive time points of the recurrent neural network share the same quantization parameters.

[0010] In a fifth aspect, this disclosure provides a computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform the method described in any embodiment of the fourth aspect of this disclosure.

[0011] By utilizing the quantization method, computing device, neural network chip, board, and computer-readable storage medium provided above, the disclosed solution significantly reduces the computation time required to calculate quantization parameters while maintaining a certain level of network accuracy by time-divisionally decomposing the recurrent neural network and sharing the same quantization parameters across all layers of the recurrent neural network at one or more consecutive time points. Compared to existing online quantization methods, the solution disclosed in this embodiment significantly reduces the computation time required to calculate quantization parameters while maintaining quantization accuracy; and compared to existing offline quantization methods, it increases quantization accuracy without increasing the computation time required to calculate quantization parameters. Attached Figure Description

[0012] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0013] Figure 1 An exemplary structural diagram of a neural network to which embodiments of this disclosure can be applied is shown;

[0014] Figure 2 This illustrates an exemplary concept of a quantization method according to embodiments of this disclosure;

[0015] Figure 3 An exemplary flowchart of a method 300 for quantizing a recurrent neural network according to an embodiment of this disclosure is shown;

[0016] Figure 4 A hardware configuration block diagram of a computing device 400 that can implement the quantized neural network scheme of the present disclosure embodiments is shown;

[0017] Figure 5 A structural diagram of a combined processing apparatus 500 according to an embodiment of the present disclosure is shown; and

[0018] Figure 6 A schematic diagram of the structure of a board 600 according to an embodiment of the present disclosure is shown. Detailed Implementation

[0019] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] It should be understood that the terms "first," "second," and "third," etc., that may be used in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or sets thereof.

[0021] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0022] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0023] First, we provide explanations of the technical terms that may be used in this disclosure.

[0024] Floating-point numbers: The IEEE floating-point standard represents a number in the form V = (-1)∧sign*mantissa*2∧E. Here, sign is the sign bit, 0 for positive and 1 for negative; E represents the exponent, which is a weighted sum of 2 raised to the power of E (possibly a negative power); mantissa represents the mantissa, a binary fraction ranging from 1 to 2⁻ε, or 0 to ε. The representation of a floating-point number in a computer consists of three fields, each encoded separately:

[0025] (1) A single sign bit s directly encodes the sign s.

[0026] (2) The exponent field of the k-bit is encoded as exp=e(k-1)......e(1)e(0).

[0027] (3) The n-bit mantissa is used to encode the mantissa. However, the encoding result depends on whether the exponent stage is all 0.

[0028] Fixed-point numbers consist of three parts: a shared exponent, a sign bit, and a mantissa. The shared exponent means that the exponent is shared within a set of real numbers to be quantized; the sign bit indicates whether the fixed-point number is positive or negative; and the mantissa determines the number of significant digits, i.e., the precision. Taking an 8-bit fixed-point number as an example, its numerical calculation method is as follows:

[0029] value = (-1)sign ×(mantissa)×2 (exponent-127)

[0030] Data bit width: How many bits are used to represent the data.

[0031] Quantization is the process of converting high-precision numbers previously expressed in 32-bit or 64-bit form into fixed-point numbers that occupy less memory space, typically 16-bit or 8-bit. This conversion process inevitably results in some loss of precision.

[0032] Training set: also known as training dataset, is a set of samples typically used to train a model or determine model parameters (e.g., weights of a neural network).

[0033] Validation set: also known as validation dataset, is a sample set typically used to determine the model structure or control the complexity of the model.

[0034] Test set: also known as test dataset, is usually used to test the performance of a trained model, such as its generalization ability.

[0035] The following is a brief introduction to the neural network environments in which the embodiments disclosed herein can be applied.

[0036] Figure 1 An exemplary structural diagram of a neural network to which embodiments of this disclosure can be applied is shown.

[0037] like Figure 1 As shown, neural network 100 is a recurrent neural network, which can include three layers: an input layer, a hidden layer, and an output layer. The hidden layer can further include multiple layers. Figure 1 The diagram shows L hidden layers.

[0038] Recurrent neural networks are structures that repeat over time, taking the form of chains of repeating modules. Figure 1 The left side shows a repeating module of a recurrent neural network, where X is the input, O is the output, and H is the output. (1) H (2) H (L) These represent the states of each hidden layer. Figure 1 It can be seen that the repetitive module has a loop or cyclic body, which allows information to be passed from the current time step to the next time step. In other words, the state of each hidden layer at each time step is related to the input at the current time step and the state at the previous time step. Expanding this repetitive module temporally yields... Figure 1 The network structure shown on the right.

[0039] In each layer of a recurrent neural network, a series of inputs are calculated cyclically, and each cycle is called a time node. Figure 1 The right side shows an example of T time points, or T cycles. Figure 1 The inputs X1, X2, ..., X at T time points are shown. T And the corresponding outputs O1, O2, ..., O at the T time points. T However, it should be noted that output is not required at every point in time. Figure 1 The right side also shows the states of each hidden layer. For example, the states of the first hidden layer at T time points are H1 and H2 respectively. (1) H2 (1) H T (1) The states of the second hidden layer at T time points are H1 and H2 respectively. (2) H2 (2) H T (2) The states of the Lth hidden layer at time T are H1 and H2 respectively. (L) H2 (L) H T (L) The unfolded structure clearly shows that the state of each hidden layer at each time step is related to the input at the current time step and the state at the previous time step. In recurrent neural networks (RNNs), the input data exhibits temporal correlation; that is, the data at the previous time step influences the data at the next time step. Due to the repetitive block chain structure of RNNs, the parameters (including but not limited to weights, biases, activation functions, etc.) can be shared across different time steps.

[0040] There are many types of recurrent neural networks, including but not limited to Long Short-Term Memory (LSTM) networks, bidirectional recurrent neural networks, deep recurrent neural networks, and gated recurrent unit (GRU) networks. These recurrent neural networks have been successfully applied in many fields such as speech recognition, image captioning, and natural language processing.

[0041] The above combination Figure 1 An example structural diagram of a neural network that can implement the quantization method of the embodiments of this disclosure is described. It is understood that the above description of the neural network is merely exemplary, and the structure of the neural network is not limited to the structure shown in the diagram. Those skilled in the art can modify the structure shown in the diagram as needed, such as adding one or more new layers in the hidden layers, changing the operations in the loop body, etc.

[0042] Quantization is the process of mapping a set of input values ​​to a smaller set of values. For example, input values ​​can be rounded to a given unit of precision. Specifically, in one example, the conversion from floating-point numbers to fixed-point numbers can be a quantization process. According to computer architecture, for floating-point and fixed-point operations of the same length, floating-point operations are more complex and require more logic devices to construct the floating-point unit. Thus, floating-point units are larger than fixed-point units. Furthermore, floating-point units consume more resources, making the power consumption difference between fixed-point and floating-point operations typically orders of magnitude, resulting in significant differences in computational cost. However, experiments have shown that fixed-point operations are faster than floating-point operations with minimal precision loss. Therefore, using fixed-point operations to handle large amounts of neural network operations (such as convolution and fully connected operations) in artificial intelligence chips is a feasible solution. For example, floating-point data involving the inputs, weights, biases, activation functions, and gradients of forward convolution, forward fully connected, backward convolution, and backward fully connected operators can be quantized and then subjected to fixed-point arithmetic. After the operator operation is completed, the low-precision data can be converted into high-precision data.

[0043] As mentioned earlier, existing online quantization for recurrent neural networks (RNNs) significantly increases computational complexity, leading to higher power consumption and longer computation time, because the quantization parameters are calculated for each layer of the RNN at each time point, and the quantization parameters for each layer are different at each time point. Offline quantization for RNNs, on the other hand, uses the same set of quantization parameters for each layer or the entire RNN regardless of the time point, resulting in a decrease in the accuracy of the quantized network.

[0044] In practical engineering, the inventors discovered that the parameter characteristics of a recurrent neural network (RNN) differ significantly between later time points and those at the beginning. Therefore, to maintain this functional difference across time points, thereby preserving network accuracy while reducing computational load, this disclosure provides a quantization method for RNNs. According to this disclosure, by time-divisionally decomposing the RNN, all RNN layers share the same quantization parameters at one or more consecutive time points, significantly reducing the computation time required to calculate the quantization parameters while maintaining network accuracy.

[0045] Figure 2 An exemplary concept of a quantization method according to an embodiment of this disclosure is shown.

[0046] like Figure 2As shown, an exemplary recurrent neural network includes T time nodes, where T > 1. These T time nodes can be divided into N time periods Pi, i = 1...N, where 1 < N <= T, and each time period includes one or more consecutive time nodes. All recurrent neural network layers at all time nodes within each time period Pi share the same quantization parameter Qi, i = 1...N. Different time periods can correspond to different quantization parameters.

[0047] Each time period can contain a different number of time nodes. Figure 2 Exemplarily, it is shown that the first time period P1 includes j time nodes, and the Nth time period P N includes k time nodes. There can be multiple ways to divide the time nodes. For example, depending on the required network accuracy requirements and / or the parameter feature differences / similarities between individual time nodes in the recurrent neural network, there can be multiple different division schemes.

[0048] In some embodiments, N = T, that is, one time node constitutes one time period. In these embodiments, the same time nodes use the same quantization parameters, and different time nodes use different quantization parameters. That is, the first moment of all recurrent neural network layers uses the same quantization parameter, and the second moment uses another same quantization parameter.

[0049] In some other embodiments, N = T / M, that is, every M consecutive time nodes among the T time nodes constitute one time period. In one example, T = 100, M = 5, and at this time N = 20, that is, the time nodes of this recurrent neural network are divided into 20 time periods, and each time period includes 5 consecutive time nodes.

[0050] In yet some other embodiments, the number of time nodes within the N time periods can be unevenly distributed. For example, for T = 100, N = 4, the first time period P1 can include 20 time nodes, the second time period P2 can include 30 time nodes, the third time period P3 can include 30 time nodes, and the 4th time period P4 includes 20 time nodes.

[0051] Each time node / time period and its associated quantization parameter can be stored associatively in a memory through various data structures for use during quantization. These data structures can include but are not limited to lookup tables. By accessing these data structures, the quantization parameter associated with a certain time node of the recurrent neural network can be uniquely determined.

[0052] Figure 3An exemplary flowchart of a method 300 for quantizing a recurrent neural network according to an embodiment of this disclosure is shown. Method 300 may be executed by a processor, for example, to perform quantization operations on the data involved when performing inference operations using the neural network. The processor may be any processor that supports neural network processing, including but not limited to one or more types of processors such as a central processing unit (CPU), a graphics processing unit (GPU), and an artificial intelligence processor chip (IPU), both general-purpose and / or dedicated. Depending on the task scenario, the input data of the recurrent neural network may include, but is not limited to, images, videos, audio, text, etc., to achieve various tasks, including but not limited to classification, recognition, and detection of such input data.

[0053] like Figure 3 As shown, in step S310, the time node of the data to be quantized in the recurrent neural network is determined.

[0054] In a recurrent neural network (RNN), each layer iteratively computes a sequence of inputs, with each iteration representing a time point. As the RNN progresses to later time points, its parameter characteristics differ significantly from those at the beginning. Therefore, to maintain these functional differences across time points and thus preserve network accuracy, different quantization parameters can be applied to each time point. Consequently, when quantizing data, it's necessary to first determine the time point of the data to obtain the corresponding quantization parameters.

[0055] Next, in step S320, the predetermined quantization parameters associated with the time node are obtained.

[0056] As referenced above Figure 2 As described, the same quantization parameters can be shared across all layers of a recurrent neural network at one or more consecutive time points (which constitute a time interval). Each time point / time interval and its associated quantization parameters can be stored in memory in association using various data structures for use during quantization.

[0057] Therefore, in some embodiments, obtaining the quantization parameters associated with the time node may include accessing memory to retrieve the quantization parameters associated with the time node.

[0058] Furthermore, in some embodiments, when performing inference operations using a recurrent neural network, all quantization parameters can be read from the memory at once, for example, all quantization parameters can be read at a first time point, and then the corresponding quantization parameters can be applied at different time points / periods.

[0059] In some other embodiments, the associated quantization parameters can be retrieved by accessing the memory multiple times. For example, before performing quantization processing, the memory can be accessed at each time node or the first time node within each time period to obtain the quantization parameters associated with that time node or time period.

[0060] Finally, in step S330, the quantization is performed on the data to be quantized currently being processed using the obtained quantization parameters. Those skilled in the art can understand that a quantization method corresponding to the quantization parameters can be used to perform the quantization, and the embodiments of the present disclosure have no limitations in this regard.

[0061] The time nodes pre-stored in the memory and their associated quantization parameters can be determined in various ways in advance. In some embodiments, when optimizing the recurrent neural network using the validation dataset, the quantization parameters of all recurrent neural network layers at all time nodes of the recurrent neural network are generated. Subsequently, the quantization parameters shared within each time period in the recurrent neural network can be determined based on the statistical values of the pre-generated quantization parameters of all recurrent neural network layers at all time nodes within the corresponding time period. The statistical values can include but are not limited to the average value, the mode, and the maximum value.

[0062] In one example, assume that a certain recurrent neural network includes L network layers and T time nodes. When optimizing the recurrent neural network using the validation dataset, the quantization parameters Qtl of all recurrent neural network layers at all time nodes of the recurrent neural network are generated, where t = 1…T and l = 1…L. Further assume that these T time nodes of the recurrent neural network are divided into N time periods Pi, where i = 1…N, and 1 < N <= T. Each time period Pi includes Ki consecutive time nodes, and Ki = 1…(T - 1).

[0063] At this time, the quantization parameters Qi shared by each time period Pi can be calculated as follows:

[0064] Q i = F(Qpq),

[0065] where

[0066] and

[0067] That is, at each time node within the time period Pi, all recurrent neural network layers use the quantization parameter Qi calculated above to perform the quantization operation. Depending on the quantization method applied, the quantization parameters can exist in various forms. Generally, the quantization parameters can include but are not limited to: the maximum dynamic coding range, the quantization step size, the data bit width, the signed or unsigned indicator value, the point position parameter, the scaling factor, the offset, and so on.

[0068] In one embodiment, the quantization parameter is the point position parameter s. In this case, the data to be quantized can be quantized using the following formula (1) to obtain the quantized data I. x :

[0069]

[0070] Where s is the point position parameter, I x F is the n-bit binary representation of data x after quantization. x Let x be the floating-point value before quantization, and round is the rounding operation. It should be noted that this is not limited to round, but other rounding operations can also be used, such as rounding up, rounding down, and rounding to zero, to replace the round operation in formula (1).

[0071] In another embodiment, the quantization parameter is a first scaling factor f1. In this case, the data to be quantized can be quantized using the following formula (2) to obtain quantized data I. x :

[0072]

[0073] Where f1 is the first scaling factor, I x F is the n-bit binary representation of data x after quantization. x `x` represents the floating-point value of the data before quantization, and `round` performs a rounding operation. Similarly, this is not limited to `round` as the only rounding operation; other rounding methods can also be used.

[0074] In another embodiment, the quantization parameter is the point position parameter s and the second scaling factor f2. In this case, the data to be quantized can be quantized using the following formula (3) to obtain the quantized data I. x :

[0075]

[0076] Where s is the point position parameter, and f2 is the second scaling factor; I x F is the n-bit binary representation of data x after quantization. x `x` represents the floating-point value of the data before quantization, and `round` performs a rounding operation. Similarly, this is not limited to `round` as the only rounding operation; other rounding methods can also be used.

[0077] In another embodiment, the quantization parameters include point position parameters and offset. In this case, the data to be quantized can be quantized using the following formula (4) to obtain quantized data I. x :

[0078]

[0079] Where s is the point position parameter, O is the offset, and I is the offset. x F is the n-bit binary representation of data x after quantization. x `x` represents the floating-point value of the data before quantization, and `round` performs a rounding operation. Similarly, this is not limited to `round` as the only rounding operation; other rounding methods can also be used.

[0080] The above is for reference only. Figure 3 The exemplary method flow describes possible implementations of the embodiments disclosed herein. As can be seen from the above description, by sharing the same quantization parameters across all layers of the recurrent neural network at one or more consecutive time points, the computation time required to calculate the quantization parameters can be significantly reduced compared to existing online quantization methods. Furthermore, compared to existing offline quantization methods, quantization accuracy is increased without increasing the computation time for calculating the quantization parameters.

[0081] Figure 4 A hardware configuration block diagram of a computing device 400 capable of implementing the quantized neural network scheme of the embodiments disclosed herein is shown. Figure 4 As shown, the computing device 400 may include a processor 410 and a memory 420. Figure 4 In the computing device 400, only the components relevant to this embodiment are shown. Therefore, it will be apparent to those skilled in the art that the computing device 400 may also include components related to... Figure 4 The following are common components with different constituent elements, such as: arithmetic unit.

[0082] The computing device 400 can correspond to a computing device with various processing functions, such as functions for generating neural networks, training or learning neural networks, quantizing floating-point neural networks into fixed-point neural networks, retraining neural networks, or running neural networks. For example, the computing device 400 can be implemented as various types of devices, such as personal computers (PCs), server devices, mobile devices, etc.

[0083] Processor 410 controls all functions of computing device 400. For example, processor 410 controls all functions of computing device 400 by executing program instructions stored in memory 420 on computing device 400. Processor 410 may be implemented by a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), artificial intelligence processor chip (IPU), etc., provided in computing device 400. However, this disclosure is not limited thereto.

[0084] In some embodiments, the processor 410 may include an input / output (I / O) unit 411 and a computation unit 412. The I / O unit 411 may be used to receive various types of data, such as input data and quantization parameters of a neural network. The computation unit 412 may be used to quantize the input data received via the I / O unit 411 and perform inference operations using the neural network. The computation result may be output by the I / O unit 411, for example. The output data may be provided to the memory 420 for use by other devices (not shown), or it may be directly provided to other devices.

[0085] Memory 420 is hardware used to store various data processed in computing device 400. For example, memory 420 can store processed data and data to be processed in computing device 400. Memory 420 can store datasets involved in neural network operations processed or to be processed by processor 410, such as data of an untrained initial neural network, intermediate data of a neural network generated during training, data of a neural network that has completed all training, compressed neural network data, quantization parameters, etc. Furthermore, memory 420 can store applications, drivers, etc., to be driven by computing device 400. For example, memory 420 can store various programs related to training algorithms, pruning algorithms, compression algorithms, etc., of the neural network to be executed by processor 410. Memory 420 can be DRAM, but this disclosure is not limited thereto. Memory 420 can include at least one of volatile memory or non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Volatile memory may include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, ferroelectric RAM (FeRAM), etc. In an embodiment, memory 420 may include at least one of hard disk drive (HDD), solid-state drive (SSD), high-density flash memory (CF), secure digital card (SD), micro-secure digital card (Micro-SD), mini-secure digital card (Mini-SD), extreme digital card (xD), cache, or memory stick.

[0086] In summary, the specific functions implemented by the memory 420 and processor 410 of the computing device 400 provided in the embodiments of this specification can be explained in comparison with the foregoing embodiments in this specification, and can achieve the technical effects of the foregoing embodiments. Therefore, they will not be repeated here.

[0087] In this embodiment, processor 410 can be implemented in any suitable manner. For example, processor 410 can take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc.

[0088] In some embodiments, the computing device 400 may be implemented on an artificial intelligence processor chip (neural network chip). An artificial intelligence processor chip is dedicated hardware for driving neural networks. The artificial intelligence processor chip may correspond to, for example, a neural processing unit (NPU), a tensor processing unit (TPU), a neural engine, etc., which are dedicated chips for driving neural networks, but this disclosure is not limited thereto.

[0089] In this disclosed embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the above-described method for quantizing a neural network.

[0090] Figure 5 This is a structural diagram illustrating a combined processing apparatus 500 according to an embodiment of this disclosure. Figure 5 As shown, the combined processing device 500 includes a computing processing device 502, an interface device 504, other processing devices 506, and a storage device 508. Depending on the application scenario, the computing processing device may include one or more computing devices 510, which can be configured to... Figure 4 The computing device 400 shown is used to perform the functions described herein in conjunction with the appendix. Figure 3 The described operation.

[0091] In different embodiments, the computing processing apparatus disclosed herein can be configured to perform user-specified operations. In exemplary applications, the computing processing apparatus can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included within the computing processing apparatus can be implemented as an artificial intelligence processor core or a portion of the hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as artificial intelligence processor cores or portions of the hardware structure of artificial intelligence processor cores, the computing processing apparatus disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure.

[0092] In exemplary operation, the computing processing device disclosed herein can interact with other processing devices through an interface device to jointly complete user-specified operations. Depending on the implementation, the other processing devices disclosed herein may include one or more types of processors such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and an artificial intelligence processor, both general-purpose and / or special-purpose processors. These processors may include, but are not limited to, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing processing device disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and other processing devices are considered together, they can be considered to form a heterogeneous multi-core structure.

[0093] In one or more embodiments, the other processing device may serve as an interface between the computing processing device disclosed herein (which may be specifically embodied in artificial intelligence, such as neural network operations) and external data and control, performing basic controls including but not limited to data transfer, starting and / or stopping the computing device. In another embodiment, the other processing device may also cooperate with the computing processing device to jointly complete computational tasks.

[0094] In one or more embodiments, the interface device can be used to transfer data and control commands between a computing processing device and other processing devices. For example, the computing processing device can obtain input data from other processing devices via the interface device and write it to on-chip storage (or memory) of the computing processing device. Further, the computing processing device can obtain control commands from other processing devices via the interface device and write them to on-chip control cache of the computing processing device. Alternatively or optionally, the interface device can also read data from the storage device of the computing processing device and transmit it to other processing devices.

[0095] Additionally or optionally, the combined processing apparatus disclosed herein may further include a storage device. As shown in the figures, the storage device is connected to both the computing processing device and the other processing device. In one or more embodiments, the storage device may be used to store data from the computing processing device and / or the other processing device. For example, the data may be data that cannot be fully stored in the internal or on-chip storage of the computing processing device or other processing device.

[0096] In some embodiments, this disclosure also discloses a chip (e.g. Figure 6 The chip shown is 602. In one implementation, the chip is a system-on-chip (SoC) and integrates one or more such... Figure 5 The combined processing unit shown is illustrated. This chip can be connected to external interface devices (such as...). Figure 6 The external interface device 606 shown is connected to other related components. These related components may be, for example, a camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. In some applications, the chip may integrate other processing units (e.g., video codecs) and / or interface modules (e.g., DRAM interfaces). In some embodiments, this disclosure also discloses a chip package structure that includes the aforementioned chip. In some embodiments, this disclosure also discloses a board that includes the aforementioned chip package structure. The following will be combined with… Figure 6 This board is described in detail.

[0097] Figure 6 This is a schematic diagram illustrating the structure of a board 600 according to an embodiment of this disclosure. Figure 6 As shown, the board includes a storage device 604 for storing data, which includes one or more storage cells 610. This storage device can be connected and transmit data with the controller 608 and the aforementioned chip 602 via, for example, a bus. Furthermore, the board also includes an external interface device 606, configured for data relay or switching between the chip (or a chip in a chip package) and an external device 612 (e.g., a server or computer). For example, data to be processed can be transferred from the external device to the chip via the external interface device. Alternatively, the calculation results of the chip can be transmitted back to the external device via the external interface device. Depending on the application scenario, the external interface device can have different interface forms, such as a standard PCIe interface.

[0098] In one or more embodiments, the controller in the disclosed board can be configured to regulate the state of the chip. Therefore, in one application scenario, the controller may include a microcontroller (MCU) for regulating the operating state of the chip.

[0099] Based on the above combination Figure 5 and Figure 6 Based on the description, those skilled in the art will understand that this disclosure also discloses an electronic device or apparatus that may include one or more of the aforementioned boards, one or more of the aforementioned chips, and / or one or more of the aforementioned combined processing apparatus.

[0100] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0101] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0102] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document divides them based on logical functions, but in actual implementation, there may be other division methods. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0103] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0104] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the disclosed solution is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0105] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0106] The foregoing can be better understood in accordance with the following terms:

[0107] Clause 1. A computing device comprising:

[0108] A memory storing program instructions and time nodes of a recurrent neural network and their associated quantization parameters, wherein one or more consecutive time nodes are associated with the same quantization parameter; and

[0109] A processor, coupled to the memory and configured to load the program instructions to perform:

[0110] Determine the time node at which the data to be quantized in the recurrent neural network is currently located;

[0111] Obtain quantization parameters associated with the time node from the memory; and

[0112] Perform quantization on the data to be quantized using the quantization parameters.

[0113] Clause 2. The computing device according to Clause 1, wherein the recurrent neural network includes T time nodes, where T>1, and the T time nodes are divided into N time periods, where 1<N<=T. Each time period includes one or more consecutive time nodes, and all recurrent neural network layers at all time nodes within each time period share the same quantization parameters, and different time periods correspond to different quantization parameters.

[0114] Clause 3. The computing device according to Clause 2, wherein the quantization parameters corresponding to each time period are determined based on statistical values of the quantization parameters of all recurrent neural network layers at all time nodes within the corresponding time period that are pre-generated, and the statistical values include any one of the following: average value, mode, and maximum value.

[0115] Clause 4. The computing device according to Clause 3, wherein the quantization parameters of all recurrent neural network layers at all time nodes within the corresponding time period that are pre-generated are generated when optimizing the recurrent neural network using a validation data set.

[0116] Clause 5. The computing device according to any one of Clauses 1-4, wherein the quantization parameters include at least one of the following:

[0117] Maximum dynamic coding range;

[0118] Quantization step size;

[0119] Data bit width;

[0120] Signed or unsigned indicator value;[[ID=--]] [[ID=--]]

[0121] Point position parameter;

[0122] Scaling factor; and

[0123] Offset.

[0124] Clause 6. The computing device according to any one of Clauses 1-5, wherein the data to be quantized is input data of the recurrent neural network, and includes at least one of the following: image, video, audio, text.

[0125] Clause 7. A neural network chip, characterized in that the neural network chip comprises a computing device as described in any one of Clauses 1-6.

[0126] Clause 8. A board, characterized in that the board comprises the neural network chip as described in Clause 7.

[0127] Clause 9. A method for quantifying data of a recurrent neural network executed by a processor, the method comprising:

[0128] Determining the time node at which the data to be quantified of the recurrent neural network is currently located;

[0129] Obtaining a pre-determined quantization parameter associated with the time node; and

[0130] Performing quantization on the data to be quantified by using the quantization parameter;

[0131] Wherein one or more consecutive time nodes of the recurrent neural network share the same quantization parameter.

[0132] Clause 10. The method according to Clause 9, wherein the recurrent neural network comprises T time nodes, where T>1, the T time nodes are divided into N periods, where 1<N<=T, each period comprises one or more consecutive time nodes, all time nodes within each period share the same quantization parameter, and different periods correspond to different quantization parameters.

[0133] Clause 11. The method according to Clause 10, wherein the quantization parameter corresponding to each period is determined based on a statistical value of quantization parameters of all recurrent neural network layers at all time nodes within the corresponding period, which is pre-generated, and the statistical value comprises any one of the following: average value, mode and maximum value.

[0134] Clause 12. The method according to Clause 11, wherein the quantization parameters of all recurrent neural network layers at all time nodes within the corresponding period, which are pre-generated, are generated when optimizing the recurrent neural network by using a validation data set.

[0135] Clause 13. The method according to any one of Clauses 9-12, wherein the quantization parameter comprises at least one of the following:

[0136] Maximum dynamic coding range;

[0137] Quantization step size;

[0138] Data bit width;

[0139] Signed or unsigned indicator value;

[0140] Dot position parameter;

[0141] Scaling factor; and

[0142] Offset.

[0143] Clause 14. The method according to any one of Clauses 9-13, wherein the data to be quantized is the input data of the recurrent neural network, which includes at least one of the following: image, video, audio, and text.

[0144] Clause 15. A computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform any of the methods described in Clauses 9-14.

Claims

1. A computing device, comprising: A memory storing program instructions, time nodes of a recurrent neural network, and their associated quantization parameters, where multiple consecutive time nodes are associated with the same quantization parameter; And A processor coupled to the memory and configured to load the program instructions to perform: Determine the time node at which the data to be quantized of the recurrent neural network is currently located; Obtain the quantization parameter associated with the time node from the memory; And Perform quantization on the data to be quantized using the quantization parameter; Where the data to be quantized is the input data of the recurrent neural network, which includes at least one of the following: images, videos, audio, text; Each cycle of the recurrent neural network is a time node; Where the recurrent neural network includes T time nodes, where T > 1, the T time nodes are divided into N periods, where 1 < N < T, each period includes multiple consecutive time nodes, and all recurrent neural network layers at all time nodes within each period share the same quantization parameter, and different periods correspond to different quantization parameters.

2. The computing device according to claim 1, wherein, The quantization parameter corresponding to each period is determined based on a pre-generated statistical value of the quantization parameters of all recurrent neural network layers at all time nodes within the corresponding period, and the statistical value includes any one of the following: average value, mode, and maximum value.

3. The computing device according to claim 2, wherein, The pre-generated quantization parameters of all recurrent neural network layers at all time nodes within the corresponding period are generated when optimizing the recurrent neural network using a validation data set.

4. The computing device according to any one of claims 1 - 3, where the quantization parameter includes at least one of the following: Maximum dynamic coding range; Quantization step size; Data bit width; Signed or unsigned indicator value; Point position parameter; Scaling factor; And Offset.

5. A neural network chip, characterized in that, The neural network chip includes the computing device according to any one of claims 1 - 4.

6. A circuit board, characterized in that, The board includes the neural network chip according to claim 5. ​ ​ ​ ​ ​ ​ ​ ​ ​ 8. The method according to claim 7, wherein, The quantization parameters for each time period are determined based on the statistical values ​​of the quantization parameters of all recurrent neural network layers at all time nodes within the corresponding time period, which are pre-generated and include any of the following: mean, mode, and maximum.

9. The method according to claim 8, wherein, The quantization parameters of all recurrent neural network layers at all time points within the corresponding time period are generated when the recurrent neural network is optimized using a validation dataset.

10. The method according to any one of claims 7-9, wherein the quantization parameter comprises at least one of the following: Maximum dynamic coding range; Quantization step size; Data bit width; A signed or unsigned indicator value; Point position parameters; Scaling factor; as well as Offset.

11. A computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform the method according to any one of claims 7-10.

Images