Processing method and related device of neural network model

Abstract

This application proposes a neural network model processing method and related devices, the method includes: obtaining m, Z a , M0, Z w ,q w and q b , m and M0 satisfy the relationship M=2 ―m M0, M satisfies the relationship #imgabs0# to obtain Z using the shift quantization method a , Z w ,q w and q b The #imgabs1# and #imgabs2# obtained by quantization are determined according to m and #imgabs3#, #imgabs4# is determined according to m and #imgabs5#, #imgabs6# is determined according to m and #imgabs7#, #imgabs8# is determined according to m and #imgabs9#, and #imgabs10# is determined so that the fixed-point calculation expression of the first network layer is updated to the first expression #imgabs11#. The method in the embodiment of the present application can reduce the resources consumed by the neural network inference process.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and more specifically, to a processing method, processing device, computer storage medium, and computer program product for a neural network model. Background Art

[0002] In the field of artificial intelligence, fixed-point quantization can be used to implement operations in neural network models. For example, fixed-point quantization can be used to implement matrix operations in neural network models.

[0003] When using fixed-point quantization calculation methods to implement operations in neural network models, it is necessary to increase the calculation speed and reduce the memory occupied and power consumed. Summary of the Invention

[0004] The embodiments of the present application provide a method for processing a neural network model, a processing device, a computer storage medium, and a computer program product. Various aspects of the embodiments of the present application are described below.

[0005] In the first aspect, a method for processing a neural network model is provided, comprising: obtaining m, Z a , M0, Z w ,q w and q b , m and M0 satisfy the relationship M=2 ―m M0, M satisfy the relationship Among them, S w It represents the ratio between the real number weight and the fixed point number weight of the first network layer in the neural network model. The floating point calculation expression of the first network layer is a=wx+b, S x represents the ratio between the real number input and the fixed point number input of the first network layer, S a Represents the ratio between the real number output and the fixed-point number output of the first network layer, M0 is a fixed-point number, q b represents the fixed-point number of b, Z w Indicates the fixed-point weight corresponding to 0 in the real number after quantization, Z a Indicates the fixed-point output corresponding to 0 in the real number after quantization, q w To use the scaling factor S w The fixed-point weights obtained by quantizing the real-number weights of the first network layer; obtaining the fixed-point weights obtained by quantizing Z using the shift quantization method a 、Z w ,q w and q b The quantified and According to m and Sure According to m and Sure According to m and Sure According to m and Sure So that the fixed-point calculation expression of the first network layer is updated to the first expression q a Represents the fixed-point output of the first network layer.

[0006] In a second aspect, a processing device for a neural network model is provided, comprising: an acquisition unit for acquiring m, Z a , M0, Z w ,q w and q b , m and M0 satisfy the relationship M=2 ―m M0, M satisfy the relationship Among them, S w It represents the ratio between the real number weight and the fixed point number weight of the first network layer in the neural network model. The floating point calculation expression of the first network layer is a=wx+b, S x represents the ratio between the real number input and the fixed point number input of the first network layer, S a Represents the ratio between the real number output and the fixed-point number output of the first network layer, M0 is a fixed-point number, q b represents the fixed-point number of b, Z w Indicates the fixed-point weight corresponding to 0 in the real number after quantization, Z a Indicates the fixed-point output corresponding to 0 in the real number after quantization, q w To use the scaling factor S w The fixed-point weight obtained by quantizing the real-number weight of the first network layer; the obtaining unit is used to obtain the fixed-point weight obtained by quantizing the real-number weight of Z using the shift quantization method. a , Z w ,q w and q b The quantified and Determine the unit for m and Sure According to m and Sure According to m and Sure According to m and Sure So that the fixed-point calculation expression of the first network layer is updated to the first expression q a Represents the fixed-point output of the first network layer.

[0007] In a third aspect, a processing device for a neural network model is provided, comprising a memory and a processor, wherein the memory is used to store programs, and the processor is used to call the programs in the memory to execute the method described in the first aspect.

[0008] In a fourth aspect, a chip is provided, comprising: a memory for storing code; and a processor for executing the code stored in the memory to perform the method described in the first aspect.

[0009] In a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program is used to execute the method described in the first aspect.

[0010] In an embodiment of the present application, the shift operation associated with the scaling factor in the fixed-point calculation formula of the scaling factor method of the network layer is integrated into the shift quantization operation of the network layer weight in the preprocessing stage of the neural network model, thereby reducing the shift operation of the neural network model during the inference process, and further reducing the resources consumed by the inference process.

[0011] In addition, in an embodiment of the present application, the multiplication operation associated with the scaling factor in the fixed-point calculation formula of the scaling factor method of the network layer is integrated into the weights of the network layer or other layers in the preprocessing stage of the neural network model, thereby reducing the multiplication operations of the neural network model in the reasoning process, and further reducing the resources consumed by the reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a structural diagram of a neural network model of an embodiment of the present application.

[0013] Figure 2 This is an exemplary architecture diagram of a neural network system according to one embodiment of the present application.

[0014] Figure 3 A schematic diagram of the chip hardware structure provided in an embodiment of the present application.

[0015] Figure 4 It is a flowchart of a method for processing a neural network model provided in one embodiment of the present application.

[0016] Figure 5 It is a structural diagram of a processing device for a neural network model provided in one embodiment of the present application.

[0017] Figure 6 It is a schematic diagram of the structure of the device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0019] The embodiments of the present application involve a large number of applications related to neural networks. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts of neural networks that may be involved in the embodiments of the present application are first introduced below.

[0020] (1) Neural Network

[0021] A neural network can be composed of neural units, which can be represented by x s The output of the operation unit can be shown as formula (1):

[0022]

[0023] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0024] (2) Deep Neural Networks

[0025] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. Based on the location of the different layers, the neural network within a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Each layer is fully connected, meaning that any neuron in layer i is connected to any neuron in layer i+1.

[0026] Although DNN looks complicated, the work of each layer is actually not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0027] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0028] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0029] (3) Convolutional Neural Network

[0030] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as position-independent information extraction. Convolution kernels can be initialized as matrices of random size, and during CNN training, they learn to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0031] (4) Loss function

[0032] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss as much as possible.

[0033] (5) Backpropagation algorithm

[0034] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0035] Since CNN is a very common neural network, the following Figure 1 This section focuses on a detailed introduction to the structure of CNNs. As mentioned in the basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. Deep learning architectures use machine learning algorithms to perform multiple levels of learning at different levels of abstraction. As a deep learning architecture, a CNN is a feed-forward artificial neural network, in which individual neurons respond to input data to be processed.

[0036] like Figure 1As shown, a convolutional neural network (CNN) 100 may include an input layer 110, a convolutional layer 120, a fully connected layer 130, an activation function layer 140, and an output layer 150. The following describes the relevant contents of these layers in detail.

[0037] It is understandable that Figure 1 The CNN shown is only an example. The CNN in the embodiments of the present application may include more or fewer layers. For example, it may include more convolutional layers, fully connected layers, activation function layers, and pooling layers.

[0038] The input layer 110 can obtain data to be processed, which can be an input image or other data. The data to be processed can be a matrix, for example, a two-dimensional or three-dimensional matrix.

[0039] The convolution layer 120 may include multiple convolution operators, also known as kernels. In model processing, a convolution operator acts as a filter that extracts specific information from the input data (e.g., an input image). A convolution operator is essentially a weight matrix, which is typically predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally, pixel by pixel (or two pixels by two pixels, depending on the stride), to the input data (input matrix) to extract specific features from the data. The size of the weight matrix should be related to the size of the data. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input data. During the convolution operation, the weight matrix extends to the entire depth of the input data. Therefore, convolution with a single weight matrix produces a convolved output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used, but rather multiple weight matrices of the same size (row × column), i.e., multiple homogeneous matrices, are applied. The output of each weight matrix is ​​stacked up to form the depth dimension of the convolution data. The dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the data. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unwanted noise in the image. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by the multiple weight matrices of the same size are also the same size. The multiple feature maps of the same size extracted are then merged to form the output of the convolution operation.

[0040] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input data, so that the convolutional neural network 100 can make correct predictions.

[0041] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the subsequent convolutional layers become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0042] After processing by the convolutional layers, the convolutional neural network 100 is not yet sufficient to output the required output information. As previously mentioned, the convolutional layers 120 only extract features and reduce the number of parameters associated with the input. However, to generate the final output information (the required class information or other related information), the convolutional neural network 100 needs to utilize the fully connected layers 130 to generate one or a set of outputs for the required number of classes.

[0043] The activation function layer 140 classifies the output of the fully connected layer 130 and outputs the classification result. The activation function layer 140 can be a layer with the activation function of Relu; alternatively, the activation function layer 140 can also be a layer with the activation function of Linear, Prelu, or Sigmoid.

[0044] The output layer 150 has a loss function similar to the categorical cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

[0045] It should be noted that if Figure 1 The convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0046] Figure 2 This is an exemplary architecture diagram of a neural network system 200 according to one embodiment of the present application. Figure 2 In the example, the data acquisition device 260 is used to acquire training data. For the image classification method, the training data may include training images and classification results corresponding to the training images, wherein the classification results of the training images may be manually pre-labeled results.

[0047] After collecting the training data, the data collection device 260 stores the training data in the database 230 , and the training device 220 obtains the target model 201 through training based on the training data maintained in the database 230 .

[0048] The following describes how training device 220 generates target model 201 based on the training data, using an example where the input data includes an image. Training device 220 processes the input original image and compares the output image with the original image until the difference between the output image of training device 220 and the original image is less than a certain threshold, thereby completing the training of target model 201.

[0049] The above-mentioned target model 201 can be used for data processing, for example, to perform image classification on the input image, that is, the image to be processed is input into the target model 201 after relevant pre-processing, and the classification result of the image can be obtained. The target model 201 in the embodiment of the present application can specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 230 does not necessarily all come from the collection of the data acquisition device 260, and may also be received from other devices. It should also be noted that the training device 220 does not necessarily train the target model 201 entirely based on the training data maintained by the database 230, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0050] The target model 201 obtained by training the training device 220 can be applied to different systems or devices, such as Figure 2 The execution device 210 shown in the figure can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc. It can also be a server or a cloud device. Figure 2 In the embodiment of the present application, the execution device 210 is configured with an input / output (I / O) interface 212 for data interaction with an external device. The user can input data to the I / O interface 212 through the client device 240. The input data may include: an image to be processed input by the client device.

[0051] The preprocessing module 213 and the preprocessing module 214 are used to perform preprocessing based on the input data (such as the image to be processed) received by the I / O interface 212. In an embodiment of the present application, the preprocessing module 213 and the preprocessing module 214 may be omitted (or only one of the preprocessing modules may be present), and the computing module 211 may be used directly to process the input data.

[0052] When the execution device 210 preprocesses the input data, or when the computing module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call the data, code, etc. in the data storage system 250 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 250.

[0053] Finally, the I / O interface 212 returns the processing result, such as the output result obtained above (such as the classification result of the image to be processed), to the client device 240, thereby providing it to the user.

[0054] It is worth noting that the training device 220 can generate corresponding target models 201 based on different training data for different goals or different tasks. The corresponding target models 201 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0055] exist Figure 2 In the illustrated case, the user can manually input data, which can be operated through the interface provided by I / O interface 212. In another case, client device 240 can automatically send input data to I / O interface 212. If the automatic transmission of input data by client device 240 requires user authorization, the user can set the corresponding permissions in client device 240. The user can view the results output by execution device 210 on client device 240, which can be presented in the form of display, sound, action, etc. Client device 240 can also serve as a data acquisition terminal, collecting input data input into I / O interface 212 and output results output from I / O interface 212 as new sample data and storing them in database 230. Of course, collection can also be performed without client device 240, and instead the input data input into I / O interface 212 and output results output from I / O interface 212 as new sample data can be directly stored in database 230 by I / O interface 212.

[0056] It is understandable that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0057] like Figure 2As shown, the target model 201 is obtained by training according to the training device 220. The target model 201 can be the neural network in the embodiment of the present application. Specifically, the neural network provided in the embodiment of the present application can be CNN, deep convolutional neural networks (DCNN), etc.

[0058] Figure 3 A chip hardware structure provided in an embodiment of the present application includes a neural network processor 30. The chip can be set as follows Figure 2 The execution device 210 shown in FIG. 2 is used to complete the calculation work of the calculation module 211. The chip can also be set in Figure 2 The training device 220 shown in FIG. 2 is used to complete the training work of the training device 220 and output the target model 201. Figure 1 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 3 is implemented in the chip shown.

[0059] The neural network processor NPU 30 is mounted on the host CPU as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the arithmetic circuit 303. The controller 304 controls the arithmetic circuit 303 to extract data from the memory (weight memory or input memory) and perform calculations.

[0060] In some implementations, arithmetic circuit 303 includes multiple processing engines (PEs). In some implementations, arithmetic circuit 303 is a two-dimensional systolic array. Arithmetic circuit 303 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 303 is a general-purpose matrix processor.

[0061] For example, assume there are input matrix A, weight matrix B, and output matrix C. Operation circuit 303 retrieves the corresponding data of matrix B from weight memory 302 and caches it on each PE in operation circuit 303. Operation circuit 303 retrieves the data of matrix A from input memory 301 and performs matrix operations on matrix B. The partial or final results of the matrix are stored in accumulator 308.

[0062] As an example, the operation circuit 303 can implement matrix operations through shift operations. The implementation method of implementing matrix operations through shift operations can be referred to the following content.

[0063] The vector calculation unit 307 can further process the output of the operation circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0064] In some implementations, the vector calculation unit 307 can store the processed output vector in the unified buffer 306. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 303, for example, for use in a subsequent layer in a neural network.

[0065] The unified memory 306 is used to store input data and output data.

[0066] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 306 through the direct memory access controller 305 (DMAC), the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0067] The bus interface unit (BIU) 310 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 309 through the bus.

[0068] An instruction fetch buffer 309 connected to the controller 304 for storing instructions used by the controller 304;

[0069] The controller 304 is used to call the instructions cached in the memory 309 to control the working process of the computing accelerator.

[0070] Generally, the unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0071] in, Figure 1 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 303 or the vector calculation unit 307.

[0072] The convolutional layers and fully connected layers in a convolutional network are essentially a bunch of matrix multiplications. To improve computing speed, memory usage, and power consumption, floating-point matrix operations are converted to fixed-point matrix operations, which can be simply referred to as fixed-point operations. In the embodiments of this application, fixed-point operations can also be referred to as quantization operations or fixed-point quantization operations.

[0073] One implementation of fixed-point arithmetic is scaling factor fixed-point quantization. The following describes how to implement scaling factor fixed-point quantization.

[0074] r represents a floating point real number, and q represents a quantized fixed point number. As an example, the floating point real number may be a 32-bit floating point real number, and the fixed point number may be an 8-bit fixed point number.

[0075] The conversion formula between floating point and integer is:

[0076] r=S(q―Z) (2)

[0077]

[0078] Among them, S is scale, which indicates the proportional relationship between real numbers and fixed-point numbers; Z is zero point, which indicates the fixed-point number corresponding to 0 in the real number after quantization; round means rounding to the nearest integer.

[0079]

[0080]

[0081] r max 、r min are the maximum and minimum values ​​of r, q max ,q min are the maximum and minimum values ​​of q respectively.

[0082] It is understandable that the zero point of a fixed-point integer represents the zero of a floating-point real number, and there is no loss of precision in the conversion between the two. This can be seen from formula (3), where substituting r = 0 yields q = Z. The purpose of this is to ensure that the zero of a floating-point value and the zero point of a fixed-point integer are completely equivalent during padding, ensuring consistency between fixed-point and floating-point representations.

[0083] Assuming r1 and r2 are two N×N matrices of floating-point real numbers, and r3 is the matrix obtained by multiplying r1 and r2, then r3 can be expressed as:

[0084]

[0085] in, represents the element in row i and column j of matrix r1, represents the element in row j and column k of the r2 matrix, Represents the element in the i-th row and k-th column of the r3 matrix.

[0086] Assuming that S1 and Z1 are the scale and zero point corresponding to the r1 matrix, S2 and Z2 are the scale and zero point corresponding to the r2 matrix, and S3 and Z3 are the scale and zero point corresponding to the r3 matrix, then from formula (6) we can deduce:

[0087]

[0088] in, Represents the fixed-point number obtained by quantizing the element in the i-th row and j-th column of the r1 matrix based on the scaling factor S1. Represents the fixed-point number obtained by quantizing the element in the j-th row and k-th column of the r2 matrix based on the scaling factor S2. It represents the fixed-point number obtained by quantizing the element in row i and column k of the r3 matrix based on the scaling factor S3.

[0089] After sorting out formula (7), we can get:

[0090]

[0091] Assumptions M can be expressed as M=2 ―m M0, where M0 is a fixed-point number. This way the entire process is calculated in fixed-point.

[0092] It is understandable that fixed-point numbers are not necessarily integers. The so-called fixed-point means that the position of the decimal point is fixed, that is, the number of decimal places is fixed.

[0093] The following describes an exemplary fixed-point computation method for a neural network layer that includes matrix operations. Examples of such neural network layers include convolutional layers and fully connected layers.

[0094] S and Z represent scale and zero point, r represents a floating point real number, and q represents a fixed point integer. As an example, the floating point real number can be a 32-bit floating point real number, and the fixed point integer can be an 8-bit fixed point integer.

[0095] Assuming that the weight of the layer is w, the bias is b, the input is x, and the output activation value is a, the floating-point calculation of the layer can be expressed as:

[0096]

[0097] Based on the quantization relationship between floating-point numbers and fixed-point numbers, the floating-point operation of this layer can be expressed as the following fixed-point operation expression:

[0098]

[0099] Can be adjusted to:

[0100]

[0101] Here you can directly use S w S x To replace S b , Z b If it is directly recorded as 0, then formula (11) can be adjusted as follows:

[0102]

[0103] in,

[0104] Because M can be realized by a fixed-point number plus a bit shift operation, formula (12) can be calculated by fixed-point operation.

[0105] If M is expressed as M=2 ―m M0, where M0 is a fixed-point real number, then formula (12) can be adjusted to:

[0106]

[0107] Because Z w ,q w , Z x ,q b can be calculated in advance, so It can also be calculated in advance. When actually reasoning, you only need to calculate q x and calculate and That's it.

[0108] In this embodiment, the calculation and When , fixed-point acceleration can be achieved through shift operations. The following describes an exemplary implementation method for achieving fixed-point acceleration through shift operations.

[0109] is the weight q w Set N sub-codebooks, each codebook contains R=2 B ―1 codeword, each codeword has a width of B bits, where B is the number of bits of the fixed-point number. The quantized weight can be expressed as follows:

[0110]

[0111] Among them, C n ={0,±2 ―n+1 ,±2 ―n+2 ,±2 ―n―[N / 2] +2}, C n is the nth in N subcodebooks; idx i (n) yes The corresponding index in the nth subcodebook.

[0112] In other words, the essence of achieving fixed-point acceleration through shift operations is to replace multiplication with shift. Assume that the multiplication y = q w q x ,q w It has been quantified into a number of 2 n The form of summation Then the multiplication process can be simplified to the form of shift addition.

[0113] For example, for N=2, the sub-codebook includes 2 ―1 and 2 ―2 , then the multiplication y=q w q x Can be optimized to

[0114] In summary, among the methods of achieving fixed-point acceleration through shift operations, 2 ―m M0 The calculation process can include:

[0115] Get q w All code words, based on each code word pair q w Shift and obtain the cumulative sum of the shift results corresponding to all code words to obtain Get Z w All code words, based on each code word pair q xShift and obtain the cumulative sum of all codeword shift results to obtain calculate in, Can be calculated in advance; based on 2 ―m right Shift the result of the calculation; multiply the shifted result by M0, and then add Z a .

[0116] The inventors have found through research that the above calculation process can be optimized to save resource consumption in the reasoning process and speed up the calculation.

[0117] In the technical solution proposed by the inventor, before using the neural network layer containing matrix multiplication for inference, 2 ―m Integration In each item in . In this way, in actual reasoning, it can be reduced The calculation results of 2 ―m Shift operation.

[0118] As an example, for each quantized weight Can be pre-matched with 2 ―m Fusion, the quantized weight after fusion is

[0119] With m as 1, the quantized weight is For example, the quantized weight after fusion is That is, to calculate q w q x When q x Shift right by 2 bits and right by 3 bits respectively, and then accumulate the values ​​of the two shifts.

[0120] In this way, we can calculate After that, you don't need to The result of the calculation is shifted to the right by m bits.

[0121] In the technical solution proposed by the inventor, M0 can be integrated into the weight of the network layer. As an example, M0 can be integrated into the weight of the current layer or other layers.

[0122] For example, the current layer is a convolution layer, and the next layer of the convolution layer can be an activation function layer. The activation function corresponding to the activation function layer can be linear, relu, prelu or sigmoid, etc. In this way, the calculation is After that, there is no need to multiply the calculated result by M0.

[0123] Because the number of weights in the activation function layer is less than that in the convolution layer, fusing M0 into the weights of the activation function layer can reduce the amount of calculation, increase the calculation speed, and reduce the resources consumed by the fusion compared to fusing it into the weights of the convolution layer.

[0124] Here, integrating M0 into the weight can be understood as obtaining the product of M0 and the weight, and using the product as a replacement for the original weight.

[0125] In some implementations, the present application provides ―m The fusion of M0 and M1 can be performed after the neural network model is trained. For example, after the neural network model is trained, during the quantization of the neural network model, or after the neural network model is quantized, the fusion of M1 and M2 can be performed after the neural network model is trained. ―m Fusion with M0.

[0126] For example, you can Figure 2 After the training device in the training obtains the target neural network model 201, Figure 2 The pre-processing module 213 and / or the pre-processing module 214 shown in FIG. ―m and the fusion of M0.

[0127] Figure 4 This is a method for processing a neural network model according to an embodiment of the present application. The method may include S410 to S430.

[0128] As an example, after a neural network model is obtained through training and quantized, the neural network model obtained through quantization is used as the neural network model to be processed, and the method is executed.

[0129] For example, the training device 220 trains to obtain the target model 201, and the preprocessing module 213 performs quantization preprocessing on the target model 201 to reconstruct the fixed-point calculation expression of the network layer that requires fixed-point calculation in the target model 201, and then the neural network model obtained by the preprocessing module 213 is used as the neural network model to be processed, and S410 is executed by the preprocessing module 214.

[0130] As another example, after the neural network model is quantized and trained, the neural network model obtained by quantization processing can be used as the neural network model to be processed, and the present method can be executed.

[0131] S410, obtain m, Z a , M0, Z w ,q w and q b , m and M0 satisfy the relationship M=2 ―m M0, M satisfy the relationship Among them, S wIt represents the ratio between the real number weight and the fixed point number weight of the first network layer in the neural network model to be processed. The floating point calculation expression of the first network layer is a=wx+b, S x Represents the ratio between the real number input and the fixed point number input of the first network layer, S a Represents the ratio between the real number output and the fixed-point number output of the first network layer, M0 is a fixed-point number, q b represents the fixed-point number of b, Z w Indicates the fixed-point weight corresponding to 0 in the real number after quantization, Z a Indicates the fixed-point output corresponding to 0 in the real number after quantization, q w To use the scaling factor S w The fixed-point weights are obtained by quantizing the real-number weights of the first network layer.

[0132] As an example, S w 、S x and S a It can be calculated by referring to formula (4).

[0133] As an example, Z w It can be calculated by referring to formula (5).

[0134] As an example, q b It can be calculated by referring to formula (3).

[0135] As an example, q w The weight may be obtained by quantizing each weight in the first network layer using the method shown in formula (5).

[0136] As an example, m, Z a , M0, Z w ,q w and q b The fixed-point calculation method for the first network layer is 2 ―m M0 m, Z in a , M0, Z w ,q w and q b .

[0137] As an example, the first network layer may include a convolutional layer or a fully connected layer, etc.

[0138] S420, obtain the Z using the shift quantization method a 、Z w ,q w and q b The quantified and

[0139] As an example, Z can be calculated using the method shown in formula (14) a 、Z w ,q w and q b Perform shift quantization respectively and get and

[0140] As an example, q w It is 3 o'clock, is 2 1 +2 0 .

[0141] As an example, at this time, the fixed-point expression of the first network layer is

[0142] S430, according to m and Sure According to m and Sure According to m and Sure According to m and Sure So that the fixed-point calculation expression of the first network layer is updated to the first expression q a Represents the fixed-point output of the first network layer.

[0143] As an example, is 2 ―m and The product of .

[0144] As an example, is 2 ―m and The product of .

[0145] As an example, is 2 ―m and The product of .

[0146] As an example, is 2 ―m and The product of .

[0147] With 2 ―m is 2 ―1 and is 2 1 +2 0 For example, is 2 ―1 (2 1 +2 0 )=20 +2 ―1 .

[0148] At this time, the fixed-point calculation expression of the first network layer is updated to the first expression M0

[0149] Optionally, the method 400 may further include step S440, which is as follows:

[0150] S440, determining a second weight of the second network layer according to M0 and the first weight of the second network layer, the second weight being the product of M0 and the first weight, so that the fixed-point calculation expression of the first network layer is updated to the second expression

[0151] Before executing S440 , the floating-point weights of the second network layer may be quantized using a scaling factor to obtain second weights.

[0152] As an example, when the first network layer is a convolutional layer, the second network layer can be a network layer located after the convolutional layer. Optionally, the second network layer can be an activation function layer, and the activation function corresponding to the activation function layer can be linear, relu, prelu, or sigmoid, etc. It is understandable that the second network layer can have the following characteristics: when a point is input, the calculated output is a point, and this output point does not depend on any other points other than the input point.

[0153] Optionally, the method 400 may further include step S450, which is specifically as follows:

[0154] S450, when using the neural network model to be processed for inference, based on the fixed-point input q of the first network layer x 、 and Determine the output q of the first network layer a ,q a ,q x 、 and The second expression is satisfied.

[0155] Before executing S450 , the floating-point input of the first network layer may be quantized to obtain a first fixed-point input.

[0156] Optionally, the method 400 may further include step S460, which is specifically as follows:

[0157] S460: When performing inference using the neural network model to be processed, determine the output of the second network layer based on the fixed-point input and the second weight of the second network layer.

[0158] Before executing S460 , the floating-point input of the second network layer may be quantized to obtain a second fixed-point input.

[0159] In this embodiment, optionally, more or fewer steps may be included.

[0160] For example, S430 may not be included. At this time, the fixed-point calculation expression of the first network layer is updated to

[0161] For another example, S440 may not be included. In this case, the fixed-point calculation expression of the first network layer is updated to

[0162] In this embodiment, optionally, in S440, the second weight of the first network layer can be determined according to M0 and the first weight of the first network layer, and the second weight is the product of M0 and the first weight. In this case, q in S410 w The fixed-point weight is obtained by quantizing the second weight using a scaling factor.

[0163] Combined with the above Figures 1 to 4 , describes the method embodiment of the present application in detail, and the following is combined with Figures 5 and 6 , the device embodiment of the present application is described in detail. It should be understood that the description of the device embodiment corresponds to the description of the method embodiment, so for parts not described in detail, reference can be made to the previous method embodiment.

[0164] Figure 5 Schematic diagram of the structure of the processing device of the neural network model provided in the embodiment of the present application. Figure 5 As shown, the apparatus 500 may include an acquiring unit 510 and a determining unit 520 .

[0165] Acquisition unit 410, used to obtain m, Z a , M0, Z w ,q w and q b , m and M0 satisfy the relationship M=2 ―m M0, M satisfy the relationship Among them, S w It represents the ratio between the real number weight and the fixed point number weight of the first network layer in the neural network model. The floating point calculation expression of the first network layer is a=wx+b, S x represents the ratio between the real number input and the fixed point number input of the first network layer, S a Represents the ratio between the real number output and the fixed-point number output of the first network layer, M0 is a fixed-point number, q brepresents the fixed-point number of b, Z w Indicates the fixed-point weight corresponding to 0 in the real number after quantization, Z a Indicates the fixed-point output corresponding to 0 in the real number after quantization, q w To use the scaling factor S w Fixed-point weights obtained by quantizing the real-number weights of the first network layer;

[0166] The acquisition unit 410 is used to obtain the Z a , Z w ,q w and q b The quantified and

[0167] Determine the unit for m and Sure According to m and Sure According to m and Sure According to m and Sure So that the fixed-point calculation expression of the first network layer is updated to the first expression q a Represents the fixed-point output of the first network layer.

[0168] Optionally, the determining unit 420 is further configured to determine a second weight of the second network layer according to M0 and the first weight of the second network layer of the neural network model, wherein the second weight is the product of M0 and the first weight, so that the fixed-point calculation expression of the first network layer is updated to a second expression

[0169] Optionally, the second network layer is located after the first network layer, and the output of the first network layer is the input of the second network layer.

[0170] Optionally, the first network layer is a convolutional layer, and the second network layer has the following characteristics: a point output is calculated based on a point input, and the point output does not depend on other point inputs other than the point input

[0171] Optionally, the determining unit 420 is further configured to: when performing inference using the neural network model, determine the output of the second network layer based on the fixed-point number input of the second network layer and the second weight.

[0172] Optionally, the determining unit 420 is further configured to: when using the neural network model for inference, based on the fixed-point number input q of the first network layer x 、 and Determine the output q of the first network layer a ,q a ,q x 、 and The second expression is satisfied.

[0173] Figure 6 It is a schematic structural diagram of a device provided in one embodiment of the present application. Figure 6 The dotted line in the figure indicates that the unit or module is optional. The device 600 can be used to implement the method described in the above method embodiment. The device 600 can be a processing device of a chip or a neural network model.

[0174] The device 600 may include one or more processors 610. The processor 610 may support the device 600 to implement the method described in the method embodiment above. The processor 610 may be a general-purpose processor or a special-purpose processor. For example, the processor may be a central processing unit (CPU). Alternatively, the processor may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0175] The apparatus 600 may further include one or more memories 620. The memories 620 store programs that can be executed by the processor 610, causing the processor 610 to perform the methods described in the above method embodiments. The memories 620 may be independent of the processor 610 or integrated into the processor 610.

[0176] The apparatus 600 may further include a transceiver 630. The processor 610 may communicate with other devices or chips via the transceiver 630. For example, the processor 610 may transmit and receive data with other devices or chips via the transceiver 630.

[0177] The present invention also provides a computer-readable storage medium for storing a program. The computer-readable storage medium can be applied to the apparatus provided in the present invention, and the program enables a computer to execute the method performed by the apparatus in each embodiment of the present invention.

[0178] The present application also provides a computer program product. The computer program product includes a program. The computer program product can be applied to the apparatus provided in the present application, and the program causes a computer to execute the method performed by the apparatus in each embodiment of the present application.

[0179] The embodiments of the present application also provide a computer program. The computer program can be applied to the apparatus provided in the embodiments of the present application, and the computer program enables a computer to execute the method executed by the apparatus in each embodiment of the present application.

[0180] It should be understood that in the embodiments of the present application, "B corresponding to A" means that B is associated with A and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A, but B can also be determined based on A and / or other information.

[0181] It should be understood that the term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0182] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0184] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0185] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0186] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be read by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0187] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for processing a neural network model, characterized in that: include: Get m, Z a , M0, Z w ,q w and q b , m and M0 satisfy the relationship M=2 ―m M0, M satisfy the relationship Among them, S w It represents the ratio between the real number weight and the fixed point number weight of the first network layer in the neural network model. The floating point calculation expression of the first network layer is a=wx+b, S x represents the ratio between the real number input and the fixed point number input of the first network layer, S a Represents the ratio between the real number output and the fixed-point number output of the first network layer, M0 is a fixed-point number, q b represents the fixed-point number of b, Z w Indicates the fixed-point weight corresponding to 0 in the real number after quantization, Z a Indicates the fixed-point output corresponding to 0 in the real number after quantization, q w To use the scaling factor S w Fixed-point weights obtained by quantizing the real-number weights of the first network layer, wherein the neural network model is used to perform image classification based on an input image to be processed to obtain a classification result of the image to be processed; Get the Z value using the shift quantization method a 、Z w ,q w and q b The quantified and According to m and Sure According to m and Sure According to m and Sure According to m and Sure So that the fixed-point calculation expression of the first network layer is updated to the first expression q a =M0 q a Represents the fixed-point output of the first network layer.

2. The method according to claim 1, characterized in that The method further comprises: Determine the second weight of the second network layer according to M0 and the first weight of the second network layer of the neural network model, wherein the second weight is the product of M0 and the first weight, so that the fixed-point calculation expression of the first network layer is updated to the second expression 3. The method according to claim 2, characterized in that The second network layer is located after the first network layer, and the output of the first network layer is the input of the second network layer.

4. The method according to claim 3, characterized in that The first network layer is a convolutional layer, and the second network layer has the following characteristics: point output is calculated based on point input, and the point output does not depend on other point inputs other than the point input.

5. The method according to claim 3 or 4, characterized in that The method further comprises: When the neural network model is used for inference, an output of the second network layer is determined based on the fixed-point number input of the second network layer and the second weight.

6. The method according to any one of claims 2 to 4, characterized in that The method further comprises: When using the neural network model for inference, based on the fixed-point input q of the first network layer x 、 and Determine the output q of the first network layer a ,q a ,q x 、 and The second expression is satisfied.

7. A processing device for a neural network model, characterized in that: include: Acquisition unit, used to obtain m, Z a , M0, Z w ,q w and q b , m and M0 satisfy the relationship M=2 ―m M0, M satisfy the relationship Among them, S w It represents the ratio between the real number weight and the fixed point number weight of the first network layer in the neural network model. The floating point calculation expression of the first network layer is a=wx+b, S x represents the ratio between the real number input and the fixed point number input of the first network layer, S a Represents the ratio between the real number output and the fixed-point number output of the first network layer, M0 is a fixed-point number, q b represents the fixed-point number of b, Z w Indicates the fixed-point weight corresponding to 0 in the real number after quantization, Z a Indicates the fixed-point output corresponding to 0 in the real number after quantization, q w To use the scaling factor S w Fixed-point weights obtained by quantizing the real-number weights of the first network layer, wherein the neural network model is used to perform image classification based on an input image to be processed to obtain a classification result of the image to be processed; The acquisition unit is used to obtain the Z a 、Z w ,q w and q b The quantified and Determine the unit for m and Sure According to m and Sure According to m and Sure According to m and Sure So that the fixed-point calculation expression of the first network layer is updated to the first expression q a =M0 q a Represents the fixed-point output of the first network layer.

8. A processing device for a neural network model, characterized in that: The system comprises a memory and a processor, wherein the memory is used to store a program, and the processor is used to call the program in the memory to execute the method according to any one of claims 1 to 6.

9. A chip, characterized in that: include: A memory for storing code; A processor, configured to execute the code stored in the memory to perform the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is used to execute the method according to any one of claims 1 to 6.

Images