Neural network processing method and device, storage medium and program product

Abstract

The invention relates to a neural network processing method and device, a storage medium and a program product, and relates to the technical field of network frameworks. The neural network comprises a convolutional layer; the method comprises the following steps: acquiring a first parameter and a second parameter; the first parameter comprises a quantization operator or an inverse quantization operator, the quantization operator is used for quantizing data to be quantized in the neural network to obtain quantized data, the inverse quantization operator is used for inverse quantization of the quantized data, and the second parameter is a parameter different from the first parameter; and fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer.

Description

Technical Field

[0001] This disclosure relates to the field of network framework technology, and in particular to a method, apparatus, storage medium and program product for processing neural networks. Background Technology

[0002] Currently, the model processes the input data to obtain the processing results. Taking in-vehicle images as the input data as an example, the model can be deployed on a vehicle to identify and process the in-vehicle images acquired by the vehicle. To achieve fast processing of in-vehicle images, model quantization methods have been developed. These methods can use quantization operators (Q) to quantize the data in the model, converting high-precision data into low-precision data, and dequantization operators (DQ) to dequantize the data in the model, restoring low-precision data to high-precision data. This allows the model to quickly output the processed in-vehicle image results.

[0003] In related technologies, quantization or dequantization operations are performed before the convolutional layers of the model. This increases the number of memory read and write operations, resulting in a slower final output of the model. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a neural network processing method, apparatus, storage medium, and program product.

[0005] According to a first aspect of the present disclosure, a method for processing a neural network is provided, the neural network including convolutional layers; the method includes:

[0006] Obtain a first parameter and a second parameter; the first parameter includes a quantization operator or a dequantization operator, the quantization operator is used to quantize the data to be quantized in the neural network to obtain quantized data, the dequantization operator is used to dequantize the quantized data, and the second parameter is a parameter different from the first parameter;

[0007] The calculation operations between the first parameter and the second parameter are merged into the tail calculation of the convolutional layer.

[0008] Optionally, the dequantization operator includes an activation dequantization operator and a weight dequantization operator, the first parameter is the activation dequantization operator located before the convolutional layer, and the second parameter is the weight dequantization operator located before the convolutional layer; the step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes:

[0009] The computational operations between the activation dequantization operator and the weight dequantization operator are merged into the tail computation of the convolutional layer.

[0010] Optionally, the first parameter is a first quantization operator located before the connection layer, and the second parameter is a first dequantization operator located before the connection layer; the step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes:

[0011] When the neural network is in the first network mode, the computational operations of the first quantization operator and the first dequantization operator are fused into the tail computation of the convolutional layer; in the first network mode, the first quantization operator, the first dequantization operator, and the second quantization operator are performed sequentially.

[0012] Optionally, the first parameter is a second inverse quantization operator located before the connection layer, and the second parameter is a scaling factor; the step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes:

[0013] When the neural network mode is the second network mode, the calculation operation between the second dequantization operator and the scaling factor is fused into the tail calculation of the convolutional layer; in the second network mode, the second dequantization operator and the third quantization operator are performed sequentially.

[0014] Optionally, the first parameter is a fourth quantization operator located after the convolutional layer, and the second parameter is the convolution result output by the convolutional layer; the step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes:

[0015] The computational operations between the fourth quantization operator and the convolution result output by the convolutional layer are merged into the tail computation of the convolutional layer.

[0016] Optionally, the first parameter is a third dequantization operator located before the addition layer, and the second parameter is a scaling factor; the step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes:

[0017] The calculation operation between the third dequantization operator and the scaling factor is merged into the tail calculation of the convolutional layer.

[0018] Optionally, the first parameter further includes a target scaling factor, and the second parameter includes the convolution result output by the convolutional layer; the step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes:

[0019] The calculation operation between the target scaling factor and the convolution result output by the convolutional layer is merged into the tail calculation of the convolutional layer; the target scaling factor is obtained based on the scaling factor.

[0020] Optionally, the calculation operation includes multiple calculation operations between multiple sets of first parameters and second parameters; the step of fusing the calculation operations between the first parameters and second parameters into the tail calculation of the convolutional layer includes:

[0021] The multiple computational operations are combined to obtain the target computational operation;

[0022] The target computation operation is integrated into the tail computation of the convolutional layer.

[0023] Optionally, the step of fusing the target computation operation into the tail computation of the convolutional layer includes:

[0024] Multiple addition operations are combined to obtain the target addition operation;

[0025] The target computation operation and the target addition operation are merged into the tail computation of the convolutional layer.

[0026] Optionally, after fusing the target computation operation and the target addition operation into the tail computation of the convolutional layer, the method further includes:

[0027] A first computing mode or a second computing mode after fusion is obtained; the first computing mode is a computing mode that does not include the residual network, and the second computing mode is a computing mode that includes the residual network.

[0028] Optionally, the first computation mode includes at least one of the following processing methods: tail operation combination, activation operation and output quantization operation;

[0029] The tail operation combination includes the target calculation operation and the target addition operation.

[0030] Optionally, the first computation mode includes at least one of the following processing methods: tail operation combination, activation operation, output quantization operation and residual operation;

[0031] The tail operation combination includes the target calculation operation and the target addition operation.

[0032] Optionally, the calculation operation between the first parameter and the second parameter includes a multiplication operation between the first parameter and the second parameter.

[0033] Optionally, the method further includes:

[0034] The execution order of the quantization operator or the dequantization operator is exchanged with at least one of the following network layers: unary operation layer, connection layer, addition layer.

[0035] Optionally, the method further includes:

[0036] Obtain the weights in the neural network after they have been quantized by the quantization operator;

[0037] The quantized weights are used as the weights of the convolutional layer.

[0038] Optionally, the data to be quantized includes weights; the convolutional layer includes multiple convolutional kernels; obtaining the quantization result of the weights in the neural network after being quantized by the quantization operator includes:

[0039] The weights of the multiple convolution kernels are quantized using multiple quantization operators to obtain the quantized weights of the multiple convolution kernels; wherein, the multiple convolution kernels correspond to multiple channels, and the multiple channels correspond to multiple weights.

[0040] Obtain the weights of the multiple convolutional kernels after quantization.

[0041] Optionally, the data to be quantized includes input data; the method further includes:

[0042] The input data is quantized using a quantization operator to obtain quantized input data; wherein the input data shares the quantization operator across multiple channels.

[0043] According to a second aspect of the present disclosure, a neural network processing apparatus is provided, comprising:

[0044] The acquisition module is configured to acquire a first parameter and a second parameter; the first parameter includes a quantization operator or a dequantization operator, the quantization operator is used to quantize the data to be quantized in the neural network to obtain quantized data, the dequantization operator is used to dequantize the quantized data, and the second parameter is a parameter different from the first parameter;

[0045] The fusion module is configured to fuse the calculation operations between the first parameter and the second parameter into the tail calculation of the convolutional layer.

[0046] According to a third aspect of the present disclosure, a processing apparatus for a neural network is provided, the neural network including a convolutional layer; comprising:

[0047] processor;

[0048] Memory used to store processor-executable instructions;

[0049] The processor is configured as follows:

[0050] The steps of performing the neural network processing method provided in the first aspect of the embodiments of this disclosure.

[0051] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the neural network processing method provided in the first aspect of the present disclosure.

[0052] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the neural network processing method provided in the first aspect of the present disclosure.

[0053] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0054] The calculation operations of the first and second parameters can be fused into the tail calculation of the convolutional layer, so that the calculation operations of the first and second parameters can be executed in the same stage as the tail calculation, instead of having to be executed separately. This reduces the number of global memory accesses caused by executing the calculation operations of the first and second parameters separately, thereby saving the time consumed by the calculation fusion result, and enabling the neural network to output the processing result of the input data more quickly.

[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0057] Figure 1 This is a flowchart illustrating a neural network processing method according to an exemplary embodiment.

[0058] Figure 2 This is a schematic diagram of a neural network according to an exemplary embodiment.

[0059] Figure 3 This is a schematic diagram of a neural network according to an exemplary embodiment.

[0060] Figure 4 This is a schematic diagram of a neural network according to an exemplary embodiment.

[0061] Figure 5 This is a schematic diagram of a neural network according to an exemplary embodiment.

[0062] Figure 6 This is a schematic diagram of a neural network according to an exemplary embodiment.

[0063] Figure 7 This is a schematic diagram of a neural network according to an exemplary embodiment.

[0064] Figure 8 This is a schematic diagram of a neural network according to an exemplary embodiment.

[0065] Figure 9 This is a schematic diagram of a neural network according to an exemplary embodiment.

[0066] Figure 10 This is a schematic diagram of a neural network according to an exemplary embodiment.

[0067] Figure 11 This is a block diagram illustrating a neural network processing apparatus according to an exemplary embodiment.

[0068] Figure 12 This is a block diagram illustrating a neural network processing apparatus according to an exemplary embodiment.

[0069] Figure 13 This is a block diagram illustrating a neural network processing apparatus according to an exemplary embodiment.

[0070] Figure 14 This is a block diagram illustrating a chip system according to an exemplary embodiment. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0072] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0073] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0074] Figure 1This is a flowchart illustrating a neural network processing method according to an exemplary embodiment, such as... Figure 1 As shown, it includes the following steps.

[0075] In step S11, the first parameter and the second parameter are obtained.

[0076] Pseudo-quantization operators can be inserted during the training of a neural network model. These operators can quantize the data to be quantized in the neural network or dequantize the already quantized data. During model training, the pseudo-quantization operator continuously updates its scaling factor and zero-point bias, thereby gradually reducing the model's output error until it falls below a preset value.

[0077] Pseudo-quantization operators include quantization operators and dequantization operators. Quantization operators are used to quantize input data or weights awaiting quantization in a neural network, converting high-precision data to low-precision quantized data. For example, converting a 32-bit floating-point number (float32) to 8-bit integer (INT8) quantized data. Quantizing high-precision input data or weights to low-precision quantized data has two advantages: First, 8-bit integer quantized data requires less storage space than 32-bit floating-point data (e.g., 4 bytes for a 32-bit floating-point number and 1 byte for an 8-bit integer). Second, converting floating-point data to integers simplifies computation, as integers are simpler and faster than floating-point numbers, allowing the neural network to output the processed input data more quickly. The dequantization operator is used to dequantize quantized input data or weights, etc., to restore low-precision data to high-precision data, such as converting 8-bit integers to 32-bit floating-point numbers, thereby ensuring the accuracy of the final output of the model.

[0078] The first parameter can be either a quantization operator or a dequantization operator. The second parameter differs from the first parameter; it can be any one of the dequantization operator, the convolution result, or a scaling factor. If the first parameter is a dequantization operator, the corresponding second parameter can also be a dequantization parameter or a scaling factor; if the first parameter is a quantization operator, the corresponding second parameter can be a scaling factor or the convolution result.

[0079] Current model quantization methods include Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). Post-training quantization is a quantization method performed after model training is complete. In post-training quantization, the model weights are fixed, and quantization parameters are statistically calculated using data calibration. These quantization parameters are then used to convert the model weights and activation functions from floating-point numbers to integers. Quantization-Aware Training (QAT) is a quantization method that introduces quantization during the model training phase. In QAT, pseudo-quantization operators are inserted during model training to simulate the impact of quantization operators on the model's weights and activation functions. By inputting scaling factors and / or zero-point biases into the pseudo-quantization operators, quantized data can be obtained.

[0080] Both of the above methods use quantization parameters. A quantizer is a method used to calculate statistical quantization parameters in model quantization. Quantizers include the Uniform Affine Quantizer and the Uniform Symmetric Quantizer.

[0081] The uniform affine quantizer includes two quantization operators: scaling and zero-point bias. The scaling factor maps the input data or weights to the integer range from the floating-point range. It is calculated by first determining the difference between the maximum and minimum floating-point values; then dividing this difference by the maximum and minimum integer values ​​respectively. Dividing the difference by the maximum value yields a first value, and dividing the difference by the minimum value yields a second value. The range between these two values ​​represents the range from the floating-point range to the integer range. Zero-point bias determines the integer corresponding to the zero of the floating-point number. Typically, the zero of the floating-point number is mapped to the center of the integer range, ensuring a balance between positive and negative numbers within the integer range.

[0082] Alternatively, the quantization formula for converting floating-point numbers to integers based on the scaling factor and zero-point offset is as follows:

[0083]

[0084] Where Q(x) is the quantized floating-point number; x is the unquantized floating-point number, which is the input data or weight data to be quantized mentioned above; scale is the scaling factor, which is the quantization operator mentioned above; zero_point is the zero-point bias.

[0085] The uniform symmetric quantizer is similar to the uniform affine quantizer, but the uniform symmetric quantizer includes a scaling factor, which maps the data to be quantized from the range of floating-point numbers to the range of integers.

[0086] Alternatively, the quantization formula for converting floating-point numbers to integers based on the scaling factor is as follows:

[0087]

[0088] Where Q(x) is the quantized floating-point number; x is the unquantized floating-point number, which is the input data or weight data to be quantized mentioned above; scale is the scaling factor, which is the quantization operator mentioned above.

[0089] In this embodiment of the disclosure, the scaling factor isoquantization operator in a uniform symmetric quantizer can be used to convert the input data or weight data to be quantized from floating-point numbers to integers.

[0090] Understandably, in related technologies, quantization operators are used to quantize the data to be quantized at each stage of the neural network in real time, and dequantization operators are used to dequantize the quantized data at each stage of the neural network. In this process, if the calculation of the quantization and dequantization operators is before the convolutional layer, the convolution result output by the convolutional layer needs to be written to global memory first. Since the quantization or dequantization operator needs to use the convolution result, it is necessary to read the convolution result from global memory again, perform quantization or dequantization operations on the convolution result, and finally write the quantized or dequantized data back to global memory for subsequent operations.

[0091] Global memory is a memory region accessible to all processors in a graphics processing unit (GPU) architecture. This global memory is typically located outside the GPU, while the processors reside inside. Therefore, reading and writing data from global memory is slow, and concurrent access conflicts can occur when multiple processors simultaneously read and write data from global memory, further slowing down the process. It is evident that accessing global memory is generally slower than accessing shared memory or other registers. Therefore, frequent access to global memory during quantization of data to be quantized or dequantization of quantized data results in slower final output processing, hindering the rapid processing of the model.

[0092] In this embodiment of the disclosure, after the calculation operation between the first parameter and the second parameter is fused into the tail calculation of the convolutional layer, it can also be understood that after the quantization operator or the dequantization operator is fused into the tail calculation of the convolutional layer, after the convolutional layer outputs the convolution result, it is not necessary to write the convolution result to the global memory first. Instead, the convolution result and the calculation operation are further processed, and finally the processing result is stored in the global memory. This reduces one global memory read and write operation, thereby making the model's processing speed faster.

[0093] It is understandable that pseudo-quantization operators are computational methods for quantization or dequantization, while quantization parameters or dequantization parameters are parameters used by pseudo-quantization operators during the quantization or dequantization process.

[0094] In step S12, the calculation operation between the first parameter and the second parameter is merged into the tail calculation of the convolutional layer.

[0095] Optionally, the calculation operation between the first parameter and the second parameter can be a multiplication operation between the first parameter and the second parameter.

[0096] Convolutional layers are key layers in neural networks used to process input data. A convolutional layer includes a convolutional kernel, which is used to extract local features of the input data. The convolutional kernel slides on the input data, and the dot product between the convolutional kernel and the local region of the input data is calculated to generate a feature map.

[0097] Tail computation of a convolutional layer refers to a series of computational steps after data is input into the convolutional layer and convolution is performed. These steps supplement the computation of the convolutional layer and are used to complete the output of the entire convolutional layer, thereby providing input for the next layer of the network.

[0098] In related technologies, during the training phase of the model's neural network, please refer to... Figure 2 As shown, pseudo-quantization operators can be inserted before and after the network layer, respectively. These pseudo-quantization operators include quantization and dequantization operators. Quantization operators are used to reduce data precision, while dequantization operators are used to restore data precision. Please refer to [link to relevant documentation]. Figure 2 As shown, taking a network consisting of convolutional layers and activation layers as an example, quantization operators can be inserted before the convolutional layers and after the activation layers.

[0099] During model training, updating the quantization parameters in the quantization and dequantization operators, such as updating the scaling coefficients and zero-point biases in the quantization and dequantization operators, can reduce the error of the final output of the model.

[0100] During the application phase of the model, pseudo-quantization operators bring additional memory accesses, resulting in slower output of processing results. Therefore, the computation location of these pseudo-quantization operators can be changed, shifting their computation time from before the convolutional layer to after the convolutional layer. This combines the computation process of pseudo-quantization operators with the tail computation process of the convolutional layer, reducing the number of global memory accesses and making the model output processing results faster.

[0101] The above technical solution can integrate the calculation operations of the first parameter and the second parameter into the tail calculation of the convolutional layer, so that the calculation operations of the first parameter and the second parameter can be executed in the same stage as the tail calculation, instead of performing the calculation operations of the first parameter and the second parameter separately. This reduces the number of global memory accesses, thereby saving the time consumed by the calculation fusion result, and enabling the neural network to output the processing result after processing the input data faster.

[0102] The following describes five scenarios involving the fusion of the calculation operations of the first parameter and the second parameter into the tail calculation of the convolutional layer in the embodiments of this disclosure.

[0103] For related technologies, please refer to Figure 2 As shown, please refer to Figure 2 The neural network shown comprises, from top to bottom, convolutional layers (Conv), batch normalization layers, and activation layers. In this neural network, pseudo-quantization operators are inserted before the convolutional layers, and dequantization operators include weight dequantization (WeightDQ) and activation dequantization (ActDQ).

[0104] Please see Figure 2 As shown, the data is divided into two branches before being input into the convolutional layer. The left branch, from top to bottom, consists of the weight quantization operator and the weight dequantization operator, while the right branch consists of the activation quantization operator and the activation dequantization operator for the input data.

[0105] Figure 2In the left branch, the weights are quantized using a weight quantization operator, and then the output of the weight quantization operator is dequantized using a weight dequantization operator. The dequantized weights are then used as one of the operands of the convolutional layer. Essentially, the weight quantization operator converts weights from high-precision (e.g., 32-bit floating-point numbers) to low-precision (e.g., 8-bit integers), and the weight dequantization operator converts the low-precision weights back to high-precision weights. High-precision weights are typically used during neural network training to ensure accuracy, while low-precision weights are used during application to accelerate prediction. Therefore, to reduce quantization errors, weight quantization and weight dequantization operators can be introduced during neural network training, allowing the neural network to adapt to the errors introduced by these operators during training. Furthermore, by optimizing the scaling coefficients in the weight quantization and weight dequantization operators, the errors introduced by these operators can be reduced, ensuring that the weights quantized by the weight quantization operator, after dequantization, are as close as possible to the original weights.

[0106] Figure 2 In the right-hand branch, the input data is quantized using a quantization operator, and the output of the activation quantization operator is dequantized using an activation dequantization operator. The dequantized input data is then used as another operand of the convolutional layer. The purpose of quantizing and dequantizing the weights is to introduce weight quantization and dequantization operators during the training phase of the neural network, allowing the neural network to adapt to the errors introduced by these operators during training. The two dequantized data are then input into the convolutional layer for convolution. The convolutional layer then inputs the convolution result into a batch normalization layer for processing, and the batch normalized output is input into the activation layer. The activation layer activates the batch normalized output to obtain the activation result. Finally, the activation result is quantized to obtain the quantized output, which is then input into the next convolutional layer.

[0107] As can be seen, a neural network application stage involves multiple quantization operators and multiple dequantization operators. The computation of these operators or dequantization operators involves accessing global memory. Since accessing global memory is relatively slow, frequent accesses lead to slower output of the final processing results, making it difficult for the model to respond quickly. Furthermore, directly computing quantization and dequantization operators introduces unnecessary computational overhead. Therefore, quantization operators can be fused into the tail computation of the convolutional layer preceding them, and dequantization operators can be fused into the tail computation of the convolutional layer following them. This reduces computation time and improves the inference speed of the neural network.

[0108] It is understood that in any embodiment of this disclosure, the purpose of quantization is to reduce the precision of the data, thereby reducing the storage space occupied by the data and increasing the computation speed; the purpose of dequantization is to ensure that the data input to the network layer is at its initial precision, thereby reducing the computational error caused by quantization.

[0109] In the first scenario, the first parameter is the activation dequantization operator, and the second parameter is the weight dequantization operator. In this scenario, the computational operations between the activation dequantization operator and the weight dequantization operator can be merged into the tail computation of the convolutional layer.

[0110] Please see Figure 3 The neural network shown consists of convolutional layers, batch normalization layers, and activation layers from top to bottom. A pseudo-quantization operator is inserted above the convolutional layers. The pseudo-quantization operator has two branches: the left branch is the weight dequantization operator, and the right branch is the activation dequantization operator. This allows the computational operations between the weight dequantization operator and the activation dequantization operator to be fused into the tail computation of the convolutional layer.

[0111] The activation layer is used to introduce non-linearity, enabling the network to learn more complex function mappings. The activation layer is used to apply the output of the convolutional layer or fully connected layer to the non-linear activation function.

[0112] Among them, the activation dequantization operator is a variable parameter, while the weight dequantization operator is a fixed parameter.

[0113] Optionally, a multiplication operation can be inserted after the convolutional layer, and the weight dequantization operator and the activation dequantization operator can be input into the multiplication operation to realize the calculation operation between the activation dequantization operator and the weight dequantization operator.

[0114] For example, the calculation result between the weighted dequantization operator and the activation dequantization operator can be obtained by the following formula:

[0115] scale = scale ActDQ *scale WeightDQ

[0116] Here, scale represents the fusion result, which is a vector; ActDQ It is the activation dequantization operator, which is a scalar; scale WeightDQ It is a weighted dequantization operator, which is a vector.

[0117] As can be seen, the product between the activation dequantization operator and the weight dequantization operator can be used as the result of the calculation between the two.

[0118] It is understandable that the matrix output by the convolutional layer is a 32-bit integer convolution result, but the fusion result between the activation dequantization operator and the weight dequantization operator is a 32-bit floating-point number. Therefore, in order to ensure that both can be input into the batch normalization layer with uniform precision, a cast (precision conversion) operation can be inserted after the convolutional layer to convert the convolution result output by the convolutional layer from a 32-bit integer to a 32-bit floating-point number. This makes the precision of the two operands of the batch normalization layer consistent, thus facilitating the processing by the batch normalization layer.

[0119] For related technologies, please refer to Figure 3 As shown, the computational operation between the activation dequantization operator and the weight dequantization operator is located before the convolutional layer. Therefore, the convolution result of the convolutional layer needs to be written to global memory first; then the convolution result is read from global memory, and the activation dequantization operator and the weight dequantization operator are used to dequantize the convolution result; finally, the dequantized output data is written to global memory, which requires frequent access to global memory.

[0120] In the first scenario, the neural network is Figure 3 In the case of the neural network shown, if there are activation dequantization and weight dequantization operators above the convolutional layer, the computational operations between the activation dequantization and weight dequantization operators are moved from before the convolutional layer to after the convolutional layer. Figure 3 The solid lines in the diagram represent the computational operations between the activation dequantization operator and the weight dequantization operator before the convolutional layer. Figure 3 The dashed lines in the diagram represent the computational operations between the activation dequantization operator and the weight dequantization operator after the convolutional layer. Thus, the tail computation of the convolutional layer and the computational operations between the activation and weight dequantization operators are processed in the same stage. After processing, the final output of the convolutional layer is written to global memory. Compared to the method of performing the activation and weight dequantization operations before the convolutional layer, performing the operations after the convolutional layer reduces the read-write process of writing the convolution result to global memory and then reading it back from global memory for dequantization, thereby reducing access to global memory and making the model faster.

[0121] In the second scenario, the first and second parameters are located before the connection layer, and the first and second parameters located before the connection layer will vary depending on the network mode. Therefore, the calculation process under the two different network modes will be introduced in the following two ways.

[0122] In Scheme A, the first parameter is a first quantization operator located before the connection layer, and the second parameter is a first dequantization operator located before the connection layer. In this scenario, the neural network mode is the first network mode, which can integrate the calculation operations of the first quantization operator and the first dequantization operator into the tail calculation of the convolutional layer.

[0123] In the first network mode, the first quantization operator, the first dequantization operator, and the second quantization operator are performed sequentially. In the first network mode, the first quantization operator, the first dequantization operator, and the second quantization operator are located before the connection layer, and the first quantization operator, the first dequantization operator, and the second quantization operator are performed sequentially.

[0124] Optionally, the computational operations between the first quantization operator and the first dequantization operator can be merged into the tail computation of the convolutional layer; then, the second quantization operator is converted into a specified combination of operations, including rounding, pruning, and precision conversion. Rounding is used to round the floating-point number after the dequantization operator to a closer integer representation; pruning is used to ensure that the integer is within the allowed range of quantization, and the range of the quantized integer can be [-128, 127]; precision conversion is used to convert the floating-point number into an integer. Precision conversion can also implement the quantization operator to convert the floating-point number into an integer, and similarly, precision conversion can also convert the integer into a floating-point number. It varies depending on the use case, and this disclosure does not limit it.

[0125] For example, the first network mode before fusion is represented as: Q→DQ→Q→concat, where the first Q is the first quantization operator, DQ is the first dequantization operator, the second Q is the second quantization operator, and concat is the connection layer.

[0126] The first network mode after fusion is represented as: Multiply→Round→Clip→Cast→Concat. Multiply represents the computational operation between the first quantization operator Q and the first dequantization operator DQ; Round→Clip→Cast is the combination of operations after the transformation of the second quantization operator Q. Round is the rounding process; Clip is the clipping process; Cast is the precision conversion process; and Concat is the connection layer.

[0127] Optionally, the computational operation between the first quantization operator and the first dequantization operator located before the connecting layer is fused into the tail computation of the convolutional layer located before the connecting layer.

[0128] Optionally, the computational operation between the first quantization operator and the first dequantization operator located before the connection layer includes a multiplication operation between the first quantization operator and the first dequantization operator. The multiplication operation can be inserted after the convolutional layer before the connection layer to realize the multiplication calculation between the first quantization operator and the first dequantization operator.

[0129] In related technologies, the computational operation between the first quantization operator and the first dequantization operator is located before the connection layer. The first quantization operator and the second dequantization operator still need to quantize and dequantize the convolution result of the convolutional layer before the connection layer. In this process, it is still necessary to read the convolution result output by the convolutional layer before the connection layer from the global memory, quantize the convolution result using the first quantization operator, dequantize it using the first dequantization operator, and then write the dequantized output data into the global memory. This requires frequent access to the global memory.

[0130] With scheme A, if the neural network is the first network mode, the computational operation between the first quantization operator and the first dequantization operator before the connection layer will be transferred to the convolutional layer. In this way, the computational operation of the first quantization operator and the first dequantization operator is executed at the same stage as the tail computation of the convolutional layer. After the processing is completed, the output result after the computational operation is written into the global memory, which does not require frequent access to the global memory.

[0131] Option B, where the first parameter is a second dequantization operator located before the connection layer, and the second parameter is a scaling factor, in this scenario, the neural network pattern is the second network pattern, and the calculation operation between the second dequantization operator and the scaling factor can be fused into the tail calculation of the convolutional layer.

[0132] The second network mode has one less quantization operator than the first network mode. In the second network mode, the second inverse quantization operator and the third quantization operator are performed sequentially. The second network mode has a second inverse quantization operator and a third quantization operator in front of the connection layer, and the second inverse quantization operator and the third quantization operator are performed sequentially.

[0133] Optionally, the computational operation between the second inverse quantization operator and the scaling factor is fused into the tail computation of the convolutional layer; then the third quantization operator is converted into a specified combination of operations.

[0134] For example, the second network mode before fusion is represented as: DQ→Q→concat, where DQ is the second inverse quantization operator; Q is the third quantization operator; and concat is the connection layer.

[0135] The fused second network mode is represented as: Cast→Multiply→Round→Clip→Cast→Concat. Cast converts integers to floating-point numbers; Multiply represents the calculation operation between the second inverse quantization operator and the scaling factor; Round→Clip→Cast is the combination of operations after the Q-transformation of the third quantization operator. Round is the rounding process; Clip is the clipping process; Cast is the precision conversion process; and Concat is the connection layer.

[0136] Optionally, the computational operation between the second inverse quantization operator located before the connecting layer and the scaling factor is fused into the tail computation of the convolutional layer located before the connecting layer.

[0137] Optionally, the computation operation between the second dequantization operator and the scaling factor located before the connection layer includes a multiplication operation between the second dequantization operator and the scaling factor. The multiplication operation can be inserted after the convolutional layer before the connection layer to realize the multiplication calculation between the second dequantization operator and the scaling factor.

[0138] In related technologies, the calculation operation between the second dequantization operator and the scaling factor is located before the connection layer. The second dequantization operator and the scaling factor still need to dequantize the convolution result of the convolutional layer before the connection layer. In this process, it is still necessary to read the convolution result output by the convolutional layer before the connection layer from the global memory, use the second dequantization operator and the scaling factor to dequantize the convolution result, and then write the dequantized output data into the global memory. This requires frequent access to the global memory.

[0139] With Scheme B, if the neural network is the second network mode, the calculation operation between the second dequantization operator and the scaling factor before the connection layer will be transferred to the convolutional layer. In this way, the calculation operation between the second dequantization operator and the scaling factor is executed at the same stage as the tail calculation of the convolutional layer. The second dequantization operator and the scaling factor can directly dequantize the convolution result without reading the convolution result from global memory, thus avoiding frequent access to global memory.

[0140] In the third scenario, the first parameter is the fourth quantization operator located after the convolutional layer, and the second parameter is the convolution result output by the convolutional layer. In this scenario, the computational operations between the fourth quantization operator and the convolution result output by the convolutional layer can be merged into the tail computation of the convolutional layer.

[0141] Please see Figure 4The neural network shown includes a first convolutional layer, a batch normalization layer, an activation layer, and a second convolutional layer in sequence. The left and right branches in front of the first convolutional layer are two dequantization operators, respectively. The left branch in front of the second convolutional layer has a quantization operator and a dequantization operator, respectively, and the right branch in front of the second convolutional layer also has a quantization operator and a fourth dequantization operator, respectively.

[0142] The first parameter is the parameter located after the network layer below the convolutional layer.

[0143] exist Figure 4 In the neural network shown, the fourth quantization operator after the first convolutional layer quantizes the convolution result of the first convolutional layer. Since this fourth quantization operator is located after the activation layer, the convolution result of the first convolutional layer is first written to global memory. When using the fourth quantization operator to quantize the convolution result of the first convolutional layer, the convolution result needs to be read from global memory, quantized again using the fourth quantization operator, and finally written back to global memory, resulting in frequent access to global memory. Therefore, the computational operations between the fourth quantization operator and the convolution result of the first convolutional layer can be merged into the tail operations of the first convolutional layer, reducing access to global memory.

[0144] Optionally, the computational operation between the fourth quantization operator located after the first convolutional layer and the convolution result output by the first convolutional layer includes a multiplication operation between the fourth quantization operator and the convolution result output by the first convolutional layer. The multiplication operation can be inserted after the first convolutional layer to realize the multiplication calculation between the fourth quantization operator and the convolution result output by the first convolutional layer.

[0145] For related technologies, please refer to Figure 4 As shown, the computation operation between the fourth quantization operator and the convolution result output by the first convolutional layer is located after the activation layer. Therefore, it is necessary to write the convolution result to global memory after the first convolutional layer, then read the convolution result from global memory, then use the fourth quantization operator to quantize the convolution result, and finally write the quantized result to global memory, resulting in frequent access to global memory.

[0146] In the third scenario, where the neural network is Figure 4In the case of the neural network shown, if a fourth quantization operator exists below the network layer below the convolutional layer, the computational operation between the fourth quantization operator and the convolution result output from the first convolutional layer is moved from the network layer below the convolutional layer to the tail computation of the convolutional layer. Thus, the tail computation of the first convolutional layer and the computational operation between the fourth quantization operator and the convolution result output from the first convolutional layer are processed in the same stage. After processing, the final output result of the first convolutional layer is written to global memory. Compared to the method where the computational operation between the fourth quantization operator and the convolution result is located after the network layer following the convolutional layer, this reduces the read-write process of writing the convolution result to global memory and then reading the convolution result from global memory for quantization, thereby reducing access to global memory and making the model faster.

[0147] In the fourth scenario, the first parameter is the third dequantization operator located before the addition layer, and the second parameter is the scaling factor. In this scenario, the calculation operation between the third dequantization operator and the scaling factor can be merged into the tail calculation of the convolutional layer.

[0148] Please see Figure 5 The neural network shown consists of a main branch network on the left and a residual network on the right. The main branch network includes convolutional layers, batch normalization layers, and addition layers. The residual network includes a quantization operator and a third inverse quantization operator. The quantization operator preceding the third inverse quantization operator can be eliminated in the third scenario by fusing it with the convolution result of the first convolutional layer. Therefore, the third inverse quantization operator remains uneliminated.

[0149] For the third dequantization operator, the calculation operation between the third dequantization operator and the scaling factor can be merged into the tail calculation of the convolutional layer after the third dequantization operator, so that it is not necessary to access global memory multiple times when calculating the third dequantization operator.

[0150] For example, the third dequantization operator DQ can be replaced with the Cast→Multiply operation combination, where Cast is used to convert data from integers to floating-point numbers; Multiply indicates the computational operation between the third dequantization operator and the scaling factor.

[0151] Optionally, the computation operation between the third dequantization operator and the scaling factor located before the addition layer includes a multiplication operation between the third dequantization operator and the scaling factor. The multiplication operation can be inserted after the convolutional layer located above the addition layer to realize the multiplication calculation between the third dequantization operator and the scaling factor.

[0152] In the fifth scenario, the first parameter further includes a target scaling factor, and the second parameter includes the convolution result output by the convolutional layer. In this scenario, the calculation operation between the target scaling factor and the second parameter (including the convolution result output by the convolutional layer) is fused into the tail calculation of the convolutional layer.

[0153] The target scaling factor is obtained from the scaling factor, and the target bias is obtained from the bias.

[0154] Please see Figure 3 As shown, below the convolutional layer is a batch normalization layer, and the calculation formula for the batch normalization layer is as follows:

[0155]

[0156] Where E[x] represents the moving average; x is the convolution result output by the convolutional layer; Var[x] represents the moving variance; γ represents the scaling factor shared by multiple channels; β represents the zero-point bias shared by multiple channels; ∈ is a constant added to avoid the denominator being 0; and y is the normalized data after normalization of the convolution result.

[0157] The above formula can be simplified to the following formula:

[0158] y = bn_scale * x + bn_bias

[0159] in, bn_scale is the target scaling factor; bn_bias is the target bias.

[0160] As can be seen from the above formula, the calculation formula for the batch normalization layer can be simplified to the product of the target scaling factor bn_scale and the convolution result x output by the convolutional layer, and then the target bias bn_bias is added on top of that.

[0161] As can be seen, multiplication operations also exist in the batch normalization layer. Therefore, the multiplication operation between the target scaling factor bn_scale and the convolution result x output by the convolutional layer can be merged into the tail calculation of the convolutional layer.

[0162] The batch normalization layer is used to normalize the data in each small batch, thereby adjusting and scaling the activation value.

[0163] It is understandable that both bn_scale and bn_bias are vectors.

[0164] Optionally, the calculation operation between the target scaling factor and the convolution result includes a multiplication operation between the target scaling factor and the convolution result; a multiplication operation can be inserted after the convolutional layer to realize the multiplication calculation between the target scaling factor and the convolution result.

[0165] In related technologies, when the product operation between the target scaling factor in the batch normalization layer and the convolution result x output by the convolutional layer is located in the batch normalization layer, since the target scaling factor needs to quantize the convolution result, the convolutional layer needs to first write the convolution result into global memory, and then read the convolution result from global memory when the target scaling factor needs to scale the convolution result; then use the target scaling factor to quantize the convolution result, and then save the quantized output result in global memory, which involves frequent access to global memory.

[0166] In the fifth scenario, when the calculation operation between the target scaling factor and the convolution result is transferred from the batch normalization layer to the tail calculation of the convolution layer, it is not necessary to write the convolution result to global memory after the convolution layer outputs the convolution result. Instead, the convolution result is quantized using the target scaling factor, and then the quantized output result is stored in global memory, thereby reducing access to global memory and improving the running speed of the model.

[0167] The following describes an exemplary embodiment of step S12 above, which illustrates an exemplary scheme for fusing multiple computational operations into the tail computation of a convolutional layer when multiple computational operations exist, including the following steps:

[0168] (1) Combine the multiple calculation operations to obtain the target calculation operation.

[0169] Optionally, the calculation operation is a multiplication operation, which can move multiple multiplication operations to adjacent positions and then merge adjacent multiplication operations.

[0170] For example, if a multiplication operation follows an activation operation, it can be moved to precede the activation operation, thus ensuring that the multiplication operation moved to precede the activation operation can be adjacent to other multiplication operations. This can be represented as relu → multiply ==> multiply → relu, where relu is the activation operation and multiply is the multiplication operation.

[0171] Furthermore, the prerequisite for swapping the multiplication operation and the activation operation is that the result of the multiplication operation is greater than or equal to 0, and the data precision after the multiplication operation and the activation operation are swapped is equal to the data precision before the multiplication operation and the activation operation are swapped, satisfying the mathematical equivalence principle multiply(relu(x))=relu(multiply(x)).

[0172] Alternatively, the multiplication operation can be moved into the addition operation.

[0173] For example, if a multiplication operation follows an addition operation, the multiplication operation can be moved into the addition operation. This can be represented as add → multiply ==> add, where add is the addition operation and multiply is the multiplication operation.

[0174] Furthermore, moving the multiplication operation into the addition operation must also satisfy the mathematical equivalence principle: `mutual(add(a,b),scale) = add(a*scale,b*scale)`. For example, in `x*(a+b)`, `+` is the addition operation and `*` is the multiplication operation. The addition operation precedes the multiplication operation, so it can be transformed into `xa+xb`, thus merging the multiplication operation into the addition operation while also satisfying the mathematical equivalence principle.

[0175] After moving the multiplication operation, multiple multiplication operations will be moved to adjacent positions, at which point the multiple multiplication operations can be merged into the target multiplication operation.

[0176] Optionally, the constants in multiple multiplication operations can be multiplied together while keeping the unknowns unchanged, thus obtaining a new target multiplication operation. Taking multiple adjacent multiplication operations (multiply(x,scale1) and (multiply(x,scale2)) as an example, the calculation formula after multiplying multiple multiplication operations is as follows:

[0177] multiply(multiply(x,scale1),scale2)=multiply(x,scale1*scale2)

[0178] In this context, scale1 and scale2 are both constants. For example, if scale1 is 1 and scale2 is 2, then the two multiplication operations are x*1 and x*2 respectively before multiplication, and become x*2 after multiplication and combination.

[0179] In some scenarios, the order of multiplication operations is scrambled. Therefore, the order of the multiplication operations can be rearranged, placing them in adjacent positions, ultimately merging multiple multiplication operations into a single operation. Each multiplication operation corresponds to one access to global memory, resulting in multiple global memory accesses across multiple operations. Merging multiple multiplication operations into a single operation further reduces these multiple global memory accesses to a single access, thereby reducing the number of global memory accesses and further improving the processing speed of the neural network model.

[0180] (2) Combine multiple addition operations to obtain the target addition operation.

[0181] Optionally, the neural network has a first addition operation (SubstitueBiasAdd) and a second addition operation (Add). The first addition operation can be converted into a second addition operation, and multiple second addition operations can be merged.

[0182] The first addition operation is a channel-by-channel addition operation. Assuming one of the tensors in the addition operation is [A, B, C] and the other tensor is [c], the first addition operation will expand the other tensor [c] to [C, C, C] and then add the tensor [A, B, C] to the tensor [C, C, C], resulting in the sum [A+C, B+C, C+C].

[0183] The second addition operation is an element-wise addition operation. Suppose one of the tensors in the addition operation is [A, B, C] and the other tensor is [c]. The second addition operation will not add the two tensors together. Instead, it will add the tensor [A, B, C] together if the other tensor also has a three-dimensional shape.

[0184] Since there are two different forms of addition operations in neural networks, it is necessary to unify the first addition operation into the form of the second addition operation in order to meet the requirements for tensor shape.

[0185] Tensors can be one-dimensional or multi-dimensional data, such as scalars, vectors, matrices, or higher-dimensional data. Therefore, the input data, convolution results, weights, and other data in the embodiments of this disclosure can all be called tensors.

[0186] Optionally, after unifying the form of multiple addition operations, these operations can be moved to adjacent positions. When multiple addition operations are adjacent, they can be merged. The principle of mathematical equivalence must be satisfied before and after merging. The merging formula is as follows:

[0187] add(add(x,a),b) = add(x,a+b)

[0188] In this case, both a and b are constants. For example, if a is 1 and b is 2, then the two addition operations are x+1 and x+2 respectively before being added together, and become x+3 after being added together.

[0189] (3) The target calculation operation and the target addition operation are merged into the tail calculation of the convolutional layer.

[0190] In the tail computation of the convolutional layer of a neural network, multiple scattered computational operations and addition operations are integrated. After moving and merging multiple computational operations, a single computational operation is obtained. After moving and merging multiple addition operations, a single addition operation is obtained. After merging, it means that the target computational operation and target addition operation have been integrated into the tail computation of the convolutional layer.

[0191] (4) Obtain the first or second calculation mode after fusion.

[0192] For a first computational pattern that does not include a residual network, the first computational pattern can be obtained by moving and merging multiple computational operations with multiple addition operations. The first computational pattern includes at least one of the following processes: tail operation combination, activation processing, and output quantization operation.

[0193] The tail operation combination is a fixed component of the tail computation in the convolutional layer, including the target computation operation and the target addition operation. In addition, the tail operation combination also includes first precision processing. The first precision processing converts the integer data to be quantized into floating-point data to be quantized; the target computation operation obtains the calculation result based on the floating-point data to be quantized and a first constant; the target addition operation obtains the summation result based on the calculation result and a second constant.

[0194] For example, the target computation operation multiplies the floating-point data to be quantized (which can be the convolution result output by a convolutional layer) by a first constant to obtain the computation result; the target addition operation adds the computation result to a second constant to obtain a sum. The first constant is the product of the constants in multiple computation operations, and the second constant is the sum of the constants in multiple addition operations.

[0195] The activation process is used to transform the summation result. The activation process is a non-linear activation function used to introduce non-linearity into the neural network. In the quantization process, the activation process is used to transform the summation result from one form of expression to another, thereby improving the non-linear expressive power of the quantized summation result.

[0196] The quantization operation includes rounding, clipping, and second precision processing. Rounding is used to round the sum of the quantized results; clipping is used to clip the sum of the rounded results to a preset range, which is the range allowed by quantization; and second precision processing is used to convert the sum of results within the preset range into integer sums.

[0197] The first computation mode mentioned above is one of the computation modes in the tail processing of the convolutional layer. It can be called scale-bias epilogue. It realizes the post-processing of the convolution result output by the convolutional layer and the fusion process between the target computation operation and the convolution result. Finally, after multi-layer data processing, the final output result of the convolutional layer is obtained. The output result is in integer form, which facilitates the fast processing of the next network layer.

[0198] In some scenarios, the first calculation mode described above can be represented as follows:

[0199] cast→multiply→add[→relu][→round→clip→cast]; where cast→multiply→add is a fixed tail operation combination, cast is first precision processing, multiply is the target calculation operation, add is the target addition operation; relu is activation processing; round→clip→cast is an optional quantization operation, round is rounding processing, clip is clipping processing, and cast is second precision processing.

[0200] For the second computation mode that includes the residual network, the second computation mode can be obtained by moving and merging multiple computation operations and multiple addition operations. The first computation mode includes at least one of the following processing: tail operation combination, activation operation, output quantization operation and residual operation.

[0201] The second computational mode, which includes a residual network, is based on the first computational mode by adding a residual network. The computational modes prior to the residual network will not be described in detail here.

[0202] After the second precision processing outputs the integer summation result, the integer summation result is further processed through the residual network. The residual network includes third precision processing and target calculation operation. The third precision processing is used to convert the integer summation result into a floating-point summation result. The target calculation operation then multiplies the floating-point summation result by a constant to obtain the final output result.

[0203] The first calculation mode mentioned above is another calculation mode in the tail processing of convolutional layers, which can be called scale-bias-add epilogue. It realizes the post-processing of the convolution result output by the convolutional layer and the fusion process between the target calculation operation and the convolution result when a residual network exists. Finally, after multi-layer data processing, the final output result of the convolutional layer is obtained. The output result is in floating-point form, thereby reducing the processing error of the next network layer.

[0204] In some scenarios, the second computation mode can be represented as follows:

[0205] cast→multiply→add→add[→relu][→round→clip→cast]

[0206] cast→multiply; where cast→multiply→add is a fixed tail operation combination, cast is the first precision processing, multiply is the target computation operation, and add is the target addition operation; ReLU is the activation processing; round→clip→cast is an optional quantization operation, round is the rounding processing, clip is the clipping processing, and cast is the second precision processing; cast→multiply is a residual network, cast is the third precision processing, and multiply is the target computation operation.

[0207] Through the above technical solution, multiple sets of multiplication operations between the first and second parameters can be directly obtained in the tail calculation of the convolutional layer. These multiple multiplication operations are then merged into a single multiplication operation, reducing the multiple accesses to global memory under multiple multiplication operations to a single access. This significantly reduces the number of global memory accesses and improves the model's running speed. Similarly, merging multiple addition operations into a single addition operation can reduce the computational overhead of addition operations and also improve the model's running speed.

[0208] The following describes exemplary embodiments of this disclosure, which illustrate the movement of quantization or dequantization operators, exchanging the execution order of quantization or dequantization operators with at least one network layer selected from activation layers, pooling layers, connection layers (CNNAT), and addition layers, so that the quantization or dequantization operators are placed as far after and adjacent to the convolutional layers as possible, so as to be fused with the convolution results of the convolutional layers. This includes several scenarios:

[0209] In the first scenario, the execution order of the quantization operator or the dequantization operator is swapped with that of the unary operation layer (UnaryOp). Since the unary operation layer includes a max pooling layer (MaxPool) and an activation layer (ReLU), it can be divided into the following two schemes for explanation.

[0210] Option A involves swapping the quantization or dequantization operators with the max pooling layer. It's important to note that swapping the quantization or dequantization operators with the max pooling layer must satisfy the principle of mathematical equivalence, meaning the output of the neural network before the swap must be consistent with the output of the neural network after the swap.

[0211] Alternatively, the quantization operator can be moved before the max pooling layer.

[0212] Max pooling layers can reduce the spatial dimensions of data, such as width and height, thereby reducing the computational cost and number of parameters in subsequent layers, which helps to reduce the complexity and computational cost of the model.

[0213] For example, the mathematical equivalence principle for the exchange of quantization operators and max pooling layers can be defined as:

[0214] Q(f(x))=f(Q(x))

[0215] Where Q(f(x)) represents executing the pooling function of the max pooling layer first and then the quantization operator; f(Q(x)) represents executing the quantization operator first and then the pooling function of the max pooling layer.

[0216] Alternatively, the dequantization operator can be moved after the max pooling layer.

[0217] For example, the mathematical equivalence principle for the exchange between the dequantization operator and the max pooling layer can be defined as:

[0218] f(DQ(x))=DQ(f(x))

[0219] Where f(DQ(x)) represents executing the dequantization operator first and then the pooling function; DQ(f(x)) represents executing the pooling function first and then the dequantization operator.

[0220] Please see Figure 6 The neural network shown includes, from top to bottom, a convolutional layer, a batch normalization layer, an activation layer, and a max pooling layer. The max pooling layer has a quantization operator below it. Therefore, the quantization operator can be moved before the max pooling layer and after the activation layer. In this way, the quantization operator can be executed first and then the pooling function can be executed.

[0221] Scheme A moves the quantization operator to the front of the max pooling layer. Firstly, this allows the quantization operator to be moved as far forward as possible, enabling the quantization operation to be executed earlier. This converts the data to be quantized from floating-point numbers to lower-precision integers, allowing more network layers to use the lower-precision data, reducing memory usage and improving computation speed. Secondly, moving the quantization operator before the max pooling layer allows it to be as close as possible to the convolutional layer located before the max pooling layer, facilitating the fusion of the quantization operator with the tail computation of the convolutional layer in subsequent processing.

[0222] Option B involves swapping the quantization or dequantization operator with the activation layer.

[0223] The principle of exchanging quantization or dequantization operators with the max pooling layer is similar to that in Scheme A above, and will not be repeated here.

[0224] Please note that you should participate. Figure 7 As shown, after moving the quantization operator forward to the front of the max pooling layer, the quantization operator is now located behind the activation layer. Therefore, the quantization operator can be moved forward further to the front of the activation layer, behind the batch normalization layer, thus getting closer to the convolutional layer.

[0225] By using Scheme B, moving the quantization operator before the activation layer has two advantages. First, it allows the quantization operator to be moved as far forward as possible, enabling the quantization operation to be executed earlier. This converts the data to be quantized from floating-point numbers to lower-precision integers, allowing more network layers to use the lower-precision data, reducing memory usage and improving computation speed. Second, moving the quantization operator before the activation layer allows it to be placed as close as possible to the convolutional layer located before the activation layer, facilitating the fusion of the quantization operator with the tail computation process of the convolutional layer.

[0226] In the second scenario, the execution order of the quantization operator or the dequantization operator is swapped with that of the connection layer.

[0227] Alternatively, the quantization operator can be moved before the connection layer.

[0228] Connection layers are used to combine multiple tensors into a single tensor along a specified dimension. They are used to concatenate the outputs of different layers in a sequential or specified manner, thereby increasing the depth or width of the network.

[0229] Please see Figure 8 As shown, Figure 8 Above the connection layer are two branches. The left branch, from top to bottom, consists of a batch normalization layer and an activation layer, while the right branch, from top to bottom, consists of an activation layer and an addition layer. The connection layer is used to concatenate the activation layer and the addition layer. When there is a quantization operator after the connection layer, the quantization operator can be moved forward, to the front of the connection layer. Since there are usually two network layers before the connection layer, such as an activation layer and an addition layer, the quantization operator can be moved forward to one of the two branch networks before the connection layer, for example, below the activation layer and below the addition layer.

[0230] It is worth noting that since the quantization operator also needs to be moved forward when it is below the activation layer, the quantization operator can be moved from below the activation layer to above the activation layer.

[0231] By using the above technical solution, moving the quantization operator before the connection layer has two advantages. First, it allows the quantization operator to be moved as far forward as possible, enabling the quantization operation to be executed earlier. This converts the data to be quantized from floating-point numbers to lower-precision integers, allowing more network layers to use the lower-precision data, reducing memory usage and improving computation speed. Second, moving the quantization operator before the connection layer allows it to be placed as close as possible to the convolutional layer located before the connection layer, facilitating the fusion of the quantization operator with the tail computation process of the convolutional layer.

[0232] In the third scenario, the execution order of the quantization operator or the dequantization operator is swapped with that of the addition layer.

[0233] Alternatively, the quantization operator can be moved before the addition layer.

[0234] Additive layers are used to add at least two tensors to form a new tensor. For example, in some network structures where residual networks exist, additive layers can add inputs to the outputs of other network layers to form residual connections.

[0235] Please see Figure 9 The neural network shown consists of an activation layer, an addition layer, and a connection layer from top to bottom. When the quantization operator is located behind the connection layer, it can be moved to the front of the connection layer, at which point the quantization operator is located behind the addition layer. The quantization operator located behind the addition layer can then be moved to the front of the addition layer, thus being located behind the activation layer.

[0236] By using the above technical solution, moving the quantization operator before the addition layer has two advantages. First, it allows the quantization operator to be moved as far forward as possible, enabling the quantization operation to be executed earlier. This converts the data to be quantized from floating-point numbers to lower-precision integers, allowing more network layers to use the lower-precision data, reducing memory usage and improving computation speed. Second, moving the quantization operator before the addition layer allows it to be placed as close as possible to the convolutional layer located before the addition layer, facilitating the fusion of the quantization operator with the tail computation process of the convolutional layer.

[0237] In the above scheme, if a neural network contains an activation layer, an addition layer, and a connection layer in sequence from top to bottom, and the quantization operator is located after the connection layer, the above multiple scenarios can be linked together. The quantization operator is moved from behind the connection layer to behind the addition layer, then from behind the addition layer to behind the activation layer, and then from behind the activation layer to in front of the activation layer, thereby gradually approaching the convolutional layer, so that the quantization operator is moved as far forward as possible and the quantization operation can be executed as early as possible.

[0238] Furthermore, in the aforementioned scenarios, moving the quantization operator must also satisfy the principle of mathematical equivalence, that is, the data output by the neural network of the model before and after moving the quantization operator must remain consistent.

[0239] The following describes exemplary embodiments related to this disclosure, which illustrate the use of different quantization granularities to quantize data in different scenarios.

[0240] The first scenario, where the data to be quantified is weights, includes the following steps:

[0241] (1) Obtain the weights in the neural network after they have been quantized by the quantization operator.

[0242] During the offline training of the model, the quantization result of the weights of the neural network after being quantized by the quantization operator can be calculated first; during the actual application of the model, the quantization result can be obtained without having to calculate the quantization result on the spot.

[0243] Optionally, the quantization operator includes a scaling factor, and the product of the scaling factor and the weights can be used as the quantization result, calculated as follows:

[0244] new w eights = weights * q s cale

[0245] Among them, new w eights is the quantized result, the weights obtained after quantization; weights is the weights before quantization; q s cale is the scaling factor for the quantization operator.

[0246] Please see Figure 10 As shown, Figure 10 The left input is the changing data to be quantized. This data can be the input data into the neural network, such as feature maps; it can also be the convolution result output by the convolutional layer, or the intermediate result output by other network layers, etc. This disclosure does not limit it. Figure 10 The input on the right is a fixed data to be quantized, such as weights. Weights usually do not change. Therefore, during the offline training of the model, the weights after being quantized by the scaling factor can be calculated in advance. When the model is actually applied, the quantized weights can be obtained directly without having to quantize the weights on the spot.

[0247] Optionally, multiple quantization operators can be used to quantize the weights of multiple convolution kernels to obtain the quantized weights of multiple convolution kernels; each of the multiple convolution kernels corresponds to multiple channels, and each of the multiple channels corresponds to multiple weights.

[0248] When the data to be quantized is weights, each channel or each convolutional kernel can have its own quantization operator.

[0249] For example, suppose the input data is an image with a size of 224×224×3. Here, 224×224 represents the width and height of the input image, and 3 represents the number of color channels, such as RGB three channels. After the input image is fed into a convolutional layer with 32 kernels, each kernel produces one channel of output. This means that the 32 kernels will process the input image into 32 feature maps. These 32 two-dimensional feature maps are then superimposed to form a new three-dimensional tensor with a shape of 224×224×32 and 32 channels. Each of the 32 kernels or channels then corresponds to its own quantization operator. The weights of each kernel are quantized using their respective quantization operators, thus using precise quantization operators to quantize the weights and reducing the loss of weight precision.

[0250] (2) Use the quantized weights as the weights of the convolutional layer.

[0251] Optionally, multiple weights can be obtained after being quantized by multiple operators.

[0252] Through the above technical solution, firstly, for fixed data to be quantized such as weights, the results of the weights after being quantized by the quantization operator can be stored in advance. In the actual application stage of the model, the quantized weights can be obtained directly without having to perform quantization calculations on the weights temporarily, thereby enabling the model to run faster and output processing results more quickly. Secondly, by using different quantization operators for different weights, accurate quantization of the weights can be achieved, reducing the loss of weight accuracy.

[0253] In the second scenario, when the parameter to be quantized is the input data, a unified quantization operator can be used to quantize the input data to obtain quantized input data, and the input data shares a quantization parameter across multiple channels.

[0254] Continuing with the example in the first scenario above, if the convolutional layer has 32 convolutional kernels, and each of the 32 convolutional kernels produces an output with 32 channels, then the 32 channels or 32 convolutional kernels will uniformly use a unified quantization operator to process the input data, thereby reducing the amount of computation.

[0255] It is understandable that quantization methods include asymmetric quantization (Uniform Affine Quantizer) and symmetric quantization (Uniform Symmetric Quantizer).

[0256] Asymmetric quantization refers to quantization using scaling factors and zero-point offset, and its calculation formula is as follows:

[0257]

[0258] Where Q(x) is the quantized integer; x is the original floating-point number; scale-factor is the scaling factor; and zero_point is the zero-point offset.

[0259] As can be seen, in asymmetric quantization, the floating-point number can be divided by the scaling factor and the zero-point offset to obtain the quantized integer.

[0260] Symmetric quantization refers to quantization using a scaling factor, with a zero-point offset of 0. Its calculation formula is as follows:

[0261]

[0262] Where Q(x) is the quantized integer; x is the original floating-point number; and scale-factor is the scaling factor.

[0263] As can be seen, in symmetric quantization, the value of a floating-point number divided by the scaling factor can be used as the quantized integer.

[0264] In terms of quantization method, when quantizing the input data and the weight data to be quantized in the embodiments of this disclosure, a symmetric quantization method can be adopted. Therefore, the quantization parameter used by the above-mentioned quantization operator is a scaling factor, which can reduce the amount of calculation and eliminate the need to calculate the zero-point bias.

[0265] Through the above technical solution, firstly, for the weights of each convolutional kernel in the convolutional layer, the weights of each convolutional kernel can be quantized using their respective scaling factors, thereby making the error of the quantized weights smaller; for the input data, a uniform scaling factor is used to quantize the input data, thereby reducing the amount of data processing and improving the model running speed.

[0266] Secondly, by using symmetric quantization to quantize the input data and the weights and other data to be quantized, the computational load is reduced because symmetric quantization uses scaling factors instead of zero-point bias, which makes the quantization speed of the model faster.

[0267] Figure 11 This is a block diagram of a neural network processing apparatus according to an exemplary embodiment. (Refer to...) Figure 11 The neural network includes convolutional layers; the processing device 1100 of the neural network includes an acquisition module 1110 and a fusion module 1120.

[0268] The acquisition module 1110 is configured to acquire a first parameter and a second parameter; the first parameter includes a quantization operator or a dequantization operator, the quantization operator is used to quantize the data to be quantized in the neural network to obtain quantized data, the dequantization operator is used to dequantize the quantized data, and the second parameter is a parameter different from the first parameter;

[0269] The fusion module 1120 is configured to fuse the calculation operations between the first parameter and the second parameter into the tail calculation of the convolutional layer.

[0270] Optionally, the dequantization operator includes an activation dequantization operator and a weight dequantization operator, the first parameter is the activation dequantization operator located before the convolutional layer, and the second parameter is the weight dequantization operator located before the convolutional layer; the fusion module 1120 includes:

[0271] The first fusion submodule is configured to fuse the computational operations between the activation dequantization operator and the weight dequantization operator into the tail computation of the convolutional layer.

[0272] Optionally, the first parameter is a first quantization operator located before the connection layer, and the second parameter is a first inverse quantization operator located before the connection layer; the fusion module 1120 includes:

[0273] The second fusion submodule is configured to, when the neural network mode is the first network mode, fuse the computation operations of the first quantization operator and the first dequantization operator into the tail computation of the convolutional layer; in the first network mode, the first quantization operator, the first dequantization operator and the second quantization operator are performed sequentially.

[0274] Optionally, the first parameter is a second inverse quantization operator located before the connection layer, and the second parameter is a scaling factor; the fusion module 1120 includes:

[0275] The third fusion submodule is configured to, when the neural network mode is the second network mode, fuse the calculation operation between the second dequantization operator and the scaling factor into the tail calculation of the convolutional layer; in the second network mode, the second dequantization operator and the third quantization operator are performed sequentially.

[0276] Optionally, the first parameter is a fourth quantization operator located after the convolutional layer, and the second parameter is the convolution result output by the convolutional layer; the fusion module 1120 includes:

[0277] The fourth fusion submodule is configured to fuse the computational operations between the fourth quantization operator and the convolution result output by the convolutional layer into the tail computation of the convolutional layer.

[0278] Optionally, the first parameter is a third inverse quantization operator located before the addition layer, and the second parameter is a scaling factor; the fusion module 1120 includes:

[0279] The fifth fusion submodule is configured to fuse the computational operation between the third inverse quantization operator and the scaling factor into the tail computation of the convolutional layer.

[0280] Optionally, the first parameter further includes a target scaling factor, and the second parameter includes the convolution result output by the convolutional layer; the fusion module 1120 includes:

[0281] The sixth fusion submodule is configured to fuse the calculation operation between the target scaling factor and the convolution result output by the convolutional layer into the tail calculation of the convolutional layer; the target scaling factor is obtained based on the scaling factor.

[0282] Optionally, the calculation operation includes multiple calculation operations between multiple sets of first parameters and second parameters;

[0283] Optionally, the fusion module 1120 includes:

[0284] The calculation merging module is configured to merge the multiple calculation operations to obtain the target calculation operation;

[0285] The seventh fusion submodule is configured to fuse the target computation operation into the tail computation of the convolutional layer.

[0286] Optionally, the seventh fusion submodule includes:

[0287] The addition merging submodule is configured to merge multiple addition operations to obtain the target addition operation;

[0288] The eighth fusion submodule is configured to fuse the target computation operation and the target addition operation into the tail computation of the convolutional layer.

[0289] Optionally, the processing device 1100 of the neural network includes:

[0290] The mode module is configured to obtain a first computing mode or a second computing mode after fusion; the first computing mode is a computing mode that does not include the residual network, and the second computing mode is a computing mode that includes the residual network.

[0291] Optionally, the first computation mode includes at least one of the following processing methods: tail operation combination, activation operation and output quantization operation;

[0292] The tail operation combination includes the target calculation operation and the target addition operation.

[0293] Optionally, the first computation mode includes at least one of the following processing methods: tail operation combination, activation operation, output quantization operation and residual operation;

[0294] The tail operation combination includes the target calculation operation and the target addition operation.

[0295] Optionally, the calculation operation between the first parameter and the second parameter includes a multiplication operation between the first parameter and the second parameter.

[0296] Optionally, the processing device 1100 of the neural network includes:

[0297] The switching module is configured to exchange the execution order of the quantization operator or the dequantization operator with at least one of the following network layers: unary operation layer, connection layer, and addition layer.

[0298] Optionally, the processing device 1100 of the neural network includes:

[0299] The weight acquisition module is configured to acquire the weights in the neural network after they have been quantized by the quantization operator.

[0300] The weights, as a module, are configured to use the quantized weights as the weights of the convolutional layer.

[0301] Optionally, the data to be quantized includes weights; the convolutional layer includes multiple convolutional kernels; the weight acquisition module includes:

[0302] The quantization submodule is configured to use multiple quantization operators to quantize the weights of the multiple convolutional kernels respectively, to obtain the quantized weights of the multiple convolutional kernels; wherein, the multiple convolutional kernels correspond to multiple channels, and the multiple channels correspond to multiple weights respectively;

[0303] The acquisition submodule is configured to acquire the weights after quantization of the multiple convolutional kernels.

[0304] Optionally, the data to be quantized includes input data; the processing device 1100 of the neural network includes:

[0305] The shared module is configured to quantize the input data using a quantization operator to obtain quantized input data; wherein the input data shares the quantization operator across multiple channels.

[0306] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0307] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the neural network processing method provided in this disclosure.

[0308] Figure 12 This is a block diagram illustrating a vehicle 1200 according to an exemplary embodiment. For example, vehicle 1200 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicle. Vehicle 1200 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0309] Reference Figure 12 The vehicle 1200 may include various subsystems, such as an infotainment system 1210, a perception system 1220, a decision control system 1230, a drive system 1240, and a computing platform 1250. The vehicle 1200 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the vehicle 1200 can be interconnected via wired or wireless means.

[0310] In some embodiments, the infotainment system 1210 may include a communication system, an entertainment system, and a navigation system, etc.

[0311] The perception system 1220 may include several sensors for sensing information about the environment surrounding the vehicle 1200. For example, the perception system 1220 may include a global positioning system (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU), a lidar, a millimeter-wave radar, an ultrasonic radar, and a camera device.

[0312] The decision control system 1230 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0313] The drive system 1240 may include components that provide powered motion to the vehicle 1200. In one embodiment, the drive system 1240 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0314] Some or all of the functions of the vehicle 1200 are controlled by a computing platform 1250. The computing platform 1250 may include at least one processor 1251 and a memory 1252, the processor 1251 being able to execute instructions 1253 stored in the memory 1252.

[0315] Processor 1251 can be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof.

[0316] The memory 1252 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0317] In addition to instruction 1253, memory 1252 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in memory 1252 can be used by computing platform 1250.

[0318] In this embodiment of the present disclosure, the processor 1251 may execute instructions 1253 to complete all or part of the steps of the neural network processing method described above.

[0319] Figure 13 This is a block diagram illustrating a processing apparatus 1300 for a neural network according to an exemplary embodiment. For example, apparatus 1300 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0320] Reference Figure 13 The device 1300 may include one or more of the following components: a processing component 1302, a memory 1304, a power supply component 1306, a multimedia component 13013, an audio component 1310, an input / output interface 1312, a sensor component 1314, and a communication component 1316.

[0321] Processing component 1302 typically controls the overall operation of device 1300, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1302 may include one or more processors 1320 to execute instructions to complete all or part of the steps of the neural network processing method described above. Furthermore, processing component 1302 may include one or more modules to facilitate interaction between processing component 1302 and other components. For example, processing component 1302 may include a multimedia module to facilitate interaction between multimedia component 13013 and processing component 1302.

[0322] Memory 1304 is configured to store various types of data to support the operation of device 1300. Examples of such data include instructions for any application or method operating on device 1300, contact data, phonebook data, messages, pictures, videos, etc. Memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0323] Power supply component 1306 provides power to various components of device 1300. Power supply component 1306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 1300.

[0324] Multimedia component 13013 includes a screen that provides an output interface between the device 1300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 13013 includes a front-facing camera and / or a rear-facing camera. When the device 1300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0325] Audio component 1310 is configured to output and / or input audio signals. For example, audio component 1310 includes a microphone (MIC) configured to receive external audio signals when device 1300 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1304 or transmitted via communication component 1316. In some embodiments, audio component 1310 also includes a speaker for outputting audio signals.

[0326] Input / output interface 1312 provides an interface between processing component 1302 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0327] Sensor assembly 1314 includes one or more sensors for providing status assessments of various aspects of device 1300. For example, sensor assembly 1314 may detect the on / off state of device 1300, the relative positioning of components such as the display and keypad of device 1300, changes in the position of device 1300 or a component of device 1300, the presence or absence of user contact with device 1300, the orientation or acceleration / deceleration of device 1300, and temperature changes of device 1300. Sensor assembly 1314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1314 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0328] Communication component 1316 is configured to facilitate wired or wireless communication between device 1300 and other devices. Device 1300 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 1316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1316 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0329] In an exemplary embodiment, the apparatus 1300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the neural network processing method described above.

[0330] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, which can be executed by a processor 1320 of the device 1300 to complete the neural network processing method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0331] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described neural network processing method when executed by the programmable device.

[0332] Some embodiments of this disclosure also provide a chip system, such as Figure 14 As shown, the chip system includes at least one processor 1401 and at least one interface circuit 1402. The processor 1401 and the interface circuit 1402 are interconnected via lines. For example, the interface circuit 1402 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1402 can be used to send signals to other devices (e.g., the processor 1401). Exemplarily, the interface circuit 1402 can read instructions stored in memory and send those instructions to the processor 1401. When the instructions are executed by the processor 1401, the neural network processing device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and some embodiments of this disclosure do not specifically limit this.

[0333] In some embodiments of this disclosure, the interface circuit 1402 can acquire data, program instructions, and / or information from the internal storage area of ​​the chip system; it can also acquire data, program instructions, and / or information from outside the chip system.

[0334] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0335] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0336] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0337] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0338] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for processing neural networks, characterized in that, The neural network includes convolutional layers; the method includes: Obtain a first parameter and a second parameter; the first parameter includes a quantization operator or a dequantization operator, the quantization operator is used to quantize the data to be quantized in the neural network to obtain quantized data, the dequantization operator is used to dequantize the quantized data, and the second parameter is a parameter different from the first parameter; The calculation operations between the first parameter and the second parameter are merged into the tail calculation of the convolutional layer.

2. The method according to claim 1, characterized in that, The dequantization operator includes an activation dequantization operator and a weight dequantization operator. The first parameter is the activation dequantization operator located before the convolutional layer, and the second parameter is the weight dequantization operator located before the convolutional layer. The step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes: The computational operations between the activation dequantization operator and the weight dequantization operator are merged into the tail computation of the convolutional layer.

3. The method according to claim 1, characterized in that, The first parameter is a first quantization operator located before the connect layer, and the second parameter is a first dequantization operator located before the connect layer; the step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes: When the neural network is in the first network mode, the computational operations of the first quantization operator and the first dequantization operator are fused into the tail computation of the convolutional layer; in the first network mode, the first quantization operator, the first dequantization operator, and the second quantization operator are performed sequentially.

4. The method according to claim 1, characterized in that, The first parameter is the second inverse quantization operator located before the connection layer, and the second parameter is the scaling factor; The step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes: When the neural network mode is the second network mode, the calculation operation between the second dequantization operator and the scaling factor is fused into the tail calculation of the convolutional layer; In the second network mode, the second inverse quantization operator and the third quantization operator are performed sequentially.

5. The method according to claim 1, characterized in that, The first parameter is the fourth quantization operator located after the convolutional layer, and the second parameter is the convolution result output by the convolutional layer; the step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes: The computational operations between the fourth quantization operator and the convolution result output by the convolutional layer are merged into the tail computation of the convolutional layer.

6. The method according to claim 1, characterized in that, The first parameter is the third dequantization operator located before the addition layer, and the second parameter is the scaling factor; The step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes: The calculation operation between the third dequantization operator and the scaling factor is merged into the tail calculation of the convolutional layer.

7. The method according to claim 1, characterized in that, The first parameter further includes a target scaling factor, and the second parameter includes the convolution result output by the convolutional layer; the step of fusing the calculation operation between the first parameter and the second parameter into the tail calculation of the convolutional layer includes: The calculation operation between the target scaling factor and the convolution result output by the convolutional layer is merged into the tail calculation of the convolutional layer; the target scaling factor is obtained based on the scaling factor.

8. The method according to claim 1, characterized in that, The computational operation includes multiple computational operations between multiple sets of first parameters and second parameters; the fusion of the computational operations between the first parameters and second parameters into the tail computation of the convolutional layer includes: The multiple computational operations are combined to obtain the target computational operation; The target computation operation is integrated into the tail computation of the convolutional layer.

9. The method according to claim 8, characterized in that, The step of fusing the target computation operation into the tail computation of the convolutional layer includes: Multiple addition operations are combined to obtain the target addition operation; The target computation operation and the target addition operation are merged into the tail computation of the convolutional layer.

10. The method according to claim 9, characterized in that, After fusing the target computation operation and the target addition operation into the tail computation of the convolutional layer, the method further includes: A first computing mode or a second computing mode after fusion is obtained; the first computing mode is a computing mode that does not include the residual network, and the second computing mode is a computing mode that includes the residual network.

11. The method according to claim 10, characterized in that, The first computing mode includes at least one of the following processing methods: tail operation combination, activation operation and output quantization operation; The tail operation combination includes the target calculation operation and the target addition operation.

12. The method according to claim 10, characterized in that, The first calculation mode includes at least one of the following processing methods: tail operation combination, activation operation, output quantization operation and residual operation; The tail operation combination includes the target calculation operation and the target addition operation.

13. The method according to any one of claims 1 to 12, characterized in that, The calculation operation between the first parameter and the second parameter includes the multiplication operation between the first parameter and the second parameter.

14. The method according to claim 1, characterized in that, The method further includes: The execution order of the quantization operator or the dequantization operator is exchanged with at least one of the following network layers: unary operation layer, connection layer, addition layer.

15. The method according to claim 1, characterized in that, The method further includes: Obtain the weights in the neural network after they have been quantized by the quantization operator; The quantized weights are used as the weights of the convolutional layer.

16. The method according to claim 15, characterized in that, The data to be quantized includes weights; the convolutional layer includes multiple convolutional kernels; The step of obtaining the quantization result of the weights in the neural network after being quantized by the quantization operator includes: The weights of the multiple convolution kernels are quantized using multiple quantization operators to obtain the quantized weights of the multiple convolution kernels; wherein, the multiple convolution kernels correspond to multiple channels, and the multiple channels correspond to multiple weights. Obtain the weights of the multiple convolutional kernels after quantization.

17. The method according to claim 1, characterized in that, The data to be quantized includes input data; the method further includes: The input data is quantized using a quantization operator to obtain quantized input data; wherein the input data shares the quantization operator across multiple channels.

18. A neural network processing device, characterized in that, The neural network includes convolutional layers; including: The acquisition module is configured to acquire a first parameter and a second parameter; the first parameter includes a quantization operator or a dequantization operator, the quantization operator is used to quantize the data to be quantized in the neural network to obtain quantized data, the dequantization operator is used to dequantize the quantized data, and the second parameter is a parameter different from the first parameter; The fusion module is configured to fuse the calculation operations between the first parameter and the second parameter into the tail calculation of the convolutional layer.

19. A neural network processing device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured as follows: Perform the steps of the method according to any one of claims 1 to 17.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program performs the steps of the method described in any one of claims 1 to 17.

21. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 17.

Images