Data processing method, neural network quantization method and related device

Abstract

The embodiment of the present application provides a data processing method, a quantization method of a neural network and a related device, the method comprising: obtaining data to be processed; inputting the data to be processed into a neural network, determining a feature map extracted by one or more convolutional layers of the neural network, the feature map containing m*n feature parameters, m and n being positive integers; determining the uncertainty of each feature parameter in the feature map, wherein the uncertainty of the feature parameter is used to characterize the volatility of the sign of the feature parameter close to zero during the binarization process of the feature parameter; calculating the binarization feature of the feature parameter based on the uncertainty corresponding to the feature parameter, the binarization feature is used to determine the feature matrix, the m*n binarization features contained in the feature matrix correspond one-to-one to the m*n feature parameters; obtaining the processing result of the data to be processed based on the feature matrix. The embodiment of the present application can reduce memory overhead and improve computing speed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, a neural network quantization method and related devices. Background Art

[0002] Binary Neural Network (BNN) can quantize neural network parameters such as weights and / or features to a single bit, so that the model parameters can occupy a smaller storage space. In addition, compared to the full-precision neural network that uses floating-point multiplication and accumulation to implement convolution operations, binary neural networks can implement convolution operations through bitwise XOR. Therefore, binary neural networks can reduce the amount of model calculations, speed up the model's inference process, and greatly facilitate the deployment of models on resource-constrained devices.

[0003] However, binarization will inevitably lead to information loss, and its quantization function discontinuity also brings difficulties to network optimization. Among them, weight optimization in binary neural networks is one of the main reasons for the decline in network performance. In order to solve the above problems, the prior art provides a naive binarization method of direct quantization, as well as an improved binarization method using techniques such as minimum quantization error, improved network loss function and reduced gradient error. However, the above methods all emphasize the gradient size of the weights, while ignoring the gradient direction of the weights, which can determine the optimization direction of the weights. However, unstable optimization directions may lead to slow convergence and instability of the neural network. Therefore, how to reduce and improve the stability of the optimization direction is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application improve a data processing method, a neural network quantization method and related devices, which can reduce memory overhead and improve computing speed.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which may include: obtaining data to be processed; inputting the data to be processed into a neural network, determining a feature map extracted by one or more convolutional layers of the neural network, the feature map containing m*n feature parameters, where m and n are positive integers; determining the uncertainty of each feature parameter in the feature map, wherein the uncertainty of the feature parameter is used to characterize the volatility of the sign of the feature parameter close to zero during the binarization process; calculating the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters, the binary features are used to determine a feature matrix, and the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters; and obtaining the processing result of the data to be processed based on the feature matrix.

[0006] By implementing the embodiments of the present application, the uncertainty effect caused by the binarization of the characteristic parameters in the neural network can be taken into account, so that the uncertainty of the characteristic parameters can be quantitatively calculated through the uncertainty function. In addition, in order to reduce the uncertainty in the reasoning process and improve the stability of the reasoning, the characteristic parameters are binarized according to the uncertainty of the calculated characteristic parameters. In this way, the operation speed and stability of the neural network can be improved.

[0007] In a possible implementation, the neural network is obtained based on binarized weight training, and the binarized weight is obtained by binarizing the weight parameter according to the uncertainty of the weight parameter corresponding to the binarized weight, and the uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process. It can be understood that the uncertainty effect brought about by the binarization of the weight parameter and the characteristic value in the neural network is taken into account during the training of the model, so as to quantitatively calculate the uncertainty through the uncertainty function. The neural network parameters (such as weight parameters) are binarized by the calculated uncertainty. In this way, the convergence speed and stability of the neural network can be improved.

[0008] In a possible implementation, determining the uncertainty of each feature parameter in the feature graph includes: calculating the uncertainty of each feature parameter in the feature graph according to an uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is. The uncertainty function can calculate the uncertainty of the feature parameters in the feature graph during the reasoning process and improve network performance.

[0009] In one possible implementation, the formula for the uncertainty function is:

[0010]

[0011] in, is the uncertainty of the target characteristic parameter, f(x j ) is the uncertainty of a feature parameter adjacent to the target feature parameter, the target parameter is any parameter on the feature graph, and i, j, a are all positive integers. In order to improve the stability of the uncertainty of the feature parameter, the uncertainty of one or more feature parameters in the feature graph can be jointly considered to comprehensively calculate the uncertainty of the target feature parameter.

[0012] In a possible implementation, a binary feature of a feature parameter is calculated based on the uncertainty corresponding to the feature parameter, including: when the uncertainty of the target feature parameter is less than or equal to a second preset threshold, binarizing the target feature parameter through a sign function to obtain the binary feature of the target feature parameter.

[0013] In one possible implementation, a binary feature of a feature parameter is calculated based on the uncertainty corresponding to the feature parameter, including: when the uncertainty of the target feature parameter is greater than a second preset threshold, one or more feature parameters adjacent to the target feature parameter after average pooling are binarized by a sign function to obtain a binary feature of the target position point.

[0014] The feature parameters are binarized based on the correlation calculation of uncertainty. For the feature parameters with large uncertainty (that is, the uncertainty of the target position point is greater than the second preset threshold), average pooling is used and the sign function is introduced, and the feature parameters of the target position point are binarized from the spatial dimension.

[0015] In a second aspect, an embodiment of the present application provides a quantization method for a neural network, which may include: obtaining a first weight matrix, the first weight matrix containing parameters for extracting features in the neural network, the first weight matrix containing s*k weight parameters, s and k are positive integers; calculating the uncertainty of each weight parameter in the first weight matrix, wherein the uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process; calculating the binarization weight of the weight parameter based on the uncertainty of the weight parameter, the binarization weight is used to determine a second weight matrix, and the s*k binarization weights contained in the second weight matrix correspond one-to-one to the s*k weight parameters.

[0016] By implementing the embodiments of the present application, the uncertainty effect caused by the binarization of the weight parameters in the neural network can be taken into account, so that the uncertainty can be quantitatively calculated through the uncertainty function. In addition, in order to reduce the uncertainty in the training process and improve the stability of the training, the weight parameters can be binarized by the calculated uncertainty. In this way, the convergence speed and stability of the neural network can be improved.

[0017] In one possible implementation, the uncertainty of each weight parameter in the first weight matrix is ​​calculated, including: calculating the uncertainty of each weight parameter in the first weight matrix according to an uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is.

[0018] Among them, the uncertainty function can determine the uncertainty of the weight parameters during the iterative update process, thereby improving the parameter quantization mechanism and improving network performance.

[0019] In one possible implementation, the uncertainty of each weight parameter in the first weight matrix is ​​calculated according to an uncertainty function, including: when the current number of iterations is less than or equal to a preset number of iterations, the uncertainty of each weight parameter in the first weight matrix corresponding to the current number of iterations is calculated using the uncertainty function.

[0020] In one possible implementation, the uncertainty of each weight parameter in the first weight matrix is ​​calculated according to the uncertainty function, including: when the current number of iterations is greater than the preset number of iterations, the uncertainty of each weight parameter in the first weight matrix corresponding to the current number of iterations is calculated according to the uncertainty of each weight parameter in the first weight matrix calculated within the reference number of iterations, wherein the reference number of iterations is the preset number of iterations closest to the current number of iterations.

[0021] In order to minimize the uncertainty of the neural network, the uncertainty of the weight parameter of the current iteration number can be calculated by the uncertainty of the weight parameter within the preset iteration number closest to the current iteration number.

[0022] In one possible implementation, the binary weight of the weight parameter is calculated based on the uncertainty corresponding to the weight parameter, including: when the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is less than or equal to a first value, the target weight parameter in the first weight matrix corresponding to the current number of iterations is binarized by a sign function to obtain a binary weight; wherein the first value is the minimum value between the uncertainty of the weight parameter of the previous number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

[0023] In one possible implementation, the binary weight of the weight parameter is calculated based on the uncertainty corresponding to the weight parameter, including: when the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is greater than a first value, the binary weight of the target weight parameter in the first weight matrix corresponding to the number of iterations before the current number of iterations is used as the binary weight of the target weight parameter in the first weight matrix corresponding to the current number of iterations, wherein the first value is the minimum value between the uncertainty of the weight parameter of the number of iterations before the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

[0024] In order to minimize the uncertainty of the neural network, the uncertainty of the current iteration number is compared with other values ​​(such as the uncertainty of the previous iteration number or the first preset threshold). If the conditions are met, the binarized weight of the previous iteration number is used as the binarized weight of the current iteration number.

[0025] In a possible implementation, the method may also include: also including: obtaining training data; inputting the training data into the neural network, determining the feature map extracted by one or more convolutional layers of the neural network; calculating the binary features of each feature parameter in the feature map, wherein the feature map contains m*n feature parameters, m and n are positive integers, and the feature map is the feature of the training data extracted in one or more convolutional layers of the neural network.

[0026] Taking into account the uncertainty effect brought by the binarization of the characteristic parameters in the neural network, the uncertainty is quantitatively calculated through the uncertainty function. In addition, in order to reduce the uncertainty in the reasoning process and improve the stability of the reasoning, the characteristic parameters are binarized through the calculated uncertainty. In this way, the operation speed and stability of the neural network can be improved.

[0027] In a possible implementation, the binary features of each feature parameter in the feature map are calculated, including: determining the uncertainty of each feature parameter in the feature map, wherein the uncertainty of the feature parameter is used to characterize the volatility of the sign of the feature parameter close to zero during the binarization process; the binary features of the feature parameters are calculated based on the uncertainty corresponding to the feature parameters, the binary features are used to determine the feature matrix, and the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters.

[0028] In one possible implementation, determining the uncertainty of each feature parameter in the feature graph includes: calculating the uncertainty of the feature parameter in the feature graph according to an uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is.

[0029] Among them, the uncertainty function can calculate the uncertainty of feature parameters during the iterative update process, thereby improving the parameter quantization mechanism and improving network performance.

[0030] In one possible implementation, the uncertainty function formula is:

[0031]

[0032] in, is the uncertainty of the target characteristic parameter, f(x j ) is the uncertainty of a feature parameter adjacent to the target feature parameter, i, j, a are all natural numbers. The uncertainty of the feature parameter of one or more position points in the training data is jointly considered to comprehensively calculate the uncertainty of the feature map of the target position point (or position points).

[0033] In a possible implementation, a binary feature of a feature parameter is calculated based on the uncertainty corresponding to the feature parameter, including: when the uncertainty of the target feature parameter is less than or equal to a second preset threshold, binarizing the target feature parameter through a sign function to obtain the binary feature of the target feature parameter.

[0034] In one possible implementation, a binary feature of a feature parameter is calculated based on the uncertainty corresponding to the feature parameter, including: when a target uncertainty of the target feature parameter is greater than a second preset threshold, one or more feature parameters adjacent to the target feature parameter after average pooling are binarized by a sign function to obtain a binary feature of the target feature parameter.

[0035] The feature parameters are binarized based on the correlation calculation of uncertainty. For the feature parameters with large uncertainty (that is, the uncertainty of the target position point is greater than the second preset threshold), average pooling is used and the sign function is introduced, and the feature parameters of the target position point are binarized from the spatial dimension.

[0036] In a third aspect, an embodiment of the present application provides a training method for a neural network, which may include: obtaining a first weight matrix and a feature map, wherein the first weight matrix includes parameters for extracting features in the neural network, the first weight matrix includes s*k weight parameters, the feature map is the features of the training data extracted in one or more convolutional layers of the neural network, the feature map includes m*n feature parameters, s, k, m and n are all positive integers; calculating the uncertainty of each weight parameter in the first weight matrix, wherein the uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process; calculating the binarization weight of the weight parameter based on the uncertainty corresponding to the weight parameter, the binarization weight is used to determine the second weight matrix, and the s*k binarization weights included in the second weight matrix correspond one-to-one to the s*k weight parameters; calculating the binarization features of each feature parameter in the feature map, wherein the feature map includes m*n feature parameters, m and n are positive integers, and the feature map is the features of the training data extracted in one or more convolutional layers of the neural network; training the neural network based on the second weight matrix and the feature matrix. By implementing the embodiments of the present application, the uncertainty effect caused by the binarization of weights and feature parameters in the neural network can be taken into account, so that the uncertainty can be quantitatively calculated through the uncertainty function. In addition, in order to reduce the uncertainty in the training process and improve the stability of the training, the calculated uncertainty is binarized. In this way, the convergence speed and stability of the neural network can be improved.

[0037] In one possible implementation, the uncertainty of each weight parameter in the first weight matrix is ​​calculated, including: calculating the uncertainty of each weight parameter in the first weight matrix according to an uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is.

[0038] Among them, the uncertainty function can calculate the uncertainty of weight parameters during the iterative update process, thereby improving the parameter quantization mechanism and improving network performance.

[0039] In one possible implementation, the uncertainty of each weight parameter in the first weight matrix is ​​calculated according to an uncertainty function, including: when the current number of iterations is less than or equal to a preset number of iterations, the uncertainty of each weight parameter in the first weight matrix corresponding to the current number of iterations is calculated according to the uncertainty function.

[0040] In one possible implementation, the uncertainty of each weight parameter in the first weight matrix is ​​calculated according to the uncertainty function, including: when the current number of iterations is greater than the preset number of iterations, the uncertainty of each weight parameter in the first weight matrix corresponding to the current number of iterations is calculated according to the uncertainty of each weight parameter in the first weight matrix calculated within the reference number of iterations, wherein the reference number of iterations is the preset number of iterations closest to the current number of iterations.

[0041] In order to minimize the uncertainty of the neural network, the uncertainty of the weight parameters of the current iteration number can be calculated by the uncertainty of the weight parameters within a preset iteration number.

[0042] In one possible implementation, the binary weight of the weight parameter is calculated based on the uncertainty corresponding to the weight parameter, including: when the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is less than or equal to a first value, the target weight parameter in the first weight matrix corresponding to the current number of iterations is binarized by a sign function to obtain a binary weight; wherein the first value is the minimum value between the uncertainty of the weight parameter of the previous number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

[0043] In one possible implementation, the binary weight of the weight parameter is calculated based on the uncertainty corresponding to the weight parameter, including: when the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is greater than a first value, the binary weight of the target weight parameter in the first weight matrix corresponding to the number of iterations before the current number of iterations is used as the binary weight of the target weight parameter in the first weight matrix corresponding to the current number of iterations, wherein the first value is the minimum value between the uncertainty of the weight parameter of the number of iterations before the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

[0044] In order to minimize the uncertainty of the neural network, the uncertainty of the current iteration number is compared with other values ​​(such as the uncertainty of the previous iteration number or the first preset threshold). If the conditions are met, the binarized weight of the previous iteration number is used as the binarized weight of the current iteration number.

[0045] In a possible implementation, calculating the binary features of each feature parameter in the feature map includes:

[0046] Determine the uncertainty of each feature parameter in the feature map, wherein the uncertainty of the feature parameter is used to characterize the volatility of the sign of the feature parameter close to zero during the binarization process; calculate the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters, and the binary features are used to determine the feature matrix, and the m*n binary features contained in the feature matrix correspond to the m*n feature parameters one by one.

[0047] Based on the uncertainty of the feature parameters, in order to minimize the uncertainty of the neural network, the binary features of the feature parameters are calculated according to the uncertainty.

[0048] In one possible implementation, determining the uncertainty of each feature parameter in the feature graph includes: calculating the uncertainty of the feature parameter in the feature graph according to an uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is.

[0049] Among them, the uncertainty function can calculate the uncertainty of feature parameters during the iterative update process, thereby improving the parameter quantization mechanism and improving network performance.

[0050] In one possible implementation, the uncertainty function formula is:

[0051]

[0052] in, is the uncertainty of the target characteristic parameter, f(x j ) is the uncertainty of a feature parameters adjacent to the target feature parameter, i, j, a are all natural numbers.

[0053] In a possible implementation, the binary features of the feature parameters are calculated based on the uncertainty corresponding to the feature parameters, including: when the uncertainty of the target feature parameters is less than or equal to the second preset threshold, the target feature parameters are binarized by a sign function to obtain the binary features of the target feature parameters. The uncertainty of the feature parameters of one or more position points in the feature map is jointly considered to comprehensively calculate the uncertainty of the feature parameters of the target position point (or position points).

[0054] In one possible implementation, a binary feature of a feature parameter is calculated based on the uncertainty corresponding to the feature parameter, including: when a target uncertainty of the target feature parameter is greater than a second preset threshold, one or more feature parameters adjacent to the target feature parameter after average pooling are binarized by a sign function to obtain a binary feature of the target feature parameter.

[0055] The feature parameters are binarized based on the correlation calculation of uncertainty. For the feature parameters with large uncertainty (that is, the uncertainty of the target position point is greater than the second preset threshold), average pooling is used and the sign function is introduced, and the feature parameters of the target position point are binarized from the spatial dimension.

[0056] In a possible implementation, training a neural network based on a second weight matrix and a feature matrix includes: performing a two-dimensional convolution on the second weight matrix and the feature matrix to obtain an output result of an output layer in the neural network; obtaining a loss function according to the output result; calculating the gradient of a weight parameter in the first weight matrix by the loss function; and training the neural network by updating the weight parameter according to the gradient of the weight parameter. The binary features and binary weights obtained based on uncertainty calculation can reduce the instability of the optimization direction and improve the convergence speed of the neural network.

[0057] In a fourth aspect, an embodiment of the present application provides a data processing device, which may include: an acquisition unit, used to acquire data to be processed; an input unit, used to input the data to be processed into a neural network, and determine a feature map extracted by one or more convolutional layers of the neural network; a calculation unit, used to calculate the uncertainty of feature parameters in the feature map, wherein the uncertainty of the feature parameters is used to characterize the volatility of the sign of the feature parameters close to zero during the binarization process; a quantization unit, used to calculate the binary features of the feature parameters according to the uncertainty of the feature parameters; and the calculation unit, further used to obtain the processing result of the data to be processed based on the binary features.

[0058] In one possible implementation, the neural network is trained based on binarized weights, and the binarized weights are obtained based on the uncertainty of weight parameters in the neural network. The uncertainty of the weight parameters is used to characterize the volatility of the signs of one or more weight parameters close to zero during the binarization process.

[0059] In one possible implementation, the computing unit is specifically used to: calculate the uncertainty of the feature parameters in the feature graph according to the uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is.

[0060] In one possible implementation, the calculation unit is specifically used to: calculate the uncertainty of the feature parameter corresponding to the position point on the feature map according to the uncertainty function; calculate the uncertainty of the target position point according to the uncertainty of the feature parameters of one or more position points adjacent to the target position point, any position point on the target position point feature map.

[0061] In a possible implementation, the quantization unit is specifically used to: when the uncertainty of the characteristic parameter of the target position point is less than or equal to a second preset threshold, binarize the characteristic parameter of the target position point through a sign function to obtain a binary feature of the target position point.

[0062] In one possible implementation, the quantization unit is specifically used to: when the uncertainty of the feature parameters of the target position point is greater than a second preset threshold, binarize the feature parameters of one or more position points adjacent to the target position point after average pooling through a sign function to obtain the binary features of the target position point.

[0063] In a fifth aspect, an embodiment of the present application provides a quantization device for a neural network, which may include an input unit,

[0064] The acquisition unit is also used to obtain a first weight matrix, which contains parameters used to extract features in the neural network, and the first weight matrix contains s*k weight parameters, where s and k are positive integers; the calculation unit is used to calculate the uncertainty of each weight parameter in the first weight matrix, wherein the weight parameter is any weight in the weight of the neural network, and the uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process; the quantization unit is used to calculate the binarized weight of the weight parameter based on the uncertainty corresponding to the weight parameter, and the binarized weight is used to determine the second weight matrix, and the s*k binarized weights contained in the second weight matrix correspond one-to-one to the s*k weight parameters.

[0065] In one possible implementation, the calculation unit is specifically used to: calculate the uncertainty of each weight parameter in the first weight matrix according to the uncertainty function, wherein, when the independent variable of the uncertainty function is closer to 0, the value of the uncertainty function is larger; when the absolute value of the independent variable of the uncertainty function is larger, the value of the uncertainty function is smaller.

[0066] In a possible implementation, the calculation unit is specifically used to: when the current number of iterations is less than or equal to the preset number of iterations, calculate the uncertainty of each weight parameter in the first weight matrix corresponding to the current number of iterations by using an uncertainty function.

[0067] In one possible implementation, the calculation unit is specifically used to: when the current number of iterations is greater than the preset number of iterations, calculate the uncertainty of each weight parameter of the first weight matrix corresponding to the current number of iterations based on the uncertainty of each weight parameter in the first weight matrix calculated within a reference number of iterations, wherein the reference number of iterations is the preset number of iterations closest to the current number of iterations.

[0068] In one possible implementation, the quantization unit is specifically used to: when the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is less than or equal to a first value, binarize the target weight parameter in the first weight matrix corresponding to the current number of iterations through a sign function to obtain a binary weight; wherein the first value is the minimum value between the uncertainty of the weight parameter of the previous number of iterations of the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

[0069] In one possible implementation, the quantization unit is specifically used to: when the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is greater than a first value, use the binary weight of the target weight parameter in the first weight matrix corresponding to the number of iterations before the current number of iterations as the binary weight of the target weight parameter in the first weight matrix corresponding to the current number of iterations, wherein the first value is the minimum value between the uncertainty of the weight parameter of the number of iterations before the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

[0070] In a possible implementation, the device may also include: an acquisition unit for acquiring training data; an input unit for inputting the training data into a neural network to determine a feature map extracted by one or more convolutional layers of the neural network; a quantization unit for calculating the binary features of each feature parameter in the feature map, wherein the feature map contains m*n feature parameters, m and n are positive integers, and the feature map is the feature of the training data extracted in one or more convolutional layers of the neural network. In a possible implementation, the quantization unit is specifically used to: determine the uncertainty of each feature parameter in the feature map, wherein the uncertainty of the feature parameter is used to characterize the volatility of the sign of the feature parameter close to zero during the binarization process; calculate the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters, the binary features are used to determine the feature matrix, and the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters.

[0071] In one possible implementation, the quantization unit is specifically used to: calculate the uncertainty of the feature parameters in the feature graph according to the uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is.

[0072] In one possible implementation, the uncertainty function formula is:

[0073]

[0074] in, is the uncertainty of the target characteristic parameter, f(x j ) is the uncertainty of a feature parameters adjacent to the target feature parameter, i, j, a are all natural numbers.

[0075] In a possible implementation, the quantization unit is specifically used to: when the uncertainty of the target feature parameter is less than or equal to a second preset threshold, binarize the target feature parameter through a sign function to obtain a binary feature of the target feature parameter.

[0076] In one possible implementation, the quantization unit is specifically used to: when the target uncertainty of the target feature parameter is greater than a second preset threshold, binarize one or more feature parameters adjacent to the target feature parameter after average pooling through a sign function to obtain the binary features of the target feature parameter.

[0077] In a sixth aspect, an embodiment of the present application provides a data processing device, which may include a memory and a processor, the memory being used to store a computer program that supports a training device to execute the above-mentioned method, the computer program including program instructions, and the processor being configured to call the program instructions to execute the above-mentioned method of the first aspect.

[0078] In the seventh aspect, an embodiment of the present application provides a quantization device, which may include a memory and a processor, the memory is used to store a computer program that supports a data processing device to execute the above method, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method of the above second aspect.

[0079] In an eighth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method of the first aspect described above.

[0080] In a ninth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method of the second aspect described above.

[0081] In the tenth aspect, an embodiment of the present application further provides a computer program, which includes computer software instructions, and when the computer software instructions are executed by a computer, the computer executes any method of the first aspect, the second aspect, or the third aspect.

[0082] In the eleventh aspect, an embodiment of the present application further provides a computer program product comprising instructions, and when the computer program product is run on an electronic device, the electronic device executes any one of the methods of the first aspect, the second aspect, or the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The following is an introduction to the drawings used in the embodiments of the present application.

[0084] Figure 1A A schematic diagram of the computing time required to process data through a full-precision neural network model provided in an embodiment of the present application;

[0085] Figure 1B A schematic diagram of a high-order approximation method for approximating a symbolic function provided in an embodiment of the present application;

[0086] Figure 1C A schematic diagram of the sign volatility of one or more weight parameters close to zero in a binarization process provided by an embodiment of the present application;

[0087] Figure 2A structural diagram of an artificial intelligence main framework provided in an embodiment of the present application;

[0088] Figure 3 A schematic diagram of a system architecture 100 provided in an embodiment of the present application;

[0089] Figure 4A A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0090] Figure 4B A schematic diagram of the structure of another convolutional neural network provided in an embodiment of the present application;

[0091] Figure 4C A schematic diagram of the structure of a fully connected network provided in an embodiment of the present application;

[0092] Figure 5 A schematic diagram of the hardware structure of a chip provided in an embodiment of the present application;

[0093] Figure 6 A flowchart of a data processing method provided in an embodiment of the present application;

[0094] Figure 7 A schematic diagram of a network architecture of a data processing method provided in an embodiment of the present application;

[0095] Fig. 8A A schematic diagram of a flow chart of a neural network quantization method provided in an embodiment of the present application;

[0096] Figure 8B A schematic diagram of an uncertainty function provided in an embodiment of the present application;

[0097] Figure 8C A schematic diagram of a binarization result provided in an embodiment of the present application;

[0098] Fig. 9 A schematic diagram of a flow chart of a neural network quantization method provided in an embodiment of the present application;

[0099] Fig.10 A flowchart of a neural network training method provided in an embodiment of the present application;

[0100] Fig.11 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0101] Fig.12 A schematic diagram of the structure of a neural network quantization device provided in an embodiment of the present application;

[0102] Fig.13 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0103] Fig.14 A schematic diagram of the structure of a neural network quantization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0104] The technical solutions in the embodiments of the present application are clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0105] The terms "first" and "second" in the specification and drawings of the present application are used to distinguish different objects, or to distinguish different treatments of the same object, rather than to describe the specific order of objects. In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes other steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. It should be noted that in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design method described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being superior or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way. In the embodiments of the present application, "A and / or B" means A and B, A or B. "A, and / or B, and / or C" means any one of A, B, and C, or any two of A, B, and C, or A, B, and C. The technical solution in the present application will be described below in conjunction with the accompanying drawings.

[0106] With the development of artificial intelligence (AI), the accuracy of algorithms often depends on the training and analysis of massive data, so a large amount of data needs to be input into the model to complete the training of the model. Since the training and analysis based on massive data will occupy a large amount of storage and computing, the binarization of neural networks can minimize the storage occupation and computing amount of the full-precision neural network model, quantize the original 32 floating-point parameters in the neural network to 1-bit integer parameters, reduce the storage resource consumption of the model deployment, and greatly accelerate the inference process of the neural network. Table 1 shows the computing amount and storage required for the full-precision neural network model. Among them, the visual geometry group (VGG) model is a convolutional neural network model; residual networks (ResNets) refers to the idea of ​​adding residual learning to the traditional convolutional neural network; SENet (Squeeze-and-Excitation Networks) is an image recognition structure.

[0107] Table 1. Computational and storage requirements for full-precision neural network models

[0108]

[0109] like Figure 1A Shown is a schematic diagram of the computing time required to process data through a full-precision neural network model provided by an embodiment of the present application. Figure 1A (a) is the computation time required for the VGG16 model under different batch sizes. Figure 1A (b) in the figure is the computation time required by the GoogleNet model under different batch sizes. The batch size is the number of samples selected for one training. Figure 1A It can be seen that the full-precision neural network model requires a long calculation time and is difficult to meet real-time computing requirements.

[0110] However, binarization will inevitably lead to serious information loss, and its quantization function discontinuity also brings difficulties to the optimization of deep networks. Among them, the optimization of discrete weights in binary neural networks is one of the main reasons for the performance degradation. This optimization is usually performed using a non-smooth symbolic function, whose derivative is zero everywhere except at zero, and the points around zero are called "sensitive points".

[0111] In order to deal with these "sensitive points", existing binary neural networks tend to approximate the derivative of the symbolic function or the symbolic function itself. The straight through estimator (STE) proposed to use the identity of the binary neural network to approximate the derivative of the symbolic function, but it may bring large errors. Inspired by STE, more precise approximations, including first-order approximation and exponential polynomial approximation, are introduced to replace the derivative to reduce the gradient error. It can be seen that the above methods provide the gradient of the "sensitive point", such as Figure 1B The figure shows a schematic diagram of the approximation of a symbolic function by a high-order approximation method.

[0112] in, Figure 1B (a) is the sign function sign(x) and the gradient of sign(x) Schematic diagram of . Figure 1B (b) is the gradient of the clipping function Clip(-1, x, 1) and Clip(-1, x, 1) Schematic diagram of . Figure 1B (c) in the figure is the custom sign function Approxsign(x) and the gradient of Approxsign Schematic diagram of . Figure 1B (d) in the figure is the custom sign function 3-oder-Approxsign(x) and the gradient of 3-oder-Approxsign(x) Schematic diagram of .

[0113] from Figure 1B It can be seen that in addition to direct derivative approximation, symbolic functions with adaptive or learnable parameters can also be used in model training, such as using asymptotic methods to estimate symbolic functions. As the model training proceeds, these symbolic functions expand the gradient of the "sensitive point", making the gradient large enough to change the binary weight of the "sensitive point".

[0114] In addition, the embodiments of the present application also provide the following solutions to deal with these "sensitive points".

[0115] Solution 1: BONN minimizes quantization error based on the Bayesian method and redistributes the real weights into a bimodal distribution. The effectiveness of Bayesian learning is used to build binary neural networks (1-Bit Convolutional Neural Networks, 1-Bit CNNs) in an end-to-end manner. In particular, two new Bayesian losses are introduced to optimize 1-Bit CNNs on this basis, which can improve efficiency and stability. Under a unified theoretical framework, these Bayesian losses not only consider the distribution of kernels and weights of 1-Bit CNNs, but also supervise the feature distribution. The Bayesian kernel loss improves the layer-by-layer kernel distribution of each convolutional layer, while the Bayesian feature loss introduces class compactness to alleviate the interference caused by the quantization process. It should be noted that the Bayesian feature loss is only applicable to fully connected layers.

[0116] The Bayesian loss introduced in Solution 1 increases the complexity of model training. It is not very stable in practical applications and there is a risk of gradient explosion during back propagation.

[0117] Solution 2: Anti-aliasing Convolutional Neural Network (AA-CNN). In the field of traditional information processing, the anti-aliasing technology is generally to increase the sampling frequency. However, since image processing tasks generally require downsampling, image blurring technology can also be used. According to the Nyquist sampling theory, the sampling frequency is first given, and the signal can be reconstructed by reducing the frequency of the original signal. Downsampling is performed on the blurred and unprocessed original images. Some contours can be seen in the image downsampled from the blurred original image, while the image downsampled from the unprocessed original image appears more chaotic. By combining anti-aliasing and downsampling together, the model is robust to image damage.

[0118] The purpose of binary neural networks is to quantize features and weights to 1 bit. Although the anti-aliasing operation of solution 2 can effectively deal with image damage, it cannot solve the problem of information loss caused by model binarization. The problem of quantization uncertainty still exists, and parameters near zero in the full-precision model are still not quantization robust.

[0119] Solution 3: Information preserving network (IR-Net) studies the forward and backward propagation process of binary neural network from the perspective of unified information, introduces information entropy loss, and optimizes quantization error. The network can carry enough information in the forward propagation process and the gradient obtained in the backward propagation can improve the correct information for network optimization, which is one of the keys to maintain high performance of the network. IR-Net introduces a balanced standard quantization method called Libra Parameter Binarization (Libra) binarization in forward propagation to maximize the information entropy of quantization parameters and minimize quantization error, and expands the representation capacity of weights through integer shift scalars. In the backward propagation process, an error attenuation estimator is used to calculate the gradient to ensure sufficient update at the beginning of training and accurate gradient at the end of training.

[0120] In solution 3, IR-Net cannot maximize the difference in weight parameters, resulting in inaccurate quantization. The quantization ability of "sensitive points" is poor, resulting in loss of quantization information.

[0121] In summary, 1-bit quantization of the network model may cause information loss due to quantization error. In order to reduce information loss, many solutions have been proposed, such as minimizing the distance between the real-valued weights and the binary weights or adjusting the distribution of parameters to reduce quantization errors. However, there is still a significant accuracy gap between binary neural networks and full-precision neural networks. Existing methods cannot solve the quantization problem of "sensitive points". For example, existing derivative or symbolic function approximation methods emphasize the gradient size of "sensitive points" and ignore the optimization methods of "sensitive points". Due to the instability of "sensitive points". The symbolic function may lead to unstable optimization directions. For example Figure 1C Schematic diagram showing the sign volatility of one or more weight parameters approaching zero during the binarization process. Figure 1C It can be seen that in the binarization process, weights close to the quantity are more unstable. For example, if the weight is 0.001, the sign of the weight is +1; when the weight changes slightly, for example, the weight is updated from 0.001 to -0.001, the sign of the weight is -1. Therefore, the fluctuation of weights near zero will cause frequent jumps in the direction of weight optimization due to frequent crossing of zero points. Directionally uncertain learning may cause slow convergence and instability of the neural network.

[0122] Therefore, the implementation of this application proposes to model the uncertainty of binarization and determine the optimization direction based on the uncertainty, so as to improve the relevant performance of the binarized neural network (such as accuracy, convergence speed, etc.).

[0123] First, the overall workflow of the artificial intelligence system is described. Figure 2 , Figure 2 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the main framework of artificial intelligence from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0124] (1) Infrastructure

[0125] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, GPU, NPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnection networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0126] (2) Data

[0127] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0128] (3) Data processing

[0129] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0130] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0131] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0132] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0133] (4) General ability

[0134] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0135] (5) Smart products and industry applications

[0136] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0137] The embodiments of the present application are mainly used in the fields of driving assistance, automatic driving, mobile phone terminals, etc.

[0138] Here are some application scenarios:

[0139] Application Scenario 1: Advanced Driver Assistance System (ADAS) / Autonomous Driving Solution (ADS)

[0140] In ADAS and ADS, multiple types of 2D target detection need to be performed in real time, including: dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks, buses), static obstacles (traffic cones, traffic sticks, fire hydrants, motorcycles, bicycles), traffic signs (Traffic Sign, Guide Sign, Billboard, red traffic light (Traffic Light_Red) / yellow traffic light (Traffic Light_Yellow) / green traffic light (Traffic Light_Green) / black traffic light (Traffic Light_Black), road signs (Road Sign)). In addition, in order to accurately obtain the area occupied by dynamic obstacles in 3D space, it is also necessary to perform 3D estimation of dynamic obstacles and output 3D boxes. In order to fuse with the LiDAR data, it is necessary to obtain the mask of dynamic obstacles, so as to filter out the laser point cloud that hits the dynamic obstacles; in order to accurately park, it is necessary to detect the four key points of the parking space at the same time; in order to perform composition positioning, it is necessary to detect the key points of static targets. This is a semantic segmentation problem. The camera of an autonomous driving vehicle captures the road scene, and the scene needs to be segmented to separate different objects such as the road surface, roadbed, vehicles, pedestrians, etc., so as to keep the vehicle driving in the correct area. For autonomous driving with extremely high safety requirements, it is necessary to understand the picture in real time, and a convolutional neural network that can run in real time for semantic segmentation is crucial.

[0141] Application scenario 2: Image classification scenario

[0142] After acquiring the image to be classified, the object recognition device processes the objects in the image to be classified through the classification model trained by the quantization method of the neural network based on the embodiment of the present application to obtain the category of the image to be classified, and then the image to be classified can be classified according to the object category of the object in the image to be classified. For photographers, many photos are taken every day, including animals, people, and plants. The method of the present application can quickly classify photos according to the content in the photos, which can be divided into photos containing animals, photos containing people, and photos containing plants.

[0143] When the number of images is large, manual classification is inefficient, and people are prone to fatigue when dealing with the same thing for a long time, so the classification results will have large errors.

[0144] Application scenario 3: Product classification

[0145] After the object recognition device obtains the image of the commodity, it processes the image of the commodity through the classification model trained by the quantization method of the neural network in the embodiment of the present application to obtain the category of the commodity in the image of the commodity, and then classifies the commodity according to the category of the commodity. For a wide variety of commodities in large shopping malls or supermarkets, the object recognition method of the present application can quickly complete the classification of commodities, reducing time expenditure and labor costs.

[0146] Application scenario 4: Face verification at the entrance gate

[0147] This is an image similarity comparison problem. When passengers perform facial authentication at gates at high-speed trains, airports, etc., the camera will capture the facial image, extract features using a convolutional neural network, and perform similarity calculations with the image features of the ID card stored in the system. If the similarity is high, the authentication is successful. Among them, feature extraction by the convolutional neural network is the most time-consuming. To quickly perform facial authentication, an efficient convolutional neural network is required for feature extraction.

[0148] Application scenario 5: simultaneous interpretation by machine translator

[0149] This is a speech recognition and machine translation problem. Convolutional neural networks are also a common recognition model in speech recognition and machine translation. In scenarios where simultaneous interpretation is required, real-time speech recognition and translation must be achieved. Efficient convolutional neural networks can bring a better experience to the translation machine.

[0150] The neural network model trained by the embodiment of the present application can realize the above functions.

[0151] The quantization method of the neural network provided in the embodiment of the present application may involve the processing of computer vision or natural language, etc., and may be specifically applied to data processing methods such as data training, machine learning, and deep learning, and symbolize and formalize the training data for intelligent information modeling, extraction, preprocessing, and training, etc., to finally obtain a trained neural network model (ie, target model / rule). Moreover, the data processing method provided in the embodiment of the present application may be applied to the above-mentioned trained neural network model to obtain output data (such as the recognition result of the picture). It should be noted that the training method and data processing method of the neural network provided in the embodiment of the present application are inventions based on the same concept.

[0152] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0153] (1) Neural Network

[0154] A neural network can be composed of neural units. A neural unit can be a s and intercept b as inputs, the output of the operation unit can be:

[0155]

[0156] Where, s = 1, 2, ... n, n is a natural number greater than 1, w s For x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0157] (2) Deep Neural Networks

[0158] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, b is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer simply performs such a simple operation on the input vector α to obtain the output vector Since there are many DNN layers, the number of coefficients W and offset vectors b is also large. The definitions of these parameters in DNN are as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vectors w).

[0159] (3) Convolutional Neural Network

[0160] Convolutional neural network (CNN) is a deep neural network with convolutional structure. Convolutional neural network contains a feature extractor consisting of convolution layer and subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input data (such as image data, described by taking image data as an example) or convolution feature plane (feature map). Convolution layer refers to the neuron layer in convolutional neural network that performs convolution processing on the input signal. In the convolution layer of convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0161] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0162] (4) Recurrent Neural Network

[0163] Recurrent neural networks (RNN) are used to process sequence data. In the traditional neural network model, from the input layer to the hidden layer and then to the output layer, the layers are fully connected, and the nodes within each layer are disconnected. Although this ordinary neural network has solved many difficult problems, it is still powerless for many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. The same error back propagation algorithm is used, but there is a difference: that is, if the RNN is expanded, the parameters, such as W, are shared; this is not the case with the traditional neural network mentioned above. And when using the gradient descent algorithm, the output of each step depends not only on the network at the current step, but also on the state of the network at the previous steps. This learning algorithm is called Back Propagation Through Time (BPTT).

[0164] Since we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, there is a premise that: elements are independent of each other, and input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the change of stocks over time, or someone said: I like traveling, and my favorite place is Yunnan. I must go there if I have the chance in the future (__). Fill in the blank here, humans should all know to fill in "Yunnan". Because humans will make inferences based on the content of the context, but how can machines do this? RNN came into being. RNN aims to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on current input information and historical memory information.

[0165] (5) Loss Function

[0166] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0167] (6) Back propagation algorithm

[0168] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial model during the training process, so that the reconstruction error loss of the initial model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the target model, such as the weight matrix.

[0169] (7) Model Quantization

[0170] Model quantization is one of the common means of deep learning optimization. On the one hand, model quantization can reduce memory and storage overhead, and on the other hand, it can also speed up the convergence of the model and improve the reasoning efficiency of the model. In the embodiment of the present application, quantization is the process of mapping a set of numbers within the original value range to another target value range through a mathematical transformation. For example, the model parameters of the neural network are converted from floating point numbers to integers.

[0171] (8) Binary Neural Network

[0172] Binary Neural Network (BNN) refers to a neural network obtained by binarizing the parameter values ​​in a full-precision neural network (a network with 32 floating-point parameters). That is, the 32-floating-point parameter is binarized into a 1-bit integer (1 or -1). Through binarization, the parameters can occupy a smaller storage space (the memory consumption is theoretically reduced to 1 / 32 times of the original, from float32 to 1bit), and the bit operation is used to replace the multiplication and addition operations in the network, which can reduce the operation time.

[0173] The following introduces the system architecture provided by the embodiments of the present application.

[0174] See also Figure 3 , a system architecture 100 provided in an embodiment of the present application. As shown in the system architecture 100, the data acquisition device 160 is used to collect or generate training data. In the embodiment of the present application, the training data includes: multiple images or multiple voice clips with labels, etc.; and the training data is stored in the database 130. The training device 120 can determine the feature map extracted by one or more convolutional layers in the neural network. The feature map contains m*n feature parameters, and m and n are positive integers. Then, the training device 120 can determine the uncertainty of each feature parameter in the feature map, and calculate the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters. Among them, the uncertainty of the feature parameters is used to characterize the volatility of the sign of the feature parameters close to zero during the binarization process of the feature parameters. The binarized features are used to determine the feature matrix, and the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters.

[0175] The training device 120 can obtain a first weight matrix, which contains parameters used to extract features in a neural network. The first weight matrix contains s*k weight parameters, where s and k are positive integers. Then, the training device 120 can calculate the uncertainty of each weight parameter in the first weight matrix, and calculate the binarized weight of the weight parameter based on the uncertainty corresponding to the weight parameter. Among them, the uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process. The binarized weights are used to determine the second weight matrix, and the s*k binarized weights contained in the second weight matrix correspond one-to-one to the s*k weight parameters.

[0176] Finally, the training device 120 can train the neural network based on the above-mentioned binarized weights and the above-mentioned binarized features. Specifically, the trained neural network model can be used to implement the data processing method provided in the embodiment of the present application.

[0177] It should be noted that, in actual applications, the training data maintained in the database 130 may not all come from the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 may not train the neural network model entirely based on the training data maintained by the database 130, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0178] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 3 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud. Figure 3 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include, in an embodiment of the present application: an image to be recognized, a video, or a voice segment to be recognized.

[0179] When the execution device 120 preprocesses the input data, or when the computing module 111 of the execution device 120 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 120 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0180] Finally, the I / O interface 112 returns the processing result, such as the recognition result or classification result of the image, video or voice, to the client device 140, so that the client device 140 can provide it to the user device 150. The user device 150 can be a lightweight terminal that needs to use the target model / rule 101, such as a mobile phone terminal, a laptop computer, an AR / VR terminal or a car terminal, etc., to respond to the corresponding needs of the terminal user, such as performing image recognition on the image input by the terminal user and outputting the recognition result to the terminal user, or performing text classification on the text input by the terminal user and outputting the classification result to the terminal user, etc.

[0181] It is worth noting that the training device 120 can generate a corresponding target model 101 based on different training data for different goals or different tasks, and the corresponding target model 101 can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results.

[0182] exist Figure 3 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0183] After receiving the output result, the client device 140 may transmit the result to the user device 150, which may be a terminal such as a mobile terminal, a tablet computer, a laptop computer, AR / VR, a vehicle terminal, etc. In one example, the user device 150 may run the target model / rule 101 to implement a specific function.

[0184] It is worth noting that Figure 3 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0185] like Figure 3As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be a classification model in application scenario 2 and application scenario 3, an image recognition model in application scenario 4, and a speech recognition model in application scenario 5. Specifically, the target model / rule 101 provided in the embodiment of the present application is, for example, an image recognition model; or, for example, a speech recognition model, etc. In practical applications, the image recognition model and the speech recognition model can both be convolutional neural network models.

[0186] For ease of understanding and for the convenience of explanation, in the embodiments of the present application, the neural network model may include a convolutional neural network, a fully connected network, etc. As described in the previous basic concept introduction, a convolutional neural network is a deep neural network with a convolution structure, and is a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to an image input therein.

[0187] In some possible implementations, such as Figure 4A The structural diagram of the convolutional neural network shown in FIG. 2 is a convolutional neural network (CNN) 200, which may include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230. The input layer 210 may obtain data to be processed, and the obtained data to be processed is passed to the convolutional layer / pooling layer 220 and the subsequent neural network layer 230 for processing, and the image processing result may be obtained. Figure 4A The internal layer structure of CNN 200 is introduced in detail.

[0188] Convolutional layer / pooling layer 220:

[0189] Convolutional Layer:

[0190] like Figure 4A The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0191] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0192] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise points in the image, etc. The multiple weight matrices have the same size (rows × columns), and the convolution feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted multiple convolution feature maps of the same size are then merged to form the output of the convolution operation.

[0193] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0194] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0195] Pooling layer:

[0196] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 4AEach layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. Specifically, the pooling layer is used to sample the data and reduce the amount of data. For example, taking the data as image data, the spatial size of the image can be reduced by the pooling layer during image processing. Generally, the pooling layer may include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator can calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0197] Neural Network Layer 230:

[0198] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 4A 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0199] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 4A The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 4A The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0200] It should be noted that if Figure 4A The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models. Figure 4B The structural diagram of another convolutional neural network is shown in Figure 1. Figure 4B The convolutional neural network (CNN) 300 shown may include an input layer 310, a convolutional layer / pooling layer 320 (where the pooling layer is optional), and a neural network layer 130. Figure 4A In comparison, the multiple convolutional layers / pooling layers in the convolutional layer / pooling layer 320 in FIG. 4B are operated in parallel, and the features extracted respectively are input to the neural network layer 330 for processing.

[0201] For another example, the above neural network model is a fully connected network. A fully connected network means that for n-1 layers and n layers, any node (also called a neuron) in the n-1 layer is connected to all nodes in the n layer. Figure 4C , is a schematic diagram of a fully connected layer structure provided in an embodiment of the present application, such as Figure 4C As shown, the neural network includes an input layer, a hidden layer and an output layer, wherein the two-dimensional parameter matrix of the fully connected layer between the input layer and the hidden layer is (3,4), and the two-dimensional parameter matrix (3,4) indicates that in the fully connected layer structure between the input layer and the hidden layer, the number of input neurons is 3, the number of output neurons is 4, and the number of weights is 12. It can be understood that there is a connection relationship between neurons.

[0202] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0203] Figure 5 A chip hardware structure is provided for the embodiment of the present application, and the chip includes an artificial intelligence processor 50. The chip can be set as follows Figure 3 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 3 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 4A and Figure 4B The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 5 is implemented in the chip shown.

[0204] The artificial intelligence processor 50 can be a neural network processor (network processing unit, NPU), a tensor processing unit (tensor processing unit, TPU) or a graphics processing unit (graphics processing unit, GPU) and any other processor suitable for large-scale XOR operation processing. Take NPU as an example: NPU can be mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks to it. The core part of NPU is the operation circuit 503, which is controlled by the controller 504 to extract data from the memory (weight memory or input memory) and perform operations.

[0205] In some implementations, the operation circuit 503 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 503 is a two-dimensional systolic array. The operation circuit 503 can also be a one-dimensional systolic array or other electronic circuits capable of performing digital operations such as multiplication and addition. In some implementations, the operation circuit 503 is a general-purpose matrix processor.

[0206] For example, assume that there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 503 takes the corresponding data of the matrix B from the weight memory 503 and caches it on each PE in the operation circuit 503. The operation circuit 503 takes the input data of the matrix A from the input memory 501, performs matrix operations based on the input data of the matrix A and the weight data of the matrix B, and stores the partial results or the final results of the matrix in the accumulator 508.

[0207] The unified memory 506 is used to store input data and output data. The weight data is directly transferred to the weight memory 502 through the direct memory access controller (DMAC) 505. The input data is also transferred to the unified memory 506 through the DMAC.

[0208] The bus interface unit (BIU) 510 is used for interaction between DMCA and instruction fetch buffer 509; the bus interface unit 310 is also used for the instruction fetch buffer 509 to obtain instructions from an external memory; the bus interface unit 510 is also used for the storage unit access controller 505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0209] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 506, or to transfer weight data to the weight memory 502, or to transfer input data to the input memory 501.

[0210] The vector calculation unit 507 may include multiple operation processing units, and if necessary, further process the output of the operation circuit 503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. The vector calculation unit 507 is mainly used for calculations of non-convolutional layers or fully connected layers (FC) in neural networks, and can specifically process: pooling, batch normalization, local response normalization, etc. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both.

[0211] In some implementations, the vector calculation unit 507 stores the vector of processed outputs to the unified buffer 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the vector of processed outputs can be used as an activation input to the operation circuit 503, for example, for use in a subsequent layer in a neural network.

[0212] An instruction fetch buffer 509 connected to the controller 504 is used to store instructions used by the controller 504 .

[0213] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0214] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0215] Introduced above Figure 3 The execution device 110 in the embodiment of the present application can execute the quantization method of the neural network or each step of the quantization method of the neural network. Figure 4A and Figure 4B The convolutional neural network model and Figure 5 The chip shown can also execute the neural network quantization method or each step of the neural network quantization method of the embodiment of the present application.

[0216] The embodiment of the present application provides a system architecture. The system architecture includes one or more local devices, an execution device and a data storage system. The local device is connected to the execution device via a communication network.

[0217] The execution device can be implemented by one or more servers. Optionally, the execution device can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, and other devices. The execution device can be arranged at one physical site, or distributed at multiple physical sites. The execution device can use data in a data storage system, or call program code in a data storage system to implement the quantization method of the neural network of the embodiment of the present application.

[0218] Users can operate their respective user devices (e.g., one or more local devices) to interact with the execution device. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0219] Each user's local device can interact with the execution device through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0220] In one implementation, the local device obtains relevant parameters of the target neural network from the execution device, deploys the target neural network on the local device, and uses the target neural network to perform image classification or image processing, etc. The target neural network is trained according to the quantization method of the neural network in the embodiment of the present application.

[0221] In another implementation, the target neural network can be directly deployed on the execution device, and the execution device obtains the data to be processed from the local device and the local device, and classifies or performs other types of processing on the data to be processed according to the target neural network.

[0222] The above-mentioned execution device can also be called a cloud device, and the execution device is generally deployed in the cloud.

[0223] The method involved in the embodiments of the present application is described in detail below. Figure 6 A flow chart of a data processing method provided in an embodiment of the present application is shown in FIG. 1 . The method can be executed by a data processing device, which can be Figure 3 In the system architecture 100 shown, the execution device 120, the client device 140 or the user device 150, the method includes but is not limited to the following steps:

[0224] Step S601: Obtain data to be processed.

[0225] Step S602: input the data to be processed into the neural network, and determine the feature graphs extracted by one or more convolutional layers of the neural network.

[0226] Specifically, the neural network is obtained by training based on binarized weights, wherein the binarized weights are obtained by binarizing the weight parameters according to the uncertainty of the weight parameters corresponding to the binarized weights. The uncertainty of the weight parameters is used to characterize the volatility of the signs of one or more weight values ​​close to zero during the binarization process of the weight parameters.

[0227] It is easy to understand that a neural network contains one or more convolutional layers. In one implementation, the output of a convolutional layer (i.e., the output feature map) can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation. In each convolutional layer, the data exists in a three-dimensional form, which can be regarded as many two-dimensional data stacked together, and each two-dimensional data can become a feature map. The feature map can include m*n feature parameters, where m and n are positive integers.

[0228] Step S603: determine the uncertainty of each feature parameter in the feature map.

[0229] Specifically, in order to speed up the operation of the model, generally speaking, in the process of processing each layer of the neural network, each feature parameter in the feature map can be binarized, that is, the full-precision feature is processed into a binary feature. In an embodiment of the present application, in order to improve the stability of the model, the data processing device needs to binarize the feature parameters based on the uncertainty corresponding to the feature parameters. Among them, the uncertainty of the feature parameters is used to characterize the volatility of the sign of the feature parameters close to zero during the binarization process.

[0230] It can be understood that the sign of each characteristic parameter in the characteristic graph is calculated by the sign function sign(x), when x≥0, sign(x)=1; when x<0, sign(x)=-1. Therefore, when a small change is made to the characteristic parameter near zero, the sign of the characteristic parameter near zero may jump between 1 and -1, and be in an unstable state.

[0231] In order to quantitatively measure the uncertainty of each characteristic parameter in the characteristic graph, in one implementation, the data processing device can calculate the uncertainty of each characteristic parameter in the characteristic graph according to the uncertainty function. The uncertainty function is obtained by modeling a large amount of data. When the independent variable x of the uncertainty function is closer to 0, the value f(x) of the uncertainty function is larger; when the absolute value of the independent variable x of the uncertainty function is larger, the value f(x) of the uncertainty function is smaller. Uncertainty can be represented by a variety of functions. In one implementation, the uncertainty function can be represented by a Gaussian function. The expression of the uncertainty function can be specifically shown in formula (1-2).

[0232]

[0233] Among them, σ is a hyperparameter. In one implementation, σ can be expressed as an element function of a matrix.

[0234] In the embodiment of the present application, in order to improve the processing speed of the model, the embodiment of the present application introduces the uncertainty of k consecutive feature parameters in the feature map to comprehensively estimate the uncertainty of the current position point. Among them, the value of k is the value of the preset position point, and the preset position point is determined based on human experience or historical values.

[0235] In one implementation, the mathematical expression for calculating the uncertainty of each feature parameter in the feature map according to the uncertainty function can be:

[0236]

[0237] Where i is the target position point, which can also be understood as the feature parameter currently being processed. If the data to be processed is an image, the position point can be understood as a pixel point. a is the position point adjacent to the target position point, is the uncertainty corresponding to the target feature parameter on the feature map, f(x j ) is the uncertainty corresponding to one or more feature parameters adjacent to the target feature parameter on the feature graph, and f() is the uncertainty function.

[0238] Step S604: Calculate the binary features of the feature parameters based on the uncertainties corresponding to the feature parameters, wherein the binary features are used to determine the feature matrix.

[0239] Specifically, in order to speed up the processing speed of the model and improve the stability of the model, the data processing device can perform binarization processing on the feature parameters according to the uncertainty corresponding to the feature parameters to obtain the binary features of the feature parameters. It can be understood that the feature map contains m*n feature parameters, and when the data processing device performs binarization processing on the feature map, it actually performs binarization processing on each feature value in the feature map. Therefore, the data processing device can calculate the binary features of the feature value based on the uncertainty corresponding to each feature value.

[0240] In one implementation, when the uncertainty of the target feature value is less than or equal to the second preset threshold, the data processing device may binarize the target feature parameter through a sign function to obtain the binary feature of the target feature value.

[0241] In one implementation, when the uncertainty of the target feature value is greater than the second preset threshold, that is, for features with greater uncertainty, the data processing device can use average pooling and introduce a sign function to binarize the target feature parameters from a spatial dimension. The target data feature is any feature parameter on the feature map.

[0242] That is, the data processing device can perform binarization processing on one or more feature parameters adjacent to the target feature value after average pooling through a sign function to obtain the binary features of the target position point.

[0243] In one implementation, for any feature parameter on the feature map, the mathematical expression for calculating the binary feature of the feature parameter using the uncertainty corresponding to the feature parameter can be found in formula (1-4).

[0244]

[0245] Among them, x in formula (1-4) i is the target feature parameter currently being binarized, is a feature parameter adjacent to the target feature parameter including the target feature parameter. Δ is the second preset threshold, which is artificially set based on experience, and is used to indicate the number of features that need to be binarized in each layer of the neural network. The second preset threshold can be adaptively changed according to actual needs. For example, according to actual needs, the feature parameters ranked in the top 30% from large to small need to be binarized, and the first preset threshold can be set to 0.3. In this way, when the above-mentioned feature parameter is in the top 30% of the sorting value, the data processing device can update the binary feature of the feature parameter through a sign function. When the above-mentioned feature parameter is not in the top 40% of the sorting value, the data processing device can binarize the feature parameters of one or more position points adjacent to the target position point after average pooling through a sign function to obtain the binary feature of the feature map of the target position point.

[0246] After binarizing each eigenvalue in the feature map based on the uncertainty corresponding to the eigenvalue, a feature matrix can be obtained, wherein the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters contained in the feature map.

[0247] Step S605: Obtain processing results of the data to be processed based on the feature matrix.

[0248] Specifically, in order to ensure the completeness of the information of the data to be processed, in each convolution layer except the first convolution layer and the last convolution layer, the device to be processed can perform a two-dimensional convolution operation on the extracted feature matrix and the weight matrix to extract the features in the data to be processed. It should be noted that if the size of the feature matrix is ​​the same as that of the weight matrix, the device to be processed can multiply each parameter on the weight matrix with the parameter on the feature matrix, and finally use the calculated result as the result of this convolution; if the size of the feature matrix is ​​inconsistent with that of the weight matrix, each parameter on the weight matrix can be multiplied with part of the parameters on the feature matrix, and then the device to be processed moves the weight matrix by one step and then performs the next convolution with other parameters on the feature matrix until the parameters in the entire feature matrix are traversed, and the result after traversal is the result of this convolution.

[0249] Among them, different weight matrices can be used to extract different features in the data to be processed. In one implementation, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation. After processing all convolutional layers and other processing, the device to be processed can output the result to be processed. It should be noted that the content of the processing result depends on the function of the trained neural network, and the function of the trained neural network depends on the function of the neural network to be trained. The processing result can be a classification result, a recognition result, etc. of the image.

[0250] For example, see Figure 7 , Figure 7 The figure shows a schematic diagram of the network architecture of a data processing method provided by an embodiment of the present application. The data processing method is specifically an image processing method, comprising: a data processing device obtains an image to be processed, inputs the image to be processed into a trained neural network model, and processes the image by the convolution layer / pooling layer and the subsequent neural network layer in the neural network model to obtain a processing result of the image. In order to ensure the integrity of the information of the image to be processed, the binarization operation is not performed in the first convolution layer and the last convolution layer of the convolution layer, and the binarization operation is performed in the middle convolution layer. In particular, the binarization processing module that performs the binarization operation in each layer of the middle convolution layer can be specifically a first quantization module or a second quantization module. In particular, the feature map of the image to be processed extracted by each convolution layer will be used as the input of the next layer.

[0251] In the first quantization module, the data processing device performs normalization, binarization, two-dimensional convolution, and activation on the input feature map to obtain an output result, and inputs the output result to the next layer as the input of the next layer. The binarization process can refer to the quantization method of the neural network provided in the embodiment of the present application, and the detailed description can refer to Fig. 8A Some of the contents in will not be repeated here.

[0252] In the second quantization module, the data processing device performs deviation, binarization, two-dimensional convolution, normalization, deviation, activation and deviation on the input feature map to obtain an output result, and inputs the output result to the next layer as the input of the next layer. Among them, the binarization process can refer to the quantization method of the neural network provided in the embodiment of the present application, and the detailed description can refer to Fig. 8A Some of the contents in will not be repeated here.

[0253] Finally, the last convolutional layer inputs the output feature map into the fully connected layer, and the fully connected layer obtains the processing result based on the output feature map. It should be noted that the content of the processing result depends on the function of the trained neural network, and the function of the trained neural network depends on the function of the neural network to be trained, which can be the result of image segmentation, recognition result, etc.

[0254] Fig. 8A A flowchart of a neural network quantization method provided in an embodiment of the present application is shown in FIG. The method can be performed by quantization, and the quantization device can be specifically Figure 3 In the training device 110 of the system architecture 100 shown, the method includes but is not limited to the following steps:

[0255] Step S801, obtaining a first weight matrix, wherein the first weight matrix includes parameters used to extract features in a neural network, and the first weight matrix includes s*k weight parameters.

[0256] Specifically, the first weight matrix acquired by the quantization device of the neural network may be an initialization weight matrix or a weight matrix after iterative updating, and the embodiment of the present application does not impose any limitation thereon. s and k are positive integers.

[0257] Step S802: Calculate the uncertainty of each weight parameter in the first weight matrix.

[0258] Specifically, in order to speed up the convergence speed of the neural network model during the training process, the quantization device can perform binarization processing on the first weight matrix of each layer in the neural network. Among them, the weight parameters in the first weight matrix are full-precision parameters. It can be understood that binarization processing is performed on the first weight matrix, that is, binarization processing is performed on each weight parameter in the first weight matrix. Therefore, the quantization device can calculate the uncertainty of each weight parameter in the first weight matrix.

[0259] The uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process. In one implementation, the sign of the weight parameter is calculated by the sign function sign(x), when x≥0, sign(x)=1; when x<0, sign(x)=-1. Therefore, the sign of the weight parameter near zero will frequently jump between 1 and -1, and is in an unstable state. In order to quantitatively measure the uncertainty of the weight parameter, the quantization device needs to calculate the uncertainty of the weight parameter in the neural network based on the uncertainty function. Among them, the uncertainty function is obtained by modeling a large amount of data. When the independent variable x of the uncertainty function is closer to 0, the value f(x) of the uncertainty function is larger; when the absolute value of the independent variable x of the uncertainty function is larger, the value f(x) of the uncertainty function is smaller.

[0260] Uncertainty can be represented by a variety of functions. In one implementation, the uncertainty function can be represented by a Gaussian function. Figure 8B FIG. 1 is a schematic diagram of an uncertainty function provided by an embodiment of the present application. Fig. 8A It can be seen that the value of the uncertainty function is the largest at 0, and gradually decreases as the independent variable (ie, the weight parameter) approaches +1 / -1. Therefore, the embodiment of the present application models the uncertainty function through the predicted continuous value x (-1≤x≤1) and its target (+1 and -1) through the Gaussian function as shown in formula (1-2).

[0261] The uncertainty function obtained by the Gaussian function can be used to calculate the uncertainty of each weight parameter in the weight matrix. It is easy to understand that the higher the value of the uncertainty function, the lower the confidence of the weight parameter, that is, the greater the possibility that the sign of the weight parameter is reversed. For example, a slight change to the weight parameter may cause the sign of the weight parameter to change from +1 to -1. The lower the value of the uncertainty function, the higher the confidence of the weight parameter, that is, the less likely the sign of the weight parameter is to be reversed. For example, a slight change to the weight parameter is unlikely to cause the sign of the weight parameter to change from +1 to -1.

[0262] In the embodiment of the present application, in order to maintain a stable training process and avoid the appearance of weight parameters with relatively unstable sign volatility, the embodiment of the present application introduces the uncertainty of the weight parameters in the first weight matrix corresponding to the number of consecutive m iterations in the neural network to comprehensively estimate the uncertainty of the weight parameters corresponding to the current number of iterations. Among them, the value of m is the value of the preset number of iterations, and the preset number of iterations is determined based on human experience or historical values.

[0263] Therefore, in one implementation, when the current number of iterations is less than or equal to the preset number of iterations, the quantization device may calculate the uncertainty of each weight parameter in the first weight matrix corresponding to the current number of iterations by using an uncertainty function.

[0264] In one implementation, when the current number of iterations is less than or equal to the preset number of iterations, the quantization device may comprehensively consider the uncertainty of each weight parameter in the first weight matrix calculated within the preset number of iterations. The quantization device may calculate the uncertainty of a weight parameter in the first weight matrix corresponding to the current number of iterations based on the uncertainty of each weight parameter in the first weight matrix calculated within the reference preset number of iterations. The reference preset number of iterations is the preset number of iterations closest to the current number of iterations.

[0265] In one implementation, the mathematical expression for calculating each weight parameter in the first weight matrix by using the uncertainty function may be:

[0266]

[0267] Where t is the current iteration number, which can also be understood as the current time step, and m is the preset iteration number, which can also be understood as the preset time step. It can be identified that each weight parameter in the first weight matrix corresponding to the current number of iterations is uncertain. It can represent the uncertainty of each weight parameter in the first weight matrix calculated within a preset number of times, and f() is an uncertainty function.

[0268] When t ≥ m, From 0≤f()≤1, we can get:

[0269] For example, assuming that the preset number of iterations is 5, during the training process of the neural network, if the current number of iterations is 4, the quantization device can calculate the uncertainty of each weight parameter in the first weight matrix corresponding to the current number of iterations through an uncertainty function.

[0270] If the current number of iterations is 5, the quantization device can calculate the uncertainty of each weight parameter in the first weight matrix corresponding to the fifth iteration based on the uncertainty of the weight parameters calculated in the previous five times. That is, the quantization device can calculate the uncertainty of each weight parameter in the first weight matrix corresponding to the fifth iteration based on the expression To calculate the uncertainty of each weight parameter in the first weight parameter corresponding to the 5th iteration number.

[0271] If the current number of iterations is 9, the quantization device can calculate the uncertainty of each weight parameter in the first weight matrix corresponding to the 9th iteration according to the uncertainty of each weight parameter in the first weight matrix calculated within the 5 iterations closest to the current number of iterations. The 5 iterations closest to the current number of iterations 9 can be the 5th iteration, the 6th iteration, the 7th iteration, the 8th iteration, and the 9th iteration. Therefore, the quantization device can calculate the uncertainty of each weight parameter in the first weight matrix corresponding to the 9th iteration according to the uncertainty of each weight parameter in the first weight matrix calculated within the 5 iterations closest to the current number of iterations. To calculate the uncertainty of each weight parameter in the first weight matrix corresponding to the 9th iteration.

[0272] It should be noted that uncertainty can be represented by a variety of functions, and the Gaussian function is only one of the ways of expression, and the embodiments of the present application do not impose any limitation thereto.

[0273] Step S803: Calculate the binary weight of the weight parameter based on the uncertainty corresponding to the weight parameter.

[0274] Specifically, in the binarization process, in order to improve the convergence speed and stability of the neural network, the quantization device can calculate the binarization weight of the weight parameter according to the uncertainty corresponding to the weight parameter. That is, in each iterative update process, when binarizing each weight parameter in the first weight matrix of the current iteration number, the uncertainty of each weight parameter in the first weight matrix of the current iteration number needs to be considered.

[0275] In one implementation, when the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is less than or equal to a first value, the quantization device can binarize the target weight parameter in the first weight matrix corresponding to the current number of iterations through a sign function to obtain a binary weight; wherein the first value is the minimum value between the uncertainty of the weight parameter of the previous number of iterations of the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

[0276] In one implementation, when the uncertainty of the weight parameters in the first weight matrix corresponding to the current number of iterations is greater than the above-mentioned first value, the quantization device can use the binary weights of the target weight parameters in the first weight matrix corresponding to the number of iterations before the current number of iterations as the binary weights of the target weight parameters in the first weight matrix corresponding to the current number of iterations; wherein the first value is the minimum value between the uncertainty of the weight parameters of the number of iterations before the current number of iterations and the first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

[0277] In one implementation, the mathematical expression for calculating the binary weight of the weight parameter based on the uncertainty corresponding to the weight parameter may be:

[0278]

[0279] Where t is the current iteration number, t-1 is the previous iteration number of the current iteration number, and w is the target weight parameter, that is, any weight in the first weight matrix. t is the target weight parameter in the first weight matrix corresponding to the current number of iterations, w t-1 is the target weight parameter in the first weight matrix corresponding to the previous iteration number of the current iteration number. Δ is the first preset threshold. csign() is a deterministic sign function proposed in the embodiment of the present application to minimize the uncertainty of the neural network.

[0280] See also Figure 8C , Figure 8C is a schematic diagram of a binarization result provided in an embodiment of the present application. Figure 8C (a) is a schematic diagram of the real value of the weight parameter and its uncertainty. Figure 8C (b) in is the binarization result of the real value of the weight parameter. Figure 8C As can be seen from (b) in Figure 1, the uncertainty of the weight parameters can be reduced by the csign function.

[0281] For example, assuming that the first preset threshold is 0.3, the uncertainty of the target weight parameter corresponding to the current number of iterations is 0.1, and the uncertainty of the target weight parameter in the first weight matrix corresponding to the number of iterations before the current number of iterations is 0.2. It can be seen that 0.1 is less than min (0.2, 0.3), so the quantization device can calculate the binary weight of the target weight parameter of the current number of iterations through the sign function. It is easy to understand that the iterative update process of the weight is to continuously optimize the weight parameter according to the back propagation gradient. The binary weight of the weight parameter of the current number of iterations is obtained by updating the previous iteration. According to the uncertainty function, it can be known that the target weight parameter corresponding to the current number of iterations is larger than the target weight parameter corresponding to the previous number of iterations. Therefore, after a slight change in the target weight parameter of the current number of iterations, it is unlikely to change the sign of the target weight parameter of the current number of iterations. Therefore, the quantization device can perform binary processing on the target weight parameter corresponding to the current number of iterations according to the sign function.

[0282] Assume that the first preset threshold is 0.3, the uncertainty of the target weight parameter corresponding to the current number of iterations is 0.4, and the uncertainty of the target weight parameter corresponding to the previous number of iterations of the current number of iterations is 0.5. It can be seen that 0.4 is greater than min (0.5, 0.3), so the quantization device can use the binary weight of the target weight parameter of the previous number of iterations of the current number of iterations as the binary weight of the target weight parameter of the current number of iterations. It is easy to understand that the iterative update process of the weight is to continuously optimize the weight parameter according to the back propagation gradient. The binary weight of the weight parameter of the current number of iterations is obtained by updating the previous iteration. According to the uncertainty function, it can be known that the target weight parameter corresponding to the current number of iterations is smaller than the target weight parameter corresponding to the previous number of iterations. Therefore, after a slight change in the target weight parameter of the current number of iterations, the sign of the target weight parameter of the current number of iterations may be changed. Therefore, the quantization device can use the binary weight of the target weight parameter of the previous number of iterations as the binary weight of the target weight parameter of the current number of iterations. In this way, the uncertainty of the neural network can be minimized and the convergence speed of the neural network can be accelerated.

[0283] The target weight parameter is any weight parameter in the first weight matrix.

[0284] It should be noted that in the embodiment of the present application, the quantization device determines the binary weight of the target weight parameter in the first weight matrix corresponding to the current number of iterations by comparing the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations with the first value. The first value is the minimum value between the uncertainty of the weight parameter of the previous number of iterations and the first preset threshold, so the first preset threshold also affects the binary weight of the weight parameter of the current number of iterations.

[0285] Among them, the first preset threshold is artificially set based on experience, and is used to represent the number of target weight parameters that need to be binarized in each layer of the neural network. The first preset threshold can be adaptively changed according to actual needs. For example, according to actual needs, the target weight parameters ranked in the top 30% from large to small need to be binarized, and the first preset threshold can be set to 0.3. In this way, when the target weight parameter is in the top 30% of the sorted value, the quantization device can update the binarized weight of the target weight parameter through a sign function. When the target weight parameter is not in the top 40% of the sorted value, the quantization device will not update the binarized weight of the target weight parameter, that is, the binarized weight of the target weight parameter of the previous iteration number is used as the current iteration number.

[0286] See also Fig. 9 Before executing step S801 to step S802, or after executing step S801 to step S802, the following steps may also be included:

[0287] Step S901: Obtain training data.

[0288] Step S902: input the training data into the neural network to determine the feature graphs extracted by one or more convolutional layers of the neural network.

[0289] Specifically, it is easy to understand that in each convolution layer, the data exists in a three-dimensional form, which can be regarded as many two-dimensional images stacked together. Each of them is called a feature map. In the input layer, if it is a grayscale image, there is only one feature map; if it is a color image, there are generally three feature maps (red, green, and blue). There will be several convolution kernels between layers. The previous layer and each feature map are convolved with each convolution kernel, which will produce a feature map of the next layer. Among them, the feature map can include m*n feature parameters, where m and n are positive integers.

[0290] Step S903: Calculate the binary features of each feature parameter in the feature map.

[0291] Specifically, in order to speed up the convergence of the neural network model during the training process, the neural network parameters can be binarized, for example, each feature parameter in the feature map is binarized to obtain a binary feature.

[0292] In the embodiment of the present application, the uncertainty of the characteristic parameter is used to characterize the volatility of the sign of the characteristic parameter close to zero during the binarization process. In one implementation, the sign of the characteristic parameter is calculated by the sign function sign(x), when x≥0, sign(x)=1; when x<0, sign(x)=-1. Therefore, a slight change in the characteristic parameter near zero may cause the sign of the characteristic parameter near zero to jump between 1 and -1, which is in an unstable state.

[0293] In order to quantitatively measure the uncertainty of the characteristic parameters, in one implementation, the quantization device can calculate the uncertainty of the characteristic parameters in the neural network based on the uncertainty function. The uncertainty function is obtained by modeling a large amount of data. When the independent variable x of the uncertainty function is closer to 0, the value f(x) of the uncertainty function is larger; when the absolute value of the independent variable x of the uncertainty function is larger, the value f(x) of the uncertainty function is smaller. Uncertainty can be represented by a variety of functions. In one implementation, the uncertainty function can be represented by a Gaussian function. The expression of the uncertainty function can be specifically shown in formula (1-2).

[0294] In the embodiment of the present application, in order to maintain a stable training process and avoid the occurrence of a relatively unstable feature graph with symbol fluctuations, the embodiment of the present application introduces the uncertainty of the feature parameters of consecutive k adjacent position points to comprehensively estimate the uncertainty of the feature parameters of the current position point. Among them, the value of k is the value of the preset position point, and the preset position point is determined based on human experience or historical values.

[0295] In one implementation, the mathematical expression of the uncertainty of each feature parameter in the feature map is calculated by the uncertainty function as shown in formula (1-3). Where i is the target position point, which can also be understood as the feature parameter currently being processed. If the training data is an image, the position point can be understood as a pixel point. m is the position point adjacent to the target position point, x is the feature map, x i is the feature parameter corresponding to the target position point on the feature map, and f() is the uncertainty function.

[0296] It is understandable that in order to improve the convergence speed and stability of the neural network during the training process, the quantization device can calculate the binary features of the feature parameters according to the uncertainty corresponding to the feature parameters. It is understandable that the feature map contains m*n feature parameters, and when the data processing device performs binarization processing on the feature map, it actually performs binarization processing on each feature value in the feature map. Therefore, the data processing device can calculate the binary features of each feature value based on the uncertainty corresponding to the feature value.

[0297] In one implementation, when the uncertainty of the target feature value is less than or equal to the second preset threshold, the quantization device may binarize the target feature parameter through a sign function to obtain the binary features of the target feature parameter.

[0298] In one implementation, when the uncertainty of the target feature parameter is greater than the second preset threshold, that is, for features with greater uncertainty, the quantization device can use average pooling and introduce a sign function to binarize the target feature parameter from a spatial dimension. That is, the quantization device can binarize one or more feature parameters adjacent to the target feature parameter after average pooling through a sign function to obtain the binary features of the above-mentioned target position point. Wherein, the target data feature is any feature parameter on the feature map.

[0299] In one implementation, corresponding to any feature parameter on the feature graph, the mathematical expression for calculating the binary feature of the feature parameter by using the uncertainty corresponding to the feature parameter can be as shown in formula (1-4).

[0300] Among them, x i is the feature parameter currently being binarized, are k feature parameters adjacent to the target feature parameter, including the target feature parameter. Δ is the second preset threshold, which is artificially set based on experience, and is used to indicate the number of feature parameters that need to be binarized in each layer of the neural network. The second preset threshold can be adaptively changed according to actual needs. For example, according to actual needs, the first preset threshold can be set to 0.3 if the feature parameters ranked in the top 30% from large to small need to be binarized. In this way, when the above-mentioned feature parameters are in the top 30% of the sorted value, the quantization device can update the binary features of the feature parameters through a sign function. When the feature parameters are not in the top 40% of the sorted value, the quantization device can binarize the feature parameters of one or more position points adjacent to the target position point after average pooling through a sign function to obtain the binary features of the target position point.

[0301] After binarizing each eigenvalue in the feature map based on the uncertainty corresponding to the eigenvalue, a feature matrix can be obtained, wherein the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters contained in the feature map.

[0302] It should be noted that in the training process of the neural network, Fig. 8A or Fig. 9 After training the neural network using the quantization method of the neural network shown in Figure 6 The data processing device shown is used to execute Figure 6 The data processing method shown.

[0303] See also Fig.10 , Fig.10 is a flowchart of a neural network training method provided in an embodiment of the present application. The method can be Figure 3 The training device 120 in the system architecture shown is executed.

[0304] It is easy to understand that in a convolutional neural network with L layers, the weight parameters and feature parameters of the lth layer (any layer in the L layers) are represented as W l and F l , so the operation occurring at layer l can be expressed as:

[0305] F l+1 =φ l (W l *F l ) (1-7)

[0306] Among them, * represents the convolution operation, φ l Represents the rest of the operations that occur in layer l, such as batch normalization (BN), linear rectifier function (ReLU), etc. In a binary neural network, W l and F l Every element of can be projected to {-1, +1} by the sign function.

[0307] However, the sign function discards the amplitude information in the variable and may cause a large quantization error. Therefore, in order to reduce this precision loss, the scale factor method provided by the prior art is widely used in binary neural networks to reduce the precision loss and enhance the representation ability of binary neural networks. The operation using the scale factor can be expressed as:

[0308]

[0309] Among them, W Bl It is represented by the binarization weight obtained by binarization processing, F B l represents the binary features obtained by binarization processing for Represents a binary convolution consisting of XNOR and popcount operations. In this way, the multiple accumulation operations in the real-weight convolution in formula (1-7) can be replaced with light-weight XNOR and popcount operations of simplified convolution to speed up the operation of binary neural networks and reduce storage.

[0310] from Fig.10 It can be seen that L is the number of network layers of the binary neural network. First, the training device initializes the hyperparameters of the neural network model and the weight parameters of all layers. In the forward propagation process, the training device obtains training data, inputs the data to be processed into the neural network, and determines the feature maps extracted by one or more convolutional layers of the neural network.

[0311] The training device calculates the uncertainty of the weight parameters and the uncertainty of the feature parameters in the feature map layer by layer from the 1st layer to the Lth layer. The uncertainty of the weight parameters can be calculated by referring to Fig. 8A For the relevant contents of step S801 shown in FIG. 1 , the uncertainty of the characteristic parameters in the characteristic graph can be referred to Fig. 9 The relevant contents in step S903 shown are not repeated here.

[0312] The training device can update the weight parameter to a binary weight through the csign function according to the uncertainty of the weight parameter. The relevant calculation can be referred to Fig. 8A The step S802 shown is not repeated here.

[0313] The training device can update the feature parameters into binary features through the csign function according to the uncertainty of the feature parameters in the feature map. Fig. 9 The step S903 shown is not repeated here.

[0314] After obtaining the binary weights and binary features, the training device performs a two-dimensional convolution operation on the binary weights and binary features. After the forward propagation is completed, the training device calculates the gradient of the weight parameter from the Lth layer to the 1st layer through back propagation, and updates the weight parameter W layer by layer. l , until the training is completed.

[0315] In summary, in one iteration, the weight parameters can be updated according to the csign function. Furthermore, asynchronous updates are used in the training process of the model. In the forward propagation process, the quantization device can binarize the weight parameters according to the sign function and update them to binary weights; due to the uncertainty of some weight parameters, the weight parameters will not be updated, that is, they will not be binarized. In this way, the uncertainty in the neural network can be reduced, thereby enhancing the stability of the neural network and accelerating its convergence speed.

[0316] In the embodiment of the present application, the training device is based on Fig. 8A After the neural network model is trained using the neural network quantization method shown, the trained model needs to be evaluated using validation data to ensure that the trained neural network model has good generalization properties.

[0317] In one implementation, the training device is based on the Pytorch deep learning framework, and for the CIFAR10 / 100 dataset, WideResNet-22 (WRN-22) is used as the backbone network in the neural network to verify Figure 6 The effectiveness of the method shown. Among them, the learning rate in the neural network model is initially 0.1, the momentum of the stochastic gradient descent (SGD) optimizer is 0.9, and the cosine annealing decay method is applied. On the CIFAR10 / 100 dataset, all networks can be trained for 200 epochs. Among them, WRN-22 can be a WRN network with 22 convolutional layers. Since WRN is a prototype based on ResNet, it introduces a new depth factor k, and adjusts the depth expansion of the feature map through three stages to ensure that the spatial dimension of the feature remains unchanged. In one implementation, k is set to 1. The number of first-level channels is a parameter of WRN, which is set to 16 and 64, so that the network configurations of 16-16-32-64 and 64-64-128-256 can be obtained respectively.

[0318] It should be noted that compared with the test results of other methods on the CIFAR10 / 100 dataset, Fig. 8A The model trained by the quantization method of the neural network shown has good performance. For example, different network configurations are used on CIFAR10 and CIFAR101 to obtain improvements of 0.69%, 0.51%, 0.77%, and 0.49%, respectively. Detailed data can be found in Table 2: Comparison of test results of the embodiment of the present application with other methods on the CIFA dataset.

[0319] Table 2: Comparison of test results of the embodiment of the present application with other methods on the CIFA dataset

[0320]

[0321] As shown in Table 2, W / A represents the weight and activation bit width, FP represents the full precision model, and UaBNN is Fig. 8A The neural network model trained by the quantization method of the neural network shown in FIG. 2 can be seen that, compared with other methods, the model accuracy of UaBNN is closer to the full-precision model. This shows that the neural network model obtained by the embodiment of the present application can not only accelerate the convergence speed of the model, but also ensure the accuracy of the model.

[0322] In one implementation, the training device uses ResNet18 as the backbone network in the neural network for the ImageNet dataset to verify Figure 6 The effectiveness of the method shown. The learning rate in the neural network model is initially 0.001, and the momentum of the Adma optimizer is 0.9. The learning rate adopts a linear decay strategy to reduce the learning rate in a linear manner. For ResNet18, following the device and network modifications in Bi-Real Net, the features and kernels of the backbone convolutional layers except the first and last layers are binarized.

[0323] It should be noted that compared with the test results of other models on the ImageNet dataset, Fig. 8A The model trained by the l-quantization method of the neural network shown has good performance, for example, it improves the Top-1 accuracy by 1.0% and the Top-5 accuracy by 0.6%. For detailed data, see Table 3: Comparison of test results of the embodiment of the present application with other models on the ImageNet dataset.

[0324] Table 3: Comparison of test results of the embodiment of the present application with other models on the CIFA dataset

[0325]

[0326]

[0327] As shown in Table 3, W and A represent the weight and activation bit width, respectively. The backbone of all models in Table 3 is ResNet, and UaBNN* is Fig. 8A The neural network model trained by the quantization method of the neural network shown in FIG. 3 can be seen that, compared with other methods, the model accuracy of UaBNN and UaBNN* is closer to the full-precision model. This shows that the neural network model obtained by the embodiment of the present application can not only accelerate the convergence speed of the model, but also ensure the accuracy of the model.

[0328] In summary, the embodiments of the present application take into account the uncertainty of parameters (such as weights and features) in the neural network, and model a function for quantitatively calculating uncertainty, and binarize the parameters according to the calculated uncertainty, thereby completing the training of the model. Experiments conducted on CIFA and ImageNet show that the method provided by the embodiments of the present application effectively enhances WRN and ResNet18.

[0329] Fig.11 Schematic diagram of the data processing device in the embodiment of the present application. Fig.11 As shown, the data processing device 110 may include: an acquisition unit 1101, an input unit 1102, a calculation unit 1103 and a quantization unit 1104.

[0330] An acquisition unit 1101 is used to acquire data to be processed;

[0331] An input unit 1102 is used to input the data to be processed into the neural network, and determine a feature map extracted by one or more convolutional layers of the neural network, where the feature map includes m*n feature parameters, where m and n are positive integers;

[0332] A calculation unit 1103 is used to calculate the uncertainty of the characteristic parameters in the characteristic graph, wherein the uncertainty of the characteristic parameters is used to characterize the volatility of the signs of the characteristic parameters close to zero during the binarization process of the characteristic parameters;

[0333] A quantization unit 1104, configured to calculate a binary feature of the feature parameter according to the uncertainty of the feature parameter;

[0334] The calculation unit 1103 is further used to obtain a processing result of the data to be processed based on the binary features.

[0335] In one possible implementation, the neural network is trained based on binarized weights, and the binarized weights are obtained based on the uncertainty of weight parameters in the neural network. The uncertainty of the weight parameters is used to characterize the volatility of the signs of one or more weight parameters close to zero during the binarization process.

[0336] In one possible implementation, the calculation unit 1103 is specifically used to calculate the uncertainty of the feature parameters in the feature graph based on the uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is.

[0337] In one possible implementation, the calculation unit 1103 is specifically used to calculate the uncertainty of the feature parameters corresponding to the position point on the feature map according to the uncertainty function; calculate the uncertainty of the target position point according to the uncertainty of the feature parameters of one or more position points adjacent to the target position point, any position point on the target position point feature map.

[0338] In a possible implementation, the quantization unit 1104 is specifically used to binarize the characteristic parameters of the target position point through a sign function to obtain a binary feature of the target position point when the uncertainty of the characteristic parameters of the target position point is less than or equal to a second preset threshold.

[0339] In one possible implementation, the quantization unit 1104 is specifically used to binarize the feature parameters of one or more position points adjacent to the target position point after average pooling through a sign function when the uncertainty of the feature parameters of the target position point is greater than a second preset threshold, so as to obtain the binary features of the target position point.

[0340] It should be understood that the implementation of each device can also refer to the corresponding description in the above embodiments, and the embodiments of the present application will not be repeated.

[0341] Fig.12 is a schematic framework diagram of a quantization device of a neural network in an embodiment of the present application. Fig.12 As shown, the neural network quantization device 120 may include: an acquisition unit 1201, a calculation unit 1202 and a quantization unit 1203.

[0342] An acquisition unit 1201 is used to acquire a first weight matrix, where the first weight matrix includes parameters used to extract features in a neural network, and the first weight matrix includes s*k weight parameters, where s and k are positive integers;

[0343] A calculation unit 1202 is used to calculate the uncertainty of each weight parameter in the first weight matrix, wherein the weight parameter is any weight in the weights of the neural network, and the uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process of the weight parameter;

[0344] The quantization unit 1203 is used to calculate the binarized weight of the weight parameter based on the uncertainty corresponding to the weight parameter. The binarized weight is used to determine the second weight matrix. The s*k binarized weights contained in the second weight matrix correspond to the s*k weight parameters one by one.

[0345] In one possible implementation, the calculation unit 1202 is specifically used to: calculate the uncertainty of each weight parameter in the first weight matrix according to the uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function; and the larger the absolute value of the independent variable of the uncertainty function, the smaller the value of the uncertainty function.

[0346] In a possible implementation, the calculation unit 1202 is specifically configured to: when the current number of iterations is less than or equal to a preset number of iterations, calculate the uncertainty of each weight parameter in the first weight matrix corresponding to the current number of iterations by using an uncertainty function.

[0347] In one possible implementation, the quantization unit 1203 is specifically used to: when the current number of iterations is greater than the preset number of iterations, calculate the uncertainty of each weight parameter of the first weight matrix corresponding to the current number of iterations based on the uncertainty of each weight parameter in the first weight matrix calculated within a reference number of iterations, wherein the reference number of iterations is the preset number of iterations closest to the current number of iterations.

[0348] In one possible implementation, the quantization unit 1203 is specifically used to: when the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is less than or equal to a first value, binarize the target weight parameter in the first weight matrix corresponding to the current number of iterations through a sign function to obtain a binary weight; wherein the first value is the minimum value between the uncertainty of the weight parameter of the previous number of iterations of the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

[0349] In one possible implementation, the quantization unit 1203 is specifically used to: when the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is greater than a first value, use the binary weight of the target weight parameter in the first weight matrix corresponding to the previous number of iterations of the current number of iterations as the binary weight of the target weight parameter in the first weight matrix corresponding to the current number of iterations, wherein the first value is the minimum value between the uncertainty of the weight parameter of the previous number of iterations of the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

[0350] In a possible implementation, the neural network quantization device 120 may further include an input unit 1204:

[0351] The acquisition unit 1201 is also used to acquire training data;

[0352] An input unit 1204, used to input training data into the neural network and determine feature maps extracted by one or more convolutional layers of the neural network;

[0353] The quantization unit 1203 is used to calculate the binary features of each feature parameter in the feature map, wherein the feature map contains m*n feature parameters, m and n are positive integers, and the feature map is the features of the training data extracted in one or more convolutional layers of the neural network. In a possible implementation, the quantization unit 1203 is specifically used to: determine the uncertainty of each feature parameter in the feature map, wherein the uncertainty of the feature parameter is used to characterize the volatility of the sign of the feature parameter close to zero during the binarization process of the feature parameter; calculate the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters, the binary features are used to determine the feature matrix, and the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters.

[0354] In one possible implementation, the quantization unit 1203 is specifically used to: calculate the uncertainty of the feature parameters in the feature graph according to the uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is.

[0355] In one possible implementation, the uncertainty function formula is:

[0356]

[0357] in, is the uncertainty of the target characteristic parameter, f(x j ) is the uncertainty of a feature parameters adjacent to the target feature parameter, i, j, a are all natural numbers.

[0358] In a possible implementation, the quantization unit 1203 is specifically used to: when the uncertainty of the target feature parameter is less than or equal to a second preset threshold, binarize the target feature parameter by a sign function to obtain a binary feature of the target feature parameter.

[0359] In a possible implementation, the quantization unit 1203 is specifically used to: when the target uncertainty of the target feature parameter is greater than a second preset threshold, binarize one or more feature parameters adjacent to the target feature parameter after average pooling through a sign function to obtain a binary feature of the target feature parameter.

[0360] It should be understood that the implementation of each device can also refer to the corresponding description in the above embodiments, and the embodiments of the present application will not be repeated.

[0361] like Fig.13As shown, an embodiment of the present application provides a data processing device, the data processing device 1300 may include a processor 1301 , a memory 1302 , a communication bus 1303 and a communication interface 1304 , and the processor 1301 is connected to the memory 1302 and the communication interface 1304 via the communication bus 1303 .

[0362] Processor 1301 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a neural network processor (NPU) or one or more integrated circuits to execute relevant programs to execute the data processing method of the method embodiment of the present application.

[0363] The processor 1301 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the training method of the neural network of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1301. The above-mentioned processor 1301 may also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as a hardware decoding processor for execution, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1302, and the processor 1301 reads the information in the memory 1302 and executes the data processing method of the embodiment of the method of the present application in combination with its hardware.

[0364] The memory 1302 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1302 may store programs and data, such as the program of the training method of the neural network in the embodiment of the present application. When the program stored in the memory 1302 is executed by the processor 1301, the processor 1301 and the communication interface 1304 are used to execute the various steps of the data processing method in the embodiment of the present application.

[0365] For example, the program used to implement the data processing method in the embodiments of the present application, etc.

[0366] The communication interface 1304 uses a transceiver such as, but not limited to, a transceiver to implement communication between the second device 1300 and other devices or communication networks. For example, a trained neural network can be obtained through the communication interface 1304 to implement information interaction with an execution device, a client device, a user device, or a terminal device.

[0367] Optionally, the data processing device 1300 may also include an artificial intelligence processor 1305, which may be a neural network processor (Network Processing Unit, NPU), a tensor processor (TensorProcessing Unit, TPU), or a graphics processor (Graphics Processing Unit, GPU) and other processors suitable for large-scale XOR operation processing. The artificial intelligence processor 1305 can be mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks to it. The artificial intelligence processor 1305 can implement one or more operations involved in the above-mentioned neural network training method. For example, taking NPU as an example, the core part of NPU is the operation circuit, which controls the operation circuit through the controller to extract the matrix data in the memory 1302 and perform multiplication and addition operations.

[0368] The processor 1301 is used to call the data and program code in the memory to execute the specific operations performed by the data processing device 1300 in the above method embodiment, which will not be described in detail here.

[0369] It should be understood that the implementation of each device can also refer to the corresponding description in the above data processing method embodiment, and the embodiments of the present application will not be repeated.

[0370] Fig.14 Schematic diagram of the structure of a neural network quantization device in an embodiment of the present application, such as Fig.14As shown, the neural network quantization device 1400 may include a processor 1401, a memory 1402, a communication bus 1403 and a communication interface 1404, and the processor 1401 is connected to the memory 1402 and the communication interface 1404 via the communication bus 1403.

[0371] Processor 1401 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a neural network processor (NPU) or one or more integrated circuits to execute relevant programs to execute the neural network quantization method of the method embodiment of the present application.

[0372] The processor 1401 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the quantization method of the neural network of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1401. The above-mentioned processor 1401 may also be a general-purpose processor, a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1402, and the processor 1401 reads the information in the memory 1402 and executes the neural network quantization method of the embodiment of the method of the present application in combination with its hardware.

[0373] The memory 1402 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1402 may store programs and data, such as the program of the training method of the neural network in the embodiment of the present application. When the program stored in the memory 1402 is executed by the processor 1401, the processor 1401 and the communication interface 1404 are used to execute the various steps of the quantization method of the neural network in the embodiment of the present application.

[0374] For example, a program for implementing the quantization method of the neural network in the embodiment of the present application, etc.

[0375] The communication interface 1404 uses a transceiver such as, but not limited to, a transceiver to implement communication between the neural network quantization device 1400 and other devices or communication networks. For example, a trained neural network can be obtained through the communication interface 1404 to implement information interaction with an execution device, a client device, a user device, or a terminal device.

[0376] Optionally, the quantization device of the neural network may also include an artificial intelligence processor 1405, which may be a neural network processor (Network Processing Unit, NPU), a tensor processor (TensorProcessing Unit, TPU), or a graphics processor (Graphics Processing Unit, GPU) and other processors suitable for large-scale XOR operation processing. The artificial intelligence processor 1405 can be mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks to it. The artificial intelligence processor 1405 can implement one or more operations involved in the above-mentioned neural network quantization method. For example, taking NPU as an example, the core part of NPU is the operation circuit, which controls the operation circuit through the controller to extract the matrix data in the memory 1402 and perform multiplication and addition operations.

[0377] The processor 1401 is used to call the data and program code in the memory to execute the above-mentioned neural network quantization method.

[0378] It should be understood that the implementation of each device can also refer to the corresponding description in the above-mentioned neural network training method embodiment, and the embodiments of the present application will not be repeated.

[0379] The embodiment of the present invention further provides a computer storage medium, in which instructions are stored, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above-mentioned embodiment methods. If the components of the above-mentioned device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer product is stored in a computer-readable storage medium.

[0380] The computer-readable storage medium may be an internal storage unit of the device of the aforementioned embodiment, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0381] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0382] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0383] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0384] It is to be understood that those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the various embodiments of the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0385] Those skilled in the art will appreciate that the functions described in the various illustrative logic blocks, modules, and algorithm steps disclosed in the various embodiments of the present application can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or includes any communication media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this way, computer-readable media can generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the technology described in the present application. A computer program product may include computer-readable media.

[0386] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0387] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0388] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0389] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0390] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a base station, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0391] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire data to be processed; the data includes graphics, images, voice, text, or IoT data; the IoT data includes sensory data, and the sensory data includes sensory data of force, displacement, liquid level, temperature, or humidity; Inputting the data to be processed into a neural network, determining a feature map extracted by one or more convolutional layers of the neural network, wherein the feature map includes m*n feature parameters, where m and n are positive integers; Determine the uncertainty of each characteristic parameter in the characteristic graph, wherein the uncertainty of the characteristic parameter is used to characterize the volatility of the sign of the characteristic parameter close to zero during the binarization process of the characteristic parameter; Calculate the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters, the binary features are used to determine a feature matrix, and the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters; A processing result of the data to be processed is obtained based on the feature matrix.

2. The method according to claim 1, characterized in that The neural network is obtained based on binarized weight training, and the binarized weight is obtained by binarizing the weight parameter according to the uncertainty of the weight parameter corresponding to the binarized weight, and the uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process.

3. The method according to claim 1 or 2, characterized in that: Determining the uncertainty of each characteristic parameter in the characteristic graph includes: The uncertainty of each feature parameter in the feature graph is calculated according to the uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is.

4. The method according to claim 3, characterized in that The formula for calculating the uncertainty of each feature parameter in the feature graph according to the uncertainty function is: in, is the uncertainty of the target characteristic parameters, is the uncertainty of a feature parameters adjacent to the target feature parameter, the target parameter is any parameter on the feature graph, and i, j, a are all positive integers.

5. The method according to claim 4, characterized in that The step of calculating the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters includes: When the uncertainty of the target feature parameter is less than or equal to a second preset threshold, the target feature parameter is binarized by a sign function to obtain a binary feature of the target feature parameter.

6. The method according to claim 4, characterized in that The step of calculating the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters includes: When the uncertainty of the target feature parameter is greater than a second preset threshold, one or more feature parameters adjacent to the target feature parameter after average pooling are binarized by a sign function to obtain a binary feature of the target feature parameter.

7. A method for quantizing a neural network, characterized in that: include: Obtain a first weight matrix, wherein the first weight matrix includes parameters for extracting features in a neural network, and the first weight matrix includes s*k weight parameters, where s and k are positive integers; the data input to the neural network includes graphics, images, voice, text, or IoT data; the IoT data includes sensory data, and the sensory data includes sensory data of force, displacement, liquid level, temperature, or humidity; Calculating the uncertainty of each weight parameter in the first weight matrix, wherein the uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process of the weight parameter; The binarized weights of the weight parameters are calculated based on the uncertainties corresponding to the weight parameters, and the binarized weights are used to determine a second weight matrix, wherein the s*k binarized weights contained in the second weight matrix correspond one-to-one to the s*k weight parameters.

8. The method according to claim 7, characterized in that The calculating the uncertainty of each weight parameter in the first weight matrix includes: The uncertainty of each weight parameter in the first weight matrix is ​​calculated according to the uncertainty function, wherein, when the independent variable of the uncertainty function is closer to 0, the value of the uncertainty function is larger; when the absolute value of the independent variable of the uncertainty function is larger, the value of the uncertainty function is smaller.

9. The method according to claim 8, characterized in that The calculating the uncertainty of each weight parameter in the first weight matrix according to the uncertainty function includes: When the current number of iterations is less than or equal to the preset number of iterations, the uncertainty of each weight parameter in the first weight matrix corresponding to the current number of iterations is calculated according to the uncertainty function.

10. The method according to claim 9, characterized in that The calculating the uncertainty of each weight parameter in the first weight matrix according to the uncertainty function includes: When the current number of iterations is greater than the preset number of iterations, the uncertainty of each weight parameter of the first weight matrix corresponding to the current number of iterations is calculated based on the uncertainty of each weight parameter in the first weight matrix calculated within a reference number of iterations, wherein the reference number of iterations is the preset number of iterations closest to the current number of iterations.

11. The method according to any one of claims 7 to 10, characterized in that: The step of calculating the binarized weight of the weight parameter based on the uncertainty corresponding to the weight parameter includes: When the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is less than or equal to a first value, the target weight parameter in the first weight matrix corresponding to the current number of iterations is binarized by a sign function to obtain a binary weight; wherein the first value is the minimum value between the uncertainty of the weight parameter of the previous number of iterations of the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

12. The method according to any one of claims 7 to 10, characterized in that: The step of calculating the binarized weight of the weight parameter based on the uncertainty corresponding to the weight parameter includes: When the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is greater than a first value, the binary weight of the target weight parameter in the first weight matrix corresponding to the number of iterations before the current number of iterations is used as the binary weight of the target weight parameter in the first weight matrix corresponding to the current number of iterations, wherein the first value is the minimum value between the uncertainty of the weight parameter of the number of iterations before the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

13. The method according to any one of claims 7 to 10, characterized in that Also includes: Get training data; Inputting the training data into a neural network to determine a feature map extracted by one or more convolutional layers of the neural network; Calculate the binary features of each feature parameter in the feature map, wherein the feature map contains m*n feature parameters, m and n are positive integers, and the feature map is the feature of the training data extracted in one or more convolutional layers of the neural network.

14. The method according to claim 13, characterized in that The calculating of the binary features of each feature parameter in the feature map includes: Determine the uncertainty of each characteristic parameter in the characteristic graph, wherein the uncertainty of the characteristic parameter is used to characterize the volatility of the sign of the characteristic parameter close to zero during the binarization process of the characteristic parameter; Binary features of the feature parameters are calculated based on the uncertainties corresponding to the feature parameters, and the binary features are used to determine a feature matrix, wherein the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters.

15. The method according to claim 14, characterized in that Determining the uncertainty of each characteristic parameter in the characteristic graph includes: The uncertainty of the feature parameters in the feature graph is calculated according to the uncertainty function, wherein, when the independent variable of the uncertainty function is closer to 0, the value of the uncertainty function is larger; when the absolute value of the independent variable of the uncertainty function is larger, the value of the uncertainty function is smaller.

16. The method according to claim 15, characterized in that The uncertainty function formula is: in, is the uncertainty of the target characteristic parameters, is the uncertainty of a feature parameters adjacent to the target feature parameter, and i, j, a are all natural numbers.

17. The method according to claim 16, characterized in that The step of calculating the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters includes: When the uncertainty of the target feature parameter is less than or equal to a second preset threshold, the target feature parameter is binarized by a sign function to obtain a binary feature of the target feature parameter.

18. The method according to claim 16, characterized in that The step of calculating the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters includes: When the target uncertainty of the target feature parameter is greater than a second preset threshold, one or more feature parameters adjacent to the target feature parameter after average pooling are binarized by a sign function to obtain a binary feature of the target feature parameter.

19. A data processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring data to be processed; the data includes graphics, images, voice, text, or IoT data; the IoT data includes sensory data, and the sensory data includes sensory data of force, displacement, liquid level, temperature, or humidity; An input unit, used to input the data to be processed into a neural network, and determine a feature map extracted by one or more convolutional layers of the neural network, wherein the feature map includes m*n feature parameters, where m and n are positive integers; A calculation unit, used to determine the uncertainty of each characteristic parameter in the characteristic graph, wherein the uncertainty of the characteristic parameter is used to characterize the volatility of the sign of the characteristic parameter close to zero during the binarization process of the characteristic parameter; A quantization unit, used for calculating the binary features of the feature parameters based on the uncertainty corresponding to the feature parameters, wherein the binary features are used to determine a feature matrix, wherein the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters; The calculation unit is further used to obtain a processing result of the data to be processed based on the feature matrix.

20. The device according to claim 19, characterized in that The neural network is obtained based on binarized weight training, and the binarized weight is obtained by binarizing the weight parameter according to the uncertainty of the weight parameter corresponding to the binarized weight, and the uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process.

21. The device according to claim 19 or 20, characterized in that The computing unit is specifically used for: The uncertainty of each feature parameter in the feature graph is calculated according to the uncertainty function, wherein the closer the independent variable of the uncertainty function is to 0, the larger the value of the uncertainty function is; and the larger the absolute value of the independent variable of the uncertainty function is, the smaller the value of the uncertainty function is.

22. The device according to claim 21, characterized in that The formula for calculating the uncertainty of each feature parameter in the feature graph according to the uncertainty function is: in, is the uncertainty of the target characteristic parameters, is the uncertainty of a feature parameters adjacent to the target feature parameter, the target parameter is any parameter on the feature graph, and i, j, a are all positive integers.

23. The device according to claim 22, wherein the quantization unit is specifically configured to: When the uncertainty of the target feature parameter is less than or equal to a second preset threshold, the target feature parameter is binarized by a sign function to obtain a binary feature of the target feature parameter.

24. The device according to claim 22, wherein the quantization unit is specifically configured to: When the uncertainty of the target feature parameter is greater than a second preset threshold, one or more feature parameters adjacent to the target feature parameter after average pooling are binarized by a sign function to obtain a binary feature of the target feature parameter.

25. A quantization device for a neural network, characterized in that: The device comprises: An acquisition unit is used to acquire a first weight matrix, wherein the first weight matrix includes parameters for extracting features in a neural network, and the first weight matrix includes s*k weight parameters, where s and k are positive integers; the data input to the neural network includes graphics, images, voice, text, or IoT data; the IoT data includes perception data, and the perception data includes perception data of force, displacement, liquid level, temperature, or humidity; A calculation unit, used to calculate the uncertainty of each weight parameter in the first weight matrix, wherein the weight parameter is any weight in the weights of the neural network, and the uncertainty of the weight parameter is used to characterize the volatility of the sign of the weight parameter close to zero during the binarization process of the weight parameter; A quantization unit is used to calculate the binarized weight of the weight parameter based on the uncertainty corresponding to the weight parameter, and the binarized weight is used to determine a second weight matrix, and the s*k binarized weights contained in the second weight matrix correspond one-to-one to the s*k weight parameters.

26. The device according to claim 25, characterized in that The computing unit is specifically used for: The uncertainty of each weight parameter in the first weight matrix is ​​calculated according to the uncertainty function, wherein, when the independent variable of the uncertainty function is closer to 0, the value of the uncertainty function is larger; when the absolute value of the independent variable of the uncertainty function is larger, the value of the uncertainty function is smaller.

27. The device according to claim 25, characterized in that The computing unit is specifically used for: When the current number of iterations is less than or equal to the preset number of iterations, the uncertainty of each weight parameter in the first weight matrix corresponding to the current number of iterations is calculated using an uncertainty function.

28. The device according to claim 25, characterized in that The computing unit is specifically used for: When the current number of iterations is greater than the preset number of iterations, the uncertainty of each weight parameter of the first weight matrix corresponding to the current number of iterations is calculated based on the uncertainty of each weight parameter in the first weight matrix calculated within a reference number of iterations, wherein the reference number of iterations is the preset number of iterations closest to the current number of iterations.

29. The device according to any one of claims 25 to 28, characterized in that The quantization unit is specifically used for: When the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is less than or equal to a first value, the target weight parameter in the first weight matrix corresponding to the current number of iterations is binarized by a sign function to obtain a binary weight; wherein the first value is the minimum value between the uncertainty of the weight parameter of the previous number of iterations of the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

30. The device according to any one of claims 25 to 28, characterized in that The quantization unit is specifically used for: When the uncertainty of the target weight parameter in the first weight matrix corresponding to the current number of iterations is greater than a first value, the binary weight of the target weight parameter in the first weight matrix corresponding to the number of iterations before the current number of iterations is used as the binary weight of the target weight parameter in the first weight matrix corresponding to the current number of iterations, wherein the first value is the minimum value between the uncertainty of the weight parameter of the number of iterations before the current number of iterations and a first preset threshold, and the target weight parameter is any parameter in the first weight matrix.

31. The device according to any one of claims 25 to 28, characterized in that Also includes input unit, The acquisition unit is further used to acquire training data; The input unit is used to input the training data into the neural network to determine the feature graph extracted by one or more convolutional layers of the neural network; The quantization unit is used to calculate the binary features of each feature parameter in the feature map, wherein the feature map contains m*n feature parameters, m and n are positive integers, and the feature map is the features of the training data extracted in one or more convolutional layers of the neural network.

32. The device according to claim 31, characterized in that The quantization unit is specifically used for: Determine the uncertainty of each characteristic parameter in the characteristic graph, wherein the uncertainty of the characteristic parameter is used to characterize the volatility of the sign of the characteristic parameter close to zero during the binarization process of the characteristic parameter; Binary features of the feature parameters are calculated based on the uncertainties corresponding to the feature parameters, and the binary features are used to determine a feature matrix, wherein the m*n binary features contained in the feature matrix correspond one-to-one to the m*n feature parameters.

33. The device according to claim 32, characterized in that The quantization unit is specifically used for: The uncertainty of the feature parameters in the feature graph is calculated according to the uncertainty function, wherein, when the independent variable of the uncertainty function is closer to 0, the value of the uncertainty function is larger; when the absolute value of the independent variable of the uncertainty function is larger, the value of the uncertainty function is smaller.

34. The device according to claim 33, characterized in that The uncertainty function formula is: in, is the uncertainty of the target characteristic parameters, is the uncertainty of a feature parameters adjacent to the target feature parameter, and i, j, a are all natural numbers.

35. The device according to claim 34, characterized in that The quantization unit is specifically used for: When the uncertainty of the target feature parameter is less than or equal to a second preset threshold, the target feature parameter is binarized by a sign function to obtain a binary feature of the target feature parameter.

36. The device according to claim 34, characterized in that The quantization unit is specifically used for: When the target uncertainty of the target feature parameter is greater than a second preset threshold, one or more feature parameters adjacent to the target feature parameter after average pooling are binarized by a sign function to obtain a binary feature of the target feature parameter.

37. A data processing device, characterized in that: include: A memory and a processor, wherein the memory is used for a program, and the processor executes the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the data processing method according to any one of claims 1 to 6.

38. A neural network quantization device, characterized in that: include: A memory and a processor, wherein the memory is used for a program, and the processor executes the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the quantization method of a neural network as described in any one of claims 7 to 18.

39. A computer-readable storage medium, characterized in that: The computer-readable medium stores a program code for execution by an electronic device, wherein the program code includes the method according to any one of claims 1 to 6 or claims 7 to 18.

40. A computer program product comprising instructions, characterized in that When the computer program product is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 6 or claims 7 to 18.

Images