A network model training method, device and equipment

Abstract

The present application provides a training method, device, and equipment for a network model, the method comprising: obtaining a floating-point activation gradient value corresponding to a network layer; quantizing the floating-point activation gradient value to obtain a fixed-point activation gradient value and an activation value quantization hyperparameter; the floating-point activation gradient value includes multiple floating-point values, and the quantization intermediate value corresponding to each floating-point value is determined based on the activation value quantization hyperparameter; for each quantization intermediate value, if the quantization intermediate value is not greater than a preset threshold, a probability vector corresponding to the quantization intermediate value is determined, and the fixed-point value corresponding to the quantization intermediate value is determined based on the probability vector; a fixed-point activation gradient value is generated based on the fixed-point values ​​corresponding to all the quantization intermediate values; the initial model weights of the network layer are adjusted based on the fixed-point activation gradient value, and the adjusted model weights are updated to the initial model weights of the network layer. Through the technical solution of the present application, the computational complexity can be reduced, the amount of computation is small, and the training process can be accelerated.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, apparatus, and device for training a network model. Background Art

[0002] Machine learning is a path to artificial intelligence and a multidisciplinary field, encompassing probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning focuses on algorithmic design, enabling computers to automatically learn patterns from data and use these patterns to make predictions about unknown data.

[0003] Machine learning has been widely used, such as deep learning, data mining, computer vision, natural language processing, biometric recognition, search engines, medical diagnosis, speech and handwriting recognition, etc.

[0004] In order to use machine learning to implement artificial intelligence processing, the server needs to obtain a large amount of sample data, train a machine learning model based on this sample data, and deploy the machine learning model to the terminal device (such as a camera, etc.) so that the terminal device can implement artificial intelligence processing based on the machine learning model.

[0005] However, since the training process of machine learning models involves a large number of complex calculations, the training time of machine learning models is long, the training speed is slow, the amount of calculation is large, and the resource consumption is high. Summary of the Invention

[0006] The present application provides a training method for a network model. The initial network model includes multiple network layers, each network layer includes an initial model weight, and for each network layer, the method includes:

[0007] Obtain the floating-point activation gradient value corresponding to the network layer;

[0008] quantizing the floating-point activation gradient value to obtain a fixed-point activation gradient value and an activation value quantization hyperparameter; wherein the floating-point activation gradient value includes a plurality of floating-point values, and determining a quantization intermediate value corresponding to each floating-point value based on the activation value quantization hyperparameter; for each quantization intermediate value, if the quantization intermediate value is not greater than a preset threshold, determining a probability vector corresponding to the quantization intermediate value, and determining a fixed-point value corresponding to the quantization intermediate value based on the probability vector; and generating the fixed-point activation gradient value based on the fixed-point values ​​corresponding to all the quantization intermediate values;

[0009] The initial model weights of the network layer are adjusted based on the fixed-point activation gradient values, and the adjusted model weights are updated as the initial model weights of the network layer.

[0010] The present application provides a training device for a network model. The initial network model includes multiple network layers, each network layer includes an initial model weight, and for each network layer, the device includes:

[0011] An acquisition module, configured to obtain a floating-point activation gradient value corresponding to the network layer;

[0012] a processing module, configured to quantize the floating-point activation gradient value to obtain a fixed-point activation gradient value and an activation value quantization hyperparameter; wherein the floating-point activation gradient value includes a plurality of floating-point values, and determining a quantization intermediate value corresponding to each floating-point value based on the activation value quantization hyperparameter; for each quantization intermediate value, if the quantization intermediate value is not greater than a preset threshold, determining a probability vector corresponding to the quantization intermediate value, and determining a fixed-point value corresponding to the quantization intermediate value based on the probability vector; and generating the fixed-point activation gradient value based on the fixed-point values ​​corresponding to all the quantization intermediate values;

[0013] A training module is used to adjust the initial model weights of the network layer based on the fixed-point activation gradient value, and update the adjusted model weights to the initial model weights of the network layer.

[0014] The present application provides a network model training device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned network model training method.

[0015] It can be seen from the above technical solution that in the embodiment of the present application, the floating-point activation gradient value can be quantized to obtain the fixed-point activation gradient value, and the network model training is realized based on the fixed-point activation gradient value. Since the network model training is realized based on fixed-point data, and the computational complexity of fixed-point data is low, the computational complexity can be reduced, the amount of computation is small, and the resource consumption is small. It can accelerate the training process of the network model, reduce the training time of the network model, and increase the training speed of the network model. It can realize lightweight training, and can realize forward acceleration and reverse acceleration at the same time. At the same time, the performance of the network model is relatively large, and it can be widely used in various business scenarios of artificial intelligence, such as image classification, target detection, segmentation, posture estimation and other business scenarios. By converting high-bit floating-point data into low-bit fixed-point data for storage and calculation, the resource consumption in the network model training process can be greatly reduced and the training speed can be accelerated. When converting floating-point activation gradient values ​​into fixed-point activation gradient values, if the quantized intermediate value is not greater than the preset threshold, the fixed-point value is determined based on the probability vector, thereby introducing randomness into the gradient quantization process. At the same time, a quantization strategy that is adaptive to large and small values ​​is introduced. When performing gradient quantization on small values, probabilistic quantization is used to improve quantization accuracy and enhance the training performance of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a method for training a network model in one embodiment of the present application;

[0017] Figure 2 is a schematic structural diagram of an initial network model in one embodiment of the present application;

[0018] Figure 3 Schematic diagram of the training process of the network model in one embodiment of the present application;

[0019] Figures 4A-4C This is a schematic diagram of data distribution in one embodiment of the present application;

[0020] Figure 5 It is a structural diagram of a training device for a network model in one embodiment of the present application;

[0021] Figure 6 This is a hardware structure diagram of a network model training device in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The terms used in the embodiments of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application and claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" used may also be interpreted as "at the time of" or "when" or "in response to determining".

[0024] In an embodiment of the present application, a network model training method is proposed for training an initial network model to obtain a trained target network model. The initial network model may include multiple network layers, each of which may include initial model weights (i.e., model weights that need to be adjusted and optimized).

[0025] See also Figure 1 FIG. 5 is a flow chart of the method, which may include:

[0026] Step 101: For each network layer, obtain the floating-point activation gradient value corresponding to the network layer.

[0027] Step 102: Quantize the floating-point activation gradient value to obtain a fixed-point activation gradient value and an activation value quantization hyperparameter. The floating-point activation gradient value may include multiple floating-point values, and a quantization intermediate value corresponding to each floating-point value may be determined based on the activation value quantization hyperparameter. Then, for each quantization intermediate value, if the quantization intermediate value is not greater than a preset threshold, a probability vector corresponding to the quantization intermediate value may be determined, and a fixed-point value corresponding to the quantization intermediate value may be determined based on the probability vector. On this basis, a fixed-point activation gradient value may be generated based on the fixed-point values ​​corresponding to all the quantization intermediate values.

[0028] Exemplarily, the probability vector includes a first probability and a second probability, the first probability being a probability that the fixed-point value corresponding to the quantized intermediate value is a rounded-up value of the quantized intermediate value, and the second probability being a probability that the fixed-point value corresponding to the quantized intermediate value is a rounded-down value of the quantized intermediate value. Determining the probability vector corresponding to the quantized intermediate value may include, but is not limited to, determining the first probability and the second probability corresponding to the quantized intermediate value based on the rounded-up value of the quantized intermediate value and / or the rounded-down value of the quantized intermediate value.

[0029] Exemplarily, determining the first probability and the second probability corresponding to the quantized intermediate value based on the rounded-up value of the quantized intermediate value and / or the rounded-down value of the quantized intermediate value may include, but is not limited to: determining the first probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value, and determining the second probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value; or, determining the first probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value, and determining the second probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value; or, determining the first probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value, and determining the second probability based on the first probability, wherein the sum of the first probability and the second probability is a fixed value; or, determining the first probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value, and determining the second probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value; or, determining the second probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value, and determining the first probability based on the second probability, wherein the sum of the first probability and the second probability is a fixed value.

[0030] Exemplarily, for each quantized intermediate value, if the quantized intermediate value is greater than a preset threshold, a rounding operation may be performed on the quantized intermediate value to obtain a fixed-point value corresponding to the quantized intermediate value.

[0031] Exemplarily, a method for determining the preset threshold may include, but is not limited to: determining the preset threshold based on the maximum absolute value of all quantized intermediate values ​​(the quantized intermediate values ​​corresponding to all floating-point values).

[0032] Step 103: Adjust the initial model weights of the network layer based on the fixed-point activation gradient value, and update the adjusted model weights as the initial model weights of the network layer.

[0033] Exemplarily, adjusting the initial model weights of the network layer based on the fixed-point activation gradient value may include but is not limited to: determining the fixed-point weight gradient value based on the fixed-point activation gradient value, and determining the floating-point weight gradient value based on the fixed-point weight gradient value and the activation value quantization hyperparameter; and adjusting the initial model weights of the network layer based on the floating-point weight gradient value to obtain the adjusted model weights.

[0034] In one possible implementation, if the network layer is not the first network layer, the initial model weights of the network layer can be quantized to obtain reverse fixed-point weights and reverse weight value quantization hyperparameters; based on the fixed-point activation gradient value, the reverse fixed-point weight, the activation value quantization hyperparameter, and the reverse weight value quantization hyperparameter, the floating-point activation gradient value corresponding to the previous network layer of the network layer is determined, and the floating-point activation gradient value corresponding to the previous network layer is sent to the previous network layer.

[0035] Exemplarily, based on the fixed-point activation gradient value, the inverse fixed-point weight, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter, determining the floating-point activation gradient value corresponding to the previous network layer of the network layer may include but is not limited to: determining the fixed-point activation gradient value corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value and the inverse fixed-point weight corresponding to the network layer; then, based on the fixed-point activation gradient value corresponding to the previous network layer, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter, determining the floating-point activation gradient value corresponding to the previous network layer.

[0036] Exemplarily, obtaining the floating-point activation gradient value corresponding to the network layer may include: if the network layer is the last network layer, obtaining the floating-point activation value corresponding to the network layer, determining a target loss value based on the floating-point activation value and a calibrated floating-point true value, and determining the floating-point activation gradient value corresponding to the network layer based on the target loss value. If the network layer is not the last network layer, determining the floating-point activation gradient value output by the network layer following the network layer as the floating-point activation gradient value corresponding to the network layer.

[0037] It can be seen from the above technical solution that in the embodiment of the present application, the floating-point activation gradient value can be quantized to obtain the fixed-point activation gradient value, and the network model training is realized based on the fixed-point activation gradient value. Since the network model training is realized based on fixed-point data, and the computational complexity of fixed-point data is low, the computational complexity can be reduced, the amount of computation is small, and the resource consumption is small. It can accelerate the training process of the network model, reduce the training time of the network model, and increase the training speed of the network model. It can realize lightweight training, and can realize forward acceleration and reverse acceleration at the same time. At the same time, the performance of the network model is relatively large, and it can be widely used in various business scenarios of artificial intelligence, such as image classification, target detection, segmentation, posture estimation and other business scenarios. By converting high-bit floating-point data into low-bit fixed-point data for storage and calculation, the resource consumption in the network model training process can be greatly reduced and the training speed can be accelerated. When converting floating-point activation gradient values ​​into fixed-point activation gradient values, if the quantized intermediate value is not greater than the preset threshold, the fixed-point value is determined based on the probability vector, thereby introducing randomness into the gradient quantization process. At the same time, a quantization strategy that is adaptive to large and small values ​​is introduced. When performing gradient quantization on small values, probabilistic quantization is used to improve quantization accuracy and enhance the training performance of the network model.

[0038] The following describes the technical solutions of the embodiments of the present application in conjunction with specific application scenarios.

[0039] Before introducing the technical solution of the present application, concepts related to the embodiments of the present application are first introduced.

[0040] Machine Learning: Machine learning is a method for achieving artificial intelligence. It studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Deep learning and neural networks are subcategories of machine learning. Deep learning is the process of using mathematical models to model specific real-world problems in order to solve similar problems within the same field. Neural networks are algorithmic mathematical models that mimic the behavioral characteristics of animal neural networks and perform distributed parallel information processing. These neural networks rely on the complexity of the system to adjust the interconnections between the large number of internal nodes to achieve the purpose of information processing. For ease of description, the structure and function of neural networks are used as an example. Other subcategories of machine learning have similar structures and functions.

[0041] Neural Network: Neural networks may include but are not limited to convolutional neural networks (CNN), recurrent neural networks (RNN), fully connected networks, etc. The structural units of neural networks may include but are not limited to convolutional layers (Conv), pooling layers (Pool), excitation layers, fully connected layers (FC), etc.

[0042] In practical applications, one or more convolutional layers, one or more pooling layers, one or more excitation layers, and one or more fully connected layers can be combined to construct a neural network according to different needs.

[0043] In the convolution layer, the input data features are enhanced by using the convolution kernel to perform convolution operations on the input data features. The convolution kernel can be a matrix of size m*n. The input data features of the convolution layer are convolved with the convolution kernel to obtain the output data features of the convolution layer. The convolution operation is actually a filtering process.

[0044] In the pooling layer, by taking the maximum value, minimum value, average value and other operations on the input data features (such as the output of the convolutional layer), the principle of local correlation is used to sub-sample the input data features, reduce the processing amount, and maintain feature invariance. The pooling layer operation is actually a downsampling process.

[0045] In the excitation layer, activation functions (such as nonlinear functions) can be used to map the input data features, thereby introducing nonlinear factors and enabling the neural network to enhance its expressive power through nonlinear combinations.

[0046] The activation function may include but is not limited to a ReLU (Rectified Linear Units) function, which is used to set features less than 0 to 0, while features greater than 0 remain unchanged.

[0047] In the fully connected layer, all data features input to this fully connected layer are fully connected to obtain a feature vector, which can include multiple data features.

[0048] Network model: You can pre-build a network model, called the initial network model. The initial network model is a network model that has not yet been trained. The initial network model can be a machine learning model, such as a deep learning-based machine learning model or a neural network-based machine learning model. There are no restrictions on this.

[0049] For the training process of the initial network model, the sample data can be used to train the initial network model, that is, the process of adjusting and optimizing the model weights (i.e., network parameters) in the initial network model. For example, the initial network model may include multiple network layers, each of which includes initial model weights. The initial model weights are the model weights that need to be adjusted and optimized. For example, for the initial network model based on a neural network, the initial model weights may be convolution layer parameters (such as convolution kernel parameters), pooling layer parameters, excitation layer parameters, fully connected layer parameters, etc. For the training process of the initial network model, the sample data can be used to adjust and optimize the initial model weights in multiple network layers of the initial network model.

[0050] After the initial network model training is complete, the trained network model is called the target network model. The model weights in the multiple network layers of the target network model are called target model weights. That is, the target model weights are the model weights that have been adjusted and optimized. After obtaining the target network model, it can be deployed to the terminal device, allowing the terminal device to implement artificial intelligence processing based on the target network model.

[0051] For example, the target network model can be widely applied to various AI business scenarios, such as image classification, target detection, segmentation, and pose estimation. Taking target detection as an example, functions such as face detection, human body detection, and vehicle detection can be implemented based on the target network model. For example, for face detection, an image containing a face can be input into the target network model, and the target network model performs AI processing on the image. The result of the AI ​​processing is the face detection result. For vehicle detection, an image containing a vehicle can be input into the target network model, and the target network model performs AI processing on the image. The result of the AI ​​processing is the vehicle detection result.

[0052] Quantization of the initial network model: The initial model weights (i.e., the initial model weights of the network layer, also called model parameters or network parameters), input features, and output features represented by floating-point numbers are approximated using fixed-point values ​​to improve the computing speed of the initial network model and compress the initial network model.

[0053] For example, the initial network model may include a large number of initial model weights (such as millions). These initial model weights are floating-point types. The initial model weights of floating-point types will take up a lot of storage space, and the calculation of the initial model weights of floating-point types will consume a lot of computing resources. If the initial model weights of fixed-point types can be used for calculation without affecting the accuracy of the initial network model, the calculation speed can be improved, computing resources can be saved, and storage space can be saved. Therefore, quantization technology is introduced, that is, the initial network model is compressed by reducing the number of bits required to represent each initial model weight. Based on quantization technology, the initial model weights of floating-point types can be converted into initial model weights of fixed-point types. For example, for the same network layer of the initial network model (such as convolutional layer, pooling layer, excitation layer, etc.), the initial model weights of the network layer are distributed in a small interval, such as the interval [-10, 30], where -10 represents the minimum value of the initial model weight of the network layer and 30 represents the maximum value of the initial model weight of the network layer. Based on this, when using 8-bit quantization (other options are possible), all the initial model weights of the network layer can be linearly mapped (nonlinear mapping can also be used to compress the space) to the interval [-10, 30].

[0054] The embodiment of the present application proposes a training method for a network model. By converting high-bit floating-point data (such as 32-bit floating-point data) into low-bit fixed-point data (such as 8-bit fixed-point data) for storage and calculation, it is possible to reduce resource consumption during network model training and speed up training. This embodiment proposes a weight quantization algorithm in lightweight training, which can be widely applied to various artificial intelligence business scenarios, such as training acceleration for tasks such as image classification, target detection, segmentation, and posture estimation. It can simultaneously achieve forward acceleration and reverse acceleration, and at the same time, the performance is basically unchanged compared to floating-point training.

[0055] In this embodiment, a network model can be pre-built, and this network model is called an initial network model. The initial network model can include multiple network layers, and each network layer can include an initial model weight. Figure 2 As shown, it is a structural diagram of the initial network model. The initial network model may include network layer 21, network layer 22 and network layer 23. Of course, here we only take three network layers as an example. The number of network layers in the initial network model will be much greater than three. Network layer 21 is the first network layer of the initial network model, network layer 23 is the last network layer of the initial network model, and network layer 22 is the intermediate network layer of the initial network model. The number of intermediate network layers will be much greater than one. The processing process of each intermediate network layer is the same. In this embodiment, one intermediate network layer is used as an example for explanation. In addition, the initial model weight of network layer 21 can be recorded as initial model weight a1, the initial model weight of network layer 22 can be recorded as initial model weight a2, and the initial model weight of network layer 23 can be recorded as initial model weight a3.

[0056] Exemplarily, the initial model weight a1, the initial model weight a2, and the initial model weight a3 may be high-precision weights, which may be randomly generated or imported from a pre-trained model, and there is no restriction on the source.

[0057] In this embodiment, a training set can be pre-built, which can include a large amount of sample data and calibration data corresponding to each sample data, and the initial network model is trained based on the training set. For the sake of convenience, the following description will take the processing process of a sample data as an example.

[0058] Among them, the sample data can be sample data of floating-point type, therefore, the sample data is called floating-point sample data, and the calibration data corresponding to the sample data is calibration data of floating-point type, therefore, the calibration data is called floating-point calibration data, that is, the floating-point true value of the sample data.

[0059] In the above application scenario, see Figure 3 As shown, the training method of the network model may include:

[0060] Step S11: Obtain floating-point input data corresponding to network layer 21 and quantize the floating-point input data to obtain fixed-point input data d11 and input data quantization hyperparameter d12. Quantize the initial model weight a1 of network layer 21 to obtain forward fixed-point weight d13 and forward weight value quantization hyperparameter d14.

[0061] For example, since network layer 21 is the first network layer, floating-point sample data can be used as floating-point input data corresponding to network layer 21. By quantizing the floating-point input data, fixed-point input data d11 and input data quantization hyperparameter d12 can be obtained. In the quantization process of floating-point input data, the floating-point input data includes a large number of floating-point values. The input data quantization hyperparameter d12 is first determined. This input data quantization hyperparameter d12 represents a scaling factor. For each floating-point value, the floating-point value can be converted into a fixed-point value within a specified interval based on the scaling factor, and the fixed-point values ​​corresponding to all floating-point values ​​constitute the fixed-point input data d11. Of course, the above is only an example and does not limit this quantization process.

[0062] Exemplarily, for the quantization process of the initial model weight a1, the initial model weight a1 includes a large number of floating-point values. The forward weight value quantization hyperparameter d14 corresponding to the initial model weight a1 is first determined. This forward weight value quantization hyperparameter d14 represents the scaling factor. For each floating-point value, the floating-point value can be converted into a fixed-point value within a specified interval based on the scaling factor, and the fixed-point values ​​corresponding to all floating-point values ​​constitute the forward fixed-point weight d13. Of course, the above is just an example and there is no limitation on this quantization process.

[0063] In one possible implementation, the number of input channels of the network layer 21 can be recorded as cin1, and the number of output channels of the network layer 21 can be recorded as cout1. Obviously, the number of channels of the floating-point input data corresponding to the network layer 21 is cin1, that is, the number of channels of the floating-point input data is the same as the number of input channels of the network layer 21. In addition, the initial model weight a1 of the network layer 21 can be a four-dimensional tensor, and the four dimensions of the initial model weight a1 can be cin1, cout1, m1, and n1, respectively, where m1 and n1 are the sizes of the initial model weight a1.

[0064] On this basis, the initial model weight a1 of the network layer 21 is quantized to obtain the forward fixed-point weight d13 and the forward weight value quantization hyperparameter d14, which may include but is not limited to: dividing the initial model weight a1 into cout1 sub-model weights, each sub-model weight is expressed as cin1*m1*n1, m1*n1 is the size of the sub-model weight, and cin1 is the number of channels of the sub-model weight. Then, each sub-model weight in the cout1 sub-model weights is quantized to obtain the forward fixed-point weight d13 and the forward weight value quantization hyperparameter d14 corresponding to the sub-model weight. In summary, cout1 sub-model weights, cout1 forward fixed-point weight d13 and cout1 forward weight value quantization hyperparameter d14 can be obtained. For each sub-model weight, the sub-model weight corresponds to a forward fixed-point weight d13 and a forward weight value quantization hyperparameter d14.

[0065] In summary, it can be seen that for the forward direction of the model training process, it is quantized according to the number of output channels cout1 of the initial model weight a1, and each output channel corresponds to a forward weight value quantization hyperparameter.

[0066] Step S12: For the network layer 21, the fixed-point input data d11 is processed based on the forward fixed-point weight d13 to obtain the fixed-point activation value d15, and the floating-point activation value d16 corresponding to the network layer 21 is determined based on the fixed-point activation value d15, the input data quantization hyperparameter d12 and the forward weight value quantization hyperparameter d14.

[0067] Exemplarily, the processing process for the fixed-point input data d11 may be related to the function of the network layer 21. For example, if the network layer 21 is used to implement the convolution function, a convolution operation may be performed based on the fixed-point input data d11 and the forward fixed-point weight d13 to obtain a fixed-point activation value d15. If the network layer 21 is used to implement the pooling function, a pooling operation may be performed based on the fixed-point input data d11 and the forward fixed-point weight d13 to obtain a fixed-point activation value d15, and so on. Obviously, since the operation is based on fixed-point data (such as fixed-point input data d11 and forward fixed-point weight d13) rather than floating-point data, the processing process of the network layer 21 can be accelerated to achieve the operation acceleration function.

[0068] For example, after obtaining the fixed-point activation value d15, the fixed-point activation value d15 can be converted into a floating-point activation value d16, that is, the fixed-point activation value d15 is dequantized to obtain the floating-point activation value d16. For example, the fixed-point activation value d15 is dequantized based on the input data quantization hyperparameter d12 and the forward weight value quantization hyperparameter d14 to obtain the floating-point activation value d16. The implementation method of dequantization is the reverse operation of the above-mentioned quantization method, and this embodiment does not limit the implementation method of this dequantization.

[0069] In one possible implementation, the number of input channels of network layer 21 is cin1, the number of output channels of network layer 21 is cout1, the number of channels of floating-point input data is cin1, the four dimensions of initial model weight a1 are cin1, cout1, m1, and n1, and the three dimensions of floating-point input data are cin1, h, and w, where h and w represent the size of the floating-point input data. Furthermore, the initial model weight a1 corresponds to cout1 sub-model weights, each of which is expressed as cin1*m1*n1. Based on this:

[0070] For each sub-model weight, the fixed-point input data d11 can be processed based on the forward fixed-point weight d13 corresponding to the sub-model weight to obtain a fixed-point activation value d15. Since the dimension of the forward fixed-point weight d13 is cin1*m1*n1, the dimension of the fixed-point input data d11 (the same as the dimension of the floating-point input data) is cin1*h*w, that is, the number of input channels of the two is the same, both cin1. Taking the convolution operation as an example, after the convolution operation is performed on the two, the dimension of the fixed-point activation value d15 can be h*w, that is, each sub-model weight corresponds to a fixed-point activation value d15 of h*w dimension. Further, after the fixed-point activation value d15 is converted to a floating-point activation value d16, the floating-point activation value d16 is also h*w dimension, that is, each sub-model weight corresponds to a floating-point activation value d16 of h*w dimension.

[0071] Since the initial model weight a1 corresponds to cout1 sub-model weights, cout1 sub-model weights correspond to cout1 floating-point activation values ​​d16 of h*w dimensions. That is, the dimension of the floating-point activation value d16 is cout1*h*w, where cout1 represents the number of channels of the floating-point activation value d16 and h*w represents the size.

[0072] Step S13: Obtain floating-point input data corresponding to network layer 22 and quantize the floating-point input data to obtain fixed-point input data d21 and input data quantization hyperparameter d22. Quantize the initial model weight a2 of network layer 22 to obtain forward fixed-point weight d23 and forward weight value quantization hyperparameter d24.

[0073] Since network layer 22 is the second network layer, the floating-point activation value output by the first network layer (i.e., the floating-point activation value d16) can be used as the floating-point input data corresponding to network layer 22. By quantizing the floating-point input data, fixed-point input data d21 and input data quantization hyperparameter d22 can be obtained.

[0074] In one possible implementation, the number of input channels of network layer 22 can be denoted as cin2, and the number of output channels of network layer 22 can be denoted as cout2. The number of input channels cin2 of network layer 22 is the same as the number of output channels cout1 of network layer 21, that is, the number of channels cout1 of floating-point activation value d16 (floating-point input data) is the same as the number of input channels cin21 of network layer 22. The initial model weight a2 can be a four-dimensional tensor, and the four dimensions can be cin2, cout2, m2, and n2, respectively, where m2 and n2 are the sizes of the initial model weight a2.

[0075] On this basis, the initial model weight a2 of the network layer 22 is quantized to obtain the forward fixed-point weight d23 and the forward weight value quantization hyperparameter d24, which may include but is not limited to: dividing the initial model weight a2 into cout2 sub-model weights, each sub-model weight is expressed as cin2*m2*n2, m2*n2 is the size of the sub-model weight, and cin2 is the number of channels of the sub-model weight. Then, each sub-model weight in the cout2 sub-model weights is quantized to obtain the forward fixed-point weight d23 and the forward weight value quantization hyperparameter d24 corresponding to the sub-model weight. In summary, cout2 sub-model weights, cout2 forward fixed-point weights d23 and cout2 forward weight value quantization hyperparameter d24 can be obtained.

[0076] Step S14: For the network layer 22, the fixed-point input data d21 is processed based on the forward fixed-point weight d23 to obtain the fixed-point activation value d25, and the floating-point activation value d26 corresponding to the network layer 22 is determined based on the fixed-point activation value d25, the input data quantization hyperparameter d22 and the forward weight value quantization hyperparameter d24.

[0077] For example, the processing process of step S14 can refer to step S12, and will not be repeated here.

[0078] In one possible implementation, the number of input channels of the network layer 22 is cin2, the number of output channels of the network layer 22 is cout2, the number of channels of the floating-point input data (i.e., the floating-point activation value d16) is cout1, and cout1 is the same as cin2. The number of channels of the floating-point input data is subsequently recorded as cin2. Therefore, the four dimensions of the initial model weight a2 are cin2, cout2, m2, and n2, and the three dimensions of the floating-point input data are cin2, h, and w, where h and w represent the size of the floating-point input data. The initial model weight a2 corresponds to cout2 sub-model weights, and each sub-model weight is expressed as cin2*m2*n2. On this basis:

[0079] For each sub-model weight, the fixed-point input data d21 can be processed based on the forward fixed-point weight d23 corresponding to the sub-model weight to obtain the fixed-point activation value d25. Since the dimension of the forward fixed-point weight d23 is cin2*m2*n2, and the dimension of the fixed-point input data d21 is cin2*h*w, that is, the number of input channels of both is cin2, the dimension of the fixed-point activation value d25 is h*w, that is, each sub-model weight corresponds to a fixed-point activation value d25 of h*w dimension, and the floating-point activation value d26 is also of h*w dimension, that is, each sub-model weight corresponds to a floating-point activation value d26 of h*w dimension.

[0080] Since the initial model weight a2 corresponds to cout2 sub-model weights, the cout2 sub-model weights correspond to cout2 floating-point activation values ​​d26 of h*w dimensions. That is, the dimension of the floating-point activation value d26 is cout2*h*w, where cout2 represents the number of channels of the floating-point activation value d26 and h*w represents the size.

[0081] Step S15: Obtain the floating-point input data corresponding to network layer 23 (using the floating-point activation value d26 output by the second network layer as the floating-point input data), quantize the floating-point input data to obtain fixed-point input data d31 and input data quantization hyperparameter d32. Quantize the initial model weight a3 of network layer 23 to obtain the forward fixed-point weight d33 and the forward weight value quantization hyperparameter d34.

[0082] For example, the number of input channels of network layer 23 can be recorded as cin3, and the number of output channels of network layer 23 can be recorded as cout3. The number of input channels of network layer 23 cin3 is the same as the number of output channels of network layer 22 cout2. The initial model weight a3 can be a four-dimensional tensor, and the four dimensions are cin3, cout3, m3, and n3, respectively. m3 and n3 are the sizes of the initial model weight a2. On this basis, the initial model weight a3 is divided into cout3 sub-model weights, and each sub-model weight is expressed as cin3*m3*n3, where cin3 is the number of channels of the sub-model weight. Then, each sub-model weight in the cout3 sub-model weights is quantized to obtain the forward fixed-point weight d33 and the forward weight value quantization hyperparameter d34 corresponding to the sub-model weight.

[0083] Step S16: For the network layer 23, the fixed-point input data d31 is processed based on the forward fixed-point weight d33 to obtain the fixed-point activation value d35, and the floating-point activation value d36 corresponding to the network layer 23 is determined based on the fixed-point activation value d35, the input data quantization hyperparameter d32 and the forward weight value quantization hyperparameter d34.

[0084] For example, the processing process of step S16 can refer to step S12, and will not be repeated here.

[0085] In one possible implementation, the number of input channels of the network layer 23 is cin3, the number of output channels of the network layer 23 is cout3, the number of channels of the floating-point input data (i.e., the floating-point activation value d26) is cout2, and cout2 is the same as cin3. Therefore, the four dimensions of the initial model weight a3 are cin3, cout3, m3, and n3, and the three dimensions of the floating-point input data are cin3, h, and w. The initial model weight a3 corresponds to cout3 sub-model weights, and each sub-model weight is expressed as cin3*m3*n3. On this basis:

[0086] For each sub-model weight, the fixed-point input data d31 is processed based on the forward fixed-point weight d33 corresponding to the sub-model weight to obtain a fixed-point activation value d35. Since the number of input channels of the forward fixed-point weight d33 and the number of input channels of the fixed-point input data d31 are both cin3, the fixed-point activation value d35 and the floating-point activation value d36 are both h*w dimensional. That is, each sub-model weight corresponds to a floating-point activation value d36 of h*w dimension. Since the initial model weight a3 corresponds to cout3 sub-model weights, the cout3 sub-model weights correspond to cout3 floating-point activation values ​​d36 of h*w dimension. That is, the dimension of the floating-point activation value d36 is cout3*h*w, where cout3 represents the number of channels of the floating-point activation value d36.

[0087] Step S17, network layer 23 is the last network layer. The target loss value is determined based on the floating-point activation value d36 corresponding to network layer 23 and the calibrated floating-point true value (the calibration data corresponding to the floating-point sample data in step S11). The floating-point activation gradient value e31 corresponding to network layer 23 is determined based on the target loss value.

[0088] Exemplarily, a loss function may be preconfigured, the input of which is a floating-point activation value and a floating-point true value, and the output of which is a target loss value. There are no restrictions on this loss function.

[0089] For example, since network layer 23 is the last network layer, after obtaining the floating-point activation value d36 corresponding to network layer 23, the floating-point activation value d36 and the floating-point true value corresponding to the floating-point activation value d36 can be substituted into the loss function to obtain the target loss value. There is no restriction on this process.

[0090] After obtaining the target loss value, the floating-point activation gradient value e31 corresponding to the network layer 23 is determined based on the target loss value using algorithms such as the gradient descent method. The floating-point activation gradient value e31 is a gradient (Gradient) used to adjust and optimize the initial model weight a3 of the network layer 23. There is no restriction on this process.

[0091] Step S18: Quantize the floating-point activation gradient value e31 for network layer 23 to obtain a fixed-point activation gradient value e32 and an activation value quantization hyperparameter e33. Adjust the initial model weight a3 of network layer 23 based on the fixed-point activation gradient value e32 to obtain an adjusted model weight. The adjusted model weight is then updated as the initial model weight of network layer 23, i.e., the adjusted model weight serves as the initial model weight a3.

[0092] Exemplarily, for the quantization process of the floating-point activation gradient value e31, the floating-point activation gradient value e31 includes a large number of floating-point values. The activation value quantization hyperparameter e33 is first determined. The activation value quantization hyperparameter e33 represents a scaling factor. For each floating-point value, the floating-point value can be converted into a fixed-point value within a specified interval based on the scaling factor, and the fixed-point values ​​corresponding to all floating-point values ​​constitute the fixed-point activation gradient value e32.

[0093] For example, after obtaining the fixed-point activation gradient value e32, the initial model weight a3 can be adjusted based on the fixed-point activation gradient value e32. For example, a fixed-point weight gradient value e34 can be determined based on the fixed-point activation gradient value e32, a floating-point weight gradient value e35 can be determined based on the fixed-point weight gradient value e34 and the activation value quantization hyperparameter e33, and the initial model weight a3 of the network layer 23 can be adjusted based on the floating-point weight gradient value e35 to obtain the adjusted model weight. For example, based on the fixed-point activation gradient value e32, an algorithm such as a gradient descent method can be used to determine the fixed-point weight gradient value e34. For example, the fixed-point weight gradient value e34 can be determined based on information such as the fixed-point activation gradient value e32 and fixed-point input data d31. This determination process is not limited. Since the calculation is based on fixed-point data (such as fixed-point activation gradient value e32 and fixed-point input data d31) rather than floating-point data, the processing process of network layer 23 can be accelerated to achieve calculation acceleration. After obtaining the fixed-point weight gradient value e34, the fixed-point weight gradient value e34 can be converted to a floating-point weight gradient value e35, that is, the fixed-point weight gradient value e34 is dequantized to obtain a floating-point weight gradient value e35. For example, the fixed-point weight gradient value e34 is dequantized based on the activation value quantization hyperparameter e33 to obtain a floating-point weight gradient value e35. After obtaining the floating-point weight gradient value e35, the initial model weight a3 can be adjusted based on the floating-point weight gradient value e35.

[0094] In one possible implementation, during the quantization process of a floating-point activation gradient value e31, which includes a large number of floating-point values, an activation value quantization hyperparameter e33 can be first determined. The activation value quantization hyperparameter e33 represents a scaling factor. For each floating-point value, the floating-point value is scaled based on the scaling factor. This scaling process is not restricted, resulting in a scaled intermediate quantization value. In other words, a large number of floating-point values ​​correspond to a large number of intermediate quantization values. Each intermediate quantization value can be converted into a fixed-point value, thereby obtaining a fixed-point value corresponding to the floating-point value. The fixed-point values ​​corresponding to all floating-point values ​​constitute the fixed-point activation gradient value e32.

[0095] For example, for a large number of quantized intermediate values, these quantized intermediate values ​​may have a bimodal distribution (or multimodal distribution) characteristic, for example, see Figure 4AThe figure shows a bimodal distribution of quantized intermediate values. Multiple quantized intermediate values ​​are distributed around 0, and multiple quantized intermediate values ​​are distributed around -0.0040. When converting these quantized intermediate values ​​to fixed-point values, if the large values ​​(quantized intermediate values ​​distributed around 0) are retained, the small values ​​(quantized intermediate values ​​distributed around -0.0040) will all be converted to 0. Therefore, the converted fixed-point values ​​are all distributed around 0. Figure 4B However, large and small values ​​in a bimodal distribution are equally important. Converting all small values ​​to 0 will result in poor quantization, which will reduce the training performance of the initial network model, that is, the model performance will be greatly lost.

[0096] In response to the above findings, in this embodiment, when quantizing the floating-point activation gradient value e31, when converting these quantized intermediate values ​​into fixed-point values, if the quantized intermediate value is not greater than a preset threshold, the fixed-point value is determined based on the probability vector; if the quantized intermediate value is greater than the preset threshold, the quantized intermediate value is rounded to obtain a fixed-point value. In this way, randomness is introduced into the quantization process of the floating-point activation gradient value e31, and a quantization strategy that is adaptive to large and small values ​​is introduced. When quantizing small values, probabilistic quantization is used for quantization, thereby improving quantization accuracy and improving the training performance of the network model.

[0097] For example, multiple quantized intermediate values ​​are distributed around 0, and multiple quantized intermediate values ​​are distributed around -0.0040. When converting these quantized intermediate values ​​into fixed-point values, while retaining the large values ​​(the quantized intermediate values ​​distributed around 0), the small values ​​(the quantized intermediate values ​​distributed around -0.0040) will not be converted to 0. Instead, some small values ​​will be converted to 0 based on probability, and some small values ​​will be converted to other values. The converted fixed-point values ​​are not evenly distributed around 0. See Figure 4C Obviously, the above method takes into account the large and small values ​​in the bimodal distribution, which can improve the quantization effect and thus improve the model performance.

[0098] In the embodiment of the present application, considering that the gradient of a single small value is not important, while the gradient of a large number of small values ​​plays a key role in the training direction, probability is introduced into the quantization process. In addition, large values ​​and small values ​​can be adaptively distinguished, and different quantization schemes can be used for large values ​​and small values. For example, for large values, it is more accurate to use a deterministic quantization method. Therefore, the quantized intermediate value can be converted into a fixed-point value based on a rounding operation. For small values, it is more accurate to use an uncertain quantization method. Therefore, the quantized intermediate value can be converted into a fixed-point value based on a probabilistic quantization method.

[0099] In one possible implementation, the floating-point activation gradient value e31 may include multiple floating-point values, and the quantization intermediate value corresponding to each floating-point value may be determined based on the activation value quantization hyperparameter e33, without limiting the determination process. Then, for each quantization intermediate value, if the quantization intermediate value is not greater than a preset threshold, the probability vector corresponding to the quantization intermediate value is determined, and the fixed-point value corresponding to the quantization intermediate value is determined based on the probability vector. If the quantization intermediate value is greater than the preset threshold, the quantization intermediate value may be rounded to obtain the fixed-point value corresponding to the quantization intermediate value. On this basis, a fixed-point activation gradient value e32 may be generated based on the fixed-point values ​​corresponding to all quantization intermediate values.

[0100] For example, the preset threshold can be configured based on experience, and there is no restriction on the preset threshold. For example, the preset threshold can be determined based on the maximum absolute value of all quantized intermediate values. Alternatively, the preset threshold value may be 0.75*max(|x|), where max(|x|) represents the maximum absolute value among all quantized intermediate values. Of course, the above is only an example, and the following For example.

[0101] For example, for larger intermediate quantization values, deterministic quantization can be used to ensure accuracy, and for smaller intermediate quantization values, probabilistic quantization can be used for quantization, as expressed by the following formula:

[0102]

[0103] In the above formula, Indicates that the absolute value of the quantized intermediate value is greater than the preset threshold, that is, the quantized intermediate value is a large value. quant(x) indicates that the quantized intermediate value is rounded to the nearest integer, and f(c) indicates the fixed-point value corresponding to the quantized intermediate value. else indicates that the absolute value of the quantized intermediate value is not greater than the preset threshold, that is, the quantized intermediate value is a small value. statistice_quant(x) indicates the use of probabilistic quantization, that is, determining the probability vector corresponding to the quantized intermediate value and determining the fixed-point value corresponding to the quantized intermediate value based on the probability vector.

[0104] In one possible implementation, the probability vector corresponding to the quantized intermediate value may include a first probability and a second probability, wherein the first probability is the probability that the fixed-point value corresponding to the quantized intermediate value is the rounded-up value of the quantized intermediate value, and the second probability is the probability that the fixed-point value corresponding to the quantized intermediate value is the rounded-down value of the quantized intermediate value. On this basis, the first probability and the second probability corresponding to the quantized intermediate value may be determined based on the rounded-up value of the quantized intermediate value and / or the rounded-down value of the quantized intermediate value. For example, the probability vector corresponding to the quantized intermediate value may be determined as follows:

[0105] Method 1: Determine the first probability based on the floor value of the quantized intermediate value and the quantized intermediate value, and determine the second probability based on the ceiling value of the quantized intermediate value and the quantized intermediate value; for example, the first probability and the second probability corresponding to the quantized intermediate value can be determined by the following formula.

[0106]

[0107] In the above formula, ceil(x) represents the ceiling value of the quantized intermediate value, floor(x) represents the floor value of the quantized intermediate value, p1 represents the probability of the ceiling value of the quantized intermediate value, i.e., the first probability, and p2 represents the probability of the floor value of the quantized intermediate value, i.e., the second probability. As can be seen from the above, the first probability p1 can be determined based on the quantized intermediate value x and the floor value floor(x) of the quantized intermediate value, and the second probability p2 can be determined based on the quantized intermediate value x and the ceiling value ceil(x) of the quantized intermediate value.

[0108] From the above formula, we can see that the probability that the fixed-point value statistic_quant(x) corresponding to the quantized intermediate value x is ceil(x) is p1, and the probability that the fixed-point value statistic_quant(x) corresponding to the quantized intermediate value x is floor(x) is p2. Based on p1 and p2, the fixed-point value statistic_quant(x) can be determined.

[0109] For example, assuming the quantized intermediate value x is 0.2, then floor(x) is 0, ceil(x) is 1, p1 is 0.2, and p2 is 0.8. Therefore, the probability that the fixed-point value statistic_quant(x) is 1 is 0.2, and the probability that the fixed-point value statistic_quant(x) is 0 is 0.8. Based on these probabilities, we can obtain the fixed-point value corresponding to the quantized intermediate value x. This quantized intermediate value x may be 0 or 1. For example, we can set 100 numbers, 80 of which are 0 and 20 are 1. The order of these 100 numbers can be set arbitrarily. Then, generate a random number between 1 and 100. Suppose the random number is 88, then find the 88th number from the 100 numbers. If the 88th number is 0, the fixed-point value corresponding to the quantized intermediate value x is determined to be 0. If the 88th number is 1, the fixed-point value corresponding to the quantized intermediate value x is determined to be 1.

[0110] Method 2: Determine the first probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value, and determine the second probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value; for example, the first probability and the second probability corresponding to the quantized intermediate value can be determined by the following formula.

[0111]

[0112] In the above formula, ceil(x) represents the ceiling value of the quantized intermediate value, floor(x) represents the floor value of the quantized intermediate value, p1 represents the first probability, and p2 represents the second probability. As can be seen from the above, the first probability p1 can be determined based on the quantized intermediate value x and the floor value floor(x), and the second probability p2 can be determined based on the quantized intermediate value x and the floor value floor(x). Then, based on the first probability p1 and the second probability p2, the fixed-point value statistic_quant(x) corresponding to the quantized intermediate value x can be determined.

[0113] Method 3: Determine a first probability based on the floor value of the quantized intermediate value and the quantized intermediate value, and determine a second probability based on the first probability, where the sum of the first probability and the second probability is a fixed value, such as 1; for example, the first probability and the second probability corresponding to the quantized intermediate value can be determined by the following formula.

[0114]

[0115] In the above formula, ceil(x) represents the ceiling value of the quantized intermediate value, floor(x) represents the floor value of the quantized intermediate value, p1 represents the first probability, and p2 represents the second probability. As can be seen from the above, the first probability p1 can be determined based on the quantized intermediate value x and the floor value floor(x), and the second probability p2 can be determined based on the first probability p1. This then leads to the determination of the fixed-point value statistic_quant(x) corresponding to the quantized intermediate value x.

[0116] Method 4: Determine the first probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value, and determine the second probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value; for example, the first probability and the second probability corresponding to the quantized intermediate value can be determined by the following formula.

[0117]

[0118] In the above formula, ceil(x) represents the ceiling value of the quantized intermediate value, floor(x) represents the floor value of the quantized intermediate value, p1 represents the first probability, and p2 represents the second probability. As can be seen from the above, the first probability p1 can be determined based on the quantized intermediate value x and the ceiling value ceil(x), and the second probability p2 can be determined based on the quantized intermediate value x and the ceiling value ceil(x). Then, based on the first probability p1 and the second probability p2, the fixed-point value statistic_quant(x) corresponding to the quantized intermediate value x can be determined.

[0119] Method 5: Determine the second probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value, and determine the first probability based on the second probability, where the sum of the first probability and the second probability is a fixed value, such as 1; for example, the first probability and the second probability corresponding to the quantized intermediate value can be determined by the following formula.

[0120]

[0121] In the above formula, ceil(x) represents the ceiling value of the quantized intermediate value, floor(x) represents the floor value of the quantized intermediate value, p1 represents the first probability, and p2 represents the second probability. As can be seen from the above, the second probability p2 can be determined based on the quantized intermediate value x and the ceiling value ceil(x), and the first probability p1 can be determined based on the second probability p2. This then leads to the determination of the fixed-point value statistic_quant(x) corresponding to the quantized intermediate value x.

[0122] Of course, the above are just a few examples of determining the first probability and the second probability, and there is no limitation thereto.

[0123] Step S19: Since network layer 23 is not the first network layer, the initial model weight a3 of network layer 23 is quantized for network layer 23 to obtain the reverse fixed-point weight e36 and the reverse weight value quantization hyperparameter e37; based on the fixed-point activation gradient value e32, the reverse fixed-point weight e36, the activation value quantization hyperparameter e33 and the reverse weight value quantization hyperparameter e37, the floating-point activation gradient value e21 corresponding to the previous network layer 22 of network layer 23 is determined, and the floating-point activation gradient value e21 corresponding to network layer 22 is output to network layer 22.

[0124] Exemplarily, for the quantization process of the initial model weight a3, the initial model weight a3 includes a large number of floating-point values. The reverse weight value quantization hyperparameter e37 corresponding to the initial model weight a3 is first determined. The reverse weight value quantization hyperparameter e37 represents the scaling factor. Each floating-point value can be converted into a fixed-point value within a specified interval based on the scaling factor, and the fixed-point values ​​corresponding to all floating-point values ​​constitute the reverse fixed-point weight e36.

[0125] Among them, the number of input channels of the network layer 23 is cin3, the number of output channels of the network layer 23 is cout3, the initial model weight a3 of the network layer 23 can be a four-dimensional tensor, and the four dimensions of the initial model weight a3 can be cin3, cout3, m3, and n3 respectively, where m3 and n3 are the sizes of the initial model weight a3.

[0126] On this basis, the initial model weight a3 of the network layer 23 is quantized to obtain the inverse fixed-point weight e36 and the inverse weight value quantization hyperparameter e37, which may include but is not limited to: dividing the initial model weight a3 into cin3 sub-model weights, each sub-model weight is expressed as cout3*m3*n3, m3*n3 is the size of the sub-model weight, and cout3 is the number of channels of the sub-model weight. Then, each sub-model weight in the cin3 sub-model weights is quantized to obtain the inverse fixed-point weight e36 and the inverse weight value quantization hyperparameter e37 corresponding to the sub-model weight. In summary, cin3 sub-model weights, cin3 inverse fixed-point weights e36 and cin3 inverse weight value quantization hyperparameter e37 can be obtained. For each sub-model weight, the sub-model weight corresponds to a inverse fixed-point weight e36 and a inverse weight value quantization hyperparameter e37.

[0127] In summary, it can be seen that for the reverse direction of the model training process, the initial model weight a3 is quantized according to the number of input channels cin3, and each input channel corresponds to a reverse weight value quantization hyperparameter.

[0128] Exemplarily, the process of "determining the floating-point activation gradient value e21 corresponding to the network layer 22 based on the fixed-point activation gradient value e32, the inverse fixed-point weight e36, the activation value quantization hyperparameter e33, and the inverse weight value quantization hyperparameter e37" may include: determining the fixed-point activation gradient value e22 corresponding to the network layer 22 based on the fixed-point activation gradient value e32 and the inverse fixed-point weight e36; determining the floating-point activation gradient value e21 based on the fixed-point activation gradient value e22, the activation value quantization hyperparameter e33, and the inverse weight value quantization hyperparameter e37.

[0129] The processing of the fixed-point activation gradient value e32 and the inverse fixed-point weight e36 may be related to the function of the network layer 23. For example, if the network layer 23 is used to implement a convolution function, a convolution operation may be performed based on the fixed-point activation gradient value e32 and the inverse fixed-point weight e36 to obtain the fixed-point activation gradient value e22 corresponding to the network layer 22, and so on. Obviously, since the operation is performed based on fixed-point data (such as the fixed-point activation gradient value e32 and the inverse fixed-point weight e36) rather than floating-point data, the processing of the network layer 23 can be accelerated, thereby achieving an operation acceleration function.

[0130] After obtaining the fixed-point activation gradient value e22, the fixed-point activation gradient value e22 can be converted to a floating-point activation gradient value e21. That is, the fixed-point activation gradient value e22 is dequantized to obtain a floating-point activation gradient value e21. For example, the fixed-point activation gradient value e22 is dequantized based on the activation value quantization hyperparameter e33 and the inverse weight value quantization hyperparameter e37 to obtain a floating-point activation gradient value e21.

[0131] In one possible implementation, the number of input channels of the network layer 23 is cin3, the number of output channels of the network layer 23 is cout3, and the floating-point activation gradient value e31 is output by the network layer 23, that is, the number of channels of the floating-point activation gradient value e31 is cout3, the four dimensions of the initial model weight a3 are cin3, cout3, m3, and n3, and the three dimensions of the floating-point activation gradient value e31 are cout3, h, and w. The initial model weight a3 corresponds to cin3 sub-model weights, and each sub-model weight is expressed as cout3*m3*n3. Based on this:

[0132] For each sub-model weight, the floating-point activation gradient value e21 can be determined based on the inverse fixed-point weight e36 and the fixed-point activation gradient value e32 corresponding to the sub-model weight. Since the dimension of the inverse fixed-point weight e36 is cout3*m3*n3, the dimension of the fixed-point activation gradient value e32 (the same as the dimension of the floating-point activation gradient value e31) is cout3*h*w, that is, the number of input channels of the two is the same, both cout3. Taking the convolution operation as an example, after the convolution operation is performed on the two, the dimension of the floating-point activation gradient value e21 can be h*w, that is, each sub-model weight corresponds to a floating-point activation gradient value e21 with an h*w dimension.

[0133] Since the initial model weight a3 corresponds to cin3 sub-model weights, the cin3 sub-model weights correspond to cin3 h*w-dimensional floating-point activation gradient values ​​e21, that is, the dimension of the floating-point activation gradient value e21 is cin3*h*w, where cin3 represents the number of channels of the floating-point activation gradient value e21 and h*w represents the size.

[0134] Step S20: For network layer 22, the floating-point activation gradient value e21 output by the network layer 23 following network layer 22 is determined as the floating-point activation gradient value e21 corresponding to network layer 22. The floating-point activation gradient value e21 is then quantized to obtain a fixed-point activation gradient value e22 and an activation value quantization hyperparameter e23. The initial model weight a2 of network layer 22 is adjusted based on the fixed-point activation gradient value e22 to obtain an adjusted model weight, which is then updated as the initial model weight a2 of network layer 22.

[0135] For example, a fixed-point weight gradient value e24 can be determined based on the fixed-point activation gradient value e22, and a floating-point weight gradient value e25 can be determined based on the fixed-point weight gradient value e24 and the activation value quantization hyperparameter e23. The initial model weight a2 can be adjusted based on the floating-point weight gradient value e25 to obtain the adjusted model weight. For example, based on the fixed-point activation gradient value e22, an algorithm such as a gradient descent method can be used to determine the fixed-point weight gradient value e24. For example, the fixed-point weight gradient value e24 can be determined based on information such as the fixed-point activation gradient value e22 and fixed-point input data d21. After obtaining the fixed-point weight gradient value e24, the fixed-point weight gradient value e24 can be converted to a floating-point weight gradient value e25, i.e., the fixed-point weight gradient value e24 can be dequantized to obtain the floating-point weight gradient value e25. After obtaining the floating-point weight gradient value e25, the initial model weight a2 can be adjusted based on the floating-point weight gradient value e25.

[0136] The quantization process of the floating-point activation gradient value e21 is described in step S18 and will not be repeated here.

[0137] Step S21: Since network layer 22 is not the first network layer, the initial model weight a2 of network layer 22 is quantized for network layer 22 to obtain the reverse fixed-point weight e26 and the reverse weight value quantization hyperparameter e27; based on the fixed-point activation gradient value e22, the reverse fixed-point weight e26, the activation value quantization hyperparameter e23 and the reverse weight value quantization hyperparameter e27, the floating-point activation gradient value e11 corresponding to the previous network layer 21 of network layer 22 is determined, and the floating-point activation gradient value e11 corresponding to network layer 21 is output to network layer 21.

[0138] Among them, the number of input channels of network layer 22 is cin2, the number of output channels is cout2, the initial model weight a2 of network layer 22 can be a four-dimensional tensor, and the four dimensions of the initial model weight a2 can be cin2, cout2, m2, and n2 respectively. On this basis, the initial model weight a2 is quantized to obtain the inverse fixed-point weight e26 and the inverse weight value quantization hyperparameter e27, which may include but is not limited to: dividing the initial model weight a2 into cin2 sub-model weights, each sub-model weight is expressed as cout2*m2*n2, and quantizing each sub-model weight in the cin2 sub-model weights to obtain the inverse fixed-point weight e26 and the inverse weight value quantization hyperparameter e27 corresponding to the sub-model weight. In summary, cin2 sub-model weights, cin2 inverse fixed-point weights e26, and cin2 inverse weight value quantization hyperparameter e27 can be obtained.

[0139] The number of channels of the floating-point activation gradient value e21 is equal to the number of input channels, cin3, of network layer 23. The number of input channels, cin3, of network layer 23 is the same as the number of output channels, cout2, of network layer 22. The number of channels of the floating-point activation gradient value e21 is denoted as cout2. In summary, the four dimensions of the initial model weight a2 are cin2, cout2, m2, and n2, and the three dimensions of the floating-point activation gradient value e21 are cout2, h, and w. The initial model weight a2 corresponds to cin2 sub-model weights, each of which is expressed as cout2*m2*n2.

[0140] For each sub-model weight, a floating-point activation gradient value e11 is determined based on the corresponding inverse fixed-point weight e26 and fixed-point activation gradient value e22. Since the inverse fixed-point weight e26 has dimensions of cout2*m2*n2, and the fixed-point activation gradient value e22 has dimensions of cout2*h*w (the same as the floating-point activation gradient value e21), i.e., the number of input channels is the same (cout2), the floating-point activation gradient value e11 has dimensions of h*w. In other words, each sub-model weight corresponds to a floating-point activation gradient value e11 of h*w dimensions. Since the initial model weight a2 corresponds to cin2 sub-model weights, these cin2 sub-model weights correspond to cin2 floating-point activation gradient values ​​e11 of h*w dimensions. That is, the dimensions of the floating-point activation gradient value e11 are cin2*h*w, where cin2 represents the number of channels of the floating-point activation gradient value e11.

[0141] Step S22: For network layer 21, the floating-point activation gradient value e11 output by the network layer 22 following network layer 21 is determined as the floating-point activation gradient value e11 corresponding to network layer 21. The floating-point activation gradient value e11 is then quantized to obtain a fixed-point activation gradient value e12 and an activation value quantization hyperparameter e13. The initial model weight a1 of network layer 21 is adjusted based on the fixed-point activation gradient value e12 to obtain an adjusted model weight, which is then updated as the initial model weight a1 of network layer 21.

[0142] Exemplarily, the fixed-point weight gradient value e14 can be determined based on the fixed-point activation gradient value e12, the floating-point weight gradient value e15 can be determined based on the fixed-point weight gradient value e14 and the activation value quantization hyperparameter e13, and the initial model weight a1 can be adjusted based on the floating-point weight gradient value e15 to obtain the adjusted model weight.

[0143] The quantization process of the floating-point activation gradient value e11 is described in step S18 and will not be repeated here.

[0144] Exemplarily, since network layer 21 is already the first network layer, the floating-point activation gradient value corresponding to the previous network layer of network layer 21 is no longer determined, and the transmission of the floating-point activation gradient value is terminated.

[0145] At this point, the initial model weights of all network layers have been adjusted to obtain adjusted model weights. Then, based on the adjusted model weights, the process returns to step S11 and repeats the above steps.

[0146] In one possible implementation, it is possible to determine whether the initial network model has converged. If it has converged, the converged network model is used as the target network model to complete the network model training process. If it has not converged, the above steps are repeated to continue adjusting the initial model weights in each network layer.

[0147] In one possible implementation, after obtaining the target network model, the target network model can also be deployed on the terminal device to process the application data (such as image data, etc.) through the target network model, that is, the application data is processed by the target network model using artificial intelligence. For example, for the application data to be processed, the application data can be input into the target network model, and the application data is processed by the target network model to obtain the processing result (such as the artificial intelligence processing result). For example, assuming that the target network model is used to implement target detection (such as face detection, human body detection, vehicle detection, etc.), after the application data is provided to the target network model, the target network model determines the target detection result.

[0148] It can be seen from the above technical solutions that in the embodiment of the present application, the floating-point activation gradient value can be quantized to obtain the fixed-point activation gradient value, the initial model weight can be quantized to obtain the reverse fixed-point weight, and the network model training is realized based on the fixed-point activation gradient value and the reverse fixed-point weight. Since the network model training is realized based on fixed-point data, and the computational complexity of fixed-point data is low, the computational complexity can be reduced, the amount of computation is small, and the resource consumption is small. The training process of the network model can be accelerated, the training time of the network model can be reduced, and the training speed of the network model can be increased. Lightweight training can be realized, and forward acceleration and reverse acceleration can be realized at the same time. At the same time, the performance of the network model is relatively large, and it can be widely used in various business scenarios of artificial intelligence, such as image classification, target detection, segmentation, posture estimation and other business scenarios. By converting high-bit floating-point data into low-bit fixed-point data for storage and calculation, the resource consumption in the network model training process can be greatly reduced and the training speed can be accelerated. The above method is a lightweight training algorithm that can quantize parameters such as model weights, model activation values, weight gradients, and activation gradients. In the forward and reverse processes, the weights are quantized according to two different dimensions, cout and cin, so that forward and reverse acceleration can be achieved simultaneously, greatly improving efficiency.

[0149] When converting floating-point activation gradient values ​​into fixed-point activation gradient values, if the quantized intermediate value is not greater than the preset threshold, the fixed-point value is determined based on the probability vector, thereby introducing randomness into the gradient quantization process. At the same time, a quantization strategy that is adaptive to large and small values ​​is introduced. When performing gradient quantization on small values, probabilistic quantization is used to improve quantization accuracy and enhance the training performance of the network model.

[0150] Based on the same application concept as the above method, a network model training device is proposed in the embodiment of the present application. The initial network model includes multiple network layers, each network layer includes an initial model weight, see Figure 5 FIG. 1 is a schematic diagram of the structure of the apparatus. For each network layer, the apparatus may include:

[0151] An acquisition module 51 is used to obtain a floating-point activation gradient value corresponding to the network layer;

[0152] A processing module 52 is configured to quantize the floating-point activation gradient value to obtain a fixed-point activation gradient value and an activation value quantization hyperparameter; wherein the floating-point activation gradient value includes a plurality of floating-point values, and determine a quantization intermediate value corresponding to each floating-point value based on the activation value quantization hyperparameter; for each quantization intermediate value, if the quantization intermediate value is not greater than a preset threshold, determine a probability vector corresponding to the quantization intermediate value, and determine a fixed-point value corresponding to the quantization intermediate value based on the probability vector; and generate the fixed-point activation gradient value based on the fixed-point values ​​corresponding to all the quantization intermediate values;

[0153] The training module 53 is used to adjust the initial model weights of the network layer based on the fixed-point activation gradient value, and update the adjusted model weights to the initial model weights of the network layer.

[0154] Exemplarily, the probability vector includes a first probability and a second probability, the first probability being the probability that the fixed-point value corresponding to the quantized intermediate value is the rounded-up value of the quantized intermediate value, and the second probability being the probability that the fixed-point value corresponding to the quantized intermediate value is the rounded-down value of the quantized intermediate value; when the processing module 52 determines the probability vector corresponding to the quantized intermediate value, it is specifically used to: determine the first probability and the second probability corresponding to the quantized intermediate value based on the rounded-up value of the quantized intermediate value and / or the rounded-up value of the quantized intermediate value.

[0155] Exemplarily, when the processing module 52 determines the first probability and the second probability corresponding to the quantized intermediate value based on the rounded-up value of the quantized intermediate value and / or the rounded-up value of the quantized intermediate value, it is specifically configured to: determine the first probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value, and determine the second probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value; or, determine the first probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value, and determine the second probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value; or, determine the first probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value, and determine the second probability based on the first probability, wherein the sum of the first probability and the second probability is a fixed value; or, determine the first probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value, and determine the second probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value; or, determine the second probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value, and determine the first probability based on the second probability, wherein the sum of the first probability and the second probability is a fixed value.

[0156] Exemplarily, the processing module 52 is further configured to round off the quantized intermediate value if the quantized intermediate value is greater than a preset threshold value, so as to obtain a fixed-point value corresponding to the quantized intermediate value.

[0157] Exemplarily, the processing module 52 is further configured to determine the preset threshold value. When determining the preset threshold value, the preset threshold value is determined based on the maximum absolute value of all quantized intermediate values.

[0158] Exemplarily, the processing module 52 is also used to quantize the initial model weights of the network layer when the network layer is not the first network layer, to obtain reverse fixed-point weights and reverse weight value quantization hyperparameters; based on the fixed-point activation gradient value, the reverse fixed-point weight, the activation value quantization hyperparameter and the reverse weight value quantization hyperparameter, determine the floating-point activation gradient value corresponding to the previous network layer of the network layer, and send the floating-point activation gradient value corresponding to the previous network layer to the previous network layer.

[0159] Exemplarily, when the acquisition module 51 acquires the floating-point activation gradient value corresponding to the network layer, it is specifically used to: if the network layer of the initial network model is the last network layer, then obtain the floating-point activation value corresponding to the network layer, determine the target loss value based on the floating-point activation value and the calibrated floating-point true value, and determine the floating-point activation gradient value corresponding to the network layer based on the target loss value; if the network layer is not the last network layer of the initial network model, then determine the floating-point activation gradient value output by the next network layer of the network layer as the floating-point activation gradient value corresponding to the network layer.

[0160] Based on the same application concept as the above method, the present application embodiment proposes a network model training device, see Figure 6 As shown, the network model training device includes a processor 61 and a machine-readable storage medium 62, and the machine-readable storage medium 62 stores machine-executable instructions that can be executed by the processor 61; the processor 61 is used to execute the machine-executable instructions to implement the network model training method disclosed in the above example.

[0161] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the training method of the network model disclosed in the above example of the present application can be implemented.

[0162] The machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0163] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0164] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0165] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] Furthermore, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0168] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A training method for a network model, characterized in that: The initial network model includes multiple network layers, each network layer includes initial model weights, the input data of the initial network model is a floating-point sample image, and after the floating-point sample image is input to the initial network model, the floating-point sample image is processed by the initial network model; wherein, for each network layer, the method includes: Obtaining a floating-point activation gradient value of the floating-point sample image corresponding to the network layer; The floating-point activation gradient value of the floating-point sample image is quantized to obtain the fixed-point activation gradient value of the floating-point sample image and the activation value quantization hyperparameter of the floating-point sample image; wherein the floating-point activation gradient value of the floating-point sample image includes a plurality of floating-point values, and each floating-point value is scaled based on the activation value quantization hyperparameter of the floating-point sample image to obtain a quantization intermediate value corresponding to each floating-point value; for each quantization intermediate value, if the quantization intermediate value is not greater than a preset threshold, a probability vector corresponding to the quantization intermediate value is determined, and a fixed-point value corresponding to the quantization intermediate value is determined based on the probability vector. ; generating a fixed-point activation gradient value of the floating-point sample image based on the fixed-point numerical values ​​corresponding to all quantized intermediate values; wherein the probability vector includes a first probability and a second probability, the first probability being a probability that the fixed-point numerical value corresponding to the quantized intermediate value is a rounded-up value of the quantized intermediate value, and the second probability being a probability that the fixed-point numerical value corresponding to the quantized intermediate value is a rounded-down value of the quantized intermediate value; determining the probability vector corresponding to the quantized intermediate value includes: determining the first probability and the second probability corresponding to the quantized intermediate value based on the rounded-up value of the quantized intermediate value and / or the rounded-down value of the quantized intermediate value; Adjusting the initial model weights of the network layer based on the fixed-point activation gradient values ​​of the floating-point sample image, and updating the adjusted model weights as the initial model weights of the network layer; Among them, after adjusting the initial model weights of all network layers to obtain adjusted model weights, if the initial network model has converged, the converged network model is used as the target network model, and the target network model is deployed on the terminal device to process the image data through the target network model to obtain a processing result for the image data; wherein, if the target network model is used to realize face detection, the image including the face is input into the target network model, and the image is processed by artificial intelligence through the target network model to obtain a face detection result; if the target network model is used to realize vehicle detection, the image including the vehicle is input into the target network model, and the image is processed by artificial intelligence through the target network model to obtain a vehicle detection result.

2. The method according to claim 1, characterized in that The determining, based on the rounded-up value of the quantized intermediate value and / or the rounded-down value of the quantized intermediate value, the first probability and the second probability corresponding to the quantized intermediate value includes: determining a first probability based on a floor value of the quantized intermediate value and the quantized intermediate value, and determining a second probability based on a ceiling value of the quantized intermediate value and the quantized intermediate value; or, determining a first probability based on a floor-rounded value of the quantized intermediate value and the quantized intermediate value, and determining a second probability based on the floor-rounded value of the quantized intermediate value and the quantized intermediate value; or, determining a first probability based on a floor-rounded value of the quantized intermediate value and the quantized intermediate value, and determining a second probability based on the first probability, wherein the sum of the first probability and the second probability is a fixed value; or determining a first probability based on a rounded-up value of the quantized intermediate value and the quantized intermediate value, and determining a second probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value; or, A second probability is determined based on a rounded-up value of the quantized intermediate value and the quantized intermediate value, and a first probability is determined based on the second probability, wherein a sum of the first probability and the second probability is a fixed value.

3. The method according to claim 1, characterized in that The method further comprises: For each quantized intermediate value, if the quantized intermediate value is greater than a preset threshold, a rounding operation is performed on the quantized intermediate value to obtain a fixed-point value corresponding to the quantized intermediate value.

4. The method according to any one of claims 1 to 3, characterized in that: The method for determining the preset threshold includes: determining the preset threshold based on the maximum absolute value of all quantized intermediate values.

5. The method according to claim 1, wherein After quantizing the floating-point activation gradient value of the floating-point sample image to obtain the fixed-point activation gradient value of the floating-point sample image and the activation value quantization hyperparameter of the floating-point sample image, the method further includes: If the network layer is not the first network layer, the initial model weights of the network layer are quantized to obtain inverse fixed-point weights and inverse weight value quantization hyperparameters; based on the fixed-point activation gradient value, the inverse fixed-point weight, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter, the floating-point activation gradient value corresponding to the previous network layer of the network layer is determined, and the floating-point activation gradient value corresponding to the previous network layer is sent to the previous network layer.

6. The method according to claim 1 or 5, characterized in that The obtaining of the floating-point activation gradient value of the floating-point sample image corresponding to the network layer includes: If the network layer is the last network layer, obtaining a floating-point activation value corresponding to the network layer, determining a target loss value based on the floating-point activation value and a calibrated floating-point true value, and determining a floating-point activation gradient value corresponding to the network layer based on the target loss value; If the network layer is not the last network layer, the floating-point activation gradient value output by the next network layer after the network layer is determined as the floating-point activation gradient value corresponding to the network layer.

7. A network model training device, characterized in that: The initial network model includes multiple network layers, each network layer includes initial model weights, the input data of the initial network model is a floating-point sample image, and after the floating-point sample image is input to the initial network model, the floating-point sample image is processed by the initial network model; wherein, for each network layer, the device includes: An acquisition module, configured to acquire a floating-point activation gradient value of the floating-point sample image corresponding to the network layer; A processing module, configured to quantize the floating-point activation gradient value of the floating-point sample image to obtain the fixed-point activation gradient value of the floating-point sample image and the activation value quantization hyperparameter of the floating-point sample image; wherein the floating-point activation gradient value of the floating-point sample image includes a plurality of floating-point values, and each floating-point value is scaled based on the activation value quantization hyperparameter of the floating-point sample image to obtain a quantization intermediate value corresponding to each floating-point value; for each quantization intermediate value, if the quantization intermediate value is not greater than a preset threshold, a probability vector corresponding to the quantization intermediate value is determined, and a fixed-point value corresponding to the quantization intermediate value is determined based on the probability vector; based on The fixed-point values ​​corresponding to all quantized intermediate values ​​generate the fixed-point activation gradient value of the floating-point sample image; wherein the probability vector includes a first probability and a second probability, the first probability is the probability that the fixed-point value corresponding to the quantized intermediate value is the rounded-up value of the quantized intermediate value, and the second probability is the probability that the fixed-point value corresponding to the quantized intermediate value is the rounded-down value of the quantized intermediate value; wherein, when the processing module determines the probability vector corresponding to the quantized intermediate value, it is specifically used to: determine the first probability and the second probability corresponding to the quantized intermediate value based on the rounded-up value of the quantized intermediate value and / or the rounded-up value of the quantized intermediate value; A training module, configured to adjust the initial model weights of the network layer based on the fixed-point activation gradient values ​​of the floating-point sample images, and update the adjusted model weights as the initial model weights of the network layer; Among them, after adjusting the initial model weights of all network layers to obtain adjusted model weights, if the initial network model has converged, the converged network model is used as the target network model, and the target network model is deployed on the terminal device to process the image data through the target network model to obtain a processing result for the image data; wherein, if the target network model is used to realize face detection, the image including the face is input into the target network model, and the image is processed by artificial intelligence through the target network model to obtain a face detection result; if the target network model is used to realize vehicle detection, the image including the vehicle is input into the target network model, and the image is processed by artificial intelligence through the target network model to obtain a vehicle detection result.

8. The device according to claim 7, It is characterized in that in, The processing module is configured to determine, based on the rounded-up value of the quantized intermediate value and / or the rounded-up value of the quantized intermediate value, the first probability and the second probability corresponding to the quantized intermediate value: determine the first probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value, and determine the second probability based on the rounded-up value of the quantized intermediate value and the quantized intermediate value; or determine the first probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value, and determine the second probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value; or determine the first probability based on the rounded-down value of the quantized intermediate value and the quantized intermediate value, and determine the second probability based on the first probability, wherein the sum of the first probability and the second probability is a fixed value; Alternatively, a first probability is determined based on a rounded-up value of the quantized intermediate value and the quantized intermediate value, and a second probability is determined based on the rounded-up value of the quantized intermediate value and the quantized intermediate value; or a second probability is determined based on the rounded-up value of the quantized intermediate value and the quantized intermediate value, and the first probability is determined based on the second probability, where the sum of the first probability and the second probability is a fixed value; The processing module is further configured to round off the quantized intermediate value to obtain a fixed-point value corresponding to the quantized intermediate value if the quantized intermediate value is greater than a preset threshold value; The processing module is further configured to determine the preset threshold value, and when determining the preset threshold value, the preset threshold value is determined based on the maximum absolute value of all quantized intermediate values; The processing module is further configured to, when the network layer is not the first network layer, quantize the initial model weights of the network layer to obtain inverse fixed-point weights and inverse weight value quantization hyperparameters; determine the floating-point activation gradient value corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value, the inverse fixed-point weight, the activation value quantization hyperparameter, and the inverse weight value quantization hyperparameter, and send the floating-point activation gradient value corresponding to the previous network layer to the previous network layer; Among them, when the acquisition module obtains the floating-point activation gradient value of the floating-point sample image corresponding to the network layer, it is specifically used to: if the network layer of the initial network model is the last network layer, then obtain the floating-point activation value corresponding to the network layer, determine the target loss value based on the floating-point activation value and the calibrated floating-point true value, and determine the floating-point activation gradient value corresponding to the network layer based on the target loss value; if the network layer is not the last network layer of the initial network model, then determine the floating-point activation gradient value output by the next network layer of the network layer as the floating-point activation gradient value corresponding to the network layer.

9. A network model training device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method steps described in any one of claims 1-6.

Images