A network model training method, device and equipment

Abstract

The present application provides a training method, device and equipment for a network model, which includes: obtaining a floating-point activation gradient value corresponding to a network layer, quantizing the floating-point activation gradient value, and obtaining a fixed-point activation gradient value and an activation value quantization hyperparameter; adjusting the initial model weight of the network layer based on the fixed-point activation gradient value, and updating the adjusted model weight to the initial model weight of the network layer; if the network layer is not the first network layer, quantizing the initial model weight of the network layer to obtain a reverse fixed-point weight and a reverse weight value quantization hyperparameter; based on the fixed-point activation gradient value, the reverse fixed-point weight, the activation value quantization hyperparameter and the reverse weight value quantization hyperparameter, determining the floating-point activation gradient value corresponding to the previous network layer of the network layer. Through the technical solution of the present application, the computational complexity can be reduced, the amount of computation is small, the resource consumption is small, and the training process can be accelerated.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method, apparatus, and device for training a network model. Background Art

[0002] Machine learning is a path to artificial intelligence and a multidisciplinary field, encompassing probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning focuses on algorithmic design, enabling computers to automatically learn patterns from data and use these patterns to make predictions about unknown data.

[0003] Machine learning has been widely used, such as deep learning, data mining, computer vision, natural language processing, biometric recognition, search engines, medical diagnosis, speech and handwriting recognition, etc.

[0004] In order to use machine learning to implement artificial intelligence processing, the server needs to obtain a large amount of sample data, train a machine learning model based on this sample data, and deploy the machine learning model to the terminal device (such as a camera, etc.) so that the terminal device can implement artificial intelligence processing based on the machine learning model.

[0005] However, since the training process of machine learning models involves a large number of complex calculations, the training time of machine learning models is long, the training speed is slow, the amount of calculation is large, and the resource consumption is high. Summary of the Invention

[0006] The present application provides a method for training a network model. The initial network model includes multiple network layers, each network layer includes an initial model weight, and for each network layer, the method includes:

[0007] Obtaining a floating-point activation gradient value corresponding to the network layer, and quantizing the floating-point activation gradient value to obtain a fixed-point activation gradient value and an activation value quantization hyperparameter;

[0008] Adjusting the initial model weights of the network layer based on the fixed-point activation gradient value, and updating the adjusted model weights as the initial model weights of the network layer;

[0009] If the network layer is not the first network layer, the initial model weights of the network layer are quantized to obtain inverse fixed-point weights and inverse weight value quantization hyperparameters; based on the fixed-point activation gradient value, the inverse fixed-point weight, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter, the floating-point activation gradient value corresponding to the previous network layer of the network layer is determined.

[0010] The present application provides a training device for a network model. The initial network model includes multiple network layers, each network layer includes an initial model weight, and for each network layer, the device includes:

[0011] An acquisition module is used to obtain a floating-point activation gradient value corresponding to the network layer, quantize the floating-point activation gradient value, and obtain a fixed-point activation gradient value and an activation value quantization hyperparameter;

[0012] A training module, configured to adjust the initial model weights of the network layer based on the fixed-point activation gradient value, and update the adjusted model weights to the initial model weights of the network layer;

[0013] A processing module is used to quantize the initial model weights of the network layer if the network layer is not the first network layer, to obtain inverse fixed-point weights and inverse weight value quantization hyperparameters; and determine the floating-point activation gradient value corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value, the inverse fixed-point weight, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter.

[0014] The present application provides a network model training device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned network model training method.

[0015] It can be seen from the above technical solutions that in the embodiment of the present application, the floating-point activation gradient value can be quantized to obtain the fixed-point activation gradient value, the initial model weight can be quantized to obtain the reverse fixed-point weight, and the network model training is realized based on the fixed-point activation gradient value and the reverse fixed-point weight. Since the network model training is realized based on fixed-point data, and the computational complexity of fixed-point data is low, the computational complexity can be reduced, the amount of computation is small, and the resource consumption is small. The training process of the network model can be accelerated, the training time of the network model can be reduced, the training speed of the network model can be increased, lightweight training can be realized, and forward acceleration and reverse acceleration can be realized at the same time. At the same time, the performance of the network model is relatively large and can be widely used in various business scenarios of artificial intelligence, such as image classification, target detection, segmentation, posture estimation and other business scenarios. By converting high-bit floating-point data into low-bit fixed-point data for storage and calculation, the resource consumption in the network model training process can be greatly reduced and the training speed can be accelerated. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0017] Figure 1 It is a flowchart of a method for training a network model in one embodiment of the present application;

[0018] Figure 2 is a schematic structural diagram of an initial network model in one embodiment of the present application;

[0019] Figure 3 This is a schematic diagram of the training process of the network model in one embodiment of the present application;

[0020] Figure 4 It is a structural diagram of a training device for a network model in one embodiment of the present application;

[0021] Figure 5 This is a hardware structure diagram of a network model training device in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The terms used in the embodiments of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application and claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" used may also be interpreted as "at the time of" or "when" or "in response to determining".

[0024] In an embodiment of the present application, a network model training method is proposed for training an initial network model to obtain a trained target network model. The initial network model may include multiple network layers, each of which may include initial model weights (i.e., model weights that need to be adjusted and optimized).

[0025] See also Figure 1 FIG. 5 is a flow chart of the method, which may include:

[0026] Step 101: For each network layer, obtain the floating-point activation gradient value corresponding to the network layer, quantize the floating-point activation gradient value, and obtain the fixed-point activation gradient value and activation value quantization hyperparameter.

[0027] Step 102: Adjust the initial model weights of the network layer based on the fixed-point activation gradient value, and update the adjusted model weights to the initial model weights of the network layer.

[0028] Exemplarily, adjusting the initial model weights of the network layer based on the fixed-point activation gradient value may include but is not limited to: determining the fixed-point weight gradient value based on the fixed-point activation gradient value, and determining the floating-point weight gradient value based on the fixed-point weight gradient value and the activation value quantization hyperparameter; and adjusting the initial model weights of the network layer based on the floating-point weight gradient value to obtain the adjusted model weights.

[0029] Step 103: If the network layer is not the first network layer, the initial model weights of the network layer can be quantized to obtain inverse fixed-point weights and inverse weight value quantization hyperparameters; based on the fixed-point activation gradient value, the inverse fixed-point weight, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter, the floating-point activation gradient value corresponding to the previous network layer of the network layer is determined.

[0030] Exemplarily, the initial model weights of the network layer are quantized to obtain inverse fixed-point weights and inverse weight value quantization hyperparameters, which may include but are not limited to: if the number of input channels of the network layer is cin and the number of output channels is cout, then the initial model weights of the network layer are divided into cin sub-model weights; wherein each sub-model weight is expressed as cout*m*n, m*n is the size of the sub-model weight, and cout is the number of channels of the sub-model weight; then, each sub-model weight in the cin sub-model weights is quantized to obtain the inverse fixed-point weights and inverse weight value quantization hyperparameters corresponding to the sub-model weight.

[0031] Exemplarily, based on the fixed-point activation gradient value, the inverse fixed-point weight, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter, determining the floating-point activation gradient value corresponding to the previous network layer of the network layer may include but is not limited to: determining the fixed-point activation gradient value corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value and the inverse fixed-point weight corresponding to the network layer; then, based on the fixed-point activation gradient value corresponding to the previous network layer, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter, determining the floating-point activation gradient value corresponding to the previous network layer.

[0032] Exemplarily, obtaining the floating-point activation gradient value corresponding to the network layer may include: if the network layer is the last network layer, obtaining the floating-point activation value corresponding to the network layer, determining a target loss value based on the floating-point activation value and a calibrated floating-point true value, and determining the floating-point activation gradient value corresponding to the network layer based on the target loss value. If the network layer is not the last network layer, determining the floating-point activation gradient value output by the network layer following the network layer as the floating-point activation gradient value corresponding to the network layer.

[0033] In one possible implementation, before obtaining the floating-point activation gradient value corresponding to the network layer, floating-point input data corresponding to the network layer may also be obtained, the floating-point input data may be quantized to obtain fixed-point input data and input data quantization hyperparameters; the initial model weights of the network layer may be quantized to obtain forward fixed-point weights and forward weight value quantization hyperparameters. The fixed-point input data may then be processed based on the forward fixed-point weights to obtain fixed-point activation values, and the floating-point activation value corresponding to the network layer may be determined based on the fixed-point activation values, the input data quantization hyperparameters, and the forward weight value quantization hyperparameters.

[0034] Exemplarily, if the network layer is not the last network layer, the floating-point activation value corresponding to the network layer is used as the floating-point input data corresponding to the next network layer after the network layer.

[0035] Exemplarily, the initial model weights of the network layer are quantized to obtain forward fixed-point weights and forward weight value quantization hyperparameters, which may include but is not limited to: if the number of input channels of the network layer is cin and the number of output channels is cout, then the initial model weights of the network layer are divided into cout sub-model weights; wherein each sub-model weight is expressed as cin*m*n, where m*n is the size of the sub-model weight and cin is the number of channels of the sub-model weight. Then, each of the cout sub-model weights is quantized to obtain the forward fixed-point weights and forward weight value quantization hyperparameters corresponding to the sub-model weight.

[0036] It can be seen from the above technical solutions that in the embodiment of the present application, the floating-point activation gradient value can be quantized to obtain the fixed-point activation gradient value, the initial model weight can be quantized to obtain the reverse fixed-point weight, and the network model training is realized based on the fixed-point activation gradient value and the reverse fixed-point weight. Since the network model training is realized based on fixed-point data, and the computational complexity of fixed-point data is low, the computational complexity can be reduced, the amount of computation is small, and the resource consumption is small. The training process of the network model can be accelerated, the training time of the network model can be reduced, the training speed of the network model can be increased, lightweight training can be realized, and forward acceleration and reverse acceleration can be realized at the same time. At the same time, the performance of the network model is relatively large and can be widely used in various business scenarios of artificial intelligence, such as image classification, target detection, segmentation, posture estimation and other business scenarios. By converting high-bit floating-point data into low-bit fixed-point data for storage and calculation, the resource consumption in the network model training process can be greatly reduced and the training speed can be accelerated.

[0037] The following describes the technical solutions of the embodiments of the present application in conjunction with specific application scenarios.

[0038] Before introducing the technical solution of the present application, concepts related to the embodiments of the present application are first introduced.

[0039] Machine Learning: Machine learning is a method for achieving artificial intelligence. It studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Deep learning and neural networks are subcategories of machine learning. Deep learning is the process of using mathematical models to model specific real-world problems in order to solve similar problems within the same field. Neural networks are algorithmic mathematical models that mimic the behavioral characteristics of animal neural networks and perform distributed parallel information processing. These neural networks rely on the complexity of the system to adjust the interconnections between the large number of internal nodes to achieve the purpose of information processing. For ease of description, the structure and function of neural networks are used as an example. Other subcategories of machine learning have similar structures and functions.

[0040] Neural Network: Neural networks may include but are not limited to convolutional neural networks (CNN), recurrent neural networks (RNN), fully connected networks, etc. The structural units of neural networks may include but are not limited to convolutional layers (Conv), pooling layers (Pool), excitation layers, fully connected layers (FC), etc.

[0041] In practical applications, one or more convolutional layers, one or more pooling layers, one or more excitation layers, and one or more fully connected layers can be combined to construct a neural network according to different needs.

[0042] In the convolution layer, the input data features are enhanced by using the convolution kernel to perform convolution operations on the input data features. The convolution kernel can be a matrix of size m*n. The input data features of the convolution layer are convolved with the convolution kernel to obtain the output data features of the convolution layer. The convolution operation is actually a filtering process.

[0043] In the pooling layer, by taking the maximum value, minimum value, average value and other operations on the input data features (such as the output of the convolutional layer), the principle of local correlation is used to sub-sample the input data features, reduce the processing amount, and maintain feature invariance. The pooling layer operation is actually a downsampling process.

[0044] In the excitation layer, activation functions (such as nonlinear functions) can be used to map the input data features, thereby introducing nonlinear factors and enabling the neural network to enhance its expressive power through nonlinear combinations.

[0045] The activation function may include but is not limited to a ReLU (Rectified Linear Units) function, which is used to set features less than 0 to 0, while features greater than 0 remain unchanged.

[0046] In the fully connected layer, all data features input to this fully connected layer are fully connected to obtain a feature vector, which can include multiple data features.

[0047] Network model: You can pre-build a network model, called the initial network model. The initial network model is a network model that has not yet been trained. The initial network model can be a machine learning model, such as a deep learning-based machine learning model or a neural network-based machine learning model. There are no restrictions on this.

[0048] For the training process of the initial network model, the sample data can be used to train the initial network model, that is, the process of adjusting and optimizing the model weights (i.e., network parameters) in the initial network model. For example, the initial network model may include multiple network layers, each of which includes initial model weights. The initial model weights are the model weights that need to be adjusted and optimized. For example, for the initial network model based on a neural network, the initial model weights may be convolution layer parameters (such as convolution kernel parameters), pooling layer parameters, excitation layer parameters, fully connected layer parameters, etc. For the training process of the initial network model, the sample data can be used to adjust and optimize the initial model weights in multiple network layers of the initial network model.

[0049] After the initial network model training is complete, the trained network model is called the target network model. The model weights in the multiple network layers of the target network model are called target model weights. That is, the target model weights are the model weights that have been adjusted and optimized. After obtaining the target network model, it can be deployed to the terminal device, allowing the terminal device to implement artificial intelligence processing based on the target network model.

[0050] For example, the target network model can be widely applied to various AI business scenarios, such as image classification, target detection, segmentation, and pose estimation. Taking target detection as an example, functions such as face detection, human body detection, and vehicle detection can be implemented based on the target network model. For example, for face detection, an image containing a face can be input into the target network model, and the target network model performs AI processing on the image. The result of the AI ​​processing is the face detection result. For vehicle detection, an image containing a vehicle can be input into the target network model, and the target network model performs AI processing on the image. The result of the AI ​​processing is the vehicle detection result.

[0051] Quantization of the initial network model: The initial model weights (i.e., the initial model weights of the network layer, also called model parameters or network parameters), input features, and output features represented by floating-point numbers are approximated using fixed-point values ​​to improve the computing speed of the initial network model and compress the initial network model.

[0052] For example, the initial network model may include a large number of initial model weights (such as millions). These initial model weights are floating-point types. The initial model weights of floating-point types will take up a lot of storage space, and the calculation of the initial model weights of floating-point types will consume a lot of computing resources. If the initial model weights of fixed-point types can be used for calculation without affecting the accuracy of the initial network model, the calculation speed can be improved, computing resources can be saved, and storage space can be saved. Therefore, quantization technology is introduced, that is, the initial network model is compressed by reducing the number of bits required to represent each initial model weight. Based on quantization technology, the initial model weights of floating-point types can be converted into initial model weights of fixed-point types. For example, for the same network layer of the initial network model (such as convolutional layer, pooling layer, excitation layer, etc.), the initial model weights of the network layer are distributed in a small interval, such as the interval [-10, 30], where -10 represents the minimum value of the initial model weight of the network layer and 30 represents the maximum value of the initial model weight of the network layer. Based on this, when using 8-bit quantization (other options are possible), all the initial model weights of the network layer can be linearly mapped (nonlinear mapping can also be used to compress the space) to the interval [-10, 30].

[0053] The embodiment of the present application proposes a training method for a network model. By converting high-bit floating-point data (such as 32-bit floating-point data) into low-bit fixed-point data (such as 8-bit fixed-point data) for storage and calculation, it is possible to reduce resource consumption during network model training and speed up training. This embodiment proposes a weight quantization algorithm in lightweight training, which can be widely applied to various artificial intelligence business scenarios, such as training acceleration for tasks such as image classification, target detection, segmentation, and posture estimation. It can simultaneously achieve forward acceleration and reverse acceleration, and at the same time, the performance is basically unchanged compared to floating-point training.

[0054] In this embodiment, a network model can be pre-built, and this network model is called an initial network model. The initial network model can include multiple network layers, and each network layer can include an initial model weight. Figure 2 As shown, it is a structural diagram of the initial network model. The initial network model may include network layer 21, network layer 22 and network layer 23. Of course, here we only take three network layers as an example. The number of network layers in the initial network model will be much greater than three. Network layer 21 is the first network layer of the initial network model, network layer 23 is the last network layer of the initial network model, and network layer 22 is the intermediate network layer of the initial network model. The number of intermediate network layers will be much greater than one. The processing process of each intermediate network layer is the same. In this embodiment, one intermediate network layer is used as an example for explanation. In addition, the initial model weight of network layer 21 can be recorded as initial model weight a1, the initial model weight of network layer 22 can be recorded as initial model weight a2, and the initial model weight of network layer 23 can be recorded as initial model weight a3.

[0055] Exemplarily, the initial model weight a1, the initial model weight a2, and the initial model weight a3 may be high-precision weights, which may be randomly generated or imported from a pre-trained model, and there is no restriction on the source.

[0056] In this embodiment, a training set can be pre-built, which can include a large amount of sample data and calibration data corresponding to each sample data, and the initial network model is trained based on the training set. For the sake of convenience, the following description will take the processing process of a sample data as an example.

[0057] Among them, the sample data can be sample data of floating-point type, therefore, the sample data is called floating-point sample data, and the calibration data corresponding to the sample data is calibration data of floating-point type, therefore, the calibration data is called floating-point calibration data, that is, the floating-point true value of the sample data.

[0058] In the above application scenario, see Figure 3 As shown, the training method of the network model may include:

[0059] Step S11: Obtain floating-point input data corresponding to network layer 21 and quantize the floating-point input data to obtain fixed-point input data d11 and input data quantization hyperparameter d12. Quantize the initial model weight a1 of network layer 21 to obtain forward fixed-point weight d13 and forward weight value quantization hyperparameter d14.

[0060] For example, since network layer 21 is the first network layer, floating-point sample data can be used as floating-point input data corresponding to network layer 21. By quantizing the floating-point input data, fixed-point input data d11 and input data quantization hyperparameter d12 can be obtained. In the quantization process of floating-point input data, the floating-point input data includes a large number of floating-point values. The input data quantization hyperparameter d12 is first determined. This input data quantization hyperparameter d12 represents a scaling factor. For each floating-point value, the floating-point value can be converted into a fixed-point value within a specified interval based on the scaling factor, and the fixed-point values ​​corresponding to all floating-point values ​​constitute the fixed-point input data d11. Of course, the above is only an example and does not limit this quantization process.

[0061] Exemplarily, for the quantization process of the initial model weight a1, the initial model weight a1 includes a large number of floating-point values. The forward weight value quantization hyperparameter d14 corresponding to the initial model weight a1 is first determined. This forward weight value quantization hyperparameter d14 represents the scaling factor. For each floating-point value, the floating-point value can be converted into a fixed-point value within a specified interval based on the scaling factor, and the fixed-point values ​​corresponding to all floating-point values ​​constitute the forward fixed-point weight d13. Of course, the above is just an example and there is no limitation on this quantization process.

[0062] In one possible implementation, the number of input channels of the network layer 21 can be recorded as cin1, and the number of output channels of the network layer 21 can be recorded as cout1. Obviously, the number of channels of the floating-point input data corresponding to the network layer 21 is cin1, that is, the number of channels of the floating-point input data is the same as the number of input channels of the network layer 21. In addition, the initial model weight a1 of the network layer 21 can be a four-dimensional tensor, and the four dimensions of the initial model weight a1 can be cin1, cout1, m1, and n1, respectively, where m1 and n1 are the sizes of the initial model weight a1.

[0063] On this basis, the initial model weight a1 of the network layer 21 is quantized to obtain the forward fixed-point weight d13 and the forward weight value quantization hyperparameter d14, which may include but is not limited to: dividing the initial model weight a1 into cout1 sub-model weights, each sub-model weight is expressed as cin1*m1*n1, m1*n1 is the size of the sub-model weight, and cin1 is the number of channels of the sub-model weight. Then, each sub-model weight in the cout1 sub-model weights is quantized to obtain the forward fixed-point weight d13 and the forward weight value quantization hyperparameter d14 corresponding to the sub-model weight. In summary, cout1 sub-model weights, cout1 forward fixed-point weight d13 and cout1 forward weight value quantization hyperparameter d14 can be obtained. For each sub-model weight, the sub-model weight corresponds to a forward fixed-point weight d13 and a forward weight value quantization hyperparameter d14.

[0064] In summary, it can be seen that for the forward direction of the model training process, it is quantized according to the number of output channels cout1 of the initial model weight a1, and each output channel corresponds to a forward weight value quantization hyperparameter.

[0065] Step S12: For the network layer 21, the fixed-point input data d11 is processed based on the forward fixed-point weight d13 to obtain the fixed-point activation value d15, and the floating-point activation value d16 corresponding to the network layer 21 is determined based on the fixed-point activation value d15, the input data quantization hyperparameter d12 and the forward weight value quantization hyperparameter d14.

[0066] Exemplarily, the processing process for the fixed-point input data d11 may be related to the function of the network layer 21. For example, if the network layer 21 is used to implement the convolution function, a convolution operation may be performed based on the fixed-point input data d11 and the forward fixed-point weight d13 to obtain a fixed-point activation value d15. If the network layer 21 is used to implement the pooling function, a pooling operation may be performed based on the fixed-point input data d11 and the forward fixed-point weight d13 to obtain a fixed-point activation value d15, and so on. Obviously, since the operation is based on fixed-point data (such as fixed-point input data d11 and forward fixed-point weight d13) rather than floating-point data, the processing process of the network layer 21 can be accelerated to achieve the operation acceleration function.

[0067] For example, after obtaining the fixed-point activation value d15, the fixed-point activation value d15 can be converted into a floating-point activation value d16, that is, the fixed-point activation value d15 is dequantized to obtain the floating-point activation value d16. For example, the fixed-point activation value d15 is dequantized based on the input data quantization hyperparameter d12 and the forward weight value quantization hyperparameter d14 to obtain the floating-point activation value d16. The implementation method of dequantization is the reverse operation of the above-mentioned quantization method, and this embodiment does not limit the implementation method of this dequantization.

[0068] In a possible implementation, the above process may be expressed using the following formula.

[0069]

[0070] In the above formula, W represents the initial model weight a1, X represents floating-point input data, y represents the floating-point activation value d16, and ∑WX represents processing based on the initial model weight a1 and the floating-point input data, which is related to the function of the network layer 21. For example, when the network layer 21 is used to implement the convolution function, ∑WX represents a convolution operation based on the initial model weight a1 and the floating-point input data to obtain the floating-point activation value d16.

[0071] Referring to the above embodiment, the initial model weight a1 can be quantized to obtain the forward fixed-point weight d13 and the forward weight value quantization hyperparameter d14. Therefore, the floating-point W can be converted into alpha and W. q , alpha represents the positive weight value quantization hyperparameter d14, W q Represents the forward fixed-point weight d13.

[0072] Referring to the above embodiment, the floating-point input data can be quantized to obtain fixed-point input data d11 and input data quantization hyperparameter d12. Therefore, the floating-point input data X can be converted to and X q , Indicates the input data quantization hyperparameter d12, X q Indicates fixed-point input data d11.

[0073] Obviously, based on the above conversion relationship, the calculation process of the initial model weight a1 and floating-point input data can be converted into the calculation process of fixed-point input data d11 and forward fixed-point weight d13, that is, the calculation process of floating-point data is converted into the calculation process of fixed-point data, thereby simplifying the calculation complexity.

[0074] For example, in In the q X qIt represents the operation process of fixed-point input data d11 and forward fixed-point weight d13. This operation process is the operation process of fixed-point data, and the operation result is the fixed-point activation value d15. Then, based on the input data quantization hyperparameter d12 and the forward weight value quantization hyperparameter d14, the fixed-point activation value d15 is dequantized to obtain the floating-point activation value d16.

[0075] In one possible implementation, the number of input channels of network layer 21 is cin1, the number of output channels of network layer 21 is cout1, the number of channels of floating-point input data is cin1, the four dimensions of initial model weight a1 are cin1, cout1, m1, and n1, and the three dimensions of floating-point input data are cin1, h, and w, where h and w represent the size of the floating-point input data. Furthermore, the initial model weight a1 corresponds to cout1 sub-model weights, each of which is expressed as cin1*m1*n1. Based on this:

[0076] For each sub-model weight, the fixed-point input data d11 can be processed based on the forward fixed-point weight d13 corresponding to the sub-model weight to obtain a fixed-point activation value d15. Since the dimension of the forward fixed-point weight d13 is cin1*m1*n1, the dimension of the fixed-point input data d11 (the same as the dimension of the floating-point input data) is cin1*h*w, that is, the number of input channels of the two is the same, both cin1. Taking the convolution operation as an example, after the convolution operation is performed on the two, the dimension of the fixed-point activation value d15 can be h*w, that is, each sub-model weight corresponds to a fixed-point activation value d15 of h*w dimension. Further, after the fixed-point activation value d15 is converted to a floating-point activation value d16, the floating-point activation value d16 is also h*w dimension, that is, each sub-model weight corresponds to a floating-point activation value d16 of h*w dimension.

[0077] Since the initial model weight a1 corresponds to cout1 sub-model weights, cout1 sub-model weights correspond to cout1 floating-point activation values ​​d16 of h*w dimensions. That is, the dimension of the floating-point activation value d16 is cout1*h*w, where cout1 represents the number of channels of the floating-point activation value d16 and h*w represents the size.

[0078] Step S13: Obtain floating-point input data corresponding to network layer 22 and quantize the floating-point input data to obtain fixed-point input data d21 and input data quantization hyperparameter d22. Quantize the initial model weight a2 of network layer 22 to obtain forward fixed-point weight d23 and forward weight value quantization hyperparameter d24.

[0079] Since network layer 22 is the second network layer, the floating-point activation value output by the first network layer (i.e., the floating-point activation value d16) can be used as the floating-point input data corresponding to network layer 22. By quantizing the floating-point input data, fixed-point input data d21 and input data quantization hyperparameter d22 can be obtained.

[0080] In one possible implementation, the number of input channels of network layer 22 can be denoted as cin2, and the number of output channels of network layer 22 can be denoted as cout2. The number of input channels cin2 of network layer 22 is the same as the number of output channels cout1 of network layer 21, that is, the number of channels cout1 of floating-point activation value d16 (floating-point input data) is the same as the number of input channels cin21 of network layer 22. The initial model weight a2 can be a four-dimensional tensor, and the four dimensions can be cin2, cout2, m2, and n2, respectively, where m2 and n2 are the sizes of the initial model weight a2.

[0081] On this basis, the initial model weight a2 of the network layer 22 is quantized to obtain the forward fixed-point weight d23 and the forward weight value quantization hyperparameter d24, which may include but is not limited to: dividing the initial model weight a2 into cout2 sub-model weights, each sub-model weight is expressed as cin2*m2*n2, m2*n2 is the size of the sub-model weight, and cin2 is the number of channels of the sub-model weight. Then, each sub-model weight in the cout2 sub-model weights is quantized to obtain the forward fixed-point weight d23 and the forward weight value quantization hyperparameter d24 corresponding to the sub-model weight. In summary, cout2 sub-model weights, cout2 forward fixed-point weights d23 and cout2 forward weight value quantization hyperparameter d24 can be obtained.

[0082] Step S14: For the network layer 22, the fixed-point input data d21 is processed based on the forward fixed-point weight d23 to obtain the fixed-point activation value d25, and the floating-point activation value d26 corresponding to the network layer 22 is determined based on the fixed-point activation value d25, the input data quantization hyperparameter d22 and the forward weight value quantization hyperparameter d24.

[0083] For example, the processing process of step S14 can refer to step S12, and will not be repeated here.

[0084] In one possible implementation, the number of input channels of the network layer 22 is cin2, the number of output channels of the network layer 22 is cout2, the number of channels of the floating-point input data (i.e., the floating-point activation value d16) is cout1, and cout1 is the same as cin2. The number of channels of the floating-point input data is subsequently recorded as cin2. Therefore, the four dimensions of the initial model weight a2 are cin2, cout2, m2, and n2, and the three dimensions of the floating-point input data are cin2, h, and w, where h and w represent the size of the floating-point input data. The initial model weight a2 corresponds to cout2 sub-model weights, and each sub-model weight is expressed as cin2*m2*n2. On this basis:

[0085] For each sub-model weight, the fixed-point input data d21 can be processed based on the forward fixed-point weight d23 corresponding to the sub-model weight to obtain the fixed-point activation value d25. Since the dimension of the forward fixed-point weight d23 is cin2*m2*n2, and the dimension of the fixed-point input data d21 is cin2*h*w, that is, the number of input channels of both is cin2, the dimension of the fixed-point activation value d25 is h*w, that is, each sub-model weight corresponds to a fixed-point activation value d25 of h*w dimension, and the floating-point activation value d26 is also of h*w dimension, that is, each sub-model weight corresponds to a floating-point activation value d26 of h*w dimension.

[0086] Since the initial model weight a2 corresponds to cout2 sub-model weights, the cout2 sub-model weights correspond to cout2 floating-point activation values ​​d26 of h*w dimensions. That is, the dimension of the floating-point activation value d26 is cout2*h*w, where cout2 represents the number of channels of the floating-point activation value d26 and h*w represents the size.

[0087] Step S15: Obtain the floating-point input data corresponding to network layer 23 (using the floating-point activation value d26 output by the second network layer as the floating-point input data), quantize the floating-point input data to obtain fixed-point input data d31 and input data quantization hyperparameter d32. Quantize the initial model weight a3 of network layer 23 to obtain the forward fixed-point weight d33 and the forward weight value quantization hyperparameter d34.

[0088] For example, the number of input channels of network layer 23 can be recorded as cin3, and the number of output channels of network layer 23 can be recorded as cout3. The number of input channels of network layer 23 cin3 is the same as the number of output channels of network layer 22 cout2. The initial model weight a3 can be a four-dimensional tensor, and the four dimensions are cin3, cout3, m3, and n3, respectively. m3 and n3 are the sizes of the initial model weight a2. On this basis, the initial model weight a3 is divided into cout3 sub-model weights, and each sub-model weight is expressed as cin3*m3*n3, where cin3 is the number of channels of the sub-model weight. Then, each sub-model weight in the cout3 sub-model weights is quantized to obtain the forward fixed-point weight d33 and the forward weight value quantization hyperparameter d34 corresponding to the sub-model weight.

[0089] Step S16: For the network layer 23, the fixed-point input data d31 is processed based on the forward fixed-point weight d33 to obtain the fixed-point activation value d35, and the floating-point activation value d36 corresponding to the network layer 23 is determined based on the fixed-point activation value d35, the input data quantization hyperparameter d32 and the forward weight value quantization hyperparameter d34.

[0090] For example, the processing process of step S16 can refer to step S12, and will not be repeated here.

[0091] In one possible implementation, the number of input channels of the network layer 23 is cin3, the number of output channels of the network layer 23 is cout3, the number of channels of the floating-point input data (i.e., the floating-point activation value d26) is cout2, and cout2 is the same as cin3. Therefore, the four dimensions of the initial model weight a3 are cin3, cout3, m3, and n3, and the three dimensions of the floating-point input data are cin3, h, and w. The initial model weight a3 corresponds to cout3 sub-model weights, and each sub-model weight is expressed as cin3*m3*n3. On this basis:

[0092] For each sub-model weight, the fixed-point input data d31 is processed based on the forward fixed-point weight d33 corresponding to the sub-model weight to obtain a fixed-point activation value d35. Since the number of input channels of the forward fixed-point weight d33 and the number of input channels of the fixed-point input data d31 are both cin3, the fixed-point activation value d35 and the floating-point activation value d36 are both h*w dimensional. That is, each sub-model weight corresponds to a floating-point activation value d36 of h*w dimension. Since the initial model weight a3 corresponds to cout3 sub-model weights, the cout3 sub-model weights correspond to cout3 floating-point activation values ​​d36 of h*w dimension. That is, the dimension of the floating-point activation value d36 is cout3*h*w, where cout3 represents the number of channels of the floating-point activation value d36.

[0093] Step S17, network layer 23 is the last network layer. The target loss value is determined based on the floating-point activation value d36 corresponding to network layer 23 and the calibrated floating-point true value (the calibration data corresponding to the floating-point sample data in step S11). The floating-point activation gradient value e31 corresponding to network layer 23 is determined based on the target loss value.

[0094] Exemplarily, a loss function may be preconfigured, the input of which is a floating-point activation value and a floating-point true value, and the output of which is a target loss value. There are no restrictions on this loss function.

[0095] For example, since network layer 23 is the last network layer, after obtaining the floating-point activation value d36 corresponding to network layer 23, the floating-point activation value d36 and the floating-point true value corresponding to the floating-point activation value d36 can be substituted into the loss function to obtain the target loss value. There is no restriction on this process.

[0096] After obtaining the target loss value, the floating-point activation gradient value e31 corresponding to the network layer 23 is determined based on the target loss value using algorithms such as the gradient descent method. The floating-point activation gradient value e31 is a gradient (Gradient) used to adjust and optimize the initial model weight a3 of the network layer 23. There is no restriction on this process.

[0097] Step S18: Quantize the floating-point activation gradient value e31 for network layer 23 to obtain a fixed-point activation gradient value e32 and an activation value quantization hyperparameter e33. Adjust the initial model weight a3 of network layer 23 based on the fixed-point activation gradient value e32 to obtain an adjusted model weight. The adjusted model weight is then updated as the initial model weight of network layer 23, i.e., the adjusted model weight serves as the initial model weight a3.

[0098] Exemplarily, for the quantization process of the floating-point activation gradient value e31, the floating-point activation gradient value e31 includes a large number of floating-point values. The activation value quantization hyperparameter e33 is first determined. The activation value quantization hyperparameter e33 represents a scaling factor. For each floating-point value, the floating-point value can be converted into a fixed-point value within a specified interval based on the scaling factor, and the fixed-point values ​​corresponding to all floating-point values ​​constitute the fixed-point activation gradient value e32.

[0099] For example, after obtaining the fixed-point activation gradient value e32, the initial model weight a3 can be adjusted based on the fixed-point activation gradient value e32. For example, the fixed-point weight gradient value e34 can be determined based on the fixed-point activation gradient value e32, and the floating-point weight gradient value e35 can be determined based on the fixed-point weight gradient value e34 and the activation value quantization hyperparameter e33. Then, the initial model weight a3 of the network layer 23 can be adjusted based on the floating-point weight gradient value e35 to obtain the adjusted model weight.

[0100] For example, based on the fixed-point activation gradient value e32, an algorithm such as a gradient descent method can be used to determine the fixed-point weight gradient value e34. For example, the fixed-point weight gradient value e34 can be determined based on information such as the fixed-point activation gradient value e32 and the fixed-point input data d31. There are no restrictions on this determination process. Because the calculation is based on fixed-point data (such as the fixed-point activation gradient value e32 and the fixed-point input data d31) rather than floating-point data, the processing of network layer 23 can be accelerated, achieving computational acceleration.

[0101] For example, after obtaining the fixed-point weight gradient value e34, the fixed-point weight gradient value e34 can be converted to a floating-point weight gradient value e35, i.e., the fixed-point weight gradient value e34 can be dequantized to obtain the floating-point weight gradient value e35. For example, the fixed-point weight gradient value e34 can be dequantized based on the activation value quantization hyperparameter e33 to obtain the floating-point weight gradient value e35. The implementation method of dequantization is the reverse operation of the above-mentioned quantization method, and this embodiment does not limit the implementation method of this dequantization.

[0102] For example, after obtaining the floating-point weight gradient value e35, the initial model weight a3 can be adjusted based on the floating-point weight gradient value e35, and there is no restriction on this adjustment process.

[0103] In a possible implementation, the above process may be expressed using the following formula.

[0104]

[0105] In the above formula, W g Used to represent the floating-point weight gradient value e35, Y g is used to represent the floating-point activation gradient value e31, X is used to represent the floating-point input data corresponding to the network layer 23, ∑Y g X is used to indicate processing based on floating-point activation gradient values ​​e31 and floating-point input data, such as convolution operations.

[0106] In the above formula, X qUsed to represent the fixed-point input data d31, Y corresponding to the floating-point input data X gq Used to represent the fixed-point activation gradient value e32 corresponding to the floating-point activation gradient value e31, ∑X q Y gq It indicates that the fixed-point weight gradient value e34 is determined based on the fixed-point activation gradient value e32 and the fixed-point input data d31.

[0107] Indicates the data quantization hyperparameter d32 corresponding to the input fixed-point input data d31, Represents the activation value quantization hyperparameter e33 corresponding to the fixed-point activation gradient value e32. Based on the data quantization hyperparameter d32 and the activation value quantization hyperparameter e33, the fixed-point weight gradient value e34 can be converted to a floating-point weight gradient value e35.

[0108] Step S19: Since network layer 23 is not the first network layer, the initial model weight a3 of network layer 23 is quantized for network layer 23 to obtain the reverse fixed-point weight e36 and the reverse weight value quantization hyperparameter e37; based on the fixed-point activation gradient value e32, the reverse fixed-point weight e36, the activation value quantization hyperparameter e33 and the reverse weight value quantization hyperparameter e37, the floating-point activation gradient value e21 corresponding to the previous network layer 22 of network layer 23 is determined, and the floating-point activation gradient value e21 corresponding to network layer 22 is output to network layer 22.

[0109] Exemplarily, for the quantization process of the initial model weight a3, the initial model weight a3 includes a large number of floating-point values. The reverse weight value quantization hyperparameter e37 corresponding to the initial model weight a3 is first determined. The reverse weight value quantization hyperparameter e37 represents the scaling factor. Each floating-point value can be converted into a fixed-point value within a specified interval based on the scaling factor, and the fixed-point values ​​corresponding to all floating-point values ​​constitute the reverse fixed-point weight e36.

[0110] Among them, the number of input channels of the network layer 23 is cin3, the number of output channels of the network layer 23 is cout3, the initial model weight a3 of the network layer 23 can be a four-dimensional tensor, and the four dimensions of the initial model weight a3 can be cin3, cout3, m3, and n3 respectively, where m3 and n3 are the sizes of the initial model weight a3.

[0111] On this basis, the initial model weight a3 of the network layer 23 is quantized to obtain the inverse fixed-point weight e36 and the inverse weight value quantization hyperparameter e37, which may include but is not limited to: dividing the initial model weight a3 into cin3 sub-model weights, each sub-model weight is expressed as cout3*m3*n3, m3*n3 is the size of the sub-model weight, and cout3 is the number of channels of the sub-model weight. Then, each sub-model weight in the cin3 sub-model weights is quantized to obtain the inverse fixed-point weight e36 and the inverse weight value quantization hyperparameter e37 corresponding to the sub-model weight. In summary, cin3 sub-model weights, cin3 inverse fixed-point weights e36 and cin3 inverse weight value quantization hyperparameter e37 can be obtained. For each sub-model weight, the sub-model weight corresponds to a inverse fixed-point weight e36 and a inverse weight value quantization hyperparameter e37.

[0112] From the above, it can be seen that for the reverse direction of the model training process, the initial model weight a3 is quantized according to the number of input channels cin3, and each input channel corresponds to a reverse weight value quantization hyperparameter.

[0113] Exemplarily, the process of "determining the floating-point activation gradient value e21 corresponding to the network layer 22 based on the fixed-point activation gradient value e32, the inverse fixed-point weight e36, the activation value quantization hyperparameter e33, and the inverse weight value quantization hyperparameter e37" may include: determining the fixed-point activation gradient value e22 corresponding to the network layer 22 based on the fixed-point activation gradient value e32 and the inverse fixed-point weight e36; determining the floating-point activation gradient value e21 based on the fixed-point activation gradient value e22, the activation value quantization hyperparameter e33, and the inverse weight value quantization hyperparameter e37.

[0114] The processing of the fixed-point activation gradient value e32 and the inverse fixed-point weight e36 may be related to the function of the network layer 23. For example, if the network layer 23 is used to implement a convolution function, a convolution operation may be performed based on the fixed-point activation gradient value e32 and the inverse fixed-point weight e36 to obtain the fixed-point activation gradient value e22 corresponding to the network layer 22, and so on. Obviously, since the operation is performed based on fixed-point data (such as the fixed-point activation gradient value e32 and the inverse fixed-point weight e36) rather than floating-point data, the processing of the network layer 23 can be accelerated, thereby achieving an operation acceleration function.

[0115] After obtaining the fixed-point activation gradient value e22, the fixed-point activation gradient value e22 can be converted to a floating-point activation gradient value e21. That is, the fixed-point activation gradient value e22 is dequantized to obtain a floating-point activation gradient value e21. For example, the fixed-point activation gradient value e22 is dequantized based on the activation value quantization hyperparameter e33 and the inverse weight value quantization hyperparameter e37 to obtain a floating-point activation gradient value e21.

[0116] In a possible implementation, the above process may be expressed using the following formula.

[0117]

[0118] In the above formula, X g It is used to represent the floating-point activation gradient value e21 corresponding to the network layer 22, W is used to represent the initial model weight a3, Y g Represents the floating-point activation gradient value e31 corresponding to network layer 23. ∑Y g W represents processing based on the initial model weight a3 and the floating-point activation gradient value e31. This processing process is related to the function of the network layer 23. For example, when the network layer 23 is used to implement the convolution function, it means that a convolution operation is performed based on the initial model weight a3 and the floating-point activation gradient value e31 to obtain the floating-point activation gradient value e21.

[0119] Referring to the above embodiment, the initial model weight a3 can be quantized to obtain the reverse fixed-point weight e36 and the reverse weight value quantization hyperparameter e37. Therefore, the floating-point W can be converted to alpha in and W q , alpha in Represents the reverse weight value quantization hyperparameter e37, W q Represents the inverse fixed-point weight e36.

[0120] Referring to the above embodiment, the floating-point activation gradient value e31 can be quantized to obtain the fixed-point activation gradient value e32 and the activation value quantization hyperparameter e33. Therefore, the floating-point Y g Can be converted to and Y gq , Represents the activation value quantization hyperparameter e33, Y gq Represents the fixed-point activation gradient value e32.

[0121] Obviously, based on the above conversion relationship, the calculation process of the initial model weight a3 and the floating-point activation gradient value e31 can be converted into the calculation process of the reverse fixed-point weight e36 and the fixed-point activation gradient value e32, that is, the calculation process of floating-point data is converted into the calculation process of fixed-point data, thereby simplifying the calculation complexity.

[0122] For example, in In the q Y gq It represents the calculation process of the inverse fixed-point weight e36 and the fixed-point activation gradient value e32. Obviously, this calculation process is the calculation process of fixed-point data. The calculation result is the fixed-point activation gradient value e22 corresponding to the network layer 22. Then, based on the inverse weight value quantization hyperparameter e37 and the activation value quantization hyperparameter e33, the fixed-point activation gradient value e22 is inverse quantized to obtain the floating-point activation gradient value e21 corresponding to the network layer 22.

[0123] In one possible implementation, the number of input channels of the network layer 23 is cin3, the number of output channels of the network layer 23 is cout3, and the floating-point activation gradient value e31 is output by the network layer 23, that is, the number of channels of the floating-point activation gradient value e31 is cout3, the four dimensions of the initial model weight a3 are cin3, cout3, m3, and n3, and the three dimensions of the floating-point activation gradient value e31 are cout3, h, and w. The initial model weight a3 corresponds to cin3 sub-model weights, and each sub-model weight is expressed as cout3*m3*n3. Based on this:

[0124] For each sub-model weight, the floating-point activation gradient value e21 can be determined based on the inverse fixed-point weight e36 and the fixed-point activation gradient value e32 corresponding to the sub-model weight. Since the dimension of the inverse fixed-point weight e36 is cout3*m3*n3, the dimension of the fixed-point activation gradient value e32 (the same as the dimension of the floating-point activation gradient value e31) is cout3*h*w, that is, the number of input channels of the two is the same, both cout3. Taking the convolution operation as an example, after the convolution operation is performed on the two, the dimension of the floating-point activation gradient value e21 can be h*w, that is, each sub-model weight corresponds to a floating-point activation gradient value e21 with an h*w dimension.

[0125] Since the initial model weight a3 corresponds to cin3 sub-model weights, the cin3 sub-model weights correspond to cin3 h*w-dimensional floating-point activation gradient values ​​e21, that is, the dimension of the floating-point activation gradient value e21 is cin3*h*w, where cin3 represents the number of channels of the floating-point activation gradient value e21 and h*w represents the size.

[0126] Step S20: For network layer 22, the floating-point activation gradient value e21 output by the network layer 23 following network layer 22 is determined as the floating-point activation gradient value e21 corresponding to network layer 22. The floating-point activation gradient value e21 is then quantized to obtain a fixed-point activation gradient value e22 and an activation value quantization hyperparameter e23. The initial model weight a2 of network layer 22 is adjusted based on the fixed-point activation gradient value e22 to obtain an adjusted model weight, which is then updated as the initial model weight a2 of network layer 22.

[0127] For example, a fixed-point weight gradient value e24 can be determined based on the fixed-point activation gradient value e22, and a floating-point weight gradient value e25 can be determined based on the fixed-point weight gradient value e24 and the activation value quantization hyperparameter e23. The initial model weight a2 can be adjusted based on the floating-point weight gradient value e25 to obtain the adjusted model weight. For example, based on the fixed-point activation gradient value e22, an algorithm such as a gradient descent method can be used to determine the fixed-point weight gradient value e24. For example, the fixed-point weight gradient value e24 can be determined based on information such as the fixed-point activation gradient value e22 and fixed-point input data d21. After obtaining the fixed-point weight gradient value e24, the fixed-point weight gradient value e24 can be converted to a floating-point weight gradient value e25, i.e., the fixed-point weight gradient value e24 can be dequantized to obtain the floating-point weight gradient value e25. After obtaining the floating-point weight gradient value e25, the initial model weight a2 can be adjusted based on the floating-point weight gradient value e25.

[0128] Step S21: Since network layer 22 is not the first network layer, the initial model weight a2 of network layer 22 is quantized for network layer 22 to obtain the reverse fixed-point weight e26 and the reverse weight value quantization hyperparameter e27; based on the fixed-point activation gradient value e22, the reverse fixed-point weight e26, the activation value quantization hyperparameter e23 and the reverse weight value quantization hyperparameter e27, the floating-point activation gradient value e11 corresponding to the previous network layer 21 of network layer 22 is determined, and the floating-point activation gradient value e11 corresponding to network layer 21 is output to network layer 21.

[0129] Among them, the number of input channels of network layer 22 is cin2, the number of output channels is cout2, the initial model weight a2 of network layer 22 can be a four-dimensional tensor, and the four dimensions of the initial model weight a2 can be cin2, cout2, m2, and n2 respectively. On this basis, the initial model weight a2 is quantized to obtain the inverse fixed-point weight e26 and the inverse weight value quantization hyperparameter e27, which may include but is not limited to: dividing the initial model weight a2 into cin2 sub-model weights, each sub-model weight is expressed as cout2*m2*n2, and quantizing each sub-model weight in the cin2 sub-model weights to obtain the inverse fixed-point weight e26 and the inverse weight value quantization hyperparameter e27 corresponding to the sub-model weight. In summary, cin2 sub-model weights, cin2 inverse fixed-point weights e26, and cin2 inverse weight value quantization hyperparameter e27 can be obtained.

[0130] The number of channels of the floating-point activation gradient value e21 is equal to the number of input channels, cin3, of network layer 23. The number of input channels, cin3, of network layer 23 is the same as the number of output channels, cout2, of network layer 22. The number of channels of the floating-point activation gradient value e21 is denoted as cout2. In summary, the four dimensions of the initial model weight a2 are cin2, cout2, m2, and n2, and the three dimensions of the floating-point activation gradient value e21 are cout2, h, and w. The initial model weight a2 corresponds to cin2 sub-model weights, each of which is expressed as cout2*m2*n2.

[0131] For each sub-model weight, a floating-point activation gradient value e11 is determined based on the corresponding inverse fixed-point weight e26 and fixed-point activation gradient value e22. Since the inverse fixed-point weight e26 has dimensions of cout2*m2*n2, and the fixed-point activation gradient value e22 has dimensions of cout2*h*w (the same as the floating-point activation gradient value e21), i.e., the number of input channels is the same (cout2), the floating-point activation gradient value e11 has dimensions of h*w. In other words, each sub-model weight corresponds to a floating-point activation gradient value e11 of h*w dimensions. Since the initial model weight a2 corresponds to cin2 sub-model weights, these cin2 sub-model weights correspond to cin2 floating-point activation gradient values ​​e11 of h*w dimensions. That is, the dimensions of the floating-point activation gradient value e11 are cin2*h*w, where cin2 represents the number of channels of the floating-point activation gradient value e11.

[0132] Step S22: For network layer 21, the floating-point activation gradient value e11 output by the network layer 22 following network layer 21 is determined as the floating-point activation gradient value e11 corresponding to network layer 21. The floating-point activation gradient value e11 is then quantized to obtain a fixed-point activation gradient value e12 and an activation value quantization hyperparameter e13. The initial model weight a1 of network layer 21 is adjusted based on the fixed-point activation gradient value e12 to obtain an adjusted model weight, which is then updated as the initial model weight a1 of network layer 21.

[0133] Exemplarily, the fixed-point weight gradient value e14 can be determined based on the fixed-point activation gradient value e12, the floating-point weight gradient value e15 can be determined based on the fixed-point weight gradient value e14 and the activation value quantization hyperparameter e13, and the initial model weight a1 can be adjusted based on the floating-point weight gradient value e15 to obtain the adjusted model weight.

[0134] Exemplarily, since network layer 21 is already the first network layer, the floating-point activation gradient value corresponding to the previous network layer of network layer 21 is no longer determined, and the transmission of the floating-point activation gradient value is terminated.

[0135] At this point, the initial model weights of all network layers have been adjusted to obtain adjusted model weights. Then, based on the adjusted model weights, the process returns to step S11 and repeats the above steps.

[0136] In one possible implementation, it is possible to determine whether the initial network model has converged. If it has converged, the converged network model is used as the target network model to complete the network model training process. If it has not converged, the above steps are repeated to continue adjusting the initial model weights in each network layer.

[0137] In one possible implementation, after obtaining the target network model, the target network model can also be deployed on the terminal device to process the application data (such as image data, etc.) through the target network model, that is, the application data is processed by the target network model using artificial intelligence. For example, for the application data to be processed, the application data can be input into the target network model, and the application data is processed by the target network model to obtain the processing result (such as the artificial intelligence processing result). For example, assuming that the target network model is used to implement target detection (such as face detection, human body detection, vehicle detection, etc.), after the application data is provided to the target network model, the target network model determines the target detection result.

[0138] It can be seen from the above technical solutions that in the embodiment of the present application, the floating-point activation gradient value can be quantized to obtain the fixed-point activation gradient value, the initial model weight can be quantized to obtain the reverse fixed-point weight, and the network model training is realized based on the fixed-point activation gradient value and the reverse fixed-point weight. Since the network model training is realized based on fixed-point data, and the computational complexity of fixed-point data is low, the computational complexity can be reduced, the amount of computation is small, and the resource consumption is small. The training process of the network model can be accelerated, the training time of the network model can be reduced, the training speed of the network model can be increased, lightweight training can be realized, and forward acceleration and reverse acceleration can be realized at the same time. At the same time, the performance of the network model is relatively large and can be widely used in various business scenarios of artificial intelligence, such as image classification, target detection, segmentation, posture estimation and other business scenarios. By converting high-bit floating-point data into low-bit fixed-point data for storage and calculation, the resource consumption in the network model training process can be greatly reduced and the training speed can be accelerated. The above method is a lightweight training algorithm that can quantize parameters such as model weights, model activation values, weight gradients, and activation gradients. In the forward and reverse processes, the weights are quantized according to two different dimensions, cout and cin, so that forward and reverse acceleration can be achieved simultaneously, greatly improving efficiency.

[0139] Based on the same application concept as the above method, a network model training device is proposed in the embodiment of the present application. The initial network model includes multiple network layers, each network layer includes an initial model weight, see Figure 4 FIG. 1 is a schematic diagram of the structure of the apparatus. For each network layer, the apparatus may include:

[0140] An acquisition module 41 is configured to acquire a floating-point activation gradient value corresponding to the network layer, quantize the floating-point activation gradient value, and obtain a fixed-point activation gradient value and an activation value quantization hyperparameter;

[0141] A training module 42 is configured to adjust the initial model weights of the network layer based on the fixed-point activation gradient value, and update the adjusted model weights to the initial model weights of the network layer;

[0142] The processing module 43 is used to quantize the initial model weights of the network layer if the network layer is not the first network layer, to obtain inverse fixed-point weights and inverse weight value quantization hyperparameters; and determine the floating-point activation gradient value corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value, the inverse fixed-point weight, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter.

[0143] Exemplarily, the processing module 43 quantizes the initial model weights of the network layer to obtain inverse fixed-point weights and inverse weight value quantization hyperparameters, which are specifically used for: if the number of input channels of the network layer is cin, and the number of output channels is cout, then the initial model weights of the network layer are divided into cin sub-model weights; wherein each sub-model weight is expressed as cout*m*n, m*n is the size of the sub-model weight, and cout is the number of channels of the sub-model weight; each sub-model weight in the cin sub-model weights is quantized to obtain the inverse fixed-point weights and inverse weight value quantization hyperparameters corresponding to the sub-model weight.

[0144] Exemplarily, when the training module 42 adjusts the initial model weight of the network layer based on the fixed-point activation gradient value, it is specifically used to: determine the fixed-point weight gradient value based on the fixed-point activation gradient value; determine the floating-point weight gradient value based on the fixed-point weight gradient value and the activation value quantization hyperparameter; and adjust the initial model weight of the network layer based on the floating-point weight gradient value.

[0145] Exemplarily, the processing module 43 determines the floating-point activation gradient value corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value, the inverse fixed-point weight, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter, and is specifically used to: determine the fixed-point activation gradient value corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value and the inverse fixed-point weight corresponding to the network layer; determine the floating-point activation gradient value corresponding to the previous network layer based on the fixed-point activation gradient value, the activation value quantization hyperparameter and the inverse weight value quantization hyperparameter corresponding to the previous network layer.

[0146] Exemplarily, when the acquisition module 41 acquires the floating-point activation gradient value corresponding to the network layer, it is specifically used to: if the network layer is the last network layer, obtain the floating-point activation value corresponding to the network layer, determine the target loss value based on the floating-point activation value and the calibrated floating-point true value, and determine the floating-point activation gradient value corresponding to the network layer based on the target loss value; or, if the network layer is not the last network layer of the initial network model, determine the floating-point activation gradient value output by the next network layer after the network layer as the floating-point activation gradient value corresponding to the network layer.

[0147] Exemplarily, the acquisition module 41 is also used to obtain the floating-point input data corresponding to the network layer, quantize the floating-point input data to obtain fixed-point input data and input data quantization hyperparameters; quantize the initial model weights of the network layer to obtain forward fixed-point weights and forward weight value quantization hyperparameters; the processing module 43 is also used to process the fixed-point input data based on the forward fixed-point weights to obtain fixed-point activation values, and determine the floating-point activation value corresponding to the network layer based on the fixed-point activation value, the input data quantization hyperparameters and the forward weight value quantization hyperparameters; wherein, if the network layer is not the last network layer of the initial network model, the floating-point activation value corresponding to the network layer is used as the floating-point input data corresponding to the next network layer of the network layer.

[0148] Exemplarily, the processing module 43 quantizes the initial model weights of the network layer to obtain forward fixed-point weights and forward weight value quantization hyperparameters, which is specifically used for: if the number of input channels of the network layer is cin, and the number of output channels is cout, then the initial model weights of the network layer are divided into cout sub-model weights; wherein each sub-model weight is expressed as cin*m*n, m*n is the size of the sub-model weight, and cin is the number of channels of the sub-model weight; each sub-model weight of the cout sub-model weights is quantized to obtain the forward fixed-point weights and forward weight value quantization hyperparameters corresponding to the sub-model weight.

[0149] Based on the same application concept as the above method, the present application embodiment proposes a network model training device, see Figure 5 As shown, the network model training device includes a processor 51 and a machine-readable storage medium 52, and the machine-readable storage medium 52 stores machine-executable instructions that can be executed by the processor 81; the processor 51 is used to execute the machine-executable instructions to implement the network model training method disclosed in the above example.

[0150] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the training method of the network model disclosed in the above example of the present application can be implemented.

[0151] The machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0152] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0153] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0154] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0155] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] Furthermore, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0158] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A training method for a network model, characterized in that: The initial network model includes multiple network layers, each network layer includes initial model weights, the input data of the initial network model is a floating-point sample image, and after the floating-point sample image is input to the initial network model, the floating-point sample image is processed by the initial network model; wherein, for each network layer, the method includes: Obtaining a floating-point activation gradient value of the floating-point sample image corresponding to the network layer, quantizing the floating-point activation gradient value of the floating-point sample image, and obtaining a fixed-point activation gradient value of the floating-point sample image and an activation value quantization hyperparameter of the floating-point sample image; Adjusting the initial model weights of the network layer based on the fixed-point activation gradient values ​​of the floating-point sample image, and updating the adjusted model weights as the initial model weights of the network layer; If the network layer is not the first network layer, quantizing the initial model weights of the network layer to obtain the inverse fixed-point weights of the floating-point sample image and the inverse weight value quantization hyperparameters of the floating-point sample image; determining the floating-point activation gradient value of the floating-point sample image corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value of the floating-point sample image, the inverse fixed-point weight of the floating-point sample image, the activation value quantization hyperparameter of the floating-point sample image and the inverse weight value quantization hyperparameter of the floating-point sample image; Wherein, the determining the floating-point activation gradient value of the floating-point sample image corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value of the floating-point sample image, the inverse fixed-point weight of the floating-point sample image, the activation value quantization hyperparameter of the floating-point sample image and the inverse weight value quantization hyperparameter of the floating-point sample image includes: determining the fixed-point activation gradient value corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value of the floating-point sample image corresponding to the network layer and the inverse fixed-point weight of the floating-point sample image; determining the floating-point activation gradient value of the floating-point sample image corresponding to the previous network layer based on the fixed-point activation gradient value corresponding to the previous network layer, the activation value quantization hyperparameter of the floating-point sample image and the inverse weight value quantization hyperparameter of the floating-point sample image; Among them, after adjusting the initial model weights of all network layers to obtain adjusted model weights, if the initial network model has converged, the converged network model is used as the target network model, and the target network model is deployed on the terminal device to process the image data through the target network model to obtain a processing result for the image data; wherein, if the target network model is used to realize face detection, the image including the face is input into the target network model, and the image is processed by artificial intelligence through the target network model to obtain a face detection result; if the target network model is used to realize vehicle detection, the image including the vehicle is input into the target network model, and the image is processed by artificial intelligence through the target network model to obtain a vehicle detection result.

2. The method according to claim 1, characterized in that The quantizing of the initial model weights of the network layer to obtain the inverse fixed-point weights of the floating-point sample images and the inverse weight value quantization hyperparameters of the floating-point sample images includes: If the number of input channels of the network layer is cin and the number of output channels is cout, then the initial model weight of the network layer is divided into cin sub-model weights; wherein each sub-model weight is expressed as cout*m*n, where m*n is the size of the sub-model weight and cout is the number of channels of the sub-model weight; Each sub-model weight in the cin sub-model weights is quantized to obtain an inverse fixed-point weight and an inverse weight value quantization hyperparameter corresponding to the sub-model weight.

3. The method according to claim 1, characterized in that The adjusting the initial model weight of the network layer based on the fixed-point activation gradient value of the floating-point sample image includes: Determining a fixed-point weight gradient value based on the fixed-point activation gradient value; Determine a floating-point weight gradient value based on the fixed-point weight gradient value and the activation value quantization hyperparameter; The initial model weights of the network layer are adjusted based on the floating-point weight gradient values.

4. The method according to claim 1, wherein The obtaining of the floating-point activation gradient value of the floating-point sample image corresponding to the network layer includes: If the network layer is the last network layer, obtaining a floating-point activation value corresponding to the network layer, determining a target loss value based on the floating-point activation value and a calibrated floating-point true value, and determining a floating-point activation gradient value corresponding to the network layer based on the target loss value; If the network layer is not the last network layer, the floating-point activation gradient value output by the next network layer after the network layer is determined as the floating-point activation gradient value corresponding to the network layer.

5. The method according to any one of claims 1 to 4, characterized in that Before obtaining the floating-point activation gradient value of the floating-point sample image corresponding to the network layer, the method further includes: Obtain floating-point input data corresponding to the network layer, quantize the floating-point input data to obtain fixed-point input data and input data quantization hyperparameters; quantize the initial model weights of the network layer to obtain forward fixed-point weights and forward weight value quantization hyperparameters; Processing the fixed-point input data based on the forward fixed-point weight to obtain a fixed-point activation value, and determining a floating-point activation value corresponding to the network layer based on the fixed-point activation value, the input data quantization hyperparameter, and the forward weight value quantization hyperparameter; If the network layer is not the last network layer, the floating-point activation value corresponding to the network layer is used as the floating-point input data corresponding to the next network layer after the network layer.

6. The method according to claim 5, characterized in that The initial model weights of the network layer are quantized to obtain forward fixed-point weights and forward weight value quantization hyperparameters, including: If the number of input channels of the network layer is cin and the number of output channels is cout, then the initial model weight of the network layer is divided into cout sub-model weights; wherein each sub-model weight is expressed as cin*m*n, m*n is the size of the sub-model weight, and cin is the number of channels of the sub-model weight; Each sub-model weight in the cout sub-model weights is quantized to obtain a forward fixed-point weight and a forward weight value quantization hyperparameter corresponding to the sub-model weight.

7. A network model training device, characterized in that: The initial network model includes multiple network layers, each network layer includes initial model weights, the input data of the initial network model is a floating-point sample image, and after the floating-point sample image is input to the initial network model, the floating-point sample image is processed by the initial network model; wherein, for each network layer, the device includes: an acquisition module, configured to acquire a floating-point activation gradient value of the floating-point sample image corresponding to the network layer, quantize the floating-point activation gradient value of the floating-point sample image, and obtain a fixed-point activation gradient value of the floating-point sample image and an activation value quantization hyperparameter of the floating-point sample image; A training module, configured to adjust the initial model weights of the network layer based on the fixed-point activation gradient values ​​of the floating-point sample images, and update the adjusted model weights as the initial model weights of the network layer; a processing module configured to quantize the initial model weights of the network layer if the network layer is not the first network layer, to obtain the inverse fixed-point weights of the floating-point sample image and the inverse weight value quantization hyperparameters of the floating-point sample image; and determine the floating-point activation gradient value of the floating-point sample image corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value of the floating-point sample image, the inverse fixed-point weight of the floating-point sample image, the activation value quantization hyperparameter of the floating-point sample image, and the inverse weight value quantization hyperparameter of the floating-point sample image; Wherein, the processing module determines the floating-point activation gradient value of the floating-point sample image corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value of the floating-point sample image, the inverse fixed-point weight of the floating-point sample image, the activation value quantization hyperparameter of the floating-point sample image and the inverse weight value quantization hyperparameter of the floating-point sample image, and is specifically used to: determine the fixed-point activation gradient value corresponding to the previous network layer of the network layer based on the fixed-point activation gradient value of the floating-point sample image corresponding to the network layer and the inverse fixed-point weight of the floating-point sample image; determine the floating-point activation gradient value of the floating-point sample image corresponding to the previous network layer based on the fixed-point activation gradient value corresponding to the previous network layer, the activation value quantization hyperparameter of the floating-point sample image and the inverse weight value quantization hyperparameter of the floating-point sample image; Among them, after adjusting the initial model weights of all network layers to obtain adjusted model weights, if the initial network model has converged, the converged network model is used as the target network model, and the target network model is deployed on the terminal device to process the image data through the target network model to obtain a processing result for the image data; wherein, if the target network model is used to realize face detection, the image including the face is input into the target network model, and the image is processed by artificial intelligence through the target network model to obtain a face detection result; if the target network model is used to realize vehicle detection, the image including the vehicle is input into the target network model, and the image is processed by artificial intelligence through the target network model to obtain a vehicle detection result.

8. The device according to claim 7, It is characterized in that in, The processing module quantizes the initial model weights of the network layer to obtain the inverse fixed-point weights of the floating-point sample image and the inverse weight value quantization hyperparameters of the floating-point sample image, and is specifically used to: if the number of input channels of the network layer is cin and the number of output channels is cout, then divide the initial model weights of the network layer into cin sub-model weights; wherein each sub-model weight is expressed as cout*m*n, m*n is the size of the sub-model weight, and cout is the number of channels of the sub-model weight; quantize each sub-model weight of the cin sub-model weights to obtain the inverse fixed-point weight and inverse weight value quantization hyperparameters corresponding to the sub-model weight; Wherein, when the training module adjusts the initial model weight of the network layer based on the fixed-point activation gradient value of the floating-point sample image, it is specifically used to: determine the fixed-point weight gradient value based on the fixed-point activation gradient value; determine the floating-point weight gradient value based on the fixed-point weight gradient value and the activation value quantization hyperparameter; and adjust the initial model weight of the network layer based on the floating-point weight gradient value; Wherein, when the acquisition module acquires the floating-point activation gradient value of the floating-point sample image corresponding to the network layer, it is specifically used to: if the network layer is the last network layer, then acquire the floating-point activation value corresponding to the network layer, determine the target loss value based on the floating-point activation value and the calibrated floating-point true value, and determine the floating-point activation gradient value corresponding to the network layer based on the target loss value; or, if the network layer is not the last network layer of the initial network model, then determine the floating-point activation gradient value output by the next network layer of the network layer as the floating-point activation gradient value corresponding to the network layer; Wherein, the acquisition module is further used to obtain the floating-point input data corresponding to the network layer, quantize the floating-point input data, and obtain fixed-point input data and input data quantization hyperparameters; quantize the initial model weights of the network layer to obtain forward fixed-point weights and forward weight value quantization hyperparameters; the processing module is further used to process the fixed-point input data based on the forward fixed-point weights to obtain fixed-point activation values, and determine the floating-point activation value corresponding to the network layer based on the fixed-point activation value, the input data quantization hyperparameters and the forward weight value quantization hyperparameters; wherein, if the network layer is not the last network layer of the initial network model, the floating-point activation value corresponding to the network layer is used as the floating-point input data corresponding to the next network layer of the network layer; Among them, the processing module quantizes the initial model weights of the network layer to obtain forward fixed-point weights and forward weight value quantization hyperparameters, which are specifically used for: if the number of input channels of the network layer is cin, and the number of output channels is cout, then the initial model weights of the network layer are divided into cout sub-model weights; wherein each sub-model weight is expressed as cin*m*n, m*n is the size of the sub-model weight, and cin is the number of channels of the sub-model weight; each sub-model weight of the cout sub-model weights is quantized to obtain the forward fixed-point weights and forward weight value quantization hyperparameters corresponding to the sub-model weight.

9. A network model training device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method steps described in any one of claims 1-6.

Images