A neural network model training method and device
By performing integer fixed-point encoding of the activation amount and weight of the neural network model, the problem of hardware resource overhead caused by single-precision floating-point data is solved, and efficient neural network model training is realized, suitable for devices with resource limitations.
Patent Information
- Application Number
- CN201910808066.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-08-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-08-29
AI Technical Summary
During the training process of existing neural network models, due to the high bit width of single-precision floating-point data, hardware resource overhead is high and it is difficult to run efficiently.
Integer fixed-point encoding is used to encode the activation amount and weight in the neural network model, convert it into integer fixed-point data with a specified bit width, perform forward and reverse operations, and reduce the resource overhead of matrix multiplication and addition operations.
It significantly reduces the hardware resource overhead required for neural network model training, improves training efficiency, and is suitable for resource-constrained edge devices.
Smart Images

Figure CN112446461B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning technology, and in particular to a neural network model training method and device. Background Art
[0002] As an emerging field in machine learning research, deep neural networks parse data by imitating the mechanism of the human brain. They are intelligent models that analyze and learn by building and simulating the human brain. At present, deep neural networks, such as convolutional neural networks, recurrent neural networks, and long short-term memory networks, have been well applied in target detection and segmentation, behavior detection and recognition, and speech recognition.
[0003] At present, the training of neural network models usually uses single-precision floating-point data for calculation to ensure the accuracy of neural network model convergence. However, due to the high bit width of single-precision floating-point data, the amount of data involved in the calculation is large, resulting in a high hardware resource overhead for running neural network models. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a neural network model training method and device to reduce the hardware resource overhead required to run the neural network model. The specific technical solution is as follows:
[0005] In a first aspect, an embodiment of the present application provides a neural network model training method, the method comprising:
[0006] Get training samples;
[0007] The neural network model is trained using the training samples. When the neural network model is trained, the following steps are performed for each network layer in the neural network model:
[0008] Obtain the first activation of the input network layer and the network weight of the network layer;
[0009] Performing integer fixed-point encoding on the first activation amount and the network weight, encoding the first activation amount and the network weight into integer fixed-point data with a specified bit width;
[0010] According to the encoded first activation amount and the network weight, the second activation amount output by the network layer is calculated.
[0011] Optionally, the method is applied to a camera; the training sample is a training sample with a specified target; the neural network model is a target detection model for detecting the specified target;
[0012] Before the step of training the neural network model using the training samples, the method further includes:
[0013] Enable target detection function;
[0014] Determine whether to start the model online training function;
[0015] The step of training the neural network model using the training samples includes:
[0016] If the model online training function is enabled, the target detection model is trained using training samples with specified targets.
[0017] Optionally, the step of training the neural network model using the training samples includes:
[0018] Input the training samples into the neural network model, perform forward operation on the training samples in the order of each network layer in the neural network model from front to back, and obtain the forward operation result of the neural network model, wherein, when performing the forward operation, for each network layer, respectively perform integer fixed-point encoding on the first activation amount input to the network layer and the network weight of the network layer, encode the first activation amount and the network weight into integer fixed-point data with a specified bit width, and calculate the second activation amount output by the network layer according to the encoded first activation amount and network weight, and calculate the second activation amount as the first activation amount input to the next network layer, until the second activation amount output by the last network layer is determined as the forward operation result;
[0019] Compare the forward operation result with the preset nominal value to obtain the loss value;
[0020] Input the loss value into the neural network model, perform reverse operation on the loss value in the order of each network layer in the neural network model from back to front, and obtain the weight gradient of each network layer in the neural network model, wherein when performing the reverse operation, for each network layer, respectively perform integer fixed-point encoding on the first activation amount, the first activation amount gradient and the network weight of the network layer input into the network layer, encode the first activation amount, the first activation amount gradient and the network weight into integer fixed-point data with a specified bit width, and calculate the second activation amount gradient and the weight gradient output by the network layer according to the encoded first activation amount, the first activation amount gradient and the network weight, and use the second activation amount gradient as the first activation amount gradient input into the next network layer for calculation, until the weight gradients of all network layers are calculated;
[0021] According to the weight gradient of each network layer, the network weight of each network layer is adjusted.
[0022] Optionally, the step of adjusting the network weights of each network layer according to the weight gradient of each network layer includes:
[0023] Performing integer fixed-point encoding on the weight gradient of each network layer, encoding the weight gradient of each network layer into integer fixed-point data with a specified bit width;
[0024] According to the weight gradients of each network layer after encoding and the network weights of each network layer after encoding, the network weights of each network layer after adjustment are calculated using a preset optimization algorithm.
[0025] Optionally, after calculating the second activation amount output by the network layer according to the encoded first activation amount and the network weight, the method provided in the embodiment of the present application further includes:
[0026] The second activation amount is encoded into integer fixed-point data having a specified bit width.
[0027] Optionally, the step of performing integer fixed-point encoding on the first activation amount and the network weight, encoding the first activation amount and the network weight into integer fixed-point data with a specified bit width, includes:
[0028] The scalar values in the first activation and the network weights are respectively encoded as the product of a parameter value representing a global dynamic range and an integer fixed-point value of a specified bit width.
[0029] Optionally, if the network layer is a convolutional layer, the size of the network weights is C×R×R×N, and for each scalar value in each three-dimensional tensor of size C×R×R, the corresponding parameter value is the same;
[0030] If the network layer is a fully connected layer, the size of the network weight is M×N, and for each scalar value in each column vector of size 1×N, the corresponding parameter value is the same;
[0031] The parameter values corresponding to the scalar values in the first activation amount are the same.
[0032] In a second aspect, an embodiment of the present application provides a neural network model training device, the device comprising:
[0033] An acquisition module, used to acquire training samples;
[0034] The training module is used to train the neural network model using the training samples. When training the neural network model, the training module performs the following steps for each network layer in the neural network model:
[0035] Obtain the first activation of the input network layer and the network weight of the network layer;
[0036] Performing integer fixed-point encoding on the first activation amount and the network weight, encoding the first activation amount and the network weight into integer fixed-point data with a specified bit width;
[0037] According to the encoded first activation and the network weight, the second activation output by the network layer is calculated.
[0038] Optionally, the device is applied to a camera; the training sample is a training sample with a specified target; the neural network model is a target detection model for detecting the specified target;
[0039] The device also includes:
[0040] Enable module, used to enable target detection function;
[0041] A judgment module is used to determine whether to start the model online training function;
[0042] Training modules are specifically used for:
[0043] If the judgment result of the judgment module is to start the model online training function, the target detection model is trained using training samples with specified targets.
[0044] Optional training module, specifically for:
[0045] Input the training samples into the neural network model, perform forward operation on the training samples in the order of each network layer in the neural network model from front to back, and obtain the forward operation result of the neural network model, wherein, when performing the forward operation, for each network layer, respectively perform integer fixed-point encoding on the first activation amount input to the network layer and the network weight of the network layer, encode the first activation amount and the network weight into integer fixed-point data with a specified bit width, and calculate the second activation amount output by the network layer according to the encoded first activation amount and network weight, and calculate the second activation amount as the first activation amount input to the next network layer, until the second activation amount output by the last network layer is determined as the forward operation result;
[0046] Compare the forward operation result with the preset nominal value to obtain the loss value;
[0047] Input the loss value into the neural network model, perform reverse operation on the loss value in the order of each network layer in the neural network model from back to front, and obtain the weight gradient of each network layer in the neural network model, wherein when performing the reverse operation, for each network layer, respectively perform integer fixed-point encoding on the first activation amount, the first activation amount gradient and the network weight of the network layer input into the network layer, encode the first activation amount, the first activation amount gradient and the network weight into integer fixed-point data with a specified bit width, and calculate the second activation amount gradient and the weight gradient output by the network layer according to the encoded first activation amount, the first activation amount gradient and the network weight, and use the second activation amount gradient as the first activation amount gradient input into the next network layer for calculation, until the weight gradients of all network layers are calculated;
[0048] According to the weight gradient of each network layer, the network weight of each network layer is adjusted.
[0049] Optionally, the training module, when used to adjust the network weights of each network layer according to the weight gradients of each network layer, is specifically used to:
[0050] Performing integer fixed-point encoding on the weight gradient of each network layer, encoding the weight gradient of each network layer into integer fixed-point data with a specified bit width;
[0051] According to the weight gradients of each network layer after encoding and the network weights of each network layer after encoding, the network weights of each network layer after adjustment are calculated using a preset optimization algorithm.
[0052] Optionally, the training module is also used to:
[0053] The second activation amount is encoded into integer fixed-point data having a specified bit width.
[0054] Optionally, the training module, when used to perform integer fixed-point encoding on the first activation amount and the network weight, and encode the first activation amount and the network weight into integer fixed-point data with a specified bit width, is specifically used to:
[0055] The scalar values in the first activation and the network weights are respectively encoded as the product of a parameter value representing a global dynamic range and an integer fixed-point value of a specified bit width.
[0056] Optionally, if the network layer is a convolutional layer, the size of the network weights is C×R×R×N, and for each scalar value in each three-dimensional tensor of size C×R×R, the corresponding parameter value is the same;
[0057] If the network layer is a fully connected layer, the size of the network weight is M×N, and for each scalar value in each column vector of size 1×N, the corresponding parameter value is the same;
[0058] The parameter values corresponding to the scalar values in the first activation amount are the same.
[0059] In a third aspect, an embodiment of the present application provides a computer device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is prompted by the machine-executable instructions to implement the method provided in the first aspect of the embodiment of the present application.
[0060] In a fourth aspect, an embodiment of the present application provides a machine-readable storage medium storing machine-executable instructions, which, when called and executed by a processor, implements the method provided in the first aspect of the embodiment of the present application.
[0061] The embodiment of the present application provides a neural network model training method and device, which obtains training samples and uses the training samples to train the neural network model. When training the neural network model, for each network layer in the neural network model, respectively perform: obtain the first activation amount input to the network layer and the network weight of the network layer, perform integer fixed-point encoding on the first activation amount and the network weight, encode the first activation amount and the network weight into integer fixed-point data with a specified bit width, and calculate the second activation amount output by the network layer according to the encoded first activation amount and the network weight. When training the neural network model, the first activation amount input to each network layer and the network weight of each network layer are integer fixed-point encoded, and the encoded first activation amount and the network weight are integer fixed-point data with a specified bit width. When performing operations, the operations involved, such as matrix multiplication and matrix addition, all use integer fixed-point format, and the bit width of integer fixed-point data is significantly less than the bit width of single-precision floating-point data. Therefore, the hardware resource overhead required to run the neural network model can be greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0063] Figure 1 A flowchart of a neural network model training method according to an embodiment of the present application;
[0064] Figure 2 A schematic diagram of a neural network model training process according to an embodiment of the present application;
[0065] Figure 3 A schematic diagram of the execution flow of each network layer in a neural network model during training of the neural network model according to an embodiment of the present application;
[0066] Figure 4 A schematic diagram of a tensor space structure corresponding to a four-dimensional tensor convolution kernel of size C×R×R×N in an embodiment of the present application;
[0067] Figure 5 A schematic diagram of an encoding method for each scalar value in a three-dimensional tensor of size C×R×R according to an embodiment of the present application;
[0068] Figure 6 A schematic diagram of the tensor space structure corresponding to a two-dimensional matrix of size M×N according to an embodiment of the present application;
[0069] Figure 7A schematic diagram of an encoding method for each scalar value in a column vector of size 1×N according to an embodiment of the present application;
[0070] Figure 8 A schematic diagram of an encoding method for each scalar value in a three-dimensional tensor of activation amount and activation amount gradient according to an embodiment of the present application;
[0071] Fig. 9 A schematic diagram of a flow chart of a method for training a target detection model applied to a camera according to an embodiment of the present application;
[0072] Fig.10 A schematic diagram of the structure of a neural network model training device according to an embodiment of the present application;
[0073] Fig.11 A schematic diagram of the structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION
[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0075] In order to reduce the hardware resource overhead required to run a neural network model, the present application embodiment provides a neural network model training method, apparatus, computer equipment and machine-readable storage medium. Below, the neural network model training method provided by the present application embodiment is first introduced.
[0076] The execution subject of a neural network training method provided in an embodiment of the present application may be a computer device with a neural network model training function, or a computer device that realizes functions such as target detection and segmentation, behavior detection and recognition, and speech recognition, or a camera with functions such as target detection and segmentation, behavior detection and recognition, or a microphone with speech recognition function, etc. The execution subject at least includes a core processing chip with data processing capabilities. The method of implementing a neural network training method provided in an embodiment of the present application may be at least one of software, hardware circuits, and logic circuits set in the execution subject.
[0077] like Figure 1 As shown, a neural network model training method provided in an embodiment of the present application may include the following steps.
[0078] S101, obtaining training samples.
[0079] When training a neural network, it is usually necessary to collect a large number of training samples. The collected training samples are different based on the different functions that the neural network model needs to achieve. For example, if you are training a detection model for face detection, the collected training samples are face samples; if you are training a tracking model for vehicle tracking, the collected training samples are vehicle samples.
[0080] S102, using the training samples to train the neural network model.
[0081] Input the training samples into the neural network model, use the BP (Back Propagation) algorithm or other model training algorithms to calculate the training samples, compare the calculation results with the set nominal values, and adjust the network weights of the neural network model based on the comparison results. By inputting different training samples into the neural network model in sequence, iteratively executing the above steps, and continuously adjusting the network weights, the output of the neural network model will become closer and closer to the nominal value, until the difference between the output of the neural network model and the nominal value is small enough, or when the output of the neural network model converges, the training of the neural network model is considered to be completed.
[0082] Taking the BP algorithm as an example, the main calculation operations and data flows in the neural network model training process are as follows: Figure 2 As shown, each network layer mainly performs convolution operation Y during forward operation. i =W i *Y i-1 , each network layer mainly performs convolution operation dY when performing reverse operation i-1 =dY i-1 *W i , and the matrix multiplication operation dW i =dY i *Y i-1 , where the forward operation refers to the operation sequence from the first network layer to the back, and the reverse operation refers to the operation sequence from the last network layer to the front. i represents the network weights of the i-th network layer, such as the convolutional layer or fully connected layer parameters, Y i Represents the activation of the input or output of the i-th network layer, dW i Represents the weight gradient corresponding to the i-th network layer, dY i Represents the activation gradient of the input i-th network layer.
[0083] like Figure 2As shown in the figure, in the process of training the neural network model using the BP algorithm, the training sample X is input into the neural network model. After the forward operation of the neural network model, the k-layer network layer performs convolution operation from front to back in sequence to obtain the model output Y k , the model output is compared with the nominal value through the loss function to obtain the loss value dY k , and then through the reverse operation of the neural network model, the k layers of the network are sequentially convolved and multiplied from back to front to obtain the weight gradient corresponding to each network layer, and the network weight is adjusted according to the weight gradient. After continuous iteration, the output of the neural network model is getting closer and closer to the nominal value.
[0084] In the embodiment of the present application, in the process of training the neural network model, for each network layer in the neural network model, the following operations need to be performed respectively: Figure 3 The steps shown.
[0085] S301, obtaining a first activation of an input network layer and a network weight of the network layer.
[0086] When performing the forward operation, the first activation of the input i-th network layer is Y i When performing the reverse operation, the first activation of the input layer i is dY i .
[0087] S302, performing integer fixed-point encoding on the first activation amount and the network weight, encoding the first activation amount and the network weight into integer fixed-point data with a specified bit width.
[0088] For the i-th network layer, the first activation Y of the input network layer needs to be i ,dY i , and the network weight W of the network layer i Perform integer fixed-point encoding. Integer fixed-point encoding is to encode floating-point format data into integer fixed-point format data.
[0089] Optionally, S302 may specifically be:
[0090] The scalar values in the first activation and the network weights are respectively encoded as the product of a parameter value representing a global dynamic range and an integer fixed-point value of a specified bit width.
[0091] The specific encoding method can be to encode the scalar values in the first activation and the network weights as the product of the parameter value sp representing the global dynamic range and the integer fixed-point value ip of the specified bit width, where sp = 2 E, E is a signed binary number with a bit width of EB, EB is the set bit width, ip is a signed binary number with a bit width of IB, IB is the bit width set according to the size of the original floating-point data. The calculation method of the integer fixed-point value ip and the parameter value sp is:
[0092]
[0093]
[0094] Among them, s is the sign bit of the binary number x, which can be 0 or 1. i is the value of the i-th digit of the binary number x, which can be 0 or 1.
[0095] Optionally, if the network layer is a convolutional layer, the size of the network weight is C×R×R×N, and for each scalar value in each three-dimensional tensor of size C×R×R, the corresponding parameter value is the same; if the network layer is a fully connected layer, the size of the network weight is M×N, and for each scalar value in each column vector of size 1×N, the corresponding parameter value is the same; the parameter value corresponding to each scalar value in the first activation amount is the same.
[0096] W i is the network weight corresponding to the i-th layer of the neural network model, and the network layer type is a convolutional layer or a fully connected layer. If the i-th layer is a convolutional layer, then W i is a four-dimensional tensor convolution kernel of size C×R×R×N, and the corresponding tensor space structure is as follows Figure 4 As shown, C represents the dimension size of the convolution kernel input channel direction, R represents the dimension size of the convolution kernel space, and N represents the dimension size of the convolution kernel output channel direction. For each three-dimensional tensor W of size C×R×R i p Each scalar value w in can be expressed as:
[0097] w=sp·ip (3)
[0098] Among them, each W i p The three-dimensional tensor shares a single sp, and each scalar value w corresponds to an integer fixed-point value ip. The encoding of each scalar value in a three-dimensional tensor of size C×R×R is as follows Figure 5 As shown in the figure, the calculation methods of ip and sp are as shown in formulas (1) and (2), which will not be repeated here.
[0099] Similarly, if the i-th layer is a fully connected layer, then W i is a two-dimensional matrix of size M×N, and the corresponding tensor space structure is as follows Figure 6As shown, the matrix can be divided into the following structure, dividing the two-dimensional matrix of size M×N into M column vectors of size 1×N. For each column vector W of size 1×N i p Each scalar value w in is expressed by the above formula (3). i p The column vectors share a single sp, and each scalar value w corresponds to an integer fixed-point value ip. The encoding of each scalar value in a column vector of size 1×N is as follows Figure 7 As shown in the figure, the calculation methods of ip and sp are as shown in formulas (1) and (2), which will not be repeated here.
[0100] Y i and dY i is the activation and activation gradient corresponding to the i-th layer of the neural network model, which is a three-dimensional tensor of size C×H×W. i Or dY i Each scalar value y or dy in can be expressed as:
[0101] y=sp·ip (4)
[0102] dy=sp·ip (5)
[0103] Among them, each three-dimensional tensor Y i Or dY i Sharing a sp, each scalar value y or dy corresponds to an integer fixed-point value ip. The encoding method of each scalar value in the three-dimensional tensor of activation and activation gradient is as follows Figure 8 As shown in the figure, the calculation methods of ip and sp are as shown in formulas (1) and (2), which will not be repeated here.
[0104] S303, calculating a second activation amount output by the network layer according to the encoded first activation amount and the network weight.
[0105] As mentioned above, integer fixed-point encoding is performed on the scalar values in the first activation amount and the network weights. The encoded values are integer fixed-point values, so that the operations with the highest computing resource overhead during forward and reverse operations, such as convolution operations and matrix multiplication operations, are converted from floating-point operations to integer fixed-point operations, which greatly improves the training efficiency of neural networks on the hardware platform.
[0106] Optionally, S102 may be implemented through the following steps:
[0107] In the first step, the training samples are input into the neural network model, and the forward operation is performed on the training samples in the order of each network layer in the neural network model from front to back to obtain the forward operation result of the neural network model, wherein, when performing the forward operation, for each network layer, the first activation amount input into the network layer and the network weight of the network layer are respectively encoded as integer fixed-point data, and the first activation amount and the network weight are encoded as integer fixed-point data with a specified bit width, and the second activation amount output by the network layer is calculated according to the encoded first activation amount and network weight of each network layer, and the second activation amount is used as the first activation amount input into the next network layer for calculation, until the second activation amount output by the last network layer is determined as the forward operation result.
[0108] In the second step, the forward operation result is compared with the preset nominal value to obtain the loss value.
[0109] The third step is to input the loss value into the neural network model, and perform reverse operation on the loss value in the order of each network layer in the neural network model from back to front to obtain the weight gradient of each network layer in the neural network model. When performing the reverse operation, for each network layer, the first activation amount, the first activation amount gradient and the network weight of the network layer are respectively encoded as integer fixed-point data, and the first activation amount, the first activation amount gradient and the network weight are encoded as integer fixed-point data with a specified bit width. According to the encoded first activation amount, the first activation amount gradient and the network weight, the second activation amount gradient and the weight gradient of the network layer output are calculated, and the second activation amount gradient is used as the first activation amount gradient of the next network layer for calculation, until the weight gradients of all network layers are calculated.
[0110] The fourth step is to adjust the network weights of each network layer according to the weight gradient of each network layer.
[0111] The process from the first step to the fourth step is the operation process of the BP algorithm. By continuously executing these four steps, the training of the neural network model is realized. The forward operation process is to multiply the first activation value by the network weight Y i =W i *Y i-1 Calculate the second activation amount Y i The reverse operation process is to multiply the gradient of the first activation by the network weight dY i-1 =dY i-1 *W i Calculate the second activation gradient dY i-1 , and multiplying the first activation by the first activation gradient dW i =dY i *Y i-1 Calculate the weight gradient dW i, through the above integer fixed-point encoding, the above floating-point operations become integer fixed-point operations:
[0112] f 32 (Y k+1 ) = f 32 (Y k )*f 32 (W k ) → int YB (Y k+1 ) = int YB (Y k )*int WB (W k ) (6)
[0113] f 32 (dY k-1 ) = f 32 (dY k )*f 32 (W k ) → int dYB (dY k-1 ) = int dYB (dY k )*int WB (W k ) (7)
[0114] f 32 (dW k ) = f 32 (dY k )*f 32 (Y k-1 ) → int dWB (dW k ) = int dYB (dY k )*int YB (Y k-1 ) (8)
[0115] Among them, YB, WB, dYB, dWB are integer bit width values, f 32 () and int() represent 32-bit floating-point format and integer fixed-point format.
[0116] Optionally, the fourth step may be implemented by the following steps:
[0117] The weight gradient of each network layer is encoded into integer fixed-point data with a specified bit width; according to the encoded weight gradient of each network layer and the encoded network weight of each network layer, a preset optimization algorithm is used to calculate the adjusted network weights of each network layer.
[0118] After calculating the weight gradient of each network layer, the weight gradient can be encoded. The specific encoding process can refer to the above-mentioned process of encoding the network weight, which will not be repeated here. After encoding, the network weight needs to be adjusted based on the weight gradient. The adjustment process mainly involves matrix addition. Specifically, an optimization algorithm such as SGD (Stochastic Gradient Descent) is used to convert the network weight from a floating point format to an integer fixed point format. Taking the SGD optimization algorithm as an example, the conversion of the network weight is shown in formulas (9) to (11).
[0119] f 32 (dW) = f 32 (dW)+f 32 (λ w )·f 32 (W)→int dWB (dW) = int dWB (dW)+int λB (λ w )·int WB (W) (9)
[0120] f 32 (W old ) = f 32 (m)·f 32 (dW old )+f 32 (η)·f 32 (dW)→int WB (W old ) = int mB (m) int dWB (dW old )+int ηB (η) int dWB (dW) (10)
[0121] f 32 (W) = f 32 (W)+f 32 (W old ) → int WB (W) = int WB (W)+int WB (W old ) (11)
[0122] Among them, dW is the weight gradient of the network layer at the current moment, dW old is the weight gradient of the network layer at the previous moment, W is the network weight of the network layer at the current moment, λ w , η and m are training hyperparameters (which can be set).
[0123] Optionally, after executing S303, the neural network model training method provided in the embodiment of the present application may further execute:
[0124] The second activation amount is encoded into integer fixed-point data having a specified bit width.
[0125] After the operation of each network layer, the bit width of the obtained integer fixed-point data will generally become longer. When it is input into the subsequent network layer for operation, the operation efficiency may be reduced due to the longer bit width. In order to ensure the operation efficiency, the calculated second activation amount can be encoded into integer fixed-point again. The purpose is to reduce the bit width of the second activation amount so that the bit width of the second activation amount can meet the calculation requirements of the next network layer.
[0126] The embodiment of the present application is applied to obtain training samples, and the neural network model is trained using the training samples. When training the neural network model, for each network layer in the neural network model, respectively execute: obtain the first activation amount input to the network layer and the network weight of the network layer, perform integer fixed-point encoding on the first activation amount and the network weight, encode the first activation amount and the network weight into integer fixed-point data with a specified bit width, and calculate the second activation amount output by the network layer according to the encoded first activation amount and the network weight. When training the neural network model, the first activation amount input to each network layer and the network weight of each network layer are integer fixed-point encoded, and the encoded first activation amount and the network weight are integer fixed-point data with a specified bit width. When performing operations, the operations involved, such as matrix multiplication and matrix addition, all use integer fixed-point format, and the bit width of integer fixed-point data is significantly less than the bit width of single-precision floating-point data. Therefore, the hardware resource overhead required to run the neural network model can be greatly reduced.
[0127] The above neural network model training method is mainly suitable for resource-constrained edge devices, such as cameras. For cameras, the intelligent reasoning functions of cameras mainly include target detection, target tracking, face recognition, etc. The following takes target detection as an example to introduce the training method of the target detection model deployed on the camera. Fig. 9 As shown, it mainly includes the following steps:
[0128] S901, enable the target detection function.
[0129] The camera can enable the target detection function based on the user's selection results when target detection is needed according to the user's actual needs.
[0130] S902, determine whether to start the model online training function, if so, execute S903, otherwise wait for starting the model online training function.
[0131] Before using the target detection model for target detection, the target detection model needs to be trained. Whether to perform online training is up to the user. Usually, the camera will only perform online training after the model online training function is enabled. Figure 1 The steps of the embodiment shown are to train the target detection model.
[0132] S903: Train the target detection model using the acquired training samples with the specified target.
[0133] When training the target detection model, the training samples input to the target detection model are training samples with specified targets, so that the trained target detection model can detect the specified targets. Figure 3 The method of training the neural network model in the illustrated embodiment is the same and will not be repeated here.
[0134] Since the camera uses Figure 3 The training method in the illustrated embodiment trains the target detection model. During the training process, the first activation amount of each network layer and the network weight of each network layer are encoded as integer fixed-point data. The encoded first activation amount and network weight are integer fixed-point data with a specified bit width. When performing operations, the operations involved, such as matrix multiplication and matrix addition, all use integer fixed-point format. The bit width of integer fixed-point data is significantly less than that of single-precision floating-point data. Therefore, the hardware resource overhead of the camera can be greatly reduced. Online training of the target detection model on the camera enables the camera to have scene adaptation function.
[0135] Corresponding to the above method embodiment, the present application embodiment provides a neural network model training device, such as Fig.10 As shown, the device may include:
[0136] An acquisition module 1010 is used to acquire training samples;
[0137] The training module 1020 is used to train the neural network model using the training samples. When training the neural network model, the training module 1020 performs the following steps for each network layer in the neural network model:
[0138] Obtain the first activation of the input network layer and the network weight of the network layer;
[0139] Performing integer fixed-point encoding on the first activation amount and the network weight, encoding the first activation amount and the network weight into integer fixed-point data with a specified bit width;
[0140] According to the encoded first activation and the network weight, the second activation output by the network layer is calculated.
[0141] Optionally, the device is applied to a camera; the training sample is a training sample with a specified target; the neural network model is a target detection model for detecting the specified target;
[0142] The device may also include:
[0143] Enable module, used to enable target detection function;
[0144] A judgment module is used to determine whether to start the model online training function;
[0145] The training module 1020 may be specifically used for:
[0146] If the judgment result of the judgment module is to start the model online training function, the target detection model is trained using training samples with specified targets.
[0147] Optionally, the training module 1020 may be specifically used for:
[0148] Input the training samples into the neural network model, perform forward operation on the training samples in the order of each network layer in the neural network model from front to back, and obtain the forward operation result of the neural network model, wherein, when performing the forward operation, for each network layer, respectively perform integer fixed-point encoding on the first activation amount input to the network layer and the network weight of the network layer, encode the first activation amount and the network weight into integer fixed-point data with a specified bit width, and calculate the second activation amount output by the network layer according to the encoded first activation amount and network weight, and calculate the second activation amount as the first activation amount input to the next network layer, until the second activation amount output by the last network layer is determined as the forward operation result;
[0149] Compare the forward operation result with the preset nominal value to obtain the loss value;
[0150] Input the loss value into the neural network model, perform reverse operation on the loss value in the order of each network layer in the neural network model from back to front, and obtain the weight gradient of each network layer in the neural network model, wherein when performing the reverse operation, for each network layer, respectively perform integer fixed-point encoding on the first activation amount, the first activation amount gradient and the network weight of the network layer input into the network layer, encode the first activation amount, the first activation amount gradient and the network weight into integer fixed-point data with a specified bit width, and calculate the second activation amount gradient and the weight gradient output by the network layer according to the encoded first activation amount, the first activation amount gradient and the network weight, and use the second activation amount gradient as the first activation amount gradient input into the next network layer for calculation, until the weight gradients of all network layers are calculated;
[0151] According to the weight gradient of each network layer, the network weight of each network layer is adjusted.
[0152] Optionally, the training module 1020, when used to adjust the network weights of each network layer according to the weight gradient of each network layer, can be specifically used to:
[0153] Performing integer fixed-point encoding on the weight gradient of each network layer, encoding the weight gradient of each network layer into integer fixed-point data with a specified bit width;
[0154] According to the weight gradients of each network layer after encoding and the network weights of each network layer after encoding, the network weights of each network layer after adjustment are calculated using a preset optimization algorithm.
[0155] Optionally, the training module 1020 may also be used to:
[0156] The second activation amount is encoded into integer fixed-point data having a specified bit width.
[0157] Optionally, the training module 1020, when used to perform integer fixed-point encoding on the first activation amount and the network weight, and encode the first activation amount and the network weight into integer fixed-point data with a specified bit width, can be specifically used to:
[0158] The scalar values in the first activation and the network weights are respectively encoded as the product of a parameter value representing a global dynamic range and an integer fixed-point value of a specified bit width.
[0159] Optionally, if the network layer is a convolutional layer, the size of the network weights is C×R×R×N, and for each scalar value in each three-dimensional tensor of size C×R×R, the corresponding parameter value is the same;
[0160] If the network layer is a fully connected layer, the size of the network weight is M×N, and for each scalar value in each column vector of size 1×N, the corresponding parameter value is the same;
[0161] The parameter values corresponding to the scalar values in the first activation amount are the same.
[0162] The embodiment of the present application is applied to obtain training samples, and the neural network model is trained using the training samples. When training the neural network model, for each network layer in the neural network model, respectively execute: obtain the first activation amount input to the network layer and the network weight of the network layer, perform integer fixed-point encoding on the first activation amount and the network weight, encode the first activation amount and the network weight into integer fixed-point data with a specified bit width, and calculate the second activation amount output by the network layer according to the encoded first activation amount and the network weight. When training the neural network model, the first activation amount input to each network layer and the network weight of each network layer are integer fixed-point encoded, and the encoded first activation amount and the network weight are integer fixed-point data with a specified bit width. When performing operations, the operations involved, such as matrix multiplication and matrix addition, all use integer fixed-point format, and the bit width of integer fixed-point data is significantly less than the bit width of single-precision floating-point data. Therefore, the hardware resource overhead required to run the neural network model can be greatly reduced.
[0163] The present application embodiment provides a computer device, such as Fig.11 As shown, it may include a processor 1101 and a machine-readable storage medium 1102, wherein the machine-readable storage medium 1102 stores machine-executable instructions that can be executed by the processor 1101, and the processor 1101 is prompted by the machine-executable instructions to implement all the steps of the above-mentioned neural network model training method.
[0164] The machine-readable storage medium may include a RAM (Random Access Memory) or a NVM (Non-Volatile Memory), such as at least one disk storage. Optionally, the machine-readable storage medium may also be at least one storage device located away from the processor.
[0165] The above-mentioned processor can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it can also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0166] Data can be transmitted between the machine-readable storage medium 1102 and the processor 1101 via a wired connection or a wireless connection, and the computer device can communicate with other devices via a wired communication interface or a wireless communication interface. Fig.11 What is shown is merely an example of data transmission between the processor 1101 and the machine-readable storage medium 1102 via the bus, and is not intended to be a limitation on a specific connection method.
[0167] In this embodiment, the processor 1101 can achieve the following by reading the machine executable instructions stored in the machine readable storage medium 1102 and running the machine executable instructions: obtaining training samples, and using the training samples to train the neural network model. When training the neural network model, for each network layer in the neural network model, respectively executing: obtaining the first activation amount input to the network layer and the network weight of the network layer, performing integer fixed-point encoding on the first activation amount and the network weight, encoding the first activation amount and the network weight into integer fixed-point data with a specified bit width, and calculating the second activation amount output by the network layer according to the encoded first activation amount and the network weight. When training a neural network model, the first activation value of each network layer and the network weight of each network layer are encoded as integer fixed-point data. The encoded first activation value and network weight are integer fixed-point data with a specified bit width. When performing calculations, the matrix multiplication, matrix addition and other operations involved all use integer fixed-point format. The bit width of integer fixed-point data is significantly smaller than the bit width of single-precision floating-point data. Therefore, the hardware resource overhead required to run the neural network model can be greatly reduced.
[0168] The embodiment of the present application also provides a machine-readable storage medium storing machine-executable instructions, which, when called and executed by a processor, implement all the steps of the above-mentioned neural network model training method.
[0169] In this embodiment, the machine-readable storage medium stores machine executable instructions for executing the neural network model training method provided by the embodiment of the present application at runtime, so that the following can be achieved: obtaining training samples, using the training samples to train the neural network model. When training the neural network model, for each network layer in the neural network model, respectively execute: obtaining the first activation amount input to the network layer and the network weight of the network layer, performing integer fixed-point encoding on the first activation amount and the network weight, encoding the first activation amount and the network weight into integer fixed-point data with a specified bit width, and calculating the second activation amount output by the network layer according to the encoded first activation amount and the network weight. When training the neural network model, the first activation amount input to each network layer and the network weight of each network layer are integer fixed-point encoded, and the encoded first activation amount and the network weight are integer fixed-point data with a specified bit width. When performing operations, the operations involved, such as matrix multiplication and matrix addition, all use integer fixed-point format, and the bit width of the integer fixed-point data is significantly less than the bit width of the single-precision floating-point data, so the hardware resource overhead required to run the neural network model can be greatly reduced.
[0170] As for the computer device and machine-readable storage medium embodiments, since the method contents involved are basically similar to the aforementioned method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0171] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0172] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, computer equipment, and machine-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0173] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A neural network model training method, It is characterized in that The method is applied to a camera; the method comprises: Acquire a training sample, wherein the training sample is an image with a specified target; Enable target detection function; Determine whether to start the model online training function; If the model online training function is started, the neural network model is trained using the image with the specified target, wherein the neural network model is a target detection model for detecting the specified target; Wherein, the use of the image with the specified target to train the neural network model includes: inputting the training sample into the neural network model, and performing forward operation on the training sample in the order of each network layer in the neural network model from front to back to obtain the forward operation result of the neural network model, wherein, when performing the forward operation, for each network layer, the first activation amount input into the network layer and the network weight of the network layer are respectively integer fixed-point encoded, the first activation amount and the network weight are encoded into integer fixed-point data with a specified bit width, and the second activation amount output by the network layer is calculated according to the encoded first activation amount and the network weight, and the second activation amount is used as the first activation amount input into the next network layer for calculation, until the second activation amount output by the last network layer is determined as the forward operation result; Comparing the forward operation result with a preset nominal value to obtain a loss value; Input the loss value into the neural network model, perform reverse operation on the loss value in the order of the network layers in the neural network model from back to front, and obtain the weight gradient of each network layer in the neural network model, wherein when performing the reverse operation, for each network layer, respectively perform integer fixed-point encoding on the first activation amount, the first activation amount gradient and the network weight of the network layer input into the network layer, encode the first activation amount, the first activation amount gradient and the network weight into integer fixed-point data with a specified bit width, and calculate the second activation amount gradient and the weight gradient output by the network layer according to the encoded first activation amount, the first activation amount gradient and the network weight, and use the second activation amount gradient as the first activation amount gradient input into the next network layer for calculation, until the weight gradients of all network layers are calculated; Adjusting the network weights of each network layer according to the weight gradients of each network layer; The method of performing integer fixed-point encoding on the first activation amount, the first activation amount gradient and the network weight includes: The scalar values in the first activation, the first activation gradient and the network weight are respectively encoded as the product of the parameter value sp representing the global dynamic range and the integer fixed-point value ip of the specified bit width; wherein, s represents the sign bit of the scalar value, which can be 0 or 1. i Indicates the value of the i-th bit of the scalar value, which can be 0 or 1. EB and IB are the set bit widths.
2. The method according to claim 1, It is characterized in that The adjusting the network weights of each network layer according to the weight gradients of each network layer includes: Performing integer fixed-point encoding on the weight gradients of each network layer, encoding the weight gradients of each network layer into integer fixed-point data with a specified bit width; According to the weight gradients of the encoded network layers and the network weights of the encoded network layers, a preset optimization algorithm is used to calculate the adjusted network weights of the network layers.
3. The method according to claim 1, It is characterized in that After calculating the second activation amount output by the network layer according to the encoded first activation amount and the network weight, the method further includes: The second activation amount is encoded into integer fixed-point data having a specified bit width.
4. The method according to claim 1, It is characterized in that If the network layer is a convolutional layer, the network weight is C×R×R×N. 1 For each scalar value in each three-dimensional tensor of size C×R×R, the corresponding parameter value is the same, where C represents the dimension size of the convolution kernel input channel direction, R represents the spatial dimension size of the convolution kernel, and N 1 Indicates the dimension size of the convolution kernel output channel direction; If the network layer is a fully connected layer, the network weight is M×N 2 For each two-dimensional matrix of size 1×N 2 Each scalar value in the column vector of has the same corresponding parameter value, where M represents the number of columns of the two-dimensional matrix, and N 2 represents the number of rows of the two-dimensional matrix; The parameter values corresponding to the scalar values in the first activation amount are the same.
5. A neural network model training device, It is characterized in that The device is applied to a camera, and comprises: An acquisition module, used for acquiring a training sample, wherein the training sample is an image with a specified target; A training module is used to input the training samples into a neural network model, perform forward operations on the training samples in the order of the network layers in the neural network model from front to back, and obtain the forward operation results of the neural network model, wherein, when performing the forward operation, for each network layer, respectively, the first activation amount input to the network layer and the network weight of the network layer are integer fixed-point encoded, the first activation amount and the network weight are encoded as integer fixed-point data with a specified bit width, and the second activation amount output by the network layer is calculated based on the encoded first activation amount and the network weight, and the second activation amount is used as the first activation amount input to the next network layer for calculation, until the second activation amount output by the last network layer is determined as the forward operation result; the forward operation result is compared with a preset nominal value to obtain a loss value; the loss value is input to the neural network model, and the neural network model is used to calculate the second activation amount of the next network layer, until the second activation amount of the last network layer is determined as the forward operation result; the forward operation result is compared with a preset nominal value to obtain a loss value; the loss value is input to the neural network model, and the neural network model is used to calculate the second activation amount of the next network layer, and the second activation amount of the next network layer ... and the second activation amount of the next network layer is used to calculate the second activation amount of the In the order of each network layer from back to front, the loss value is reversely calculated to obtain the weight gradient of each network layer in the neural network model, wherein when performing the reverse calculation, for each network layer, the first activation amount, the first activation amount gradient and the network weight of the network layer are respectively encoded as integer fixed-point data with a specified bit width, and the first activation amount, the first activation amount gradient and the network weight are encoded as integer fixed-point data with a specified bit width, and the second activation amount gradient and the weight gradient output by the network layer are calculated according to the encoded first activation amount, the first activation amount gradient and the network weight, and the second activation amount gradient is used as the first activation amount gradient input to the next network layer for calculation until the weight gradients of all network layers are calculated; according to the weight gradients of each network layer, the network weights of each network layer are adjusted, wherein the neural network model is a target detection model for detecting the specified target; The device also includes: Enable module, used to enable target detection function; A judgment module is used to determine whether to start the model online training function; The training module is specifically used for: If the judgment result of the judgment module is to start the online training function of the model, the target detection model is trained using the image with the specified target; The method of performing integer fixed-point encoding on the first activation amount, the first activation amount gradient and the network weight includes: The scalar values in the first activation, the first activation gradient and the network weight are respectively encoded as the product of the parameter value sp representing the global dynamic range and the integer fixed-point value ip of the specified bit width; wherein, s represents the sign bit of the scalar value, which can be 0 or 1. i Indicates the value of the i-th bit of the scalar value, which can be 0 or 1. EB and IB are the set bit widths.
6. The device according to claim 5, It is characterized in that The training module, when used to adjust the network weights of each network layer according to the weight gradients of each network layer, is specifically used to: Performing integer fixed-point encoding on the weight gradients of each network layer, encoding the weight gradients of each network layer into integer fixed-point data with a specified bit width; According to the weight gradients of the encoded network layers and the network weights of the encoded network layers, a preset optimization algorithm is used to calculate the adjusted network weights of the network layers.
7. The device according to claim 5, It is characterized in that The training module is also used for: The second activation amount is encoded into integer fixed-point data having a specified bit width.
8. The device according to claim 5, It is characterized in that If the network layer is a convolutional layer, the network weight is C×R×R×N. 1 For each scalar value in each three-dimensional tensor of size C×R×R, the corresponding parameter value is the same, where C represents the dimension size of the convolution kernel input channel direction, R represents the spatial dimension size of the convolution kernel, and N 1 Indicates the dimension size of the convolution kernel output channel direction; If the network layer is a fully connected layer, the network weight is M×N 2 For each two-dimensional matrix of size 1×N 2 Each scalar value in the column vector of has the same corresponding parameter value, where M represents the number of columns of the two-dimensional matrix, and N 2 represents the number of rows of the two-dimensional matrix; The parameter values corresponding to the scalar values in the first activation amount are the same.
