Model training method and apparatus, storage medium, and program product
By iteratively training and adaptively adjusting the cutoff value of the neural network model, and optimizing the quantization of activation values, the quantization error problem was solved, and the model performance and efficiency were improved.
Patent Information
- Application Number
- CN201980102629.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-12-27
- Publication Date
- 2026-01-09
- Estimated Expiration
- 2039-12-27
AI Technical Summary
In existing technologies, neural network models suffer from significant quantization errors during activation value quantization, which affects model performance.
Through multiple iterations of training, the cutoff value in the neural network model is adaptively adjusted to optimize the upper and lower limits of quantization of activation values during model training and reduce quantization error.
It improves the performance of neural network models, reduces quantization errors, and enhances the efficiency and accuracy of model operation.
Smart Images
Figure CN114730367B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, and particularly relates to a model training method and device, a storage medium and a program product. BACKGROUND
[0002] A neural network model is a network system formed by a large number of simple processing units (referred to as neurons) that are widely interconnected. The neural network model can be applied to scenarios such as image classification, image detection, single image super resolution (SISR) tasks, and the like. The training process of the neural network model can include a forward propagation process and a backward propagation process.
[0003] In the forward propagation process, sample data is input into the neural network model, and the sample data is processed according to the weight values in the neural network model to obtain output data. In the backward propagation process, the weight values in the neural network model are adjusted according to the loss value between the output data and sample labels.
[0004] The intermediate result in the processing process of the neural network model can be referred to as an activation value. The activation value in the neural network model generally adopts a high-precision data format. In order to reduce the storage space occupied by the neural network model, and reduce the occupation of hardware bandwidth and cache in the operation process of the neural network model, and improve the running efficiency of the neural network, the quantization technology is often used for the activation value in the forward propagation process.
[0005] At present, when the activation value in the neural network model is quantized, a fixed clipping value is set for the network layer in the neural network model before model training, and then the activation value of the network layer is quantized according to the clipping value of the network layer in the model training process. However, due to the uncertainty of the sample data, the uncertainty of the activation value in the neural network model is caused, so the quantization of the activation value can cause a large quantization error, thereby affecting the performance of the neural network model finally trained. SUMMARY
[0006] The present application provides a model training method and device, a storage medium and a program product, which can solve the problem of poor performance of the neural network model trained in the related art. The technical solution is as follows:
[0007] In a first aspect, a model training method is provided. In the method, a neural network model is trained for multiple iterations using training samples. In one iteration of the multiple iterations, in a forward propagation process, sample data in the training samples is processed according to weights in the neural network model and a current clipping value of a network layer to obtain output data, where the clipping value of the network layer is used to quantize activation values of the network layer; in a backward propagation process, weights in the neural network model are adjusted according to a loss value between the output data and sample labels in the training samples, and the clipping value of the network layer is adjusted according to the loss value, the current clipping value of the network layer, and the activation values.
[0008] In this application, the clipping value in the neural network model is obtained through training, that is, the upper and lower limits for quantizing the activation values can be adaptively adjusted during the model training process, thereby reducing the quantization error and improving the performance of the neural network model.
[0009] It should be noted that the training samples can be set in advance, and the training samples can include sample data and sample labels. For example, the training samples can include an image (sample data) and a label (sample label) of the image, and the label of the image can be the type or identity of an object contained in the image; or the training samples can include a low resolution (LR) image (sample data) and a high resolution (HR) image (sample label) corresponding to the LR image.
[0010] In addition, the network layer can include m parts, each part can share a clipping value, and m is a positive integer. When m is 1, the network layer shares one clipping value, that is, all activation values in the network layer are quantized according to the one clipping value; when m is an integer greater than or equal to 2, the network layer includes multiple parts, each part shares a clipping value, that is, the activation values of each part are quantized according to the corresponding clipping value.
[0011] In a feasible implementation, the m parts of the network layer means that the input of the network layer can be defined as m parts according to the number of output neurons or the number of output channels of the network layer. Specifically, when the network layer has m output neurons or m output channels, the input of the network layer can be divided into m parts corresponding to the m output neurons or the m output channels. In other words, the m parts of the network layer are m groups of input neurons corresponding to the m output neurons of the network layer, or the m parts of the network layer are m groups of input channels corresponding to the m output channels of the network layer.
[0012] According to the loss value, the current clipping value and the activation value of the network layer, the operation of adjusting the clipping value of the network layer can be: determining a first adjustment degree according to the loss value and the dequantization value of the network layer; determining a second adjustment degree according to the size relationship between the current clipping value and the activation value of the network layer; multiplying the first adjustment degree and the second adjustment degree to obtain a target adjustment degree; subtracting the product of the learning rate and the target adjustment degree from the current clipping value of the network layer to obtain the adjusted clipping value of the network layer.
[0013] It should be noted that the operation of adjusting the clipping value of the network layer according to the loss value is to obtain the partial derivative of the loss function of the neural network model with respect to the clipping value (referred to as the target adjustment degree in the present application).
[0014] In the present application, the partial derivative of the loss function with respect to the clipping value is obtained according to the loss value, the current clipping value and the activation value of the network layer. Specifically, the partial derivative of the loss function with respect to the clipping value is defined as the product of the partial derivative of the loss function with respect to the dequantization value of the network layer (referred to as the first adjustment degree in the present application) and the partial derivative of the quantization function of the network layer with respect to the clipping value of the network layer (referred to as the second adjustment degree in the present application).
[0015] In the process of obtaining the partial derivative of the quantization function with respect to the clipping value, the partial derivative of the quantization function with respect to the clipping value is actually approximated to the partial derivative of the clipping function with respect to the clipping value in the present application. The partial derivative of the clipping function with respect to the clipping value depends on the size relationship between the current clipping value of the network layer and the activation value of the network layer.
[0016] Specifically, according to the size relationship between the current clipping value of the network layer and the activation value of the network layer, the operation of determining the second adjustment degree can be: when the activation value of the network layer is less than or equal to the inverse of the current clipping value of the network layer, determining the second adjustment degree as -1; when the activation value of the network layer is greater than the inverse of the current clipping value of the network layer and less than the current clipping value of the network layer, determining the second adjustment degree as 0; when the activation value of the network layer is greater than or equal to the current clipping value of the network layer, determining the second adjustment degree as 1.
[0017] It should be understood that the implementation can also be that when the activation value of the network layer is less than the inverse of the current clipping value of the network layer, the second adjustment degree is determined as -1; when the activation value of the network layer is greater than or equal to the inverse of the current clipping value of the network layer and less than or equal to the current clipping value of the network layer, the second adjustment degree is determined as 0; when the activation value of the network layer is greater than the current clipping value of the network layer, the second adjustment degree is determined as 1. Or other similar conditional segmentation methods, which will not be described here.
[0018] Further, the threshold value in the neural network model can be initialized before the threshold value in the neural network model is adjusted according to the loss value of the neural network model. That is, the threshold value in the neural network model can be initialized before the neural network model is iteratively trained multiple times using the training sample.
[0019] Specifically, the operation of initializing the threshold value in the neural network model can be: iteratively training the neural network model t times using the training sample, and then determining the initial threshold value of the network layer according to the activation values of the m parts of the network layer in the t times of iterative training. Wherein, t can be set in advance, and t can be a positive integer.
[0020] In this application, the threshold value is initialized according to the statistical characteristics of the activation value in the neural network model, so as to improve the model stability and accelerate the convergence.
[0021] Wherein, the operation of determining the initial threshold value of the network layer according to the activation values of the m parts of the network layer in the t times of iterative training can be: in the first iteration training of the t times of iterative training, obtaining the maximum activation value in the activation value of each part of the m parts of the network layer, and taking the average value of the obtained m maximum activation values as the first threshold value; in the i-th iteration training of the t times of iterative training, obtaining the maximum activation value in the activation value of each part of the m parts of the network layer, and taking the average value of the obtained m maximum activation values and the (i-1)th threshold value as the i-th threshold value, i is an integer greater than or equal to 2 and less than or equal to t; the t-th threshold value is taken as the initial threshold value corresponding to each part of the m parts of the network layer.
[0022] Secondly, a model training device is provided, which has the function of realizing the behavior of the model training method in the first aspect. The model training device includes at least one module for realizing the model training method provided in the first aspect.
[0023] Thirdly, a model training device is provided, which includes a processor and a memory. The memory is used to store programs supporting the model training device to execute the model training method provided in the first aspect, and to store data involved in the model training method in the first aspect. The processor is configured to execute the programs stored in the memory. The model training device can also include a communication bus for establishing a connection between the processor and the memory.
[0024] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores instructions which, when executed on a computer, cause the computer to perform the model training method of the first aspect.
[0025] In a fifth aspect, a computer program product is provided, and the computer program product contains instructions which, when executed on a computer, cause the computer to perform the model training method of the first aspect.
[0026] The technical effects obtained by the second aspect, the third aspect, the fourth aspect and the fifth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be described here.
[0027] The technical solutions provided in the present application can at least bring the following beneficial effects:
[0028] The cut-off value in the neural network model in the present application is obtained by training, that is, the upper and lower limits when quantizing the activation value can be adaptively adjusted during the model training process, so as to reduce the quantization error and improve the performance of the finally trained neural network model. BRIEF DESCRIPTION OF DRAWINGS
[0029] Figure 1 is a structural schematic diagram of a computer device provided by an embodiment of the present application;
[0030] Figure 2 is a flowchart of a model training method provided by an embodiment of the present application;
[0031] Figure 3 is a flowchart of an iterative training operation provided by an embodiment of the present application;
[0032] Figure 4 is a structural schematic diagram of a model training device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.
[0034] Figure 1 is a structural schematic diagram of a computer device provided by an embodiment of the present application. Referring to Figure 1 The computer device includes at least one processor 101, a communication bus 102, a memory 103 and at least one communication interface 104.
[0035] The processor 101 can be a microprocessor (including a central processing unit (CPU) and the like), an application-specific integrated circuit (ASIC), or can be one or more integrated circuits for controlling program execution of the schemes of the present application.
[0036] The communication bus 102 can include a path for transmitting information between the above-mentioned components.
[0037] The memory 103 can be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disk (including a compact disc read-only memory (CD-ROM), a compact disc, a laser disc, a digital versatile disc, a Blu-ray disc, and the like), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but is not limited thereto. The memory 103 can exist independently and be connected to the processor 101 through the communication bus 102. The memory 103 can also be integrated with the processor 101.
[0038] The communication interface 104 uses any transceiver-like device for communicating with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), and the like.
[0039] In a specific implementation, as an embodiment, the processor 101 can include one or more CPUs, such as the CPU0 and the CPU1 shown in FIG. 1. Figure 1
[0040] In a specific implementation, as an embodiment, the computer device can include multiple processors, such as the processor 101 and the processor 105 shown in FIG. 1. Each of the processors can be a single-core processor or a multi-core processor. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Figure 1
[0041] In a specific implementation, as an embodiment, the computer device can further include an output device 106 and an input device 107. The output device 106 is in communication with the processor 101 and can display information in a variety of ways. For example, the output device 106 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 107 is in communication with the processor 101 and can receive user input in a variety of ways. For example, the input device 107 can be a mouse, a keyboard, a touch screen device, or a sensor device, etc.
[0042] The computer device described above can be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device can be a desktop computer, a laptop computer, a network server, a palmtop computer, a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device, and the embodiments of the present application do not limit the type of computer device.
[0043] The memory 103 is configured to store program code 110 for implementing the scheme of the present application, and the processor 101 is configured to execute the program code 110 stored in the memory 103. The computer device can implement the following steps by means of the processor 101 and the program code 110 in the memory 103. Figure 2 The embodiments provide a model training method.
[0044] Figure 2 is a flowchart of a model training method provided by an embodiment of the present application. Referring to Figure 2 The method includes the following steps.
[0045] Step 201: iteratively training the neural network model using training samples.
[0046] It should be noted that the training samples can be set in advance, and the training samples can include sample data and sample labels.
[0047] In addition, the neural network model can be a network system formed by a large number of simple processing units (called neurons) widely interconnected. The neural network model can include a plurality of network layers, including an input layer, a hidden layer, and an output layer. The input layer is responsible for receiving sample data; the output layer is responsible for outputting processed data; the hidden layer is located between the input layer and the output layer and is responsible for processing data. The hidden layer is invisible to the outside. For example, the neural network model can be a deep neural network, and can be a convolutional neural network in the deep neural network, etc.
[0048] It should be noted that the neural network model trained by the embodiments of the present application can be applied to various scenarios, such as image classification, image detection, SISR task, etc. The goal of SISR task is to reconstruct the corresponding HR image from the LR image.
[0049] When applied to image classification scenarios or image detection scenarios, the training sample can include an image (sample data) and a label of the image (sample label), which can be the type, identity, etc. of the object contained in the image. When applied to SISR task scenarios, the training sample can include a LR image (sample data) and a HR image corresponding to the LR image (sample label).
[0050] Each iteration of the multiple iterations can include at least a forward propagation process, in which the sample data can be processed to obtain output data. After completing the forward propagation process, if the loss value between the output data of the neural network model at this time and the sample label does not exceed the specified range, the iteration training is ended, and a neural network model meeting the requirements is obtained; if the loss value between the output data of the neural network model at this time and the sample label exceeds the specified range, the reverse propagation process is continued to adjust the parameters in the neural network model, and after completing the reverse propagation process, the next iteration training can be continued.
[0051] Specifically, referring to Figure 3 The operation of one iteration of the multiple iterations can include steps 2011-2014.
[0052] Step 2011: In the forward propagation process, the sample data in the training sample is processed according to the weight value in the neural network model and the current truncation value of the network layer to obtain the output data.
[0053] It should be noted that the truncation value of the network layer is used to quantize the activation value of the network layer.
[0054] In the forward propagation process, the intermediate result in the processing process of the neural network model can be referred to as the activation value. Specifically, in the forward propagation process, for the input layer, the sample data can be directly taken as the activation value of the input layer; for any network layer except the output layer, the activation value of the network layer can be processed to obtain the activation value of the next network layer.
[0055] The activation values in a neural network model are generally in a high-precision data format (such as FP32, which is an IEEE 754 defined data representation format standard). In order to reduce the storage space occupied by the neural network model, and reduce the occupation of hardware bandwidth and cache by the neural network model in the operation process, and improve the neural network operation efficiency, a quantization technology is often used for the activation values in the forward propagation process.
[0056] In order to obtain good neural network acceleration performance, one possible quantization technology is to use symmetric linear quantization of a truncation value, which can be implemented according to the following quantization function:
[0057] The quantization function is:
[0058] where x is the activation value; n is the number of quantization bits, which can be pre-set; f(x) is a truncation function, f(x) = max(min(x, a), -a), which means that x is limited to [-a, a], that is, when x is greater than a, x is truncated to a, and when x is less than -a, x is truncated to -a; a is a truncation value, a is a positive number; s(n) is a quantization unit, or <> indicates rounding to the nearest integer.
[0059] It is worth noting that in the embodiments of the present application, the quantization technology is used for the activation values in the forward propagation process, which can be specifically as follows: for a network layer in the neural network model except the output layer, the activation values of the network layer are quantized according to the current truncation value of the network layer to obtain quantized values of the network layer; the quantized values of the network layer are processed to obtain processed quantized values; the processed quantized values are dequantized to obtain dequantized values of the network layer as activation values of the next network layer.
[0060] It should be noted that the network layer can include m parts, and each part can share one truncation value, m is a positive integer. When m is 1, the network layer shares one truncation value, that is, all the activation values in the network layer are quantized according to the one truncation value; when m is an integer greater than or equal to 2, the network layer includes multiple parts, and each part shares one truncation value, that is, the activation values of each part are quantized according to the corresponding truncation value.
[0061] In an implementation, the m parts of the network layer refer to that the input of the network layer can be defined as m parts according to the number of output neurons or the number of output channels of the network layer. Specifically, when the network layer has m output neurons or m output channels, the input of the network layer can be divided into m parts corresponding to the m output neurons or the m output channels. In other words, the m parts of the network layer are m groups of input neurons corresponding to the m output neurons of the network layer, or the m parts of the network layer are m groups of input channels corresponding to the m output channels of the network layer. Each group of input neurons can include one or more input neurons, and each group of input channels can include one or more input channels.
[0062] The operation of quantizing the activation value of the network layer according to the current clipping value of the network layer to obtain the quantized value of the network layer can be implemented according to the quantization function of the network layer, and the number of quantization bits and the quantization unit in the quantization function of the network layer are pre-set. Specifically, the current clipping value and the activation value of the network layer are substituted into to obtain the quantized value of the network layer.
[0063] The operation of processing the quantized value of the network layer to obtain the processed quantized value can be different according to the type of the network layer, and the specific operation can be referred to related technologies, which will not be described in detail in the embodiments of the present application. For example, when the network layer has weights and an activation function, the quantized value of the network layer can be processed according to the weights in the network layer to obtain a first processing result, and then the first processing result is processed according to the activation function in the network layer to obtain a second processing result as the processed quantized value.
[0064] The operation of dequantizing the processed quantized value to obtain the dequantized value of the network layer can be implemented according to the quantization function of the network layer, and the number of quantization bits and the quantization unit in the quantization function of the network layer are pre-set. Specifically, the processed quantized value is multiplied by s(n) to obtain the dequantized value of the network layer.
[0065] Step 2012: determining whether the loss value between the output data and the sample label in the training sample exceeds a specified range. If not, performing the following step 2013; if yes, performing the following step 2014.
[0066] Step 2013: ending the iterative training to obtain a neural network model meeting the requirements.
[0067] Step 2014: In the back propagation process, the weight value in the neural network model is adjusted according to the loss value between the output data and the sample label in the training sample, and the truncation value of the network layer is adjusted according to the loss value, the current truncation value and the activation value of the network layer.
[0068] It should be noted that after step 2014 is performed, step 2011 can be returned to for the next iteration of training.
[0069] It should be noted that in the prior art, in the back propagation process, the truncation value in the neural network model is kept unchanged, and only the weight value in the neural network model is adjusted. In the embodiment of the present application, in the back propagation process, the truncation value in the neural network model can be adjusted at the same time as the weight value in the neural network model is adjusted. In this way, the truncation value in the neural network model is obtained through training, i.e., the upper and lower limits for quantizing the activation value can be adaptively adjusted during model training, thereby reducing the quantization error and improving the performance of the neural network model.
[0070] It should be noted that in the embodiment of the present application, the weight value and the truncation value in the neural network model can be referred to as the parameters in the neural network model. That is, the embodiment of the present application actually adjusts the parameters in the neural network model according to the loss value between the output data of the neural network model and the sample label of the training sample.
[0071] In addition, the loss value between the output data and the sample label of the training sample can be obtained by a loss function of the neural network model. The loss function can be a general loss function, such as a cross-entropy loss function, a mean square error loss function, etc. Alternatively, the loss function can be a regularization loss function, which is the sum of a general loss function and a regularization function.
[0072] Wherein, the operation of adjusting the weight value in the neural network model according to the loss value between the output data and the sample label in the training sample can refer to related technologies, and the embodiment of the present application does not perform detailed description.
[0073] For example, for any weight value in the neural network model, the partial derivative of the loss function of the neural network model with respect to this weight value can be obtained according to the loss value and this weight value; the weight value is subtracted by the product of the learning rate and the partial derivative of this weight value, to obtain the adjusted weight value. It should be noted that the learning rate can be set in advance. For example, the learning rate can be 0.001, 0.000001, etc.
[0074] Wherein, according to the loss value, the operation of adjusting the clipping value of the network layer is crucial in obtaining the partial derivative of the loss function of the neural network model with respect to the clipping value (referred to as the target adjustment degree in the embodiments of the present application).
[0075] In the embodiments of the present application, the partial derivative of the loss function with respect to the clipping value is obtained according to the loss value, the current clipping value and the activation value of the network layer. Specifically, the partial derivative of the loss function with respect to the clipping value is defined as the product of the partial derivative of the loss function with respect to the dequantization value of the network layer (referred to as the first adjustment degree in the embodiments of the present application) and the partial derivative of the quantization function of the network layer with respect to the clipping value of the network layer (referred to as the second adjustment degree in the embodiments of the present application).
[0076] Specifically, according to the loss value, the current clipping value and the activation value of the network layer, the operation of adjusting the clipping value of the network layer can be: determining the first adjustment degree according to the loss value and the dequantization value of the network layer; determining the second adjustment degree according to the size relationship between the current clipping value and the activation value of the network layer; multiplying the first adjustment degree and the second adjustment degree to obtain the target adjustment degree; subtracting the product of the learning rate and the target adjustment degree from the current clipping value of the network layer to obtain the adjusted clipping value of the network layer.
[0077] It should be noted that the learning rate can be set in advance, and the learning rate can be the same as or different from the learning rate when adjusting the weight value in the neural network model. For example, the learning rate can be 0.001, 0.000001, etc.
[0078] Wherein, in the embodiments of the present application, the partial derivative of the loss function with respect to the dequantization value of the network layer is obtained by determining the first adjustment degree according to the loss value and the dequantization value of the network layer. In other words, the partial derivative of the loss function with respect to the dequantization value is obtained as the first adjustment degree according to the loss value and the dequantization value of the network layer.
[0079] It should be noted that for the quantization function , let , that is the result of the activation value after being truncated. At this time, the quantization function can be reconstructed as: or
[0080] Wherein, when obtaining the partial derivative of the quantization function with respect to the clipping value, that is, obtaining , let is not derivable, but it can be approximated to 1 by the straight-through estimator method. Thus can be approximated to At this time, the actual value is The partial derivative with respect to a is taken as x q The partial derivative with respect to a. The partial derivative with respect to a depends on the size relationship between a (the current truncation value of the network layer) and x (the activation value of the network layer).
[0081] That is, in the embodiment of the application, the partial derivative of the quantization function with respect to the truncation value is obtained, that is, according to the size relationship between the current truncation value and the activation value of the network layer, the second adjustment degree is determined.
[0082] Specifically, when the activation value of the network layer is less than or equal to the negative of the current truncation value of the network layer, the second adjustment degree is determined to be -1; when the activation value of the network layer is greater than the negative of the current truncation value of the network layer and less than the current truncation value of the network layer, the second adjustment degree is determined to be 0; when the activation value of the network layer is greater than or equal to the current truncation value of the network layer, the second adjustment degree is determined to be 1.
[0083] It should be understood that the embodiment can also be that when the activation value of the network layer is less than the negative of the current truncation value of the network layer, the second adjustment degree is determined to be -1; when the activation value of the network layer is greater than or equal to the negative of the current truncation value of the network layer and less than or equal to the current truncation value of the network layer, the second adjustment degree is determined to be 0; when the activation value of the network layer is greater than the current truncation value of the network layer, the second adjustment degree is determined to be 1. Or other similar conditional segmentation methods, which will not be described here.
[0084] It is worth noting that when the network layer includes m parts and each part shares a truncation value, for any one of the m parts, the first adjustment degree corresponding to the part can be determined according to the loss value and the dequantization value of the part; for any one of all activation values of the part, the second adjustment degree corresponding to the activation value can be determined according to the size relationship between the current truncation value corresponding to the part and the activation value; the average of the second adjustment degrees corresponding to all activation values of the part is taken as the second adjustment degree corresponding to the part; the product of the first adjustment degree and the second adjustment degree corresponding to the part is taken as the target adjustment degree corresponding to the part; the current truncation value corresponding to the part is subtracted by the product of the learning rate and the target adjustment degree corresponding to the part to obtain the adjusted truncation value corresponding to the part.
[0085] Further, before adjusting the truncation value in the neural network model according to the loss value of the neural network model, the truncation value in the neural network model can be initialized. That is, before step 201, the truncation value in the neural network model can be initialized.
[0086] Specifically, the operation of initializing the clipping value in the neural network model can be: performing t times of iterative training on the neural network model using the training sample, and then determining the initial clipping value of the network layer according to the activation values of the m parts of the network layer in the t times of iterative training. Wherein, t can be set in advance, and t can be a positive integer.
[0087] It is worth noting that in the embodiments of the present application, the clipping value is initialized according to the statistical characteristics of the activation values in the neural network model, so as to improve the model stability and accelerate the convergence.
[0088] Wherein, the operation of each iteration training in the t times of iterative training can be: in the forward propagation process, processing the sample data in the training sample according to the weight value in the neural network model to obtain output data; in the back propagation process, adjusting the weight value in the neural network model according to the loss value between the output data and the sample label in the training sample.
[0089] Wherein, the operation of determining the initial clipping value of the network layer according to the activation values of the m parts of the network layer in the t times of iterative training can be: in the first iteration training in the t times of iterative training, obtaining the maximum activation value in the activation value of each part of the m parts of the network layer, and taking the average value of the m maximum activation values as the first clipping value; in the i th iteration training in the t times of iterative training, obtaining the maximum activation value in the activation value of each part of the m parts of the network layer, and taking the average value of the m maximum activation values and the i-1 th clipping value as the i th clipping value, i is an integer greater than or equal to 2 and less than or equal to t; the t th clipping value is taken as the initial clipping value corresponding to each part of the m parts of the network layer.
[0090] It should be noted that the weight of the average value of the m maximum activation values and the weight of the i-1 th clipping value can be set in advance, and the sum of the two weights is 1. For example, the weight of the i-1 th clipping value can be set to 0.9997. Then, the average value of the m maximum activation values is multiplied by its weight to obtain a first value; the i-1 th clipping value is multiplied by its weight to obtain a second value; the first value and the second value are added to obtain the i th clipping value.
[0091] In the embodiments of the present application, the neural network model is trained multiple times using the training samples. For one of the multiple times of training, in the forward propagation process, the sample data in the training samples is processed according to the weights in the neural network model and the current threshold value of the network layer to obtain output data. In the backward propagation process, the weights in the neural network model are adjusted according to the loss value between the output data and the sample labels in the training samples, and the threshold value of the network layer is adjusted according to the loss value, the current threshold value of the network layer and the activation value. In this way, the threshold value in the neural network model is obtained by training, that is, the upper and lower limits of quantizing the activation value can be adaptively adjusted during the model training process, so as to reduce the quantization error and improve the performance of the neural network model finally obtained.
[0092] It is worth noting that after the model training is completed by the above-mentioned model training method, the neural network model obtained by training can be applied, such as using the neural network model for image classification, image detection, SISR task, etc. Among them, the weights and threshold values in the neural network model are obtained by training.
[0093] For example, in the SISR scenario, the low-resolution image to be reconstructed can be input into the neural network model to obtain the corresponding high-resolution image. Since the threshold value in the neural network model is obtained by training, the quantization error of the neural network model is small and the performance is good, so the quality of the high-resolution image reconstructed by the neural network model is high.
[0094] Figure 4 is a structural schematic diagram of a model training device provided by an embodiment of the present application. The model training device can be realized by software, hardware or a combination of the two to become part or all of a computer device. The computer device can be the computer device shown in Figure 1 . Referring to Figure 4 , the device comprises a training module 401.
[0095] The first training module 401 is configured to perform step 201 in the above Figure 2 embodiment.
[0096] The first training module 401 comprises:
[0097] The processing unit 4011 is configured to perform step 2011 in the above Figure 2 embodiment.
[0098] The adjustment unit 4012 is configured to perform step 2014 in the above Figure 2 embodiment.
[0099] Optionally, the adjustment unit 4012 is configured to:
[0100] determine a first adjustment degree according to the loss value and the dequantization value of the network layer;
[0101] determine a second adjustment degree according to a size relationship between the current clipping value of the network layer and the activation value of the network layer;
[0102] multiply the first adjustment degree and the second adjustment degree to obtain a target adjustment degree;
[0103] subtract the product of the learning rate and the target adjustment degree from the current clipping value of the network layer to obtain an adjusted clipping value of the network layer.
[0104] Optionally, the adjusting unit 4012 is configured to:
[0105] when the activation value of the network layer is less than or equal to the opposite of the current clipping value of the network layer, determine the second adjustment degree as -1;
[0106] when the activation value of the network layer is greater than the opposite of the current clipping value of the network layer and less than the current clipping value of the network layer, determine the second adjustment degree as 0;
[0107] when the activation value of the network layer is greater than or equal to the current clipping value of the network layer, determine the second adjustment degree as 1.
[0108] Optionally, the apparatus further comprises:
[0109] a second training module configured to perform t times of iterative training on the neural network model using the training samples, t being a positive integer;
[0110] a determining module configured to determine an initial clipping value of the network layer according to activation values of m parts of the network layer in the t times of iterative training, m being a positive integer.
[0111] Optionally, the m parts of the network layer are m groups of input neurons corresponding to m output neurons of the network layer, or the m parts of the network layer are m groups of input channels corresponding to m output channels of the network layer.
[0112] Optionally, the sample data is a low-resolution image, and the sample label is a high-resolution image corresponding to the low-resolution image.
[0113] In the embodiments of the present application, the neural network model is trained multiple times using the training samples. For one of the multiple times of training, in the forward propagation process, the sample data in the training samples is processed according to the weights in the neural network model and the current threshold value of the network layer to obtain output data. In the backward propagation process, the weights in the neural network model are adjusted according to the loss value between the output data and the sample labels in the training samples, and the threshold value of the network layer is adjusted according to the loss value, the current threshold value of the network layer and the activation value. In this way, the threshold value in the neural network model is obtained by training, that is, the upper and lower limits of quantizing the activation value can be adaptively adjusted during the model training process, so as to reduce the quantization error and improve the performance of the neural network model finally obtained.
[0114] It should be noted that the model training device provided in the above embodiments is only used as an example for the division of the above functional modules during model training. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the model training device and the model training method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.
[0115] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (for example: coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example: infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium accessible by a computer, or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example: floppy disk, hard disk, magnetic tape), optical media (for example: digital versatile disc (DVD)) or semiconductor media (for example: solid state disk (SSD)) and the like.
[0116] The above is an embodiment provided by the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A model training method, characterized in that, The method comprises: using training samples to perform multiple iteration training on a neural network model; wherein one iteration training in the multiple iteration training comprises: in a forward propagation process, processing sample data in the training samples according to weights in the neural network model and a current threshold value of a network layer to obtain output data, wherein the threshold value of the network layer is used to quantize activation values of the network layer; in a backward propagation process, adjusting the weights in the neural network model according to a loss value between the output data and sample labels in the training samples, and adjusting the threshold value of the network layer according to the loss value, the current threshold value and the activation values of the network layer; wherein the adjusting the threshold value of the network layer according to the loss value, the current threshold value and the activation values of the network layer comprises: determining a first adjustment degree according to the loss value and a dequantization value of the network layer; determining a second adjustment degree according to a size relationship between the current threshold value and the activation values of the network layer; multiplying the first adjustment degree and the second adjustment degree to obtain a target adjustment degree; subtracting a product of a learning rate and the target adjustment degree from the current threshold value of the network layer to obtain an adjusted threshold value of the network layer; wherein the sample data is a low-resolution image, and the sample label is a high-resolution image corresponding to the low-resolution image, or the sample data is an image, and the sample label is a type or identity of an object contained in the image.
2. The method of claim 1, wherein, The determining the second adjustment degree according to the size relationship between the current threshold value and the activation values of the network layer comprises: when the activation value of the network layer is less than or equal to an inverse number of the current threshold value of the network layer, determining the second adjustment degree as -1; when the activation value of the network layer is greater than the inverse number of the current threshold value of the network layer and less than the current threshold value of the network layer, determining the second adjustment degree as 0; when the activation value of the network layer is greater than or equal to the current threshold value of the network layer, determining the second adjustment degree as 1.
3. The method of claim 1, wherein, Before the using training samples to perform multiple iteration training on a neural network model, the method further comprises: performing t times of iteration training on the neural network model using the training samples, t being a positive integer; determining an initial threshold value of the network layer according to activation values of m parts of the network layer in the t times of iteration training, m being a positive integer.
4. The method of claim 3, wherein, The m parts of the network layer are m groups of input neurons corresponding to m output neurons of the network layer, or the m parts of the network layer are m groups of input channels corresponding to m output channels of the network layer.
5. A model training apparatus characterized by comprising: The apparatus comprises: a first training module configured to perform multiple iteration training on a neural network model using training samples; wherein the first training module comprises: a processing unit configured to, in a forward propagation process, process sample data in the training samples according to weights in the neural network model and a current threshold value of a network layer to obtain output data, wherein the threshold value of the network layer is used to quantize activation values of the network layer; The adjusting unit is configured to adjust a weight value in the neural network model according to a loss value between the output data and a sample label in the training sample during back propagation, and adjust a clipping value of the network layer according to the loss value, the current clipping value of the network layer, and an activation value of the network layer; The adjusting unit is configured to: determine a first adjustment degree according to the loss value and a dequantization value of the network layer; determine a second adjustment degree according to a size relationship between the current clipping value of the network layer and the activation value of the network layer; multiply the first adjustment degree and the second adjustment degree to obtain a target adjustment degree; subtract a product of a learning rate and the target adjustment degree from the current clipping value of the network layer to obtain an adjusted clipping value of the network layer. The sample data is a low-resolution image, and the sample label is a high-resolution image corresponding to the low-resolution image, or the sample data is an image, and the sample label is a type or an identity of an object contained in the image.
6. The apparatus of claim 5, wherein, The adjusting unit is configured to: determine the second adjustment degree as -1 when the activation value of the network layer is less than or equal to an inverse number of the current clipping value of the network layer; determine the second adjustment degree as 0 when the activation value of the network layer is greater than the inverse number of the current clipping value of the network layer and less than the current clipping value of the network layer; determine the second adjustment degree as 1 when the activation value of the network layer is greater than or equal to the current clipping value of the network layer.
7. The apparatus of claim 5, wherein, The apparatus further includes: a second training module configured to perform t times of iterative training on the neural network model using the training sample, t being a positive integer; a determining module configured to determine an initial clipping value of the network layer according to activation values of m parts of the network layer in the t times of iterative training, m being a positive integer.
8. The apparatus of claim 7, wherein, The m parts of the network layer are m groups of input neurons corresponding to m output neurons of the network layer, or the m parts of the network layer are m groups of input channels corresponding to m output channels of the network layer.
9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1-4.
10. A computer program product comprising instructions, characterized in that, When executed on a computer, cause the computer to perform the method of any one of claims 1-4.