Model training method, device, electronic device and computer-readable storage medium

Abstract

The present application provides a model training method, device, electronic device and computer-readable storage medium, the method comprising: obtaining the model parameters obtained for the original model in this round of training; quantizing the model parameters to obtain quantized parameters; updating the model parameters of the parallel model to the quantized parameters, and inputting the input data of the original model in this round of training into the parallel model for training; wherein the model structure of the parallel model is consistent with that of the original model; determining the comprehensive loss value of this round of training according to the loss value of the original model in this round of training and the loss value of the parallel model in this round of training; when the comprehensive loss value does not meet the preset end condition, updating the model parameters of the original model. The scheme takes into account the loss generated after the model parameters are quantized during the training process, so that the loss of the model finally obtained by training is also within an acceptable range after being quantized, thereby reducing the accuracy loss after the model is quantized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a model training method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the continuous development of AI (Artificial Intelligence) technology, deep neural networks are also being continuously applied to scenarios such as face recognition, intelligent navigation, intelligent medical consultation, remote communication, etc.

[0003] In order to meet the requirements of various AI application scenarios for detection accuracy, the width, number of layers, depth, and various parameters of the deep neural network structure have increased rapidly, resulting in a greater demand for space for deep learning models and lower reasoning efficiency. At the same time, businesses are increasingly inclined to deploy model applications from the cloud to the edge. Limited by the computing resources of edge devices, it is necessary to consider issues such as device storage, memory, power consumption, and latency, especially in application scenarios such as mobile terminals and embedded devices, which require more optimization.

[0004] As one of the general means of deep learning optimization, model quantization quantizes deep learning models into smaller fixed-point models with faster inference speed, which is suitable for the vast majority of models and usage scenarios.

[0005] Model quantization linearly maps the floating-point parameters (weights or tensors) with continuous or discrete values ​​in the network into fixed-point approximate discrete values ​​at the expense of reasoning accuracy, replacing the original float32 format data while keeping the input and output as floating-point type, thereby achieving the goals of reducing model size, reducing model memory consumption, and speeding up model reasoning.

[0006] However, the current quantization method is to directly quantize the trained model. Model quantization is achieved by converting the model parameters from fp32 to fp16 with the same distribution, or by converting the fp32 model to int8 through a pre-prepared data set combined with the weight value distribution. However, there will be a certain degree of accuracy loss after directly quantizing the trained model. Summary of the invention

[0007] The purpose of the embodiments of the present application is to provide a model training method, device, electronic device and computer-readable storage medium to alleviate the problem that the model obtained by the existing method will suffer a certain degree of accuracy loss after being quantized.

[0008] An embodiment of the present application provides a model training method, including: obtaining model parameters for an original model obtained in this round of training; quantizing the model parameters to obtain quantized parameters; updating the model parameters of a parallel model to the quantized parameters, and inputting the input data of the original model in this round of training into the parallel model for training; wherein the parallel model has a consistent model structure with that of the original model; determining a comprehensive loss value for this round of training based on a loss value of the original model in this round of training and a loss value of the parallel model in this round of training; and updating the model parameters of the original model when the comprehensive loss value does not meet a preset termination condition.

[0009] In the above implementation process, a parallel model with the same structure as the original model is used during the training process to execute the quantized parameters obtained in each training, and then a comprehensive loss value is determined based on the loss values ​​of the two models, and then the model parameters of the original model are updated based on the comprehensive loss value. In this way, the loss caused by the quantization of the model parameters can be taken into account during the training process, so that the loss of the model obtained by the final training is within an acceptable range after the model parameters are quantized, thereby alleviating the problem that the model obtained by the existing method will have a certain degree of accuracy loss after being quantized.

[0010] Furthermore, the method also includes: when the comprehensive loss value meets a preset termination condition, outputting the model parameters as the original model of the quantization parameters, or outputting the parallel model.

[0011] In the above implementation process, when the comprehensive loss value meets the preset end condition, the output model parameters are the original model of the quantized parameters, or the output parallel model, which can directly obtain the quantized model after the trained model parameters are quantized, thereby eliminating the need for users to perform subsequent quantization processing based on the trained original model, thereby improving the quantization efficiency.

[0012] Furthermore, the comprehensive loss value is a value obtained by weighted summing the loss value of the original model in this round of training and the loss value of the parallel model in this round of training.

[0013] In the above implementation process, the comprehensive loss value is obtained by weighted summing the loss value of the original model in this round of training and the loss value of the parallel model in this round of training, so that the integration between the two loss values ​​can be effectively achieved.

[0014] Furthermore, the original model and the parallel model use the same loss function.

[0015] In the above implementation process, by using the same loss function for the original model and the parallel model, it is possible to ensure that the loss values ​​of the original model and the parallel model have the same evaluation dimension, so that the two loss values ​​have higher comprehensiveness.

[0016] Furthermore, the model parameters include activation values ​​and weight values; the model parameters are quantized to obtain quantization parameters, including: mapping the activation values ​​and weight values ​​to activation values ​​and weight values ​​of a target type to obtain the quantization parameters; wherein the target type is a pre-set data type of the quantization parameter.

[0017] In the above implementation process, the activation value and the weight value are mapped to the activation value and the weight value of the target type to obtain the quantization parameters, which can ensure the quantization integrity of the model parameters.

[0018] Furthermore, mapping the activation value and the weight value to the activation value and the weight value of the target type includes: mapping the activation value and the weight value to the activation value and the weight value of the target type according to a preset quantization accuracy.

[0019] In the above implementation process, quantization is performed according to a preset quantization precision, and the quantization precision affects the precision and required space of the quantization model. Generally, the higher the quantization precision, the higher the precision of the quantization model, but the larger the space required for the quantization model. On the contrary, the lower the quantization precision, the lower the precision of the quantization model, but the smaller the space required for the quantization model. Thus, by setting the quantization precision, the obtained quantization model can achieve the desired balance between precision and occupied space.

[0020] The embodiment of the present application also provides a model training device, including: an acquisition module, a quantization module, a parallel model control module and a model update module; the acquisition module is used to acquire the model parameters obtained for the original model in this round of training; the quantization module is used to quantize the model parameters to obtain quantization parameters; the parallel model control module is used to update the model parameters of the parallel model to the quantization parameters, and input the input data of the original model in this round of training into the parallel model for training; wherein the model structure of the parallel model is consistent with that of the original model; the model update module is used to determine the comprehensive loss value of this round of training based on the loss value of the original model in this round of training and the loss value of the parallel model in this round of training; when the comprehensive loss value does not meet the preset end condition, the model parameters of the original model are updated.

[0021] Furthermore, the model training device also includes: an output module, which is used to output the original model whose model parameters are the quantization parameters, or output the parallel model when the comprehensive loss value meets the preset end condition.

[0022] An embodiment of the present application also provides an electronic device, including a processor, a memory and a communication bus; the communication bus is used to realize connection and communication between the processor and the memory; the processor is used to execute one or more programs stored in the memory to implement any of the above-mentioned model training methods.

[0023] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any of the above-mentioned model training methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the overall architecture of a model training process provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0030] Embodiment 1:

[0031] In order to reduce the accuracy loss after model quantization, a model training method is provided in the embodiment of the present application. Figure 1 As shown, Figure 1 A flow chart of the model training method provided in the embodiment of the present application includes:

[0032] S101: Obtain model parameters of the original model obtained in this round of training.

[0033] It should be understood that during training, a training sample set is constructed, and the original model is trained based on the training samples in the training sample set. There are many rounds of model training process. In the embodiment of the present application, the model parameters obtained by the original model in each round of training process are obtained.

[0034] S102: Quantize the model parameters to obtain quantized parameters.

[0035] In the embodiment of the present application, the model parameters include activation values ​​and weight values. The activation values ​​and weight values ​​can be mapped to the activation values ​​and weight values ​​of the target type to obtain quantization parameters.

[0036] It should be noted that the target type described in the embodiment of the present application refers to the data type of the pre-set quantization parameter. Exemplarily, the target type can be fp16 type, int8 type, uint8 type, etc., which is not limited in the embodiment of the present application.

[0037] It should be noted that in the embodiment of the present application, the user or engineer can pre-set a quantization accuracy, so that the activation value and the weight value are mapped to the activation value and the weight value of the target type according to the quantization accuracy.

[0038] It should be understood that in the embodiment of the present application, mapping the activation value and the weight value to the activation value and the weight value of the target type can be achieved by discretizing continuous decimals into approximate integers.

[0039] For example, assuming that an activation value is 0.125677, a weight value is 2.186511, and the quantization accuracy is 1 decimal place, then it can be quantized to 0.1 and 2.2 to obtain the quantized activation value and weight value.

[0040] It should be noted that the above is only an optional model exemplified in the embodiment of the present application to map activation values ​​and weight values ​​to activation values ​​and weight values ​​of the target type. In addition, the embodiment of the present application can also be implemented in various other quantization methods, which are not limited here.

[0041] S103: updating the model parameters of the parallel model to the quantized parameters, and inputting the input data of the original model in this round of training into the parallel model for training.

[0042] It should be noted that in the embodiments of the present application, the parallel model should be consistent with the model structure of the original model, so that the effect after the model parameters are quantized can be fully simulated.

[0043] It should be understood that in the embodiments of the present application, the original model and the parallel model can be any known or future model structure, such as a deep neural network model, and there is no limitation on this in the present application.

[0044] In an embodiment of the present application, after the model parameters of the parallel model are updated to quantized parameters, the input data of the original model in this round of training (i.e., the training samples input to the original model in this round of training) will also be input into the parallel model, so that the parallel model can simulate the execution effect after quantization.

[0045] S104: Determine a comprehensive loss value of this round of training based on the loss value of the original model in this round of training and the loss value of the parallel model in this round of training.

[0046] It should be understood that when the model is trained, a loss function is used to calculate the loss value of each round of model output, so that the training effect is determined based on the loss value.

[0047] In the embodiment of the present application, similarly, a corresponding loss function is set for the original model and the parallel model, so that the loss value of the original model and the loss value of the parallel model are obtained during this round of training according to the loss functions corresponding to the original model and the parallel model respectively.

[0048] It should be understood that in an optional implementation of the embodiment of the present application, the loss functions corresponding to the original model and the parallel model can be the same, thereby ensuring that the loss values ​​of the original model and the parallel model have the same evaluation dimension, thereby making the two loss values ​​more comprehensive.

[0049] Of course, the loss functions corresponding to the original model and the parallel model may also be different, which is not limited in the embodiments of the present application.

[0050] In a feasible implementation of the embodiment of the present application, the loss value of the original model in this round of training and the loss value of the parallel model in this round of training can be weighted and summed to obtain the comprehensive loss value of this round of training.

[0051] For example, the comprehensive loss value of this round of training can be obtained according to the formula L3=αL1+(1-α)L2, where L1 is the loss value of the original model in this round of training, L2 is the loss value of the parallel model in this round of training, L3 is the comprehensive loss value of this round of training, and α is a preset constant greater than 0 and less than 1.

[0052] It should be understood that in the above example, the value of α can be set by the engineer according to actual needs, for example, it can be set to 0.6.

[0053] It should also be understood that in the embodiments of the present application, in addition to obtaining the comprehensive loss value in this round of training by weighted summing the loss value of the original model in this round of training and the loss value of the parallel model in this round of training, the comprehensive loss value in this round of training can also be obtained by other means, which are not limited in the present application.

[0054] S105: When the comprehensive loss value does not meet the preset termination condition, the model parameters of the original model are updated.

[0055] It should be understood that in the embodiment of the present application, the preset termination condition may be that the comprehensive loss value converges, or the comprehensive loss value is less than a preset loss threshold, etc.

[0056] In an embodiment of the present application, when the comprehensive loss value does not meet the preset termination condition, the model parameters of the original model can be updated by back propagation.

[0057] After updating the model parameters of the original model, the next round of training is carried out. During the training process, the above steps are repeated. Figure 1 The steps shown.

[0058] In the embodiment of the present application, when the comprehensive loss value meets the preset end condition, it indicates that a model that meets the expectations has been obtained. At this time, the model parameters in the original model can be replaced with the quantization parameters in the last round of training, and then the original model whose model parameters are the quantization parameters can be directly output to obtain the quantization model required by the user.

[0059] In addition, since the model structure of the parallel model is consistent with that of the original model, the parallel model can also be directly output when the comprehensive loss value meets the preset end condition. At this time, since the model parameters in the parallel model are the quantization parameters in the last round of training, the parallel model at this time is the quantization model required by the user. Therefore, by directly outputting the parallel model at the end of training, the quantization model required by the user can be directly obtained.

[0060] In addition, in the embodiment of the present application, when the comprehensive loss value meets the preset end condition, the trained original model can also be output. Therefore, subsequent users or engineers can quantize the original model when a quantized model is needed according to actual needs, and can directly use the unquantized original model when a quantized model is not needed, so that a more flexible use effect can be achieved.

[0061] It should be understood that the model training method provided in the embodiments of the present application can be executed and implemented by electronic devices with data processing capabilities (such as computers, servers, etc.), or by electronic devices with data processing capabilities (such as server clusters, etc.), without limitation here.

[0062] It is also necessary to understand that the model training method provided in the embodiments of the present application can be applied to various scenarios with model usage and training needs, for example, it can be applied to various AI scenarios such as face recognition, defect detection, intelligent driving, and intelligent voice, providing higher quantitative model accuracy for models in various AI scenarios.

[0063] The model training method provided in the embodiment of the present application uses a parallel model with the same structure as the original model to execute the quantization parameters obtained in each training during the training process, and then determines a comprehensive loss value based on the loss values ​​of the two models, and then updates the model parameters of the original model based on the comprehensive loss value. In this way, the loss caused by the quantization of the model parameters can be taken into account during the training process, so that the loss of the model obtained by the final training is within an acceptable range after the model parameters are quantized, thereby alleviating the problem that the model obtained by the existing method will have a certain degree of accuracy loss after being quantized.

[0064] Embodiment 2:

[0065] Based on the first embodiment, this embodiment takes a process of quantizing an original model with a data type of fp32 into a quantized model with a data type of int8 as an example to further illustrate this application.

[0066] See also Figure 2 As shown, Figure 2 This is the overall architecture diagram of the solution of this embodiment.

[0067] In any training epoch:

[0068] First, the training sample is input into the original model to obtain the floating-point activation value and floating-point weight value of the original model, and the loss value L1 is obtained through the loss function.

[0069] Then, the floating-point activation values ​​and floating-point weight values ​​are quantized, and the parallel model is updated with the quantized activation values ​​and weight values.

[0070] Among them, the model structure of the parallel model is completely consistent with the model structure of the original model.

[0071] Then, the training samples of this round that were input into the original model are also input into the parallel model, and the loss value L2 is obtained through the loss function.

[0072] Next, L1 and L2 are weighted and calculated as follows:

[0073] L3 = αL1 + (1-α)L2.

[0074] Among them, α is the quantization loss factor, which indicates the importance of the quantization operation on the loss function of the original model.

[0075] Finally, according to the loss value L3, it is determined whether the loss value converges.

[0076] If convergence has not occurred, the weight values ​​of the original model are optimized through back propagation, and training continues according to the above process.

[0077] If converged, the parallel model obtained during the last round of training is output.

[0078] In this way, the purpose of quantizing the weight value can be achieved and the precision loss can be maintained at a certain value.

[0079] Embodiment three:

[0080] Based on the same inventive concept, the present application also provides a model training device 300. Figure 3 As shown, Figure 3 Shows the use of Figure 1 The model training device of the method shown. It should be understood that the specific functions of the device 300 can be referred to the description above. To avoid repetition, the detailed description is appropriately omitted here. The device 300 includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system of the device 300. Specifically:

[0081] See also Figure 3 As shown, the device 300 includes: an acquisition module 301, a quantization module 302, a parallel model control module 303 and a model updating module 304. Among them:

[0082] The acquisition module 301 is used to obtain the model parameters of the original model obtained in this round of training;

[0083] The quantization module 302 is used to quantize the model parameters to obtain quantization parameters;

[0084] The parallel model control module 303 is used to update the model parameters of the parallel model to the quantization parameters, and input the input data of the original model in this round of training into the parallel model for training; wherein the model structure of the parallel model is consistent with that of the original model;

[0085] The model updating module 304 is used to determine the comprehensive loss value of this round of training based on the loss value of the original model in this round of training and the loss value of the parallel model in this round of training; when the comprehensive loss value does not meet the preset end condition, the model parameters of the original model are updated.

[0086] In an embodiment of the present application, the model training device 300 may further include an output module for outputting the original model whose model parameters are the quantization parameters, or outputting the parallel model when the comprehensive loss value meets a preset termination condition.

[0087] In an embodiment of the present application, the comprehensive loss value is a value obtained by weighted summing the loss value of the original model in this round of training and the loss value of the parallel model in this round of training.

[0088] In an embodiment of the present application, the loss function adopted by the original model and the parallel model is the same.

[0089] In a feasible implementation of an embodiment of the present application, the model parameters include activation values ​​and weight values; the quantization module 302 is specifically used to map the activation values ​​and weight values ​​to activation values ​​and weight values ​​of a target type to obtain the quantization parameters; wherein the target type is a pre-set data type of the quantization parameters.

[0090] In the above feasible implementation manner, the quantization module 302 is specifically used to map the activation value and weight value to the activation value and weight value of the target type according to a preset quantization accuracy.

[0091] It should be understood that, for the sake of brevity, some of the contents described in the first embodiment will not be repeated in this embodiment.

[0092] Embodiment 4:

[0093] This embodiment provides an electronic device, see Figure 4 As shown, it includes a processor 401, a memory 402 and a communication bus 403. Among them:

[0094] The communication bus 403 is used to realize the connection and communication between the processor 401 and the memory 402 .

[0095] The processor 401 is used to execute one or more first programs stored in the memory 402 to implement the model training method in the above-mentioned embodiment 1 and / or embodiment 2.

[0096] Understandably, Figure 4 The structure shown is for illustration only. The electronic device may also include Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown.

[0097] It should be noted that the electronic device described in the embodiments of the present application may be a device with data processing capabilities such as a computer, a server, etc.

[0098] This embodiment also provides a computer-readable storage medium, such as a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash disk, an SD (Secure Digital Memory Card) card, an MMC (Multimedia Card) card, etc., in which one or more programs for implementing the above steps are stored, and the one or more programs can be executed by one or more processors to implement the model training method in the above embodiment 1 and / or embodiment 2. No further details will be given here.

[0099] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0100] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0102] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0103] As used herein, a plurality refers to two or more than two.

[0104] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: include: Obtaining model parameters of the original model used in the target scene obtained in this round of training; the target scene includes a face recognition scene; quantizing the model parameters to obtain quantized parameters; Updating the model parameters of the parallel model to the quantized parameters, and inputting the input data of the original model in this round of training into the parallel model for training; wherein the model structure of the parallel model is consistent with that of the original model; Determine a comprehensive loss value for this round of training based on the loss value of the original model in this round of training and the loss value of the parallel model in this round of training; When the comprehensive loss value does not meet the preset termination condition, the model parameters of the original model are updated.

2. The model training method according to claim 1, characterized in that: The method further comprises: When the comprehensive loss value meets the preset termination condition, the output model parameter is the original model of the quantization parameter, or the parallel model is output.

3. The model training method according to claim 1, characterized in that: The comprehensive loss value is a value obtained by weighted summing the loss value of the original model in this round of training and the loss value of the parallel model in this round of training.

4. The model training method according to any one of claims 1 to 3, characterized in that: The original model and the parallel model use the same loss function.

5. The model training method according to any one of claims 1 to 3, characterized in that: The model parameters include activation values ​​and weight values; The model parameters are quantized to obtain quantized parameters, including: The activation value and the weight value are mapped to the activation value and the weight value of the target type to obtain the quantization parameter; wherein the target type is a preset data type of the quantization parameter.

6. The model training method according to claim 5, characterized in that: Mapping the activation value and the weight value to the activation value and the weight value of the target type includes: According to the preset quantization accuracy, the activation value and the weight value are mapped to the activation value and the weight value of the target type.

7. A model training device, characterized in that: include: Acquisition module, quantization module, parallel model control module and model update module; The acquisition module is used to acquire the model parameters of the original model used in the target scene obtained in this round of training; the target scene includes a face recognition scene; The quantization module is used to quantize the model parameters to obtain quantization parameters; The parallel model control module is used to update the model parameters of the parallel model to the quantization parameters, and input the input data of the original model in this round of training into the parallel model for training; wherein the model structure of the parallel model is consistent with that of the original model; The model updating module is used to determine the comprehensive loss value of this round of training based on the loss value of the original model in this round of training and the loss value of the parallel model in this round of training; when the comprehensive loss value does not meet the preset end condition, the model parameters of the original model are updated.

8. The model training device according to claim 7, characterized in that: The model training device also includes: An output module is used to output the model parameters as the original model of the quantization parameters, or output the parallel model when the comprehensive loss value meets the preset end condition.

9. An electronic device, characterized in that: include: Processor, memory and communication bus; The communication bus is used to realize the connection and communication between the processor and the memory; The processor is used to execute one or more programs stored in the memory to implement the model training method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the model training method as described in any one of claims 1 to 6.

Images