Model checkpoint parameter domain average method, device, electronic equipment and storage medium

Abstract

The application provides a model checkpoint parameter domain average method and device, electronic equipment and storage medium, the method relates to the artificial intelligence technical field, and comprises the following steps: in the training process of a to-be-trained deep neural network model, a plurality of checkpoints and a plurality of model parameters corresponding to the plurality of checkpoints are saved; the checkpoint is used to indicate the model at different stages in the training process; the total loss value of the loss function of the model corresponding to each checkpoint is determined; based on the total loss value, a to-be-averaged checkpoint is determined from the plurality of checkpoints; the plurality of model parameters corresponding to the to-be-averaged checkpoint are averaged to obtain the parameter average value of the model. The method provided by the application combines the total loss value of the loss function of the model on the training set and the validation set, considers the bias and variance of the model, and improves the performance of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model checkpoint parameter domain averaging method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, deep learning has become one of the driving forces for the progress of artificial intelligence technology. However, the non-convex property of deep neural network models based on deep learning makes the model eventually converge to only a local optimum point; moreover, due to the randomness of model initialization and the randomness introduced by gradient descent in the model training process, the loss value of the model fluctuates constantly during the training process; especially when the model has approached convergence, the loss value of the model will have a large number of local minimum points. Therefore, how to use these local optimal points to improve the performance of the model is an important task in the field of deep learning.

[0003] The training process of a deep learning model requires multiple rounds of training data (Epoch), and during the training process, checkpoints (Checkpoint) can be set as needed to save the model training parameters at the current stage to evaluate the performance of the model at that checkpoint.

[0004] In related technologies, model averaging based on checkpoint parameter domains has become a basic configuration method for improving the performance of deep learning models; among them, the mainstream methods include: parameter averaging of the last K checkpoints (Last K-Checkpoint Averaging, LKCA) and parameter averaging of the K best checkpoints (K-Best Checkpoint Averaging, KBCA); among them, the LKCA strategy is generally used together with the Early Stop strategy, and the selected checkpoints are the K consecutive checkpoint models close to convergence on the training set for averaging, while the BKCA strategy selects the K checkpoint models with the smallest loss function values on the validation set for averaging. Since the loss function value on the training set represents the fitting degree of the model to the training data, the smaller the value, the better the fitting, which usually means that the bias of the model is smaller, and the loss function value on the validation set represents the generalization ability of the model, the smaller the value, the better the generalization ability, which usually means that the variance of the model is smaller. According to the definition of the LKCA method, its essence is to select K checkpoints with smaller bias for averaging, while the KBCA method essentially tends to select K checkpoints with smaller variance for averaging. According to the bias-variance decomposition theory, the final performance of the model is determined by both the variance and the bias.

[0005] However, LKCA mainly considers bias, and KBCA mainly considers variance, neither of which considers bias and variance of the model at the same time, so that the performance of the model is poor. SUMMARY

[0006] The application provides a model checkpoint parameter domain average method, device, electronic equipment and storage medium, which aims to solve the poor performance of the model in the prior art and improve the performance of the model.

[0007] The application provides a model checkpoint parameter domain average method, which comprises the following steps:

[0008] During the training process of a to-be-trained deep neural network model, a plurality of checkpoints and a plurality of model parameters corresponding to the plurality of checkpoints are saved; the checkpoint is used to represent the model at different stages in the training process;

[0009] The total loss value of the loss function of the model corresponding to each checkpoint is determined;

[0010] Based on the total loss value, a to-be-averaged checkpoint is determined from the plurality of checkpoints;

[0011] The parameter average of the plurality of model parameters corresponding to the to-be-averaged checkpoint is performed to obtain the parameter average value of the model.

[0012] According to the model checkpoint parameter domain average method provided by the application, the total loss value of the loss function of the model corresponding to each checkpoint is determined, which comprises the following steps:

[0013] The loss value of the loss function of each checkpoint on the training set and the validation set is calculated respectively;

[0014] Based on the loss value of the loss function of each checkpoint on the training set and the validation set, the total loss value of the loss function of the model corresponding to each checkpoint is determined.

[0015] According to the model checkpoint parameter domain average method provided by the application, the total loss value of the loss function of the model corresponding to each checkpoint is determined based on the loss value of the loss function of each checkpoint on the training set and the validation set, which comprises the following steps:

[0016] Based on the loss value of the loss function of each checkpoint on the validation set, the sum of the loss values of the loss functions corresponding to the continuous K checkpoints is determined; the K is a positive integer;

[0017] Based on the sum of the loss values of the loss functions, the continuous K checkpoints with the minimum sum of the loss values of the loss functions are determined;

[0018] Determine a total loss value of a loss function of the model corresponding to each of the checkpoints based on the K consecutive checkpoints.

[0019] According to the model checkpoint parameter domain average method provided by the application, the total loss value of the loss function of the model corresponding to each of the checkpoints is determined based on the K consecutive checkpoints, and the method comprises the following steps:

[0020] Calculate a contribution rate of the first checkpoint based on a loss value of the loss function of the first checkpoint on the validation set and a loss value of the loss function of the first checkpoint on the training set.

[0021] Calculate a penalty factor of each of the checkpoints according to a current position of each of the checkpoints.

[0022] Calculate the total loss value of the loss function of the model corresponding to each of the checkpoints based on the contribution rate and the penalty factor.

[0023] According to the model checkpoint parameter domain average method provided by the application, the total loss value is used to determine the to-be-averaged checkpoint from the plurality of checkpoints.

[0024] Determine a sum of the total loss values corresponding to W consecutive checkpoints based on the total loss values; the W is a positive integer.

[0025] Determine the W consecutive checkpoints with the smallest sum of the total loss values based on the sum of the total loss values.

[0026] Determine the to-be-averaged checkpoint from the plurality of checkpoints based on the W consecutive checkpoints.

[0027] According to the model checkpoint parameter domain average method provided by the application, the plurality of model parameters corresponding to the to-be-averaged checkpoint are parameter-averaged to obtain a parameter average value of the model.

[0028] Determine a plurality of model parameters corresponding to each of the to-be-averaged checkpoints.

[0029] Calculate an average value of the plurality of model parameters based on the plurality of model parameters to obtain the parameter average value of the model.

[0030] The application further provides a model checkpoint parameter domain average device, comprising:

[0031] A storage module is configured to save a plurality of checkpoints and a plurality of model parameters corresponding to the plurality of checkpoints during a training process of a to-be-trained deep neural network model; the checkpoints are used to represent models at different stages in the training process.

[0032] The first determination module is configured to determine a total loss value of a loss function of the model corresponding to each of the checkpoints;

[0033] The second determination module is configured to determine a to-be-averaged checkpoint from the plurality of checkpoints based on the total loss value.

[0034] The parameter averaging module is configured to perform parameter averaging on a plurality of model parameters corresponding to the to-be-averaged checkpoint to obtain a parameter average value of the model.

[0035] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for model checkpoint parameter domain averaging according to any one of the above when executing the program.

[0036] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the method for model checkpoint parameter domain averaging according to any one of the above.

[0037] The present application also provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the method for model checkpoint parameter domain averaging according to any one of the above.

[0038] The present application provides a method, device, electronic device and storage medium for model checkpoint parameter domain averaging, wherein a plurality of checkpoints and a plurality of model parameters corresponding to the plurality of checkpoints are saved during the training of a to-be-trained deep neural network model, then a total loss value of a loss function of the model is determined according to each checkpoint, a to-be-averaged checkpoint is determined from the plurality of checkpoints, and a parameter average value of the model is obtained by performing parameter averaging on a plurality of model parameters corresponding to the to-be-averaged checkpoint. The method provided by the present application combines the total loss value of the loss function of the model on the training set and the validation set, and considers the bias and variance of the model, thereby improving the performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0040] Figure 1 is one of the flowcharts of the method for model checkpoint parameter domain averaging provided by the present application;

[0041] Figure 2is the second flowchart of the model checkpoint parameter domain averaging method provided by the application;

[0042] Figure 3 is the third flowchart of the model checkpoint parameter domain averaging method provided by the application;

[0043] Figure 4 is the fourth flowchart of the model checkpoint parameter domain averaging method provided by the application;

[0044] Figure 5 is the structural diagram of the model checkpoint parameter domain averaging device provided by the application;

[0045] Figure 6 is the structural diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described in detail below with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0047] The model checkpoint parameter domain averaging method provided by the present application will be described in detail below with reference to the drawings and some embodiments and their application scenarios.

[0048] The present application provides a model checkpoint parameter domain averaging method, which is suitable for performance evaluation scenarios of deep learning network models. In the training process of a to-be-trained deep neural network model, multiple checkpoints and multiple model parameters corresponding to the multiple checkpoints are saved. The checkpoints are used to represent the model at different stages in the training process. The total loss value of the loss function of the model corresponding to each checkpoint is determined. Based on the total loss value, a to-be-averaged checkpoint is determined from the multiple checkpoints. The multiple model parameters corresponding to the to-be-averaged checkpoint are averaged to obtain the parameter average value of the model. The method provided by the present application combines the total loss value of the loss function of the model on the training set and the validation set, and considers the bias and variance of the model, thereby improving the performance of the model.

[0049] The model checkpoint parameter domain averaging method of the present application will be described below. Figures 1-4 The model checkpoint parameter domain averaging method of the present application will be described below.

[0050] Figure 1 is the first flowchart of the model checkpoint parameter domain averaging method provided by the application, as Figure 1As shown, the method comprises steps 101-104, wherein:

[0051] Step 101, in the training process of the to-be-trained deep neural network model, a plurality of checkpoints and a plurality of model parameters corresponding to the plurality of checkpoints are saved; the checkpoint is used to represent the model at different stages in the training process.

[0052] It should be noted that the model checkpoint parameter domain averaging method provided by the present application can be applied to the performance evaluation scene of the deep learning network model. The execution subject of the method can be a model checkpoint parameter domain averaging device, such as an electronic device, or a control module in the model checkpoint parameter domain averaging device for executing the model checkpoint parameter domain averaging method.

[0053] Specifically, an arbitrary to-be-trained deep neural network model is selected, and the selected to-be-trained deep neural network model is configured with hyperparameters for model training. For example, a speech recognition model (Conformer) based on an open source toolkit (WeNet) in the speech recognition field is selected as the to-be-trained deep neural network model; in order to verify more possibilities, the number of iterations (Epoch) for model training is configured to be 360 times, and the settings of the remaining hyperparameters are all in accordance with the default hyperparameters of the WeNet toolkit.

[0054] In practice, training data, an optimizer, and a loss function are selected for the to-be-trained deep neural network model, the selected to-be-trained deep neural network model is trained using the training data, and according to the configured hyperparameters, a plurality of checkpoints and a plurality of model parameters corresponding to each checkpoint are saved during the training of the to-be-trained deep neural network model; wherein the checkpoint is used to represent the model at different stages in the training process, one checkpoint corresponds to a plurality of model parameters, a plurality of checkpoints correspond to a plurality of model parameters, the number of checkpoints can be a positive integer, for example, the number of checkpoints is 360; the number of iterations for model training is the same as the number of checkpoints, that is, the model completes one iteration in the training process, saves one checkpoint and a plurality of model parameters corresponding to the checkpoint, each checkpoint uses a corresponding subscript to represent the position of the current checkpoint when saved, for example, the first checkpoint uses subscript 1, the second checkpoint uses subscript 2, the third checkpoint uses subscript 3, and so on, until all the checkpoints are saved.

[0055] Optionally, the training data uses the training set of AISHELL-1 commonly used in the speech recognition field as the training data, the validation set as the validation data, and the test set as the test data, the selected optimizer is the adaptive moment estimation (Adam) optimizer, the loss function is the cross-entropy with label smoothing, and other related configurations refer to the default configurations of WeNet.

[0056] For example, for a data set with 2000 training samples, the 2000 training samples are divided into batches of 500, and the selected deep neural network model to be trained is trained. During the model training process, the data set of 2000 training samples is trained once, completing an epoch, at which time a checkpoint and the multiple model parameters of the model corresponding to the checkpoint are saved. In the embodiments of the present application, a total of 360 epochs are trained, and a checkpoint and the multiple model parameters of the model corresponding to the checkpoint are saved every time an epoch is completed. Specifically, the training process is as follows: first, the selected deep learning model to be trained is trained using the 500 training samples in the first batch, then the deep learning model to be trained is further trained using the 500 training samples in the second batch, then the deep learning model to be trained is further trained using the 500 training samples in the third batch, and finally the deep learning model to be trained is further trained using the 500 training samples in the fourth batch. When the 500 training samples in the fourth batch are trained, the first checkpoint and the multiple model parameters of the model corresponding to the first checkpoint are saved. According to the training process described above, the data set of 2000 training samples is trained for the second time, and when the second training is completed, the second checkpoint and the multiple model parameters of the model corresponding to the second checkpoint are saved. Similarly, the third checkpoint and the multiple model parameters of the model corresponding to the third checkpoint are saved, and so on, until the data set of 2000 training samples is trained for 360 times, and a total of 360 checkpoints and the multiple model parameters of the model corresponding to the 360 checkpoints are saved.

[0057] Step 102, determining the total loss value of the loss function of the model corresponding to each checkpoint.

[0058] Specifically, during the training of the deep learning model to be trained, multiple checkpoints are saved, and the total loss value of the loss function of the model corresponding to each checkpoint is determined according to the loss function of the deep learning model.

[0059] Step 103, determining the checkpoint to be averaged from the multiple checkpoints based on the total loss value.

[0060] Specifically, the checkpoint to be averaged is determined from the saved multiple checkpoints according to the determined total loss value of the loss function of the deep learning model corresponding to each checkpoint.

[0061] Step 104, performing parameter averaging on the multiple model parameters corresponding to the checkpoint to be averaged to obtain the parameter average value of the model.

[0062] Specifically, according to the determined checkpoint to be averaged, parameter averaging is performed on the plurality of model parameters corresponding to the checkpoint to be averaged, and a parameter average value of the deep learning model is obtained, so that the final model can be obtained.

[0063] It should be noted that the model checkpoint parameter domain averaging method provided by the present application can be widely applied to deep learning tasks such as speech recognition, machine translation, and character recognition, and can be used in combination with the model checkpoint parameter domain averaging method provided by the present application, or can be used in multiple models and multiple scenarios, so as to make the model achieve better performance, and therefore has high application value.

[0064] The model checkpoint parameter domain averaging method provided by the present application saves a plurality of checkpoints and a plurality of model parameters corresponding to the plurality of checkpoints respectively during the training process of the to-be-trained deep neural network model, then determines the total loss value of the loss function of the model according to each checkpoint, determines the checkpoint to be averaged from the plurality of checkpoints, and then performs parameter averaging on the plurality of model parameters corresponding to the checkpoint to be averaged to obtain the parameter average value of the model. The method provided by the present application combines the total loss value of the loss function of the model on the training set and the validation set, and considers the bias and variance of the model, thereby improving the performance of the model.

[0065] Optionally, the specific implementation of the above step 102 comprises the following steps:

[0066] Step 1) respectively calculate the loss values of the loss function of each checkpoint on the training set and the validation set;

[0067] Step 2) determine the total loss value of the loss function of the model corresponding to each checkpoint based on the loss values of the loss function of each checkpoint on the training set and the validation set.

[0068] Specifically, in the process of training the deep learning model, the loss values of the loss function of each checkpoint on the training set and the validation set are calculated according to the loss function of the deep learning model, and the loss value set T n :{t0,…,t n} corresponding to the training set and the loss value set V n :{v0,…,v n} corresponding to the validation set are obtained, wherein n represents the number of iterations, the loss value t i of the training set and the loss value v i of the validation set corresponding to the i-th iteration, respectively.

[0069] The model checkpoint parameter domain average method provided by the application determines the total loss value of the loss function of the model corresponding to each checkpoint by calculating the loss value of the loss function of each checkpoint on the training set and the validation set, combines the total loss value of the loss function of the model on the training set and the validation set, and considers the bias and variance of the model, thereby improving the performance of the model.

[0070] Based on the description of the above embodiments, the specific implementation of determining the total loss value of the loss function of the model corresponding to each checkpoint according to the loss value of the loss function of each checkpoint on the training set and the validation set is further described as follows:

[0071] Step 1) Based on the loss value of the loss function of each checkpoint on the validation set, the sum of the loss values of the loss function of the consecutive K checkpoints is determined; K is a positive integer.

[0072] Specifically, according to the loss value of the loss function of each checkpoint on the validation set, the loss values of the loss function of the consecutive K checkpoints on the validation set are summed to determine the sum of the loss values of the loss function of the consecutive K checkpoints.

[0073] Step 2) Based on the sum of the loss values of the loss function, the consecutive K checkpoints with the minimum sum of the loss values of the loss function are determined.

[0074] Specifically, according to the sum of the loss values of the loss function of the consecutive K checkpoints, the minimum sum of the loss values is determined, and then the consecutive K checkpoints corresponding to the minimum sum of the loss values are determined according to the minimum sum of the loss values, and the index of the first checkpoint in the consecutive K checkpoints is recorded as s.

[0075] Step 3) Based on the consecutive K checkpoints, the total loss value of the loss function of the model corresponding to each checkpoint is determined.

[0076] Optionally, the determination of the total loss value of the loss function of the model corresponding to each checkpoint based on the consecutive K checkpoints comprises:

[0077] Based on the loss value of the loss function of the first checkpoint in the consecutive K checkpoints on the validation set and the loss value of the loss function of the first checkpoint on the training set, the contribution rate of the first checkpoint is calculated;

[0078] According to the current position of each checkpoint, the penalty factor of each checkpoint is calculated;

[0079] Based on the contribution rate and the penalty factor, the total loss value of the loss function of the model corresponding to each checkpoint is calculated.

[0080] Specifically, the contribution rate of the s-th checkpoint is calculated according to the loss value of the loss function of the s-th checkpoint on the validation set and the loss value of the loss function of the s-th checkpoint on the training set among the first continuous K checkpoints; wherein the calculation formula of the contribution rate adopts the following formula (1):

[0081] f = (v s x lambda) / t s (1)

[0082] Wherein, f represents the contribution rate of the s-th checkpoint, v s represents the loss value of the loss function of the s-th checkpoint on the validation set, lambda represents an adjustable hyperparameter, t s represents the loss value of the loss function of the s-th checkpoint on the training set. In the embodiment of the application, the value of lambda is set to 0.1.

[0083] In practice, the penalty factor of each checkpoint is calculated according to the current position of each checkpoint; wherein the subscript i is used as the position of each checkpoint when saving, if i < s, then p = 1, otherwise p = e 1-i / s , wherein p is the penalty factor of the i-th checkpoint, i is the subscript of the current checkpoint, and e is the natural base.

[0084] Then, the total loss value of the loss function of the model corresponding to each checkpoint is calculated according to the contribution rate of the s-th checkpoint and the penalty factor of each checkpoint, and a total loss value set A n : {a0, …, a n} is generated, wherein a i is the total loss value of the loss function corresponding to the i-th checkpoint, and the total loss value of the loss function is specifically calculated according to the following formula (2):

[0085] a i = t i x f x p + v i (2)

[0086] Wherein, t i represents the loss value of the loss function of the i-th checkpoint on the training set, f represents the contribution rate of the s-th checkpoint, p is the penalty factor of the i-th checkpoint, and v i represents the loss value of the loss function of the i-th checkpoint on the validation set.

[0087] The model checkpoint parameter domain average method provided by the application determines the sum of loss values of loss functions corresponding to consecutive K checkpoints according to the loss values of the loss functions on the verification set of each checkpoint, and then determines the consecutive K checkpoints with the minimum sum of loss values of the loss functions according to the sum of loss values of the loss functions corresponding to the consecutive K checkpoints. The contribution rate of the first checkpoint in the consecutive K checkpoints is calculated according to the loss value of the loss function on the verification set of the first checkpoint and the loss value of the loss function on the training set. The penalty factor of each checkpoint is calculated according to the current position of each checkpoint. The total loss value of the loss function of the model corresponding to each checkpoint is calculated according to the contribution rate and the penalty factor, which combines the bias and variance of the model and explicitly adjusts the combination of the bias and the variance, thereby improving the performance of the model and achieving higher performance.

[0088] Figure 2 Figure 2 is a flowchart of the model checkpoint parameter domain average method provided by the application, as shown in the figure, the method comprises steps 201-205, wherein: Figure 2

[0089] Step 201, during the training process of a to-be-trained deep neural network model, a plurality of checkpoints and a plurality of model parameters corresponding to the plurality of checkpoints are saved. The checkpoints are used to represent the model at different stages in the training process.

[0090] Step 202, the total loss value of the loss function of the model corresponding to each checkpoint is determined.

[0091] Optionally, the description and explanation of steps 201-202 can refer to the description and explanation of steps 101-102 described above, and the same technical effects can be achieved. To avoid repetition, it will not be described here.

[0092] Step 203, based on the total loss value, the sum of the total loss values corresponding to consecutive W checkpoints is determined; W is a positive integer.

[0093] Specifically, the sum of the total loss values corresponding to consecutive W checkpoints is obtained by summing the total loss values corresponding to the consecutive W checkpoints according to the total loss values of the loss function of the model corresponding to each checkpoint.

[0094] It should be noted that W and K can be the same or different. When W and K are the same, it means that the number of consecutive checkpoints is the same. When W and K are different, it means that the number of consecutive checkpoints is different, and in this case, W is less than or equal to K.

[0095] Step 204, based on the sum of the total loss values, the consecutive W checkpoints with the minimum sum of the total loss values are determined.

[0096] ​Specifically, according to the sum of total loss values corresponding to the continuous W checkpoints respectively, the continuous W checkpoints with the minimum sum of total loss values are determined, and the continuous W checkpoint set is {u, u+1,..., u+w-1}, wherein the first checkpoint in the continuous W checkpoints with the minimum sum of total loss values is u, and the formula (3) is as follows:

[0097]

[0098] Wherein, u represents the index of the first checkpoint in the continuous W checkpoints with the minimum sum of total loss values, a j is the total loss value of the loss function corresponding to the jth checkpoint.

[0099] Step 205, determining the to-be-averaged checkpoint from the plurality of checkpoints based on the continuous W checkpoints.

[0100] Specifically, the to-be-averaged checkpoint is determined from the plurality of checkpoints according to the determined continuous W checkpoints with the minimum sum of total loss values, that is, the to-be-averaged checkpoint is the determined continuous W checkpoints with the minimum sum of total loss values.

[0101] The model checkpoint parameter domain averaging method provided by the application calculates the sum of total loss values corresponding to the continuous W checkpoints respectively through the total loss values of the loss function of the model corresponding to each checkpoint, and then determines the continuous W checkpoints with the minimum sum of total loss values according to the sum of total loss values, and further determines the to-be-averaged checkpoint. The method provided by the application combines the total loss values of the loss function of the model on the training set and the validation set, considers the bias and variance of the model, and improves the performance of the model.

[0102] Figure 3 is a third flowchart of the model checkpoint parameter domain averaging method provided by the application, as shown in the figure, the method comprises steps 301-305, wherein: Figure 3

[0103] Step 301, during the training process of the to-be-trained deep neural network model, a plurality of checkpoints and a plurality of model parameters corresponding to the plurality of checkpoints are saved; the checkpoint is used to represent the model at different stages in the training process;

[0104] Step 302, determining the total loss value of the loss function of the model corresponding to each checkpoint;

[0105] Step 303, determining the to-be-averaged checkpoint from the plurality of checkpoints based on the total loss value;

[0106] ​Optionally, the description and explanation about steps 301-303 can refer to the above description and explanation for steps 101-103, and the same technical effects can be achieved, and thus the description is not repeated here.

[0107] Step 304, determining the multiple model parameters corresponding to the to-be-averaged checkpoints respectively.

[0108] Specifically, according to the to-be-averaged checkpoints determined from the multiple checkpoints, the multiple model parameters corresponding to the to-be-averaged checkpoints respectively are determined from the multiple model parameters corresponding to the saved checkpoints respectively.

[0109] Step 305, calculating the average of the multiple model parameters based on the multiple model parameters, to obtain the parameter average of the model.

[0110] Specifically, according to the multiple model parameters corresponding to the to-be-averaged checkpoints respectively, the multiple model parameters corresponding to the checkpoints of {u, u+1, …, u+w-1} are selected for averaging, to obtain the parameter average of the model, thereby obtaining the final model; wherein the implementation of the specific parameter average is shown in the following formula (4):

[0111]

[0112] Wherein, M i represents the multiple model parameters corresponding to the i-th checkpoint in the continuous W checkpoints.

[0113] It should be noted that the model checkpoint parameter domain averaging method provided by the present application is a post-processing step after model training, which only needs to perform parameter averaging on the multiple model parameters corresponding to the multiple checkpoint distributions of the model, without additional data and other resources, and has low cost and good effect.

[0114] Optionally, the final model obtained by the model checkpoint parameter domain average method provided by the present application is compared with the existing LKCA, KBCA and the final model obtained by the model checkpoint parameter domain average method provided by the present application on the AISHELL-1 dataset, and the used models are transformer (Transformer), S-Transformer, Conformer and S-Conformer, wherein the Transformer is a mainstream model in speech recognition, the S-Transformer is a model used in a streaming scenario, the Conformer is an advanced model in speech recognition, and the S-Conformer is a model used in a streaming scenario. As shown in Table 1, the results of the final model obtained by the LKCA, KBCA and the model checkpoint parameter domain average method provided by the present application in the word error rate recognition aspect show that the model obtained by the method of the present application is better than the results obtained by the existing LKCA and KBCA methods in the word error rate aspect, and has certain superiority.

[0115] Table 1. Comparison of results of LKCA, KBCA and the method of the present application (word error rate)

[0116] Model Transformer S-Transformer Conformer S-Conformer LKCA 5.04 6.23 4.68 5.46 KBCA 4.98 6.19 4.53 5.42 Method of the invention 4.91 6.14 4.44 5.35

[0117] It should be noted that the present application mainly aims at the parameter domain average field of the deep learning network model, but also has reference significance for other similar fields.

[0118] The model checkpoint parameter domain average method provided by the present application calculates the average value of the plurality of model parameters according to the plurality of model parameters respectively corresponding to the to-be-averaged checkpoints, obtains the parameter average value of the model, and further obtains the final model, thereby improving the performance of the model.

[0119] Figure 4 is a fourth flowchart of the model checkpoint parameter domain average method provided by the present application, as shown in Figure 4 The method comprises steps 401-412, wherein:

[0120] Step 401, during the training process of the to-be-trained deep neural network model, a plurality of checkpoints and a plurality of model parameters respectively corresponding to the plurality of checkpoints are saved; the checkpoint is used to represent the model at different stages in the training process;

[0121] Step 402, the loss value of the loss function of each checkpoint on the training set and the validation set is calculated respectively;

[0122] Step 403, based on the loss value of the loss function of each checkpoint on the validation set, the sum of the loss values of the loss function corresponding to the continuous K checkpoints is determined; K is a positive integer;

[0123] Step 404, determining the K consecutive checkpoints with the smallest sum of loss values based on the sum of loss values of the loss function;

[0124] Step 405, calculating the contribution rate of the first checkpoint in the K consecutive checkpoints based on the loss value of the loss function of the first checkpoint on the validation set and the loss value of the loss function of the first checkpoint on the training set;

[0125] Step 406, calculating the penalty factor of each checkpoint according to the current position of each checkpoint;

[0126] Step 407, calculating the total loss value of the loss function of the model corresponding to each checkpoint based on the contribution rate of the first checkpoint and the penalty factor of each checkpoint;

[0127] Step 408, determining the sum of the total loss values corresponding to the W consecutive checkpoints based on the total loss values; W is a positive integer;

[0128] Step 409, determining the W consecutive checkpoints with the smallest sum of total loss values based on the sum of total loss values;

[0129] Step 410, determining the to-be-averaged checkpoints from the multiple checkpoints based on the W consecutive checkpoints;

[0130] Step 411, determining the multiple model parameters corresponding to the to-be-averaged checkpoints;

[0131] Step 412, calculating the average of the multiple model parameters to obtain the average of the parameters of the model.

[0132] The model checkpoint parameter domain averaging device provided by the present application is described below. The model checkpoint parameter domain averaging device described below can be referred to in correspondence with the model checkpoint parameter domain averaging method described above.

[0133] Figure 5 is a structural schematic diagram of the model checkpoint parameter domain averaging device provided by the present application, as Figure 5 shown, the model checkpoint parameter domain averaging device 500 includes a storage module 501, a first determination module 502, a second determination module 503, and a parameter averaging module 504, wherein:

[0134] The storage module 501 is configured to save multiple checkpoints and multiple model parameters corresponding to the multiple checkpoints during the training process of a to-be-trained deep neural network model; the checkpoints are used to represent the model at different stages in the training process.

[0135] The first determination module 502 is configured to determine the total loss value of the loss function of the model corresponding to each checkpoint.

[0136] The second determination module 503 is configured to determine a to-be-averaged checkpoint from the plurality of checkpoints based on the total loss value of the loss function of the model.

[0137] The parameter averaging module 504 is configured to perform parameter averaging on the plurality of model parameters corresponding to the to-be-averaged checkpoint to obtain the parameter average value of the model.

[0138] The model checkpoint parameter domain averaging device provided by the application saves a plurality of checkpoints and a plurality of model parameters corresponding to the plurality of checkpoints during the training of the to-be-trained deep neural network model, then determines a to-be-averaged checkpoint from the plurality of checkpoints based on the total loss value of the loss function of the model, performs parameter averaging on the plurality of model parameters corresponding to the to-be-averaged checkpoint, and obtains the parameter average value of the model. The method provided by the application combines the total loss value of the loss function of the model on the training set and the validation set, considers the bias and variance of the model, and thus improves the performance of the model.

[0139] Optionally, the first determination module 502 is specifically configured to:

[0140] respectively calculate the loss value of the loss function of each checkpoint on the training set and the validation set;

[0141] determine the total loss value of the loss function of the model corresponding to each checkpoint based on the loss value of the loss function of each checkpoint on the training set and the validation set.

[0142] Optionally, the first determination module 502 is specifically configured to:

[0143] determine the sum of the loss values of the loss function corresponding to the continuous K checkpoints based on the loss value of the loss function of each checkpoint on the validation set; the K is a positive integer;

[0144] determine the continuous K checkpoints with the minimum sum of the loss values of the loss function based on the sum of the loss values of the loss function;

[0145] determine the total loss value of the loss function of the model corresponding to each checkpoint based on the continuous K checkpoints.

[0146] Optionally, the first determination module 502 is specifically configured to:

[0147] calculate the contribution rate of the first checkpoint in the continuous K checkpoints based on the loss value of the loss function of the first checkpoint on the validation set and the loss value of the loss function of the first checkpoint on the training set;

[0148] calculate the penalty factor of each checkpoint according to the current position of each checkpoint;

[0149] Based on the contribution rate and the penalty factor, calculate the total loss value of the loss function of the model corresponding to each checkpoint.

[0150] Optionally, the second determining module 503 is specifically used for:

[0151] Based on the total loss value, determine the sum of the total loss values ​​corresponding to W consecutive checkpoints; where W is a positive integer.

[0152] Based on the sum of the total loss values, determine the W consecutive checkpoints that have the smallest sum of the total loss values;

[0153] Based on the W consecutive checkpoints, determine the checkpoints to be averaged from the plurality of checkpoints.

[0154] Optionally, the parameter averaging module 504 is specifically used for:

[0155] Determine the multiple model parameters corresponding to the check points to be averaged;

[0156] Based on the multiple model parameters, the average value of the multiple model parameters is calculated to obtain the average parameter value of the model.

[0157] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device 600 may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a model checkpoint parameter domain averaging method. This method includes: during the training process of the deep neural network model to be trained, storing multiple checkpoints and multiple model parameters corresponding to each checkpoint; the checkpoints represent the model at different stages of the training process; determining the total loss value of the loss function of the model corresponding to each checkpoint; based on the total loss value, determining checkpoints to be averaged from the multiple checkpoints; and averaging the multiple model parameters corresponding to the checkpoints to be averaged to obtain the average parameter value of the model.

[0158] In addition, the logic instructions in the memory 630 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0159] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the model checkpoint parameter domain averaging method provided by the above-mentioned method. The method comprises: saving a plurality of checkpoints and a plurality of model parameters respectively corresponding to the plurality of checkpoints in a training process of a to-be-trained deep neural network model; the checkpoint is used to represent a model at different stages in the training process; determining a total loss value of a loss function of the model corresponding to each of the checkpoints; determining a to-be-averaged checkpoint from the plurality of checkpoints based on the total loss value; and performing parameter averaging on the plurality of model parameters corresponding to the to-be-averaged checkpoint to obtain a parameter average value of the model.

[0160] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the model checkpoint parameter domain averaging method provided by the above-mentioned method. The method comprises: saving a plurality of checkpoints and a plurality of model parameters respectively corresponding to the plurality of checkpoints in a training process of a to-be-trained deep neural network model; the checkpoint is used to represent a model at different stages in the training process; determining a total loss value of a loss function of the model corresponding to each of the checkpoints; determining a to-be-averaged checkpoint from the plurality of checkpoints based on the total loss value; and performing parameter averaging on the plurality of model parameters corresponding to the to-be-averaged checkpoint to obtain a parameter average value of the model.

[0161] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0162] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0163] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for averaging parameters in the model checkpoint domain, characterized in that, include: During the training process of the deep neural network model to be trained, multiple checkpoints and multiple model parameters corresponding to the multiple checkpoints are saved; the checkpoints are used to represent the model at different stages in the training process; the deep neural network model includes a speech recognition model. Determine the total loss value of the loss function of the model corresponding to each of the aforementioned checkpoints; Based on the total loss value, determine the checkpoints to be averaged from the plurality of checkpoints; The average parameter value of the model is obtained by averaging the multiple model parameters corresponding to the checkpoints to be averaged. Determining the total loss value of the model's loss function at each of the checkpoints includes: Calculate the loss value of the loss function for each checkpoint on the training set and validation set respectively; Based on the loss values ​​of the loss function of each checkpoint on the training set and validation set, determine the total loss value of the loss function of the model corresponding to each checkpoint; The step of determining the total loss value of the model's loss function corresponding to each checkpoint based on the loss value of the loss function on the training and validation sets includes: Based on the loss value of the loss function of each checkpoint on the validation set, the sum of the loss values ​​of the loss function corresponding to K consecutive checkpoints is determined; where K is a positive integer. Based on the sum of the loss values ​​of the loss function, determine the K consecutive checkpoints that have the smallest sum of the loss values ​​of the loss function; Based on the K consecutive checkpoints, determine the total loss value of the loss function of the model corresponding to each checkpoint; The step of determining the total loss value of the model's loss function for each of the K consecutive checkpoints includes: The contribution rate of the first checkpoint is calculated based on the loss value of the first checkpoint in the K consecutive checkpoints on the validation set and the loss value of the first checkpoint in the training set. Calculate the penalty factor for each checkpoint based on its current position; Based on the contribution rate and the penalty factor, calculate the total loss value of the loss function of the model corresponding to each checkpoint.

2. The model checkpoint parameter domain averaging method according to claim 1, characterized in that, The step of determining the checkpoint to be averaged from the plurality of checkpoints based on the total loss value includes: Based on the total loss value, determine the sum of the total loss values ​​corresponding to W consecutive checkpoints; where W is a positive integer. Based on the sum of the total loss values, determine the W consecutive checkpoints that have the smallest sum of the total loss values; Based on the W consecutive checkpoints, determine the checkpoints to be averaged from the plurality of checkpoints.

3. The model checkpoint parameter domain averaging method according to claim 1, characterized in that, The step of averaging multiple model parameters corresponding to the checkpoints to be averaged to obtain the average parameter value of the model includes: Determine the multiple model parameters corresponding to the check points to be averaged; Based on the multiple model parameters, the average value of the multiple model parameters is calculated to obtain the average parameter value of the model.

4. A model checkpoint parameter domain averaging device, characterized in that, include: A storage module is used to save multiple checkpoints and multiple model parameters corresponding to each checkpoint during the training process of the deep neural network model to be trained; the checkpoints are used to represent the model at different stages of the training process; the deep neural network model includes a speech recognition model; The first determining module is used to determine the total loss value of the loss function of the model corresponding to each of the checkpoints; Determining the total loss value of the model's loss function at each of the checkpoints includes: Calculate the loss value of the loss function for each checkpoint on the training set and validation set respectively; Based on the loss values ​​of the loss function of each checkpoint on the training set and validation set, determine the total loss value of the loss function of the model corresponding to each checkpoint; The step of determining the total loss value of the model's loss function corresponding to each checkpoint based on the loss value of the loss function on the training and validation sets includes: Based on the loss value of the loss function of each checkpoint on the validation set, the sum of the loss values ​​of the loss function corresponding to K consecutive checkpoints is determined; where K is a positive integer. Based on the sum of the loss values ​​of the loss function, determine the K consecutive checkpoints that have the smallest sum of the loss values ​​of the loss function; Based on the K consecutive checkpoints, determine the total loss value of the loss function of the model corresponding to each checkpoint; The step of determining the total loss value of the model's loss function for each of the K consecutive checkpoints includes: The contribution rate of the first checkpoint is calculated based on the loss value of the first checkpoint in the K consecutive checkpoints on the validation set and the loss value of the first checkpoint in the training set. Calculate the penalty factor for each checkpoint based on its current position; Based on the contribution rate and the penalty factor, calculate the total loss value of the loss function of the model corresponding to each checkpoint; The second determining module is used to determine the checkpoint to be averaged from the plurality of checkpoints based on the total loss value; The parameter averaging module is used to average the parameters of multiple model parameters corresponding to the checkpoint to be averaged, so as to obtain the average parameter value of the model.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the model checkpoint parameter domain averaging method as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model checkpoint parameter domain averaging method as described in any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the model checkpoint parameter domain averaging method as described in any one of claims 1 to 3.

Images