Neural network model training method and device, equipment, storage medium and program product
By calculating the loss change amount and historical change amount in the training of the neural network model, and dynamically adjusting the loss gating threshold, the gradient explosion and non-convergence problems caused by the sharp increase in iterative loss value are solved, and training stability and efficiency are improved.
Patent Information
- Application Number
- CN202411690656.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2024-12-24
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
During the training of neural network models, the sharp increase in the loss value of a certain iteration may cause gradient explosion or non-convergence, which increases the computational overhead of the training process and reduces efficiency.
By calculating the loss change amount and the historical loss change amount in each iteration, dynamically adjusting the loss gating threshold, deciding whether to perform backpropagation to avoid abnormal situations and multiple iterations, and improving training stability and efficiency.
Effectively avoid gradient explosion and non-convergence, reduce calculation overhead, improve training efficiency, and avoid the problem of too strict or loose fixed thresholds through a dynamic adaptive loss-gated threshold mechanism.
Smart Images
Figure CN119180319A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, computer equipment, storage medium and computer program product for a neural network model. Background Art
[0002] In the field of machine learning and artificial intelligence, the training process of a neural network model is to minimize the loss function by optimizing the model parameters so that the neural network model can make accurate predictions on the input data.
[0003] However, during the training process of the neural network model, sometimes there will be abnormal situations such as gradient explosion or non-convergence due to a sharp increase in the loss value of a certain iteration, so that more iterations are required to obtain a neural network model that meets the requirements, which increases the computational overhead of the training process and reduces the training efficiency. Summary of the invention
[0004] Based on this, it is necessary to provide a training method, device, computer equipment, storage medium and computer program product for a neural network model to address the above technical problems.
[0005] The present application provides a training method for a neural network model, the method comprising:
[0006] If the current iteration is not the first iteration, when the current iteration is performed, the training data of the current iteration is forward calculated according to the neural network model carrying the model parameters obtained after the previous iteration to obtain the loss value of the current iteration;
[0007] According to the loss value of the current iteration and the loss value of the previous iteration, the loss change corresponding to the current iteration is obtained;
[0008] The loss change corresponding to the current iteration and the loss change corresponding to the previous iterations are counted to obtain the loss gating threshold of the current iteration;
[0009] According to the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration, it is determined whether to perform back propagation based on the loss value of the current iteration to determine the model parameters obtained after the current iteration.
[0010] In one embodiment, the loss change amount corresponding to the current iteration and the loss change amount corresponding to the historical iteration are counted to obtain the loss gating threshold of the current iteration, including:
[0011] Based on the statistics of the loss changes corresponding to the historical iterations, the mean value of the loss changes obtained in the previous iteration is determined;
[0012] Determine the mean loss change amount obtained in the current iteration according to the mean loss change amount obtained in the previous iteration and the loss change amount corresponding to the current iteration;
[0013] Determine the standard deviation of the loss change obtained in the current iteration according to the mean value of the loss change obtained in the current iteration and the standard deviation of the loss change obtained in the previous iteration;
[0014] The loss gating threshold for the current iteration is obtained based on the standard deviation of the loss change obtained in the current iteration and the preset basic loss gating threshold.
[0015] In one embodiment, determining the mean loss change amount obtained in the current iteration according to the mean loss change amount obtained in the previous iteration and the loss change amount corresponding to the current iteration includes:
[0016] According to a preset mean decay factor, a first weight and a second weight are obtained; the mean decay factor and the first weight are positively correlated, and the mean decay factor and the second weight are negatively correlated;
[0017] Assign the first weight to the average loss change obtained in the previous iteration, and assign the second weight to the loss change corresponding to the current iteration, perform weighted summation on the average loss change obtained in the previous iteration and the loss change corresponding to the current iteration, and determine the average loss change obtained in the current iteration.
[0018] In one embodiment, determining the standard deviation of the loss change obtained in the current iteration according to the mean value of the loss change obtained in the current iteration and the standard deviation of the loss change obtained in the previous iteration includes:
[0019] Get the difference between the mean loss change obtained in the current iteration and the loss change corresponding to the current iteration;
[0020] Squaring the difference to obtain a square result of the difference;
[0021] According to a preset standard deviation attenuation factor, a third weight and a fourth weight are obtained; the standard deviation attenuation factor is positively correlated with the third weight, and the standard deviation attenuation factor is negatively correlated with the fourth weight;
[0022] The standard deviation of the loss change obtained in the previous iteration is assigned according to the third weight, and the squared difference result is assigned according to the fourth weight. The standard deviation of the loss change and the squared difference obtained in the previous iteration are weighted summed to determine the standard deviation of the loss change obtained in the current iteration.
[0023] In one embodiment, the loss gating threshold of the current iteration is obtained according to the standard deviation of the loss change obtained in the current iteration and the preset basic loss gating threshold, including:
[0024] The square root of the standard deviation of the loss change obtained in the current iteration is taken, multiplied by the preset standard deviation magnification factor, and then added to the preset basic loss gating threshold to obtain the loss gating threshold of the current iteration.
[0025] In one embodiment, determining whether to perform back propagation based on the loss value of the current iteration to determine the model parameters obtained after the current iteration, according to the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration, includes:
[0026] If the loss value of the current iteration is less than or equal to the loss gating threshold of the current iteration, back propagation is performed based on the loss value of the current iteration to update the model parameters obtained after the previous iteration and determine the model parameters obtained after the current iteration;
[0027] If the loss value of the current iteration is greater than the loss gating threshold of the current iteration, back propagation is not performed based on the loss value of the current iteration, and the model parameters obtained after the previous iteration are determined as the model parameters obtained after the current iteration.
[0028] The present application provides a training device for a neural network model, the device comprising:
[0029] A forward calculation module is used to perform forward calculation on the training data of the current iteration according to the neural network model carrying the model parameters obtained after the previous iteration to obtain the loss value of the current iteration if the current iteration is not the first iteration;
[0030] The loss change acquisition module is used to obtain the loss change corresponding to the current iteration based on the loss value of the current iteration and the loss value of the previous iteration;
[0031] A threshold determination module is used to collect statistics on the loss change corresponding to the current iteration and the loss change corresponding to the previous iterations to obtain the loss gating threshold of the current iteration;
[0032] The parameter determination module is used to determine whether to perform backpropagation based on the loss value of the current iteration according to the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration, so as to determine the model parameters obtained after the current iteration.
[0033] The present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the above method.
[0034] The present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to execute the above method.
[0035] The present application provides a computer program product, on which a computer program is stored, and the computer program is executed by a processor to perform the above method.
[0036] The training method, apparatus, computer equipment, storage medium and computer program product of the above-mentioned neural network model, if the current iteration is not the first iteration, when performing the current iteration, forward calculation is performed on the training data of the current iteration according to the neural network model carrying the model parameters obtained after the previous iteration to obtain the loss value of the current iteration; based on the loss value of the current iteration and the loss value of the previous iteration, the loss change corresponding to the current iteration is obtained; the loss change corresponding to the current iteration and the loss change corresponding to the historical iterations are counted to obtain the loss gating threshold of the current iteration; based on the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration, it is determined whether to perform back propagation based on the loss value of the current iteration to determine the model parameters obtained after the current iteration. With the help of this solution, it is possible to avoid abnormal situations such as gradient explosion or non-convergence caused by a sharp increase in the loss value of a certain iteration, thereby ensuring the stability of the training process, avoiding the need for more iterations, reducing computing overhead, and improving training efficiency; and the dynamic adaptive loss gating threshold mechanism provided by this solution can be dynamically adjusted according to the loss change corresponding to the current iteration and historical iterations, so as to obtain the loss gating threshold of the current iteration, avoiding the problem that the fixed loss gating threshold may be too strict or too loose, making the loss gating more reasonable and efficient. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0038] Figure 1 A flowchart of a training method for a neural network model in one embodiment;
[0039] Figure 2 A schematic diagram of a process for determining a loss gating threshold of a current iteration in one embodiment;
[0040] Figure 3 A schematic diagram of a process for determining a mean value of a loss change amount obtained in a current iteration in one embodiment;
[0041] Figure 4 is a structural block diagram of a training device for a neural network model in one embodiment;
[0042] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0044] The training method of the neural network model provided in the present application can be executed by a computer device, including: Figure 1 Steps shown.
[0045] Step S101, if the current iteration is not the first iteration, when performing the current iteration, forward calculation is performed on the training data of the current iteration according to the neural network model carrying the model parameters obtained after the previous iteration to obtain the loss value of the current iteration.
[0046] If the current iteration is not the first iteration, the current iteration may be recorded as i, where i is a positive integer greater than or equal to 2.
[0047] When performing the i-th iteration, the neural network model with the model parameters obtained after the i-1-th iteration can be obtained (which can be recorded as model[i-1]). The training data of the i-th iteration is obtained, and the training data of the i-th iteration is input into model[i-1] so that model[i-1] performs forward calculation on the training data of the i-th iteration to obtain the loss value of the i-th iteration (which can be recorded as loss[i]).
[0048] Whether to stop iteration can be determined based on the loss value loss[i] of the i-th iteration. If the iteration is stopped, steps S102 to S104 can be omitted, and model[i-1] is used as the target neural network model obtained after training; if the iteration is not stopped, steps S102 to S104 are executed.
[0049] Step S102, obtaining the loss change corresponding to the current iteration according to the loss value of the current iteration and the loss value of the previous iteration.
[0050] After obtaining the loss value loss[i] of the i-th iteration, you can also obtain the loss value loss[i-1] of the i-1-th iteration. Subtract loss[i-1] from loss[i] to get the loss change corresponding to the i-th iteration (which can be recorded as loss_change[i]), which can be expressed as loss_change[i]=loss[i]-loss[i-1].
[0051] Step S103 , statistics are collected on the loss change amount corresponding to the current iteration and the loss change amount corresponding to the historical iterations to obtain the loss gating threshold of the current iteration.
[0052] The loss change corresponding to the i-th iteration can be obtained; for each iteration before the i-th iteration, the corresponding loss change can also be obtained, for example, the loss change corresponding to the i-1-th iteration can be obtained (which can be recorded as loss_change[i-1]).
[0053] After obtaining loss_change[i] and the loss changes corresponding to each iteration before the i-th iteration, these loss changes can be counted to determine the loss gating threshold of the i-th iteration (which can be recorded as threshold[i]).
[0054] Step S104, according to the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration, determine whether to perform back propagation based on the loss value of the current iteration to determine the model parameters obtained after the current iteration.
[0055] After obtaining the loss value loss[i] of the i-th iteration and the loss gating threshold threshold[i] of the i-th iteration, we can compare loss[i] with threshold[i] to determine whether to perform backpropagation based on loss[i] and determine the model parameters obtained after the i-th iteration.
[0056] In the training method of the above neural network model, if the current iteration is not the first iteration, when performing the current iteration, the training data of the current iteration is forward calculated according to the neural network model that carries the model parameters obtained after the previous iteration to obtain the loss value of the current iteration; according to the loss value of the current iteration and the loss value of the previous iteration, the loss change corresponding to the current iteration is obtained; the loss change corresponding to the current iteration and the loss change corresponding to the historical iterations are counted to obtain the loss gating threshold of the current iteration; according to the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration, it is determined whether to perform back propagation based on the loss value of the current iteration to determine the model parameters obtained after the current iteration. With the help of this solution, it is possible to avoid abnormal situations such as gradient explosion or non-convergence caused by a sharp increase in the loss value of a certain iteration, thereby ensuring the stability of the training process, avoiding the need for more iterations, reducing computing overhead, and improving training efficiency; and the dynamic adaptive loss gating threshold mechanism provided by this solution can be dynamically adjusted according to the loss change corresponding to the current iteration and historical iterations, so as to obtain the loss gating threshold of the current iteration, avoiding the problem that the fixed loss gating threshold may be too strict or too loose, making the loss gating more reasonable and efficient.
[0057] In one embodiment, the loss change amount corresponding to the current iteration and the loss change amount corresponding to the historical iteration are counted to obtain the loss gating threshold of the current iteration, including Figure 2 The steps shown are:
[0058] Step S201, based on the statistics of loss changes corresponding to historical iterations, determine the mean loss change obtained in the previous iteration; step S202, according to the mean loss change obtained in the previous iteration and the loss change corresponding to the current iteration, determine the mean loss change obtained in the current iteration; step S203, according to the mean loss change obtained in the current iteration and the standard deviation of loss changes obtained in the previous iteration, determine the standard deviation of loss change obtained in the current iteration; step S204, according to the standard deviation of loss change obtained in the current iteration and the preset basic loss gating threshold, obtain the loss gating threshold of the current iteration.
[0059] The current iteration is recorded as i. When performing the i-th iteration, the loss change corresponding to the i-th iteration, loss_change[i], can be obtained. For each iteration before the i-th iteration, the corresponding loss change can also be obtained. For example, the loss change corresponding to the i-1-th iteration can be obtained (which can be recorded as loss_change[i-1]).
[0060] Based on the statistics of the corresponding loss changes of each iteration before the i-th iteration, the average loss change obtained in the i-1th iteration can be obtained (which can be recorded as avg_loss_change[i-1]). Based on loss_change[i] and avg_loss_change[i-1], the average loss change obtained in the i-th iteration can be determined (which can be recorded as avg_loss_change[i]).
[0061] The standard deviation of the loss change obtained at the i-1th iteration can be obtained (which can be recorded as std_loss_change[i-1]). According to avg_loss_change[i] and std_loss_change[i-1], the standard deviation of the loss change obtained at the i-th iteration can be obtained (which can be recorded as std_loss_change[i]).
[0062] Get the preset basic loss gating threshold (which can be recorded as base_threshold). According to base_threshold and std_loss_change[i], the loss gating threshold threshold[i] of the i-th iteration can be obtained.
[0063] In this embodiment, statistics are collected from the mean and standard deviation of the loss change to obtain a loss gating threshold suitable for the current iteration, thereby avoiding the missing code caused by the fixed loss gating threshold being too strict or too loose, making the loss gating more reasonable and efficient.
[0064] In one embodiment, the average loss change amount obtained in the current iteration is determined based on the average loss change amount obtained in the previous iteration and the loss change amount corresponding to the current iteration, including Figure 3 The steps shown are:
[0065] Step S301, according to the preset mean decay factor, obtain the first weight and the second weight; the mean decay factor is positively correlated with the first weight, and the mean decay factor is negatively correlated with the second weight; step S302, assign the first weight to the mean loss change obtained in the previous iteration, and assign the second weight to the loss change corresponding to the current iteration, perform weighted summation on the mean loss change obtained in the previous iteration and the loss change corresponding to the current iteration, and determine the mean loss change obtained in the current iteration.
[0066] The mean decay factor may be denoted as alpha, and the value of alpha may be preset, for example, to 0.9.
[0067] According to alpha, two weights can be obtained, which are called the first weight and the second weight. The mean decay factor is positively correlated with the first weight, and the mean decay factor is negatively correlated with the second weight.
[0068] The current iteration is recorded as i. The average loss change avg_loss_change[i-1] obtained in the i-1th iteration can be obtained; the loss change loss_change[i] corresponding to the i-th iteration can also be obtained.
[0069] Assign the first weight to avg_loss_change[i-1], assign the second weight to loss_change[i], and take the weighted sum of avg_loss_change[i-1] and loss_change[i] to determine the average loss change avg_loss_change[i] obtained at the i-th iteration.
[0070] In some scenarios, the preset value of the mean decay factor can be used as the first weight, and the result of subtracting the preset value of the mean decay factor from 1 can be used as the second weight. In this case, it can be expressed as:
[0071] avg_loss_change[i]=alpha×avg_loss_change[i-1]+(1-alpha)×loss_change[i].
[0072] In one embodiment, determining the standard deviation of the loss change obtained in the current iteration according to the mean value of the loss change obtained in the current iteration and the standard deviation of the loss change obtained in the previous iteration includes:
[0073] Obtain the difference between the mean of the loss change obtained in the current iteration and the loss change corresponding to the current iteration; square the difference to obtain the squared difference result; obtain the third weight and the fourth weight according to the preset standard deviation attenuation factor; the standard deviation attenuation factor is positively correlated with the third weight, and the standard deviation attenuation factor is negatively correlated with the fourth weight; assign the third weight to the standard deviation of the loss change obtained in the previous iteration, and assign the fourth weight to the squared difference result, perform weighted summation on the standard deviation of the loss change obtained in the previous iteration and the squared difference result, and determine the standard deviation of the loss change obtained in the current iteration.
[0074] The standard deviation attenuation factor may be denoted as beta. The value of beta may be preset, for example, to 0.9.
[0075] According to beta, two weights can be obtained, which are called the third weight and the fourth weight. The standard deviation attenuation factor is positively correlated with the third weight, and the standard deviation attenuation factor is negatively correlated with the fourth weight.
[0076] The current iteration is recorded as i. The average loss change avg_loss_change[i] obtained in the i-th iteration can be obtained, and the loss change loss_change[i] corresponding to the i-th iteration can also be obtained.
[0077] The difference between avg_loss_change[i] and loss_change[i] is squared, and the squared difference can be expressed as (avg_loss_change[i]-loss_change[i]) 2 .
[0078] The third weight is assigned to the standard deviation of the loss change std_loss_change[i-1] obtained in the i-1th iteration, and the fourth weight is assigned to the squared difference result. Std_loss_change[i-1] and the squared difference result are weighted summed to determine the standard deviation of the loss change std_loss_change[i] obtained in the i-th iteration.
[0079] In some scenarios, the preset value of the standard deviation attenuation factor can be used as the third weight, and the result of subtracting the preset value of the standard deviation attenuation factor from 1 can be used as the fourth weight. In this case, it can be expressed as:
[0080] std_loss_change[i]=beta×std_loss_change[i-1]+(1-beta)×(avg_loss_change[i]-loss_change[i]) 2 .
[0081] In one embodiment, the loss gating threshold of the current iteration is obtained according to the standard deviation of the loss change obtained in the current iteration and the preset basic loss gating threshold, including:
[0082] The square root of the standard deviation of the loss change obtained in the current iteration is taken, multiplied by the preset standard deviation magnification factor, and then added to the preset basic loss gating threshold to obtain the loss gating threshold of the current iteration.
[0083] The standard deviation magnification factor may be denoted as k, and the value of k may be preset, for example, to 1.5.
[0084] The current iteration is recorded as i. The standard deviation of the loss change obtained in the i-th iteration, std_loss_change[i], and the preset basic loss gating threshold base_threshold can be obtained. After taking the square root of std_loss_change[i], multiply it by the preset standard deviation magnification factor k, and then add it to the basic loss gating threshold base_threshold to obtain the loss gating threshold threshold[i] of the i-th iteration. The above process can be expressed as:
[0085] .
[0086] In one embodiment, determining whether to perform back propagation based on the loss value of the current iteration to determine the model parameters obtained after the current iteration, according to the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration, includes:
[0087] If the loss value of the current iteration is less than or equal to the loss gating threshold of the current iteration, back propagation is performed based on the loss value of the current iteration to update the model parameters obtained after the previous iteration and determine the model parameters obtained after the current iteration;
[0088] If the loss value of the current iteration is greater than the loss gating threshold of the current iteration, back propagation is not performed based on the loss value of the current iteration, and the model parameters obtained after the previous iteration are determined as the model parameters obtained after the current iteration.
[0089] Let the current iteration be i, and we can obtain the loss value loss[i] of the i-th iteration and the loss gating threshold threshold[i] of the i-th iteration.
[0090] If loss[i] is less than or equal to threshold[i], back propagation can be performed based on loss[i] to update the model parameters obtained after the i-1th iteration, and the updated result is determined as the model parameters obtained after the i-th iteration. In this way, a neural network model (which can be recorded as model[i]) carrying the model parameters obtained after the i-th iteration can be obtained.
[0091] If loss[i] is greater than threshold[i], backpropagation is not performed based on loss[i], and the model parameters obtained after the i-1th iteration are not updated. The model parameters obtained after the i-1th iteration can be directly determined as the model parameters obtained after the i-th iteration, thereby obtaining a neural network model (which can be recorded as model[i]) that carries the model parameters obtained after the i-th iteration.
[0092] In one embodiment, when the current iteration is the first iteration, the method provided by the present application includes the following steps: performing forward calculation on the training data of the first iteration according to the neural network model carrying the initial model parameters to obtain the loss value of the first iteration; determining the loss value of the first iteration as the loss change corresponding to the first iteration; obtaining the loss gating threshold of the first iteration according to the loss change corresponding to the first iteration; and determining whether to perform backpropagation based on the loss value of the first iteration according to the relative size between the loss value of the first iteration and the loss gating threshold of the first iteration to determine the model parameters obtained after the first iteration.
[0093] The model parameters used in the first iteration can be initialized, and the initialized model parameters are called initial model parameters. The neural network model carrying the initial model parameters can be recorded as model[init].
[0094] Input the training data of the first iteration into model[init] so that model[init] performs forward calculation on the training data of the first iteration to obtain the loss value of the first iteration (which can be recorded as loss[1]). You can determine whether to stop the iteration based on the loss value loss[1] of the first iteration. If you stop the iteration, you do not need to perform the steps described below and use model[init] as the target neural network model obtained after training. If you do not stop the iteration, perform the following steps:
[0095] Determine loss[1] as the loss change corresponding to the first iteration (which can be recorded as loss_change[1]).
[0096] By multiplying loss_change[1] by the second weight, we can determine the average loss change avg_loss_change[1] obtained in the first iteration. In some scenarios, the result of 1 minus the preset value of the mean decay factor can be used as the second weight. In this case, it can be expressed as:
[0097] avg_loss_change[1]=(1-alpha)×loss_change[1].
[0098] After squaring the difference between loss_change[1] and avg_loss_change[1], and then multiplying it by the fourth weight, we can get the standard deviation of the loss change obtained in the first iteration, std_loss_change[1]. In some scenarios, the result of 1 minus the preset value of the standard deviation attenuation factor can be used as the fourth weight. In this case, it can be expressed as:
[0099] std_loss_change[1]=(1-beta)×(avg_loss_change[i]-loss_change[1]) 2 .
[0100] After taking the square root of std_loss_change[1], multiply it by the preset standard deviation magnification factor k, and then add it to the basic loss gating threshold base_threshold to obtain the loss gating threshold threshold[1] for the first iteration. The above process can be expressed as:
[0101] .
[0102] The solution provided in this application uses a loss gating threshold to optimize the training process. When the loss value of a certain iteration exceeds the loss gating threshold of that iteration, back propagation is skipped to enter the next iteration, realizing the "checkpoint" effect, filtering out abnormally high loss values, and avoiding adverse effects on the neural network model. In addition, the loss gating threshold of each iteration will be adaptively adjusted as the iteration proceeds, which improves the stability and convergence performance of the training process, reduces the adverse effects of noisy data, and improves the generalization ability of the neural network model. It has good versatility and is easy to integrate.
[0103] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0104] Based on the same inventive concept, the embodiment of the present application also provides a training device for a neural network model for implementing the training method of the neural network model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more neural network models provided below can refer to the limitations of the training method for the neural network model above, and will not be repeated here.
[0105] In one embodiment, Figure 4 As shown, a training device for a neural network model is provided, comprising:
[0106] A forward calculation module 401 is used to perform forward calculation on the training data of the current iteration according to the neural network model carrying the model parameters obtained after the previous iteration to obtain the loss value of the current iteration if the current iteration is not the first iteration.
[0107] The loss change acquisition module 402 is used to obtain the loss change corresponding to the current iteration according to the loss value of the current iteration and the loss value of the previous iteration;
[0108] A threshold determination module 403 is used to collect statistics on the loss change corresponding to the current iteration and the loss change corresponding to the previous iterations to obtain the loss gating threshold of the current iteration;
[0109] The parameter determination module 404 is used to determine whether to perform back propagation based on the loss value of the current iteration according to the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration to determine the model parameters obtained after the current iteration.
[0110] In one embodiment, the threshold determination module 403 is also used to: determine the mean loss change obtained in the previous iteration based on the statistics of the loss change corresponding to the historical iterations; determine the mean loss change obtained in the current iteration according to the mean loss change obtained in the previous iteration and the loss change corresponding to the current iteration; determine the standard deviation of the loss change obtained in the current iteration according to the mean loss change obtained in the current iteration and the standard deviation of the loss change obtained in the previous iteration; obtain the loss gating threshold of the current iteration according to the standard deviation of the loss change obtained in the current iteration and the preset basic loss gating threshold.
[0111] In one embodiment, the threshold determination module 403 is also used to: obtain a first weight and a second weight according to a preset mean decay factor; the mean decay factor is positively correlated with the first weight, and the mean decay factor is negatively correlated with the second weight; assign the first weight to the mean loss change obtained in the previous iteration, and assign the second weight to the loss change corresponding to the current iteration, perform weighted summation on the mean loss change obtained in the previous iteration and the loss change corresponding to the current iteration, and determine the mean loss change obtained in the current iteration.
[0112] In one embodiment, the threshold determination module 403 is also used to: obtain the difference between the mean value of the loss change obtained in the current iteration and the loss change corresponding to the current iteration; square the difference to obtain a square difference result; obtain a third weight and a fourth weight according to a preset standard deviation attenuation factor; the standard deviation attenuation factor is positively correlated with the third weight, and the standard deviation attenuation factor is negatively correlated with the fourth weight; assign the third weight to the standard deviation of the loss change obtained in the previous iteration, and assign the fourth weight to the square difference result, perform weighted summation on the standard deviation of the loss change obtained in the previous iteration and the square difference result, and determine the standard deviation of the loss change obtained in the current iteration.
[0113] In one embodiment, the threshold determination module 403 is also used to: perform square root processing on the standard deviation of the loss change obtained in the current iteration, multiply it by a preset standard deviation magnification factor, and then add it to a preset basic loss gating threshold to obtain the loss gating threshold of the current iteration.
[0114] In one embodiment, the parameter determination module 404 is further used to: if the loss value of the current iteration is less than or equal to the loss gating threshold of the current iteration, then back propagation is performed based on the loss value of the current iteration to update the model parameters obtained after the previous iteration and determine the model parameters obtained after the current iteration; if the loss value of the current iteration is greater than the loss gating threshold of the current iteration, then back propagation is not performed based on the loss value of the current iteration, and the model parameters obtained after the previous iteration are determined as the model parameters obtained after the current iteration.
[0115] Each module in the training device of the above neural network model can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0116] In an exemplary embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the above method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a neural network model is implemented.
[0117] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0118] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.
[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0120] In one embodiment, a computer program product is provided, on which a computer program is stored, and the computer program is used by a processor to execute the steps in the above-mentioned various method embodiments.
[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0122] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0123] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0124] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A training method for a neural network model, characterized in that: The method comprises: If the current iteration is not the first iteration, when the current iteration is performed, the training data of the current iteration is forward calculated according to the neural network model carrying the model parameters obtained after the previous iteration to obtain the loss value of the current iteration; According to the loss value of the current iteration and the loss value of the previous iteration, the loss change corresponding to the current iteration is obtained; The loss change corresponding to the current iteration and the loss change corresponding to the previous iterations are counted to obtain the loss gating threshold of the current iteration; According to the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration, it is determined whether to perform back propagation based on the loss value of the current iteration to determine the model parameters obtained after the current iteration.
2. The method according to claim 1, characterized in that The loss change corresponding to the current iteration and the loss change corresponding to the previous iteration are counted to obtain the loss gating threshold of the current iteration, including: Based on the statistics of the loss changes corresponding to the historical iterations, the mean value of the loss changes obtained in the previous iteration is determined; Determine the mean loss change amount obtained in the current iteration according to the mean loss change amount obtained in the previous iteration and the loss change amount corresponding to the current iteration; Determine the standard deviation of the loss change obtained in the current iteration according to the mean value of the loss change obtained in the current iteration and the standard deviation of the loss change obtained in the previous iteration; The loss gating threshold for the current iteration is obtained based on the standard deviation of the loss change obtained in the current iteration and the preset basic loss gating threshold.
3. The method according to claim 2, characterized in that According to the average loss change amount obtained in the previous iteration and the loss change amount corresponding to the current iteration, the average loss change amount obtained in the current iteration is determined, including: According to a preset mean decay factor, a first weight and a second weight are obtained; the mean decay factor and the first weight are positively correlated, and the mean decay factor and the second weight are negatively correlated; Assign the first weight to the average loss change obtained in the previous iteration, and assign the second weight to the loss change corresponding to the current iteration, perform weighted summation on the average loss change obtained in the previous iteration and the loss change corresponding to the current iteration, and determine the average loss change obtained in the current iteration.
4. The method according to claim 2, characterized in that According to the mean value of the loss change obtained in the current iteration and the standard deviation of the loss change obtained in the previous iteration, the standard deviation of the loss change obtained in the current iteration is determined, including: Get the difference between the mean loss change obtained in the current iteration and the loss change corresponding to the current iteration; Squaring the difference to obtain a square result of the difference; According to a preset standard deviation attenuation factor, a third weight and a fourth weight are obtained; the standard deviation attenuation factor is positively correlated with the third weight, and the standard deviation attenuation factor is negatively correlated with the fourth weight; The standard deviation of the loss change obtained in the previous iteration is assigned according to the third weight, and the squared difference result is assigned according to the fourth weight. The standard deviation of the loss change and the squared difference obtained in the previous iteration are weighted summed to determine the standard deviation of the loss change obtained in the current iteration.
5. The method according to claim 2, characterized in that: According to the standard deviation of the loss change obtained in the current iteration and the preset basic loss gating threshold, the loss gating threshold of the current iteration is obtained, including The square root of the standard deviation of the loss change obtained in the current iteration is taken, multiplied by the preset standard deviation magnification factor, and then added to the preset basic loss gating threshold to obtain the loss gating threshold of the current iteration.
6. The method according to any one of claims 1 to 5, characterized in that: According to the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration, it is determined whether to perform back propagation based on the loss value of the current iteration to determine the model parameters obtained after the current iteration, including: If the loss value of the current iteration is less than or equal to the loss gating threshold of the current iteration, back propagation is performed based on the loss value of the current iteration to update the model parameters obtained after the previous iteration and determine the model parameters obtained after the current iteration; If the loss value of the current iteration is greater than the loss gating threshold of the current iteration, back propagation is not performed based on the loss value of the current iteration, and the model parameters obtained after the previous iteration are determined as the model parameters obtained after the current iteration.
7. A training device for a neural network model, characterized in that: The device comprises: A forward calculation module is used to perform forward calculation on the training data of the current iteration according to the neural network model carrying the model parameters obtained after the previous iteration to obtain the loss value of the current iteration if the current iteration is not the first iteration; The loss change acquisition module is used to obtain the loss change corresponding to the current iteration based on the loss value of the current iteration and the loss value of the previous iteration; A threshold determination module is used to collect statistics on the loss change corresponding to the current iteration and the loss change corresponding to the previous iterations to obtain the loss gating threshold of the current iteration; The parameter determination module is used to determine whether to perform backpropagation based on the loss value of the current iteration according to the relative size between the loss value of the current iteration and the loss gating threshold of the current iteration, so as to determine the model parameters obtained after the current iteration.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.