Model gradient accumulation training method and device, electronic equipment and storage medium

Abstract

This application provides a method, apparatus, electronic device, and non-volatile readable storage medium for training a model by accumulating gradients. The method includes: inputting training data into a deep learning model for iterative training to obtain multiple iterative training results, wherein the iterative training results include first-generation training results and second-generation training results; performing forward computation on the first-generation training results to obtain a first forward computation result; performing forward computation on the first forward computation result and the second-generation training results to obtain a second forward computation result; performing a weighted calculation on the first forward computation result and the second forward computation result to obtain a first training gradient value; and adjusting the deep learning model based on the first training gradient value to obtain a first model training result. Because the computation result carries multiple generations of training results, the diversity of training data is not limited by batch size. Even a small batch size can ensure the diversity of gradient computation results, guaranteeing the recognition accuracy of the deep learning model adjusted based on the gradient results.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model gradient accumulation training method, apparatus, electronic device, and non-volatile readable storage medium. Background Technology

[0002] Deep learning models are widely used in image recognition, text recognition, and speech recognition technologies due to their ability to quickly recognize and process information such as text and images. However, the design of deep learning models involves a large number of parameters, requiring continuous input of training data to train and adjust the model parameters to improve recognition performance. High-precision deep learning, however, consumes significant amounts of memory, computing, and bandwidth resources, resulting in high model training costs and demanding hardware and software requirements, thus hindering the widespread adoption and development of deep learning models.

[0003] To address the aforementioned issues, existing technologies often employ gradient accumulation training to train models. This method involves accumulating gradients across multiple training datasets and then updating parameters uniformly after a predetermined number of iterations, thus reducing the training requirements for deep learning models. However, this training method requires accumulating gradients across multiple batches of data, which are often set relatively small. This results in reduced training data diversity compared to traditional training methods, leading to lower accuracy in the trained deep learning models. Summary of the Invention

[0004] This application provides a model gradient accumulation training method, apparatus, electronic device, and non-volatile readable storage medium. In the model gradient accumulation training method provided in this application, the second forward calculation result is generated based on the first-generation training result and the second-generation training result. The calculation result carries the training results of multiple generations simultaneously. The diversity of training data is not limited by the batch size. Even a small batch size can ensure the diversity of gradient calculation results, thus guaranteeing the recognition accuracy of the deep learning model adjusted according to the gradient results.

[0005] Firstly, this application provides a method for training a model by accumulating gradients, the method comprising:

[0006] The training data is input into the deep learning model for iterative training to obtain multiple iterative training results, including first-generation training results and second-generation training results.

[0007] The first forward computation result is obtained by performing forward computation on the first generation training results;

[0008] The second forward calculation result is obtained by performing forward calculation on the first forward calculation result and the second generation training result;

[0009] The first training gradient value is obtained by weighting the first forward calculation result and the second forward calculation result;

[0010] The deep learning model is adjusted based on the first training gradient value to obtain the training result of the first model.

[0011] Optionally, the model gradient accumulation training method provided in this application also includes:

[0012] Based on the pre-set batch data size, the training data is divided into multiple batch training data, which include the first batch training data and the second batch training data.

[0013] The first batch of training data is input into the deep learning model to obtain the first generation of training results;

[0014] The second batch of training data is input into the deep learning model to obtain the second-generation training results.

[0015] Optionally, the model gradient accumulation training method provided in this application also includes:

[0016] The mean and variance of the multiple first training results are obtained by forward computation on multiple first training results.

[0017] Optionally, the model gradient accumulation training method provided in this application also includes:

[0018] The first forward calculation result and the second generation training result are weighted and forward calculated according to the first pre-set weight to obtain the second forward calculation result.

[0019] Optionally, the model gradient accumulation training method provided in this application also includes:

[0020] The first gradient calculation result is obtained by performing gradient calculation on the first forward calculation result;

[0021] The gradient calculation result is obtained by performing gradient calculation on the second forward calculation result;

[0022] The first training gradient value is obtained by weighting the first gradient calculation result and the second gradient calculation result according to the pre-set second weight.

[0023] Optionally, the model gradient accumulation training method provided in this application also includes:

[0024] The first forward calculation result and the second forward calculation result are reduced to obtain the reduced calculation result, wherein the reduced calculation result includes the first forward calculation result and the second forward calculation result;

[0025] The third forward calculation result is obtained by performing forward calculation on the reduction calculation result and the third generation training result;

[0026] The second training gradient value is obtained by weighting the first forward calculation result, the second forward calculation result, and the third forward calculation result according to the pre-set third weight. The sum of the first weight corresponding to the first forward calculation result, the second weight corresponding to the second forward calculation result, and the third weight corresponding to the third forward calculation result is less than or equal to 1.

[0027] The parameters of the deep learning model are updated based on the second training gradient value to obtain the training result of the second model.

[0028] Optionally, the model gradient accumulation training method provided in this application also includes:

[0029] The deep learning model is updated by inverse parameter modification based on the first training gradient value to obtain the training result of the first model.

[0030] Secondly, this application also provides a model gradient accumulation training device, comprising:

[0031] The iterative training module is used to input training data into the deep learning model for iterative training to obtain multiple iterative training results, including first-generation training results and second-generation training results.

[0032] The first forward computation module is used to perform forward computation on the first generation training results to obtain the first forward computation result;

[0033] The second forward computation module is used to perform forward computation on the first forward computation result and the second generation training result to obtain the second forward computation result;

[0034] The first training gradient calculation module is used to perform a weighted calculation on the first forward calculation result and the second forward calculation result to obtain the first training gradient value;

[0035] The first model update module is used to update the parameters of the deep learning model based on the first training gradient value to obtain the training result of the first model.

[0036] Thirdly, this application also provides an electronic device including a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the model gradient accumulation training method as described in the first aspect.

[0037] Fourthly, embodiments of this application provide a non-volatile readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements the steps of the model gradient accumulation training method as described in the first aspect.

[0038] In the model gradient accumulation training method provided in this application, the second forward calculation result is generated based on the first generation training result and the second generation training result. The calculation result carries the training results of multiple generations at the same time. The diversity of training data is not limited by the batch size. Even a small batch size can ensure the diversity of gradient calculation results and guarantee the recognition accuracy of the deep learning model adjusted according to the gradient results.

[0039] The above description is merely an overview of the technical solution provided in this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are described below. Attached Figure Description

[0040] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0041] Figure 1 This is one of the schematic diagrams of the model gradient accumulation training method provided in the embodiments of this application;

[0042] Figure 2 This is the second schematic diagram of the model gradient accumulation training method provided in the embodiments of this application;

[0043] Figure 3 This is the third schematic diagram of the model gradient accumulation training method provided in the embodiments of this application;

[0044] Figure 4 This is the fourth schematic diagram of the model gradient accumulation training method provided in the embodiments of this application;

[0045] Figure 5 This is the fifth schematic diagram of the model gradient accumulation training method provided in the embodiments of this application;

[0046] Figure 6 This is the sixth schematic diagram of the model gradient accumulation training method provided in the embodiments of this application;

[0047] Figure 7 This is the seventh schematic diagram of the model gradient accumulation training method provided in the embodiments of this application;

[0048] Figure 8 This is a schematic diagram of the model gradient accumulation training device provided in the embodiments of this application;

[0049] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0051] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0052] Current deep learning models typically include a batch normalization layer in their structural design. During each forward computation step, the batch normalization layer normalizes the input data from the batch size. In this gradient accumulation method, the diversity of data samples acquired by the batch normalization layer plays a crucial role in improving the model's training accuracy and generalization ability. Although this gradient accumulation training method increases the batch size by a factor of N through N gradient accumulation steps, the batch normalization layer still acquires the sample information from that batch size during each forward computation step. This lacks the data diversity compared to directly acquiring N*batch size data in a single training step. Furthermore, current gradient accumulation training methods do not consider the relationships between samples across different iteration steps, thus affecting the model's training accuracy.

[0053] The gradient accumulation training method provided in this application addresses the challenge of training a deep learning model with a large number of parameters and computational complexity on a computationally limited hardware platform. In each forward calculation of the batch normalization layer, the mean and variance of the data samples from the previous and current steps are reduced. The reduction result is then output to the next batch normalization calculation. This method ensures that, when training large-scale parameter models on devices with limited memory, computation, and bandwidth resources, sufficient sample information can be obtained in each iterative calculation when the batch size is relatively small. This saves memory, computation, and bandwidth resources while maintaining model accuracy. This method achieves the same training effect as direct training with a batch size of N*BS input data. Since the gradient accumulation training method in this application only requires a smaller batch size to achieve the same training effect as a larger batch size (e.g., training accuracy and speed), it saves computational resources while achieving high training model accuracy and ensuring fast training speed. This method can be applied to the training of large models in the field of natural language processing, helping users solve the problem of training large models with limited hardware resources and saving users model training costs.

[0054] The model gradient accumulation training method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0055] The first embodiment of this application relates to a model gradient accumulation training method, wherein the server is connected to a power supply, which supplies power to the server and is partially in standby mode, such as... Figure 1 As shown, it includes:

[0056] Step 101: Input the training data into the deep learning model for iterative training to obtain multiple iterative training results, including first-generation training results and second-generation training results.

[0057] Step 102: Perform forward computation on the first generation training results to obtain the first forward computation result;

[0058] Step 103: Perform forward calculation on the first forward calculation result and the second generation training result to obtain the second forward calculation result;

[0059] Step 104: Perform a weighted calculation on the first forward calculation result and the second forward calculation result to obtain the first training gradient value;

[0060] Step 105: Adjust the deep learning model according to the first training gradient value to obtain the training result of the first model.

[0061] Specifically, the gradient accumulation training method provided in this application can input training data into a deep learning model for iterative calculation, and adjust the parameters of the deep learning model as a whole based on the obtained multiple iterative results, thereby achieving the gradient accumulation training effect of the deep learning model. First, the training data is input into the deep learning model for iterative calculation, resulting in multiple iterative training results. For any generation of training results, such as the first generation training result, a forward calculation is performed to obtain the first forward calculation result. Then, based on the first forward calculation result obtained by performing forward calculation on the next generation training result (e.g., the second generation training result) and the first generation training result, a second forward calculation is performed to obtain the second forward calculation result. At this point, the second forward calculation result carries key data from both the first and second generation training results. When obtaining the first training gradient based on the second forward calculation result, it can simultaneously represent the training results of the first and second generations. For example, the first training gradient value can be obtained from the first and second forward calculation results through a weighted summation. Finally, the relevant parameters in the deep learning model are adjusted based on the first training gradient value to achieve improved data recognition after deep learning model training.

[0062] Because the gradient accumulation training method provided in this application generates the second forward calculation result based on the first generation training result and the second generation training result, the calculation result carries the training results of multiple generations at the same time. The diversity of training data is not limited by the batch size. Even a small batch size can ensure the diversity of gradient calculation results and guarantee the recognition accuracy of the deep learning model adjusted according to the gradient results.

[0063] Based on the above implementation methods, such as Figure 2 As shown, in the model gradient accumulation training method provided in this application, step 101 includes:

[0064] Step 111: According to the pre-set batch data size, the training data is divided into multiple batch training data, where the batch training data includes the first batch training data and the second batch training data.

[0065] Step 112: Input the first batch of training data into the deep learning model to obtain the first generation training results;

[0066] Step 113: Input the second batch of training data into the deep learning model to obtain the second-generation training results.

[0067] Specifically, the gradient accumulation training method provided in this application pre-sets the gradient accumulation steps and the batch size (BS). When iterative computation of the deep learning model is required, the training data used for model training, such as text data, is first segmented according to the BS data size to obtain multiple batches of training data. Each time, one batch of training data is input into the deep learning model to obtain a generation of training results; for example, the first batch of training data is input into the deep learning model to obtain a generation of training results. Furthermore, whenever the number of iterations reaches the gradient accumulation steps, iterative training stops, and the parameters of the deep learning model are updated based on the previously obtained training results. For example, if the gradient accumulation steps are N, each step inputs training data of the same size as the batch size into the deep learning model to obtain a generation of training results. When the number of inputs reaches N, the multiple generations of training results obtained from these N inputs are processed, and the deep learning model is updated based on the data processing results.

[0068] Based on the above implementation method, since the model gradient accumulation training method provided in this application can perform gradient accumulation training according to the pre-set batch data size and gradient accumulation steps, the deep learning model does not need to perform a large number of data operations for a single parameter update, thus avoiding the problem of long-term memory occupation of the deep learning model and improving the training efficiency of the deep learning model.

[0069] Based on the above implementation methods, such as Figure 3 As shown, in the model gradient accumulation training method provided in this application, step 102 includes:

[0070] Step 121: Perform forward computation on multiple first training results to obtain the mean and variance of the multiple first training results.

[0071] Specifically, in the gradient accumulation training method provided in this application, the forward computation mainly includes calculating the mean and variance of the training results. For example, forward computation can be performed through the batch nrom layer in the model to calculate the mean and variance of the training results obtained from multiple training data of that generation, such as the text data of the first generation, to obtain the mean and variance of the first training result.

[0072] Based on the above implementation methods, such as Figure 4 As shown, in the model gradient accumulation training method provided in this application, step 103 includes:

[0073] Step 131: Perform weighted forward calculation on the first forward calculation result and the second generation training result according to the pre-set first weight to obtain the second forward calculation result.

[0074] Specifically, in the model gradient accumulation training method provided in this application, the second forward computation not only needs to process the second-generation training results but also needs to process the previous-generation training results, such as the first-generation training results. For example, forward computation can be performed through the batchnrom layer in the model to calculate the mean and variance of the training results obtained from multiple training data of the previous generation, such as the text data of the first generation, to obtain the mean and variance of the first training results, which are used as the first forward computation results. Subsequently, the mean and variance of the first training results are retained, and training data of the same size as BS is input into the deep learning model to obtain the second training results. Based on the mean and variance of the first training results and the mean and variance of the second training results, the second forward computation result is calculated according to the pre-set first weights.

[0075] The calculation of the batchnrom layer can be processed in ways that are not limited to those shown in Formulas 1-4 below:

[0076]

[0077]

[0078]

[0079]

[0080] In the formula x i For each sample, the input value of this layer is BS, which is the batch size of the sample. ω1 and ω2 are the weight values ​​assigned to the sample information of the nth iteration and the previous n-1 iterations. For example, the weight of the first generation training result obtained by the first generation iteration training and the weight of the second generation training result obtained by the second generation iteration training. The sum of the weights of each generation and the next generation iteration training is 1.

[0081] Within a gradient accumulation step N, the forward computation results of each generation take into account the mean and variance obtained from the forward computation of the previous generation, ensuring the diversity of sample information obtained in each forward computation step.

[0082] Specifically, in the nth training step, in addition to calculating the mean and variance of the BS input data for that step, the mean and variance of the first n-1 calculated data are also statistically analyzed. This ensures the diversity of sample information obtained in each step of the calculation. At the same time, different weight values ​​are assigned to different sample information. The weight values ​​are flexible and adjustable. During the model training process, the influence of sample information on the model training accuracy under different iteration steps is controlled by adjusting the weight values, thereby obtaining the optimal accuracy value.

[0083] Furthermore, during the reduction calculation of the first n-1 sample information, the obtained sample information is the value calculated in the first n-1 steps, so this step will not increase additional computation and memory consumption.

[0084] Based on the above implementation methods, such as Figure 5 As shown, in the model gradient accumulation training method provided in this application, step 104 includes:

[0085] Step 141: Perform gradient calculation on the first forward calculation result to obtain the first gradient calculation result;

[0086] Step 142: Perform gradient calculation on the second forward calculation result to obtain the second gradient calculation result;

[0087] Step 143: The first training gradient value is obtained by weighting the first gradient calculation result and the second gradient calculation result according to the pre-set second weight.

[0088] Specifically, in the model gradient accumulation training method provided in this application, under the premise that all forward calculation results within a gradient accumulation step N are obtained, gradient calculation can be performed on these forward calculation results to obtain their respective gradient values, and weighted calculation is performed according to the second weights corresponding to the gradient calculation results of each generation to obtain the first training gradient value, which is used for subsequent parameter updates of the deep learning model based on the first training gradient value.

[0089] For example, the gradient value can be calculated in the manner shown in Equation 5:

[0090] ΔGn′=ω1*ΔGn+ω2*(ΔG1+ΔG2+…ΔGn-1) (5)

[0091] In the formula, ΔGn is the gradient value at the nth step. The gradient value at the nth step is weighted and accumulated with the gradient values ​​from the previous n-1 steps to obtain a new gradient value ΔGn′, which is either the final accumulated weighted total gradient value or the first training gradient value. The ω1 and ω2 in the formula are one of those found in Formula 1, ensuring the consistency of the calculated sample information, thereby guaranteeing the stability and convergence of the model training.

[0092] Based on the above implementation methods, such as Figure 6 As shown, in the model gradient accumulation training method provided in this application, after step 103, it further includes:

[0093] Step 106: Perform reduction calculation on the first forward calculation result and the second forward calculation result to obtain the reduction calculation result, wherein the reduction calculation result includes the first forward calculation result and the second forward calculation result;

[0094] Step 107: Perform forward computation on the reduction calculation result and the third generation training result to obtain the third forward computation result;

[0095] Step 108: The second training gradient value is obtained by weighting the first forward calculation result, the second forward calculation result, and the third forward calculation result according to the pre-set third weight. The sum of the first weight corresponding to the first forward calculation result, the second weight corresponding to the second forward calculation result, and the third weight corresponding to the third forward calculation result is less than or equal to 1.

[0096] Step 109: Update the parameters of the deep learning model based on the second training gradient value to obtain the training result of the second model.

[0097] Specifically, in the gradient accumulation training method provided in this application, the forward computation at each step can also summarize the forward computation results of all previous generations. When each generation is computed in the forward batchnorm layer, in addition to obtaining the sample information of the BS data of that generation, such as the mean and variance of the samples, it can also obtain the sample information of the previous n-1 generations, ensuring the diversity of sample information obtained in the forward computation of each generation.

[0098] For example, in addition to acquiring and calculating the mean and variance of the BS text data in the nth step, the batchnorm layer of the model also accumulates the mean and variance of the BS data in the previous n-1 steps. It performs reduction calculation on the mean and variance of the (n-1)*batchsize data obtained in the previous n-1 steps, and outputs the reduction result to the next calculation of the batchnorm layer in the nth step to obtain the forward calculation result of the nth step.

[0099] Furthermore, when calculating gradient values, the gradient value of each step can also be calculated by weighting the gradient value of that step with the gradient values ​​of all previous steps according to the third weight, so as to obtain the gradient value of that step. Finally, the second training gradient value of the entire gradient accumulation steps is the result of the weighted sum of the gradient values ​​of all steps.

[0100] That is, in the model gradient accumulation training method provided in this application, within a gradient accumulation step, the forward calculation and gradient value calculation of each small step can be obtained by simply using the training result of the small step and the training result of the previous small step, or the training result of the small step and the training results of all the small steps before the small step can be weighted and accumulated. The gradient value of each small step carries the key information of the training results of all the previous small steps. When training large models on devices with limited memory, computing and bandwidth resources, the same training effect as the existing technology of N times batch size can be obtained by setting the batch size by 1 times.

[0101] It should be emphasized that the third weight mentioned in this embodiment and the first and second weights mentioned in the above embodiments can be adjusted according to user needs. The first, second, and third weights can be the same or different, and this application does not impose any restrictions.

[0102] When the first, second, and third weights are consistent, the consistency of sample information obtained from multiple data processing can be ensured, thereby guaranteeing the stability and convergence of model training.

[0103] Based on the above implementation methods, such as Figure 7 As shown, in the model gradient accumulation training method provided in this application, step 105 includes:

[0104] Step 151: Update the parameters of the deep learning model in reverse according to the first training gradient value to obtain the training result of the first model.

[0105] Specifically, after obtaining the first training gradient value, the deep learning model is updated in reverse to achieve training within one gradient accumulation step and updating of the learning model from the previous year.

[0106] The gradient accumulation training method provided in this application enables rapid training of large models on devices with limited memory, computing, and bandwidth resources through gradient accumulation training optimization. It solves the problem that the training accuracy is affected by insufficient sample information obtained in each iteration due to the small batch size in traditional gradient accumulation training methods. At the same time, the gradient accumulation training method provided in this application does not increase the consumption of additional memory, computing, and bandwidth resources. When training large models on devices with limited memory, computing, and bandwidth resources, the optimization method proposed in this invention can achieve the same training effect as training with N times the batch size.

[0107] Furthermore, within a gradient accumulation step, the first few steps of the calculation process do not involve reverse parameter updates, which reduces the consumption of memory, computation, and bandwidth resources, reduces the communication time of large-scale parameters, and improves the training speed of the model.

[0108] The second embodiment of this application relates to a model gradient accumulation training device, such as... Figure 8 As shown, it includes:

[0109] The iterative training module 201 is used to input training data into the deep learning model for iterative training to obtain multiple iterative training results, including first-generation training results and second-generation training results.

[0110] The first forward computation module 202 is used to perform forward computation on the first generation training results to obtain the first forward computation result;

[0111] The second forward computation module 203 is used to perform forward computation on the first forward computation result and the second generation training result to obtain the second forward computation result;

[0112] The first training gradient calculation module 204 is used to perform a weighted calculation on the first forward calculation result and the second forward calculation result to obtain the first training gradient value.

[0113] The first model update module 205 is used to update the parameters of the deep learning model according to the first training gradient value to obtain the training result of the first model.

[0114] Based on the above embodiments, the iterative training module 201 in the model gradient accumulation training device provided in this application includes:

[0115] The segmentation processing unit 211 is used to segment the training data into multiple batches of training data according to the pre-set batch data size, wherein the batch training data includes the first batch of training data and the second batch of training data.

[0116] The first-generation training unit 212 is used to input the first batch of training data into the deep learning model to obtain the first-generation training results;

[0117] The second-generation training unit 213 is used to input the second batch of training data into the deep learning model to obtain the second-generation training results.

[0118] Based on the above embodiments, the first forward calculation module 202 in the model gradient accumulation training device provided in this application includes:

[0119] The variance calculation unit 221 is used to perform forward calculation on multiple first training results to obtain the mean and variance of the multiple first training results.

[0120] Based on the above embodiments, the second forward calculation module 203 in the model gradient accumulation training device provided in this application includes:

[0121] The first weighted calculation unit 231 is used to perform weighted forward calculation on the first forward calculation result and the second generation training result according to the preset first weight, so as to obtain the second forward calculation result.

[0122] Based on the above embodiments, the first training gradient calculation module 204 in the model gradient accumulation training device provided in this application includes:

[0123] The first gradient calculation unit 241 is used to perform gradient calculation on the first forward calculation result to obtain the first gradient calculation result.

[0124] The second gradient calculation unit 242 is used to perform gradient calculation on the second forward calculation result to obtain the second gradient calculation result.

[0125] The second weighted calculation unit 243 is used to perform weighted calculation on the first gradient calculation result and the second gradient calculation result according to the preset second weight to obtain the first training gradient value.

[0126] Based on the above embodiments, the model gradient accumulation training device provided in this application further includes:

[0127] The reduction calculation module 206 is used to perform reduction calculation on the first forward calculation result and the second forward calculation result to obtain the reduction calculation result, wherein the reduction calculation result includes the first forward calculation result and the second forward calculation result;

[0128] The third forward computation module 207 is used to perform forward computation on the reduction computation result and the third generation training result to obtain the third forward computation result;

[0129] The second training gradient calculation module 208 is used to perform weighted calculation on the first forward calculation result, the second forward calculation result and the third forward calculation result according to the pre-set third weight to obtain the second training gradient value, wherein the sum of the first weight corresponding to the first forward calculation result, the second weight corresponding to the second forward calculation result and the third weight corresponding to the third forward calculation result is less than or equal to 1.

[0130] The second model update module 209 is used to update the parameters of the deep learning model according to the second training gradient value to obtain the training result of the second model.

[0131] Based on the above embodiments, the first model update module 205 in the model gradient accumulation training device provided in this application includes:

[0132] The inverse parameter update unit 251 is used to update the parameters of the deep learning model in reverse according to the first training gradient value, so as to obtain the training result of the first model.

[0133] The third embodiment of this application relates to an electronic device, such as... Figure 9 As shown, it includes:

[0134] At least one processor 301; and,

[0135] The memory 302 is communicatively connected to the at least one processor 301; wherein,

[0136] The memory 302 stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor 301 to enable the at least one processor 301 to implement the model gradient accumulation training method described in the first embodiment of this application.

[0137] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0138] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0139] The fourth embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the model gradient accumulation training method described in the first embodiment of this application.

[0140] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0141] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0142] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A model gradient accumulation training method, applied to a computing hardware platform, the method comprising: Training data is input into a deep learning model for iterative training to obtain multiple iterative training results, including first-generation training results and second-generation training results. The training data is divided based on a pre-set batch size, which is determined based on the memory capacity of the computing hardware platform. The training data is divided into multiple batches, including a first batch and a second batch. The first-generation training result is obtained by inputting the first batch of training data into the deep learning model, and the second-generation training result is obtained by inputting the second batch of training data into the deep learning model. The first forward calculation result is obtained by performing forward calculation on the first generation training results; A second forward calculation result is obtained by performing forward calculation on the first forward calculation result and the second generation training result. The forward calculation includes mean calculation and variance calculation. The first training gradient value is obtained by weighting the first forward calculation result and the second forward calculation result; The deep learning model is adjusted based on the first training gradient value to obtain the first model training result, which is used to identify data types.

2. The method according to claim 1, characterized in that, The step of inputting training data into a deep learning model for iterative training to obtain multiple iterative training results includes: The training data is divided into multiple batches based on the pre-set batch size, wherein the batch training data includes the first batch of training data and the second batch of training data. The first batch of training data is input into the deep learning model to obtain the first generation of training results; The second batch of training data is input into the deep learning model to obtain the second-generation training results.

3. The method according to claim 1, characterized in that, The first training result includes multiple first training data sets, and the step of performing forward computation on the first generation training result to obtain the first forward computation result includes: The mean and variance of the multiple first training results are obtained by forward calculation of the multiple first training results.

4. The method according to claim 1, characterized in that, The step of performing forward computation on the first forward computation result and the second-generation training result to obtain the second forward computation result includes: The first forward calculation result and the second generation training result are weighted and forward calculated according to the first preset weight to obtain the second forward calculation result.

5. The method according to claim 1, characterized in that, The weighted calculation of the first forward calculation result and the second forward calculation result to obtain the first training gradient value includes: The first gradient calculation result is obtained by performing gradient calculation on the first forward calculation result; The second gradient calculation result is obtained by performing gradient calculation on the second forward calculation result; The first training gradient value is obtained by weighting the first gradient calculation result and the second gradient calculation result according to the pre-set second weight.

6. The method according to claim 1, characterized in that, The iterative training results also include three generations of training results. After performing forward computation on the first forward computation result and the second generation training result to obtain the second forward computation result, the process further includes: The first forward calculation result and the second forward calculation result are reduced to obtain a reduction calculation result, wherein the reduction calculation result includes the first forward calculation result and the second forward calculation result; A third forward calculation result is obtained by performing forward calculation on the reduction calculation result and the third generation training result; The second training gradient value is obtained by weighting the first forward calculation result, the second forward calculation result, and the third forward calculation result according to the pre-set third weight, wherein the sum of the first weight corresponding to the first forward calculation result, the second weight corresponding to the second forward calculation result, and the third weight corresponding to the third forward calculation result is less than or equal to 1. The parameters of the deep learning model are updated based on the second training gradient value to obtain the training result of the second model.

7. The method according to claim 1, characterized in that, The step of adjusting the deep learning model based on the first training gradient value to obtain the first model training result includes: The deep learning model is updated in reverse based on the first training gradient value to obtain the training result of the first model.

8. A model gradient accumulation training device, applied to a computing hardware platform, characterized in that, The device includes: An iterative training module is used to input training data into a deep learning model for iterative training to obtain multiple iterative training results. These iterative training results include first-generation training results and second-generation training results. The training data is divided based on a pre-set batch size, which is determined based on the memory capacity of the computing hardware platform. The training data is divided into multiple batches, including a first batch and a second batch. The first-generation training results are obtained by inputting the first batch of training data into the deep learning model, and the second-generation training results are obtained by inputting the second batch of training data into the deep learning model. The first forward computation module is used to perform forward computation on the first generation training results to obtain the first forward computation result; The second forward computation module is used to perform forward computation on the first forward computation result and the second generation training result to obtain a second forward computation result. The forward computation includes mean calculation and variance calculation. The first training gradient calculation module is used to perform a weighted calculation on the first forward calculation result and the second forward calculation result to obtain the first training gradient value; The first model update module is used to update the parameters of the deep learning model according to the first training gradient value to obtain the first model training result, which is used to identify the data type.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the model gradient accumulation training method as described in any one of claims 1-7.

10. A non-volatile readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the model gradient accumulation training method according to any one of claims 1-7.

Images