Improved method and system for reducing large model training spur problem

Abstract

The invention discloses an improved method and system for reducing a large model training spike problem, and the method comprises the steps: training a large model through employing a training data set, and employing an Adam optimizer to optimize and update the model parameters of the large model in combination with a preset loss function; a constant term # imgabs0 # used for preventing a denominator from being 0 is introduced into the Adam optimizer, so that a new constant term # imgabs2 # used for preventing the denominator from being 0 in the improved Adam optimizer tends to be continuously reduced along with the increase of the number of iterations # imgabs3 # of training, # imgabs4 # is the exponential decay rate of first-order moment estimation, and # imgabs5 # is the exponential decay rate of second-order moment estimation. The method aims at solving the problem that loss spurs are generated when an Adam optimizer is used for training a large model, so that generation of loss spurs in large model training is reduced, and the convergence speed and training efficiency of large model training are improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to an improved method and system for reducing the spurt problem in large model training. Background Art

[0002] Big models, that is, deep learning models with large-scale data processing capabilities and strong data generalization capabilities, can be used to obtain corresponding output results based on single or multiple modal input data such as text, images, videos and audio. The application scenarios in natural language processing, computer vision, industrial applications, smart office, personalized recommendations, game development, agriculture and environmental protection, education and scientific research are becoming increasingly rich. As a machine learning model, the key to a big model is to establish a mapping relationship between input and output through training. During training, it is generally necessary to use an optimizer combined with a given loss function to update and optimize the model parameters of the big model.

[0003] Adam (Adaptive Moment Estimation) optimizer is a widely used deep learning optimization algorithm. It combines the ideas of momentum method and RMSProp, and aims to adjust the learning rate of each parameter by calculating the first-order moment estimate and second-order moment estimate of the gradient, so as to achieve more efficient network training. Adam optimizer has become the mainstream optimizer and is widely used in large model training. The function expression of Adam optimizer is: , , , , In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, and are the second-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, The initial value of is 0. is the exponential decay rate of the first-order moment estimate, is the exponential decay rate of the second-order moment estimate, is a constant term used to prevent the denominator from being zero, is the update amount of the tth iteration, is the learning rate of the tth iteration, and are the model parameters for the tth and t-1th iterations respectively.

[0004] The AdamW optimizer is an improved optimizer of the Adam optimizer. The AdamW optimizer adds a weight penalty term to the Adam optimizer and proposes weight decay. The weight penalty term is conducive to the tendency of the weight to tend to zero as the weight is updated, thereby selecting the dimension of the feature vector and improving the generalization ability. The function expression of the AdamW optimizer is: , , , , , In the above formula, is the decay weight, is the penalty weight, and the rest of the symbols are the same as the Adam optimizer.

[0005] The AMSGrad optimizer improves on the Adam optimizer to address the shortcomings of Adam when dealing with large data sets. The improvement of the AMSGrad optimizer is to normalize the exponential moving average of the gradient so that the update amount of parameters with large gradients will not be too large. The AMSGrad optimizer performs well when dealing with non-convex optimization problems, especially for large data sets in deep learning. The function expression of the AMSGrad optimizer is: , , , , In the above formula, is the maximum value of the second-order moment vector at the tth iteration, used in and Choose the larger value of the two.

[0006] Other optimizers such as Nadam and AdaMax are not described here.

[0007] However, the above optimizer only noticed the training convergence and generalization problems, but did not find and solve the loss spike phenomenon during training. Loss spike refers to the sudden increase in loss during pre-training, which is especially prone to occur during pre-training of large models (more than 100B). It will lead to a series of problems, such as the model taking a long time to return to the state before the spike, or worse, the loss can no longer return, that is, the model can no longer converge. Summary of the invention

[0008] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, an improved method and system for reducing the problem of loss spikes in large model training are provided. The present invention aims to solve the problem of loss spikes when using Adam-type optimizers to train large models, thereby reducing the generation of loss spikes in large model training, thereby improving the convergence speed and training efficiency of large model training.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is: An improved method for reducing the spurt problem in large model training includes the following steps: when training a large model using a training data set and using an Adam optimizer in combination with a preset loss function to optimize and update the model parameters of the large model, a constant term used by the Adam optimizer to prevent the denominator from being zero is used. Introduction The new constant term used to prevent the denominator from being zero in the improved Adam class optimizer As the number of training iterations increases Increases and the trend is decreasing, among which is the exponential decay rate of the first-order moment estimate, is the exponential decay rate of the second-order moment estimate.

[0010] Optionally, the function expression of the improved Adam optimizer to optimize and update the model parameters of the large model is: , , , , In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, and are the second-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, is the update amount of the tth iteration, is a new constant term used to prevent the denominator from being zero, is the learning rate of the tth iteration, and are the model parameters for the tth and t-1th iterations respectively.

[0011] Optionally, the function expression of the improved Adam optimizer to optimize and update the model parameters of the large model is: , , , , , In the above formula, is the gradient of the loss function at the tth iteration, is the decay weight, and are the model parameters for the tth and t-1th iterations, respectively. and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, and are the second-order moment estimates for the t-th and t-1-th iterations, respectively, is the update amount of the tth iteration, is a new constant term used to prevent the denominator from being zero, is the learning rate of the tth iteration, is the penalty weight.

[0012] Optionally, the function expression of the improved Adam optimizer to optimize and update the model parameters of the large model is: , , , , In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, and are the second-order moment estimates for the t-th and t-1-th iterations, respectively, is the maximum value of the second-order moment vector at the tth iteration, Indicates taking the maximum value, and are the model parameters for the tth and t-1th iterations, respectively. is the learning rate for the tth iteration.

[0013] Optionally, the function expression of the improved Adam optimizer to optimize and update the model parameters of the large model is: , , , , , In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, and are the second-order moment estimates of the t-th and t-1-th iterations, respectively. The initial first-order moment estimate and second-order moment estimate are 0. is the corrected first-order moment estimate of the t-th iteration, is the corrected second-order moment estimate of the t-th iteration, and are the model parameters for the tth and t-1th iterations, respectively. is the learning rate for the tth iteration.

[0014] Optionally, the function expression of the improved Adam optimizer to optimize and update the model parameters of the large model is: , , , , In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, and are the infinite norms of the t-th and t-1-th iterations respectively, the initial first-order moment estimate and infinite norm are 0, is the learning rate, is the learning rate of the tth iteration, and are the model parameters for the tth and t-1th iterations respectively.

[0015] Optionally, the large model is a natural language processing model, the input of the natural language processing model is text, and the output is a processing result of the text.

[0016] Optionally, the large model is a multimodal model, the input of which includes data of some or all modalities among text, images, video and audio, and the output of which is the processing result of some or all modalities among text, images, video and audio.

[0017] In addition, the present invention also provides an improved system for reducing the spurt problem in large model training, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the improved method for reducing the spurt problem in large model training.

[0018] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0019] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the improved method for reducing the spurt problem of large model training through a processor.

[0020] Compared with the prior art, the present invention has the following advantages: when the method of the present invention uses the training data set to train the large model and uses the Adam optimizer in combination with the preset loss function to optimize and update the model parameters of the large model, the Adam optimizer is used to prevent the constant term with the denominator being 0. Introduction The new constant term used to prevent the denominator from being zero in the improved Adam class optimizer As the number of training iterations increases The present invention can solve the problem of loss spikes when using the Adam optimizer to train a large model, thereby reducing the generation of loss spikes in large model training and improving the convergence speed and training efficiency of the large model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the basic process of the method in Embodiment 1 of the present invention.

[0022] Figure 2 Schematic diagram of experimental comparison results in Example 1 of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below with reference to the accompanying drawings.

[0024] Embodiment 1: The main reason for the spikes is the inherent defects of the Adam optimizer algorithm. To analyze the cause of the spikes, we take the Adam optimizer as an example. First, we calculate the update parameters At t=1, , Focus on At the end of training, assuming the model converges to a certain optimal point, It should be concentrated near 0. In other words, we seem to be able to imagine the process of updating parameters as a process of converging from the two ends (non-steady state) to the middle (steady state). The central limit theorem states that one of the necessary conditions for the superposition of random events to enter a unimodal normal distribution is that the various random events should be independent of each other, but the gradient change and the change of the update parameters cannot particularly meet this condition of independence, which is precisely one of the important reasons for the oscillation of the update parameters, the appearance of loss spikes, and the non-convergence of loss. Observe the phenomena before and after the appearance of the loss spike: the current model is in a steady state (healthy state), that is, a unimodal normal distribution state, and the gradient value , at this time, the loss is stable and the training process is normal. As the training progresses, the gradient of the shallow layer (embedding layer) of the model This is generally because after a period of training, the shallow semantic knowledge representation is generally well learned. However, the gradient update of the deep network (corresponding to complex tasks) is still relatively large. It will then lead to , so that tends to 0. As a result, the shallow parameters cannot be updated (which also corresponds to the reason why the parameter update events are not independent). At this time, although the shallow (embedding) parameters are not updated for a long time, the deep parameters are still being updated. After such a long period of time, the change in sample distribution between batches may directly lead to a large gradient change in the shallow (embedding) layer again. , , so that Focus again on Nearby, it becomes a non-stable state with two peaks. This article mentions that such sudden parameter changes in the shallow layer (embedding layer) may cause a chain reaction in the model and then a loss spike phenomenon occurs (this also corresponds to the fact that changing samples and retraining may reduce the frequency of loss spikes. In fact, it is to select samples with smaller distribution changes and reduce the shallow gradient transformation amplitude). At this stage, the model is in a non-stable state, and the gradient changes are large. There is a certain degree of independence between each gradient change and the update parameter change event. Therefore, after a certain period of time, the model may enter a steady state again, and the loss will regress again (note that this re-regression is not necessarily going to occur. It is also possible that the loss is in a stable state for a long time and can no longer converge). Therefore, if Figure 1As shown, this embodiment provides an improved method for reducing the spur problem in large model training, comprising the following steps: when using a training data set to train a large model and using an Adam optimizer in combination with a preset loss function to optimize and update the model parameters of the large model, a constant term used by the Adam optimizer to prevent the denominator from being 0 Introduction The new constant term used to prevent the denominator from being zero in the improved Adam class optimizer As the number of training iterations increases Increases and the trend is decreasing, among which is the exponential decay rate of the first-order moment estimate, is the exponential decay rate of the second-order moment estimate.

[0025] As an optional implementation, the improved Adam optimizer in this embodiment is based on the basic Adam optimizer. The function expression of the improved Adam optimizer for optimizing and updating the model parameters of the large model is: , , , , In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, and are the second-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, is the update amount of the tth iteration, is a new constant term used to prevent the denominator from being zero, is the learning rate of the tth iteration, and are the model parameters for the tth and t-1th iterations respectively. Combining the above improved Adam optimizer to optimize and update the function expression of the model parameters of the large model, it can be seen that when , , hour, , As time goes by, , , because accumulation leads to subsequent , , by introducing , so that the new constant term used to prevent the denominator from being zero Following their trend, the 0, maintain shallow parameter updates, reduce large changes in subsequent gradients, and thus reduce the generation of loss spikes, thereby improving the convergence speed and training efficiency of large model training.

[0026] As an optional implementation, the large model in this embodiment is a natural language processing model, the input of the natural language processing model is text, and the output is the processing result of the text. In this embodiment, the image classification model can adopt the required network type or structure as needed. The key to this embodiment is to improve the optimizer of the training image classification model, which does not depend on the type or structure of the specific image classification model, so its implementation will not be described in detail here.

[0027] In order to verify the improved method of reducing the spur problem of large model training in this embodiment, a comparative experiment was conducted on the image classification model using the original Adam optimizer and the improved Adam class optimizer in this embodiment. The results are as follows: Figure 2 As shown, the horizontal axis is the number of iterations, the vertical axis is the loss value of the loss function, the red curve is the loss value obtained by using the original Adam optimizer, and the green curve is the loss value obtained by using the improved Adam class optimizer in this embodiment. It can be seen from the comparison that the amplitude of the number of spurs generated by using the improved Adam class optimizer in this embodiment is greatly reduced.

[0028] In summary, the improved method for reducing the spurt problem in large model training in this embodiment includes: when using the training data set to train the large model and using the Adam optimizer in combination with the preset loss function to optimize and update the model parameters of the large model, the Adam optimizer is used to prevent the constant term with the denominator being 0. Introduction The new constant term used to prevent the denominator from being zero in the improved Adam class optimizer As the number of training iterations increases The improved method for reducing the spur problem in large model training in this embodiment can solve the problem of loss spurs when using the Adam optimizer to train a large model, thereby reducing the generation of loss spurs in large model training, thereby improving the convergence speed and training efficiency of large model training.

[0029] In addition, this embodiment also provides an improved system for reducing the spurt problem in large model training, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the improved method for reducing the spurt problem in large model training.

[0030] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0031] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0032] Embodiment 2: This embodiment is basically the same as the first embodiment, the main difference being that the improved basis of the improved Adam class optimizer in this embodiment is different. The improved basis of the improved Adam class optimizer in this embodiment is the basic WAdam optimizer. The function expression of the improved Adam class optimizer for optimizing and updating the model parameters of the large model is: , , , , , In the above formula, is the gradient of the loss function at the tth iteration, is the decay weight, and are the model parameters for the tth and t-1th iterations, respectively. and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, and are the second-order moment estimates for the t-th and t-1-th iterations, respectively, is the update amount of the tth iteration, is a new constant term used to prevent the denominator from being zero, is the learning rate of the tth iteration, is the penalty weight. , and the new constant term used to prevent the denominator from being zero can also be Following their trend, the 0, keep shallow parameter updates, reduce large changes in subsequent gradients, and thus reduce the generation of loss spikes.

[0033] In addition, this embodiment also provides an improved system for reducing the spurt problem in large model training, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the improved method for reducing the spurt problem in large model training.

[0034] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0035] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0036] Embodiment three: This embodiment is basically the same as the first embodiment, the main difference being that the improved basis of the improved Adam class optimizer in this embodiment is different. The improved basis of the improved Adam class optimizer in this embodiment is the basic AMSGrad optimizer. The function expression of the improved Adam class optimizer for optimizing and updating the model parameters of the large model is: , , , , In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, and are the second-order moment estimates for the t-th and t-1-th iterations, respectively, is the maximum value of the second-order moment vector at the tth iteration, Indicates taking the maximum value, and are the model parameters for the tth and t-1th iterations, respectively. is the learning rate of the tth iteration. , and the new constant term used to prevent the denominator from being zero can also be Following their trend, the 0, keep shallow parameter updates, reduce large changes in subsequent gradients, and thus reduce the generation of loss spikes.

[0037] In addition, this embodiment also provides an improved system for reducing the spurt problem in large model training, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the improved method for reducing the spurt problem in large model training.

[0038] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0039] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0040] Embodiment 4: This embodiment is basically the same as the first embodiment, and the main difference is that the improvement basis of the improved Adam class optimizer in this embodiment is different. The improvement basis of the improved Adam class optimizer in this embodiment is the basic Nadam optimizer. The function expression of the improved Adam class optimizer for optimizing and updating the model parameters of the large model is: , , , , , In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, and are the second-order moment estimates of the t-th and t-1-th iterations, respectively. The initial first-order moment estimate and second-order moment estimate are 0. is the corrected first-order moment estimate of the t-th iteration, is the corrected second-order moment estimate of the t-th iteration, and are the model parameters for the tth and t-1th iterations, respectively. is the learning rate of the tth iteration. , and the new constant term used to prevent the denominator from being zero can also be Following their trend, the 0, keep shallow parameter updates, reduce large changes in subsequent gradients, and thus reduce the generation of loss spikes.

[0041] In addition, this embodiment also provides an improved system for reducing the spurt problem in large model training, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the improved method for reducing the spurt problem in large model training.

[0042] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0043] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0044] Embodiment five: This embodiment is basically the same as the first embodiment, the main difference being that the improved basis of the improved Adam class optimizer in this embodiment is different. The improved basis of the improved Adam class optimizer in this embodiment is the basic AdaMax optimizer. The function expression of the improved Adam class optimizer for optimizing and updating the model parameters of the large model is: , , , , In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, and are the infinite norms of the t-th and t-1-th iterations respectively, the initial first-order moment estimate and infinite norm are 0, is the learning rate, is the learning rate of the tth iteration, and are the model parameters for the tth and t-1th iterations respectively. , and the new constant term used to prevent the denominator from being zero can also be Following their trend, the 0, keep shallow parameter updates, reduce large changes in subsequent gradients, and thus reduce the generation of loss spikes.

[0045] In addition, this embodiment also provides an improved system for reducing the spurt problem in large model training, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the improved method for reducing the spurt problem in large model training.

[0046] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0047] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the improved method for reducing the spurt problem in large model training through a processor.

[0048] Embodiment six: This embodiment is basically the same as the first embodiment, the main difference being that the large model in this embodiment is a multimodal model, the input of the multimodal model includes data of some or all modalities in text, image, video and audio, and the output is the processing result of some or all modalities in text, image, video and audio. In addition, this embodiment also provides an improved system for reducing the spur problem of large model training, comprising a microprocessor and a memory connected to each other, the microprocessor being programmed or configured to execute the improved method for reducing the spur problem of large model training. In addition, this embodiment also provides a computer-readable storage medium, the computer-readable storage medium storing a computer program or instruction, the computer program or instruction being programmed or configured to execute the improved method for reducing the spur problem of large model training through a processor. In addition, this embodiment also provides a computer program product, comprising a computer program or instruction, the computer program or instruction being programmed or configured to execute the improved method for reducing the spur problem of large model training through a processor.

[0049] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0050] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An improved method for reducing the spurt problem in large model training, characterized in that: The steps include: when training a large model using a training data set and using an Adam optimizer in combination with a preset loss function to optimize and update the model parameters of the large model, a constant term used by the Adam optimizer to prevent the denominator from being zero Introduction The new constant term used to prevent the denominator from being zero in the improved Adam class optimizer As the number of training iterations increases Increases and the trend is decreasing, among which is the exponential decay rate of the first-order moment estimate, is the exponential decay rate of the second-order moment estimate.

2. The improved method for reducing the spurt problem in large model training according to claim 1 is characterized in that: The function expression of the improved Adam optimizer to optimize and update the model parameters of the large model is: ， ， ， ， In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, and are the second-order moment estimates of the t-th and t-1-th iterations, respectively. The initial first-order moment estimate and second-order moment estimate are 0. is the gradient of the loss function at the tth iteration, is the update amount of the tth iteration, is a new constant term used to prevent the denominator from being zero, is the learning rate of the tth iteration, and are the model parameters for the tth and t-1th iterations respectively.

3. The improved method for reducing the spurt problem in large model training according to claim 1 is characterized in that: The function expression of the improved Adam optimizer to optimize and update the model parameters of the large model is: ， ， ， ， ， In the above formula, is the gradient of the loss function at the tth iteration, is the decay weight, and are the model parameters for the tth and t-1th iterations, respectively. and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, and are the second-order moment estimates of the t-th and t-1-th iterations, respectively. The initial first-order moment estimate and second-order moment estimate are 0. is the update amount of the tth iteration, is a new constant term used to prevent the denominator from being zero, is the learning rate of the tth iteration, is the penalty weight.

4. The improved method for reducing the spurt problem in large model training according to claim 1, characterized in that: The function expression of the improved Adam optimizer to optimize and update the model parameters of the large model is: ， ， ， ， In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, and are the second-order moment estimates of the t-th and t-1-th iterations, respectively. The initial first-order moment estimate and second-order moment estimate are 0. is the maximum value of the second-order moment vector at the tth iteration, Indicates taking the maximum value, and are the model parameters for the tth and t-1th iterations, respectively. is the learning rate for the tth iteration.

5. The improved method for reducing the spurt problem in large model training according to claim 1, characterized in that: The function expression of the improved Adam optimizer to optimize and update the model parameters of the large model is: ， ， ， ， ， In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, and are the second-order moment estimates of the t-th and t-1-th iterations, respectively. The initial first-order moment estimate and second-order moment estimate are 0. is the corrected first-order moment estimate of the t-th iteration, is the corrected second-order moment estimate of the t-th iteration, and are the model parameters for the tth and t-1th iterations, respectively. is the learning rate for the tth iteration.

6. The improved method for reducing the spurt problem in large model training according to claim 1, characterized in that: The function expression of the improved Adam optimizer to optimize and update the model parameters of the large model is: ， ， , ， In the above formula, and are the first-order moment estimates for the t-th and t-1-th iterations, respectively, is the gradient of the loss function at the tth iteration, and are the infinite norms of the t-th and t-1-th iterations respectively, the initial first-order moment estimate and infinite norm are 0, is the learning rate, is the learning rate of the tth iteration, and are the model parameters for the tth and t-1th iterations respectively.

7. The improved method for reducing the spurt problem in large model training according to claim 1, characterized in that: The large model is a natural language processing model, the input of which is text, and the output is the processing result of the text; or the large model is a multimodal model, the input of which includes data of some or all modalities among text, image, video and audio, and the output is the processing result of some or all modalities among text, image, video and audio.

8. An improved system for reducing the spurt problem in large model training, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to perform the improved method for reducing the spurt problem in large model training as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instructions are programmed or configured to execute, through a processor, the improved method for reducing the spurt problem in large model training as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instructions are programmed or configured to execute, through a processor, the improved method for reducing the spurt problem in large model training as described in any one of claims 1 to 7.

Images