Large model accelerated training method based on staged learning
Through the stage-based learning method, dynamically evaluate and adjust the training process of large neural network models, the problem of difficulty in taking into account speed and compatibility in the existing technology is solved, and the rapid convergence and cross-domain adaptability of the model are achieved, which significantly reduces the training time .
Patent Information
- Application Number
- CN202410986288.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
There are problems in the training process of existing large neural network models that are difficult to take into account both speed and compatibility, which leads to the inability to improve training speed and adapt to different data sets and network architectures at the same time.
Using a phased learning method, by initializing a small number of initial training layers as sub-models, the growth scheduler dynamic evaluation is constructed and a new training layer is inserted, the learning rate is adjusted, the number of layers is gradually increased and the learning parameters are adjusted until the preset standard is reached. Achieve adaptive model growth.
The convergence process of the model is significantly accelerated, and the generality of the model is improved and the adaptability across architectures and fields is improved, training time is reduced, and prediction performance is maintained.
Smart Images

Figure CN118940802A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technology in the field of artificial intelligence, and in particular to a large model accelerated training method based on staged learning. Background Art
[0002] In recent years, large neural network models have been widely used in computer vision and natural language processing due to their good prediction and discrimination performance. In pursuit of higher performance, the parameter scale of network models is getting larger and larger. Training large models requires a lot of computing resources. Improving the training efficiency of neural network models has great commercial value.
[0003] At present, the acceleration methods based on multi-stage learning can be mainly divided into two directions: optimizing the initialization process before model growth and selecting the appropriate model growth time. Optimizing the initialization process before model growth can increase the model width by copying existing neurons and increase the depth by constructing an identity mapping. This method can maintain the function of the original model, but it will lead to sparsity problems, that is, there are a large number of redundant and unnecessary parameters in the model; on this basis, a clone initialization is derived, which constructs a new layer by simply cloning existing parameters to save computational overhead, but this method has limitations in terms of versatility and cannot adapt to different types of tasks or data sets. Choosing the appropriate model growth time can be achieved by optimizing the measurement function, such as using indicators based on the Lipschitz constant, but this method also has limitations and cannot adapt to different network architectures. Therefore, existing studies mainly adopt simple and fixed strategies. This method cannot take into account both the training speed of large models and the compatibility of multiple data sets at the same time. Considering only one aspect often leads to a loss of both, resulting in the failure to achieve the expected effect of improving the final training speed of large models. Summary of the invention
[0004] In order to overcome the defect of the above-mentioned prior art that both speed and compatibility cannot be taken into account, the present invention provides a large model accelerated training method based on staged learning.
[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:
[0006] A large model accelerated training method based on staged learning includes the following steps:
[0007] S1. Initialize the large model to be trained, select a small number of initial training layers from the large model as newly added training layers to form a sub-model to start training, and build a growth scheduler;
[0008] S2. Determine whether the sub-model triggers model growth based on the growth scheduler; when the model growth is triggered, construct and initialize a new training layer as a new training layer to be inserted into the sub-model;
[0009] S3, the training process of the sub-model after the learning rate is steadily increased by dynamically adjusting;
[0010] S4. Repeat steps S2 to S3 until the preset training cycle is reached to obtain a large model that has completed the training.
[0011] The present invention also proposes a large model accelerated training system based on staged learning, the system comprising:
[0012] A growth scheduling unit determines whether to trigger model growth based on the growth scheduler and a preset threshold when the sub-model is being trained;
[0013] Adaptive growth operation unit, when model growth is triggered, constructs and initializes new training layers and inserts them into the sub-model;
[0014] The learning rate adjustment unit is used to stabilize the training process after the model growth is completed.
[0015] The present invention also proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the large model accelerated training method as described in the present invention is implemented.
[0016] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the large model accelerated training method as described in the present invention is implemented.
[0017] The present invention proposes a large model accelerated training method based on staged learning, which decomposes the large model training into multiple progressive stages and adopts an incremental learning strategy from local to global to achieve it; by using a growth scheduler for dynamic evaluation, the number of layers is gradually increased and the learning parameters are adjusted until the scale and function of the sub-model reach the preset standards of the complete model. Compared with the existing acceleration methods based on learning in multiple stages, this invention can dynamically select the timing of growth, the number of new layers and the learning rate according to the state changes of the model during training while improving the training speed, so as to adaptively match the characteristics of the task. This dynamic adjustment mechanism enables the present invention to not only significantly accelerate the convergence process of the model, but also improve the versatility and cross-architecture and cross-domain adaptability of the model by adapting to various task characteristics. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 This is a flow chart of the large model accelerated training method based on staged learning in Example 1.
[0019] Figure 2 This is an architectural diagram of the large model accelerated training system based on staged learning in Example 3. DETAILED DESCRIPTION
[0020] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present invention;
[0021] It is understandable to those skilled in the art that some well-known descriptions may be omitted in the drawings.
[0022] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0023] Example 1
[0024] This embodiment proposes a large model accelerated training method based on staged learning, such as Figure 1 As shown, it is a flow chart of the large model accelerated training method based on staged learning in this embodiment.
[0025] The large model accelerated training method based on staged learning proposed in this embodiment includes the following steps:
[0026] S1. Initialize the large model to be trained, select a small number of initial training layers from the large model as newly added training layers to form a sub-model to start training, and build a growth scheduler;
[0027] S2. Determine whether the sub-model triggers model growth based on the growth scheduler; when the model growth is triggered, construct and initialize a new training layer as a new training layer to be inserted into the sub-model;
[0028] S3, the training process of the sub-model after the learning rate is steadily increased by dynamically adjusting;
[0029] S4. Repeat steps S2 to S3 until the preset training cycle is reached to obtain a large model that has completed the training.
[0030] In this embodiment, by driving progressive optimization, the large model training is decomposed into multiple progressive stages and implemented using a staged learning strategy from local to global. Specifically, in this embodiment, a small number of initial training layers will be selected from the large model as sub-models to start training. The smaller sub-models have a faster convergence speed and can provide good initialization for learning in subsequent stages. Furthermore, the gradient signal in the training process is used to guide the growth process of the model, which is conducive to achieving faster convergence of the sub-model and does not bring additional computational burden. The method is compatible with mainstream training methods, providing good versatility for the present invention, so that the present invention can not only be adapted to models of different network architectures, but also can be applied to cross-domain training tasks. In addition, by dynamically adjusting the learning rate to stabilize the growth process of the sub-model, the performance oscillation problem that may occur in the sub-model after growth can also be alleviated.
[0031] As an example, the target model for training is GPT-2, which contains a 12-layer network and is based on the Transformer network. Since the size of the input and output of each layer remains unchanged, the model is evenly divided into 4 parts, and 1, 4, 7, and 10 are selected as the initial training layers to form a sub-model to start training. As another example, a model based on the ResNet network usually consists of 4 parts, and the first layer of each part will change the size of the input data, so the first layer of each part is usually selected as the initial training layer to form a sub-model to start training.
[0032] In an optional embodiment, the step of constructing the growth scheduler is: calculating the L2 norm of the newly added training layer gradient vector in the sub-model, and accumulating the L2 norm of each newly added training layer gradient vector; its expression is:
[0033] G=Σ ω′ ∥g∥2
[0034] Among them, G represents the growth scheduler, g is the gradient vector, and ω' represents the parameters of the newly added training layer.
[0035] In this embodiment, the sum of the L2 norms of the gradient vectors of the newly added training layers is defined as the growth scheduler, and its absolute value reflects the stride of the sub-model update. Specifically, the L2 norm, also known as the Euclidean norm, is obtained by summing the squares of all elements of the gradient vector and then taking the square root. It directly reflects the overall size of the gradient. If the L2 norm of the gradient is close to zero, it may indicate that the model is approaching a local optimum. These gradient information reflects the update state of the model in the solution space and improves versatility, enabling the method to be applied to cross-architecture and cross-domain tasks.
[0036] In an optional embodiment, the step of using the growth scheduler to determine whether to trigger model growth includes: calculating the absolute value of the slope of the growth scheduler, and triggering model growth when the absolute value of the slope of the growth scheduler is lower than a preset threshold.
[0037] In this embodiment, the absolute value of the slope of the growth scheduler is used to determine whether to trigger model growth. Specifically, since the order of magnitude of the growth scheduler varies greatly in different training stages, it will be scaled according to the initial value; after each growth step, the scaling factor will also be updated with the change of the new activation layer; when the absolute value of the slope of the growth scheduler is lower than a certain threshold, it may mean that the model is close to the local optimal point. At this time, the sub-model growth will be triggered, that is, the gradient is the root cause of driving progressive optimization.
[0038] As an exemplary explanation, the preset threshold is found through a large number of experiments to be a non-sensitive parameter, and when it is set to 0.5, it can meet the needs of most situations.
[0039] In an optional embodiment, the step of constructing and initializing a new training layer includes:
[0040] S4.1: Estimate the optimal number of additional training layers;
[0041] S4.2: Initialize the parameters of the newly added training layer through soft cloning;
[0042] Furthermore, the step of estimating the optimal number of newly added training layers includes: introducing a custom parameter r to align the residual indicator RI with r ideal training layers, and outputting the optimal number of newly added training layers as Among them, the expression of the residual indicator RI is:
[0043]
[0044] Where n represents the number of newly added training layers in the previous training stage, g′ represents the gradient of the layer during back propagation, γ represents the learnable weight, and ω′ represents the parameter of the layer.
[0045] Furthermore, the step of initializing the parameters of the newly added training layer by soft cloning includes:
[0046] Initializing a new training layer based on the optimal number of newly constructed training layers, wherein the new training layer inherits the knowledge learned by the sub-model in the previous training stage, including parameters of each layer;
[0047] The weights and biases of the normalization layer in the newly added training layer are set to zero, and other parameters except the normalization layer are kept unchanged to obtain the parameters of the newly added training layer.
[0048] In this embodiment, RI is used to estimate the number of new layers to be activated in the growth step. If RI is still relatively small after sufficient updates, it means that the model cannot learn more from training and the global optimal point may be very close; therefore, activating only one layer will only show a small improvement. In this case, more layers can be activated to quickly reach the final performance; on the contrary, when RI is very high, the model may be far from the global optimal point and it will take many iterations to get close; if too many layers are activated at one time, the computational burden will increase significantly and the convergence speed will slow down; therefore, a custom parameter r is introduced to align RI with an ideal layer; each model growth will activate a new layer until the entire model is optimized.
[0049] In the step of initializing the parameters of the newly added training layer by soft cloning, the newly added training layer is initialized by knowledge reuse and function retention. Specifically, by setting the initial values of the learnable weights and bias items to zero while keeping other clone parameters unchanged, the identity can be achieved with the help of the residual structure; on the one hand, it retains the function of the sub-model as much as possible to avoid repeated training; on the other hand, its identity property is also conducive to the stability of subsequent training; further, the parameters of the newly added training layer initialized by soft cloning are inserted into the original sub-model to achieve model growth.
[0050] As an example, in the ResNet network, a single-layer structure consists of a convolutional layer and a batch normalization layer; in the Transformer network, a single-layer structure consists of a multi-head attention layer or a feedforward layer plus a normalization layer. For each newly added training layer, the soft clone initialization performs the following operations: copy all parameters of its original training layer, including the parameters of each specific layer; set the weights and bias of the normalization layer to zero, and keep the parameters of other layers unchanged.
[0051] As another exemplary illustration, the target model for training is set to GPT-2, which contains a 12-layer network. When the training of the sub-model with the selected initial training layers 1, 4, 7, and 10 is close to convergence, the absolute value of the growth scheduler slope is less than 0.5, and the estimated optimal number of growth layers is calculated to be 4. One layer is added to each part, corresponding to the 2nd, 5th, 8th, and 11th layers. The newly added training layers are initialized by soft cloning and inserted into the original sub-model to form a sub-model containing 8 layers.
[0052] In an optional embodiment, the step of stabilizing the growth process of the sub-model by dynamically adjusting the learning rate includes:
[0053] Set the initial learning rate to a certain ratio of the original learning rate;
[0054] The learning rate is gradually increased to the original learning rate through the [π, 2π] segment of the cosine function; wherein the adjustment of the learning rate follows the formula:
[0055]
[0056] Among them, η min represents the minimum learning rate, η ori is the original learning rate before growth, T cur is the number of iterations after growth, T tol is the total number of warmup iterations.
[0057] Specifically, after each growth step, as the parameters are updated, the functions of the old layers will also change, which may lead to a brief drop in prediction performance. After each model growth, the sub-model training starts with a smaller learning rate to warm up the grown sub-model, and then gradually increases to the original learning rate for subsequent training.
[0058] In this embodiment, reducing the update step by another method can also alleviate this performance fluctuation, but it also reduces the training efficiency to a certain extent; in practice, maintaining a smaller update step brings little benefit, and the model usually recovers its performance within a few training cycles; by dynamically adjusting the learning rate, not only the gradient update of the growth model is smoother, thereby reducing performance fluctuations, but also the training efficiency can be maximized.
[0059] Example 2
[0060] This embodiment applies the large model accelerated training method based on staged learning proposed in Embodiment 1 to the training of different large models for different data sets.
[0061] This embodiment uses different large models to train different data sets, as shown in Table 1 below, which compares the results of using the accelerated training method of this embodiment to train the CIFAR data set through different large models and using traditional training methods.
[0062] Table 1 Comparison of training results on the CIFAR dataset
[0063]
[0064] The CIFAR (Canadian Institute For Advanced Research) dataset used in this embodiment mainly has two versions: CIFAR-10 and CIFAR-100.
[0065] The CIFAR-10 dataset contains 60,000 32x32 color images, divided into 10 categories, with 6,000 images in each category; the categories include airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks; the number of images in each class is roughly the same, which is suitable for small-scale image classification tasks.
[0066] The CIFAR-100 dataset contains 60,000 32x32 color images, divided into 100 classes, with 600 images in each class. The small number of images in each class increases the difficulty of the classification task and is suitable for more sophisticated image classification tasks.
[0067] This embodiment uses TOP1 accuracy and TOP5 accuracy as the accuracy performance evaluation criteria. Among them, TOP1 accuracy indicates the proportion of correct predictions to the total number of predictions, and TOP5 accuracy indicates the proportion of predictions of the correct category in the top five to the total number of predictions. The higher the TOP1 accuracy and TOP5 accuracy, the higher the accuracy of this training. Furthermore, the training running time is recorded and compared, and the speed performance gap is shown by the proportion of running speed reduction.
[0068] In this embodiment, three different models, ResNet-18, ResNet-152, and SENet-152, are selected to train the CIFAR dataset. Figure 2 It can be seen that the large model accelerated training method based on staged learning proposed in the present invention has shown a significant effect in reducing the training running time, reducing the running time by 30.5% to 52.2%. In terms of Top-1 and Top-5 accuracy, in most cases, it maintains performance similar to the original method, with little overall change.
[0069] As shown in Table 2 below, the accelerated training method of this embodiment is used to train the ImageNet dataset through different large models, and the comparison results of training using traditional methods are shown.
[0070] Table 2 Comparison of the training results on the ImageNet dataset
[0071]
[0072] The ImageNet dataset used in this embodiment contains more than 1.28 million annotated images, covering about 1,000 categories, including animals, plants, transportation, daily objects, etc. The coverage is very wide, covering various scenes and objects in real life, and providing extremely high diversity and complexity.
[0073] In this embodiment, two different models, ResNet-50 and DeiT-S, are selected to train the ImageNet dataset. As shown in Table 2, the large model accelerated training method based on staged learning proposed in the present invention shows a significant effect in reducing the training running time, reducing the running time by 31.9% to 34.3%. In terms of Top-1 accuracy, in most cases, the performance is similar to that of the original method, and the overall change is not large.
[0074] Furthermore, for different large models and data sets, the accelerated training method proposed in the present invention can improve the training speed to a certain extent and ensure the accuracy of the training.
[0075] Example 3
[0076] This embodiment proposes a large model accelerated training system based on staged learning, and applies the large model accelerated training method based on staged learning proposed in Example 1. Figure 2 As shown, it is the architecture diagram of the acceleration training system of this embodiment.
[0077] This embodiment proposes a large model accelerated training system based on staged learning, including:
[0078] A growth scheduling unit determines whether to trigger model growth based on the growth scheduler and a preset threshold when the sub-model is being trained;
[0079] Adaptive growth operation unit, when model growth is triggered, builds and initializes new training layers and inserts them into the sub-model;
[0080] The learning rate adjustment unit is used to stabilize the training process after the model growth is completed.
[0081] It can be understood that the system of this embodiment corresponds to the method of the above-mentioned embodiment 1, and the options in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described repeatedly here.
[0082] Example 4
[0083] This embodiment proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor executes the steps of the large model accelerated training method based on staged learning proposed in Example 1.
[0084] Example 5
[0085] This embodiment proposes a storage medium on which computer-readable instructions are stored, wherein the computer-readable instructions, when executed by a processor, implement the steps of the large model accelerated training method based on staged learning proposed in Example 1.
[0086] Exemplarily, the storage medium includes, but is not limited to, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0087] Exemplarily, the instructions, programs, code sets or instruction sets may be implemented using conventional programming languages.
[0088] Exemplarily, the processor includes but is not limited to a smart phone, a personal computer, a server, a network device, etc., and is used to execute all or part of the steps of the large model accelerated training method based on staged learning described in Example 1.
[0089] The terms in the drawings are for illustrative purposes only and should not be construed as limiting this patent;
[0090] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A large model accelerated training method based on staged learning, characterized in that: The following steps are involved: S1. Initialize the large model to be trained, select a small number of initial training layers from the large model as newly added training layers to form a sub-model to start training, and build a growth scheduler; S2. Determine whether to trigger model growth for the sub-model based on the growth scheduler; When the model growth is triggered, a new training layer is constructed and initialized to be inserted into the sub-model as a new training layer; S3, the training process of the sub-model after the learning rate is steadily increased by dynamically adjusting; S4. Repeat steps S2 to S3 until the preset training cycle is reached to obtain a large model that has completed the training.
2. According to the large model accelerated training method based on staged learning according to claim 1, it is characterized in that: In step S1, the steps of constructing the growth scheduler are: calculating the L2 norm of the newly added training layer gradient vector in the sub-model, and accumulating the L2 norm of each newly added training layer gradient vector; Its expression is: G=∑ ω′ ∥g∥2 Among them, G represents the growth scheduler, g is the gradient vector, and ω' represents the parameters of the newly added training layer.
3. A large model accelerated training method based on staged learning according to claim 2, characterized in that: In step S2, the step of using the growth scheduler to determine whether to trigger model growth includes: updating the growth scheduler, calculating the absolute value of the slope of the growth scheduler, and triggering model growth when the absolute value of the slope of the growth scheduler is lower than a preset threshold.
4. According to the large model accelerated training method based on staged learning according to claim 1, it is characterized in that: In step S2, the steps of constructing and initializing a new training layer include: S4.1: Estimate the optimal number of additional training layers; S4.2: Initialize the parameters of the newly added training layer through soft cloning.
5. The large model accelerated training method based on staged learning according to claim 4 is characterized in that: In step S4.1, the step of estimating the optimal number of newly added training layers includes: introducing a custom parameter r to align the residual indicator RI with r ideal training layers, and outputting the optimal number of newly added training layers to be constructed as Among them, the expression of the residual indicator RI is: Where n represents the number of newly added training layers in the previous training stage, g′ represents the gradient vector of the layer during back propagation, γ represents the learnable weight, and ω′ represents the parameter of the layer.
6. A large model accelerated training method based on staged learning according to claim 4, characterized in that: In step S4.2, the step of initializing the parameters of the newly added training layer by soft cloning includes: Initializing a new training layer based on the optimal number of newly constructed training layers, wherein the new training layer inherits the knowledge learned by the sub-model in the previous training stage, and the knowledge includes parameters of each layer; The weights and biases of the normalization layer in the newly added training layer are set to zero, and other parameters except the normalization layer are kept unchanged to obtain the parameters of the newly added training layer.
7. The large model accelerated training method based on staged learning according to claim 1 is characterized in that: In the step S3, the step of stabilizing the growth process of the sub-model by dynamically adjusting the learning rate includes: Set the initial learning rate to a certain ratio of the original learning rate; The learning rate is gradually increased to the original learning rate through the [,2π] segment of the cosine function; wherein the adjustment of the learning rate follows the formula: Among them, η min represents the minimum learning rate, η ori is the original learning rate before growth, T cur is the number of iterations after growth, T tol is the total number of warmup iterations.
8. A large model accelerated training system based on staged learning, applied to the large model accelerated training method according to any one of claims 1 to 7, characterized in that: The system comprises: A growth scheduling unit determines whether to trigger model growth based on the growth scheduler and a preset threshold when the sub-model is being trained; Adaptive growth operation unit, when model growth is triggered, builds and initializes new training layers and inserts them into the sub-model; The learning rate adjustment unit is used to stabilize the training process after the model growth is completed.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the large model accelerated training method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large model accelerated training method as described in any one of claims 1 to 7 is implemented.
Citation Information
Cited By
AI large model training method based on deep learning
CN122389943A
A method for training an AI large model based on deep learning
CN122389943B
