Large model accelerated training method based on staged learning
By employing a phased learning approach and dynamic information bottleneck constraints, the problems of wasted computational resources and slow convergence speed in the early stages of large model training are solved, achieving rapid model convergence and efficient training, applicable to multiple vision and language tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING ADVANCED COMPUTING IND DEV CO LTD
- Filing Date
- 2025-12-08
- Publication Date
- 2026-05-08
AI Technical Summary
Existing large-scale model training techniques consume a large amount of computing resources in the early stages of training to learn low-value, redundant, or even noisy features, resulting in wasted computing resources and prolonged training cycles. They also lack an effective internal mechanism for learning patterns from coarse to fine and from primary to secondary.
A phased learning approach is adopted, dividing the training process into a core structure learning period, a detailed feature enrichment period, and a final fine-tuning period. Dynamic information bottleneck constraints are injected into each period. By differentiating information bottleneck constraints and adjusting the constraint strength, the model is guided to learn progressively from macro to micro, reducing redundant feature calculations.
It significantly accelerates the model convergence process, improves training efficiency, model feature learning efficiency, and generalization performance, and is particularly suitable for large-scale model training and resource-constrained application scenarios.
Smart Images
Figure CN121997986A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large model training technology, and in particular to a method for accelerating large model training based on staged learning. Background Technology
[0002] AI large-scale model training and learning refers to the ability of a neural network model to gradually optimize parameters to achieve complex tasks through massive data and computing resources. Its core lies in using deep learning frameworks for multi-level feature extraction and pattern recognition. The training process usually includes two stages: pre-training and fine-tuning. Key technologies involve distributed computing, gradient descent algorithms, and attention mechanisms. The ultimate goal is to build an artificial intelligence system with generalization, reasoning ability, and multimodal processing advantages.
[0003] With the development of deep learning, the scale of models continues to expand. While demonstrating outstanding performance in various tasks, it also brings huge computational overhead and long training cycles. Training large models usually requires several weeks or even months on thousands of GPUs, consuming a lot of computing resources. Therefore, large model training acceleration technology has become a core research focus in academia and industry.
[0004] While existing technologies can improve model performance or training stability to some extent, their core problem lies in the lack of an intrinsic mechanism that can effectively guide the model to follow the cognitive pattern of coarse-to-fine and primary-to-secondary during training. This causes the model, especially in the early stages of training, to consume a lot of computing resources to learn a large number of low-value, redundant, or even noisy features. These features will be corrected or forgotten in subsequent training, resulting in a serious waste of computing resources and an extension of the training cycle.
[0005] To address these issues, we propose a method for accelerating the training of large models based on staged learning. Summary of the Invention
[0006] The purpose of this invention is to provide a method for accelerating the training of large models based on staged learning. By combining the bottleneck constraints of injecting dynamic information in stages with a progressive feature learning mechanism, it solves the problems of wasted computing resources and slow convergence speed in the early stage of training in the prior art.
[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:
[0008] This invention provides a method for accelerating the training of large models based on staged learning, comprising the following steps:
[0009] Step a: Divide the model training process into a continuous core structure learning period, a detailed feature enrichment period, and a final fine-tuning period;
[0010] Step b: During the learning period of the core structure, a first information bottleneck constraint is injected after the first intermediate layer of the network. The loss function of the first information bottleneck constraint is:
[0011] ,
[0012] in To output features based on the first intermediate layer With variational prior distribution Calculated Divergence, calculated as follows:
[0013] ,
[0014] For feature-based The cross-entropy loss of the auxiliary classifier, Let be the first constraint strength coefficient; and let be the total loss function:
[0015] ,
[0016] Network training is conducted, among which Loss to the main task The first weighting coefficient;
[0017] Step c: During the detailed feature enrichment period, a second information bottleneck constraint is injected after the second intermediate layer of the network. The loss function for this second information bottleneck constraint is:
[0018] ,
[0019] in Based on the output features of the second intermediate layer The second constraint strength coefficient is used; and the first constraint strength coefficient is used. It linearly decays from an initial high value to a final low value; and the total loss function is:
[0020] ,
[0021] Network training is conducted, among which For time The coefficient after the change in decay, This is the second weighting coefficient;
[0022] Step d: During the final fine-tuning period, remove the first information bottleneck constraint and the second information bottleneck constraint, using the total loss function. By training the network and injecting differentiated information bottleneck constraints in stages and dynamically adjusting the constraint strength, the model is guided to learn progressively from macro to micro levels. This significantly reduces redundant feature calculations in the early stages of training, accelerates the overall convergence process, and improves the model's feature learning efficiency and generalization ability.
[0023] The present invention is further configured such that, in step a, the division of the training cycle is based on the proportion of the total training time, with the core structure learning period accounting for 20% of the total training cycle, the detailed feature enrichment period accounting for 50% of the total training cycle, and the final fine-tuning period accounting for 30% of the total training cycle. By optimizing the allocation of the training cycle proportions, it is ensured that the model has enough time to establish a stable feature base, progressively learn complex features, and fully converge performance, thereby significantly improving training efficiency and ensuring that the model can obtain the best learning effect at each stage.
[0024] The present invention is further configured such that the first constraint strength coefficient It is set to a fixed high value, specifically 10.0, to apply strong prediction constraints during the core structure learning period. The high constraint strength significantly compresses the amount of feature information, effectively reducing redundant feature learning calculations while maintaining the accuracy of the auxiliary classifier, ensuring that the model can quickly learn the most discriminative core features in the early stages of training.
[0025] The present invention is further configured such that the second constraint strength coefficient It is set to a fixed low value, specifically 1.0, to apply relatively weak constraints during the period of rich detailed features. Moderate constraint strength allows for an increase in feature information, increases feature diversity, enables the model to learn finer-grained features without overfitting, and ensures that the model can fully explore the detailed information of the data during the detailed learning stage.
[0026] The present invention is further configured such that the initial high value of the linear decay is set to 10.0 and the final low value is set to 1.0. The decay process is carried out continuously throughout the period of rich detail features. The smooth linear decay strategy avoids the instability of training caused by abrupt changes in constraint strength, realizes the natural transition from core features to detail features, and ensures that the model can smoothly transition from macro features to micro features during the learning process.
[0027] The present invention is further configured such that the variational prior distribution We adopt a standard normal distribution, which has a mean of 0 and a variance equal to the identity matrix. Using standard Gaussian priors makes Divergence has analytical solutions, which greatly improves computational efficiency, reduces computational overhead, and ensures that the model can converge quickly during training.
[0028] The present invention is further configured such that the first intermediate layer and the second intermediate layer are selected from the Vision Transformer architecture, the intermediate layer corresponding to a specific Transformer Block in the model, and injected after the Transformer Block of the ViT architecture. Constraints can directly affect the feature representation output by the self-attention mechanism, improve the effectiveness of feature selection, and ensure that the model can better capture the key features of the input data.
[0029] The present invention is further configured such that the first intermediate layer is specifically the 16th Transformer Block and the second intermediate layer is specifically the 32nd Transformer Block, which is located deeper than the first intermediate layer. The hierarchical constraint system forms a good feature abstraction hierarchy structure, which gradually reduces the feature compression rate from shallow to deep layers, ensuring that the model can learn appropriate feature representations at different levels.
[0030] The present invention is further configured such that the first weighting coefficient With the second weighting coefficient The values are the same, the and The specific values are all set to 0.1 to control the contribution of the information bottleneck constraint to the total loss. Reasonable weight settings ensure the constraint effect while avoiding dominating the optimization direction, achieving the best constraint effect and ensuring that the model can balance the learning objectives of the main task and auxiliary task during training.
[0031] The present invention is further configured such that the main task loss The cross-entropy loss function is adopted, which is used to calculate the difference between the model's main output and the true label. The cross-entropy loss is used as the main loss and the auxiliary cross-entropy loss in the IB constraint to form a co-optimization, which avoids the gradient conflict problem in multi-objective optimization, makes the training process more stable and smooth, and ensures that the model can converge to the optimal solution quickly.
[0032] The present invention has the following beneficial effects:
[0033] 1. This invention constructs a three-stage progressive training framework, from core feature learning to detailed feature enrichment and finally fine-tuning. Differentiated information bottleneck constraints are applied at each stage to guide the model to follow the feature learning law from macro to micro. This effectively avoids the computational waste on redundant features in the early stage of training, accelerates the overall convergence process of the model, and improves accuracy.
[0034] 2. This invention achieves a balance between feature compression and detail learning by dynamically adjusting the strength and position of information bottleneck constraints at different training stages. This not only improves training efficiency but also ensures the final representation quality and generalization performance of the model. This method has shown excellent acceleration and performance improvement in multiple vision and language tasks and is particularly suitable for large-scale model training and resource-constrained application scenarios. Attached Figure Description
[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.
[0036] Figure 1 This is a flowchart illustrating a method for accelerating the training of large models based on staged learning.
[0037] Figure 2 This is a flowchart illustrating the core structure of a staged learning-based method for accelerating the training of large models.
[0038] Figure 3 A flowchart illustrating the operation during the detailed feature enrichment phase of a method for accelerating the training of large models based on staged learning;
[0039] Figure 4 This is a flowchart of the final fine-tuning operation in a staged learning-based method for accelerating the training of large models.
[0040] Figure 5 This is a flowchart of the intermediate layer selection strategy in a staged learning-based method for accelerating the training of large models. Detailed Implementation
[0041] The technical solutions of the present invention will be described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments.
[0042] Example 1
[0043] Please see Figures 1-5 This invention is a method for accelerating the training of large models based on staged learning, comprising the following steps:
[0044] Step a: Divide the model training process into a continuous core structure learning period, a detailed feature enrichment period, and a final fine-tuning period;
[0045] Step b: During the core structure learning period, a first information bottleneck constraint is injected after the first intermediate layer of the network. The loss function for this first information bottleneck constraint is:
[0046] ,
[0047] in To output features based on the first intermediate layer With variational prior distribution Calculated Divergence, calculated as follows:
[0048] ,
[0049] For feature-based The cross-entropy loss of the auxiliary classifier, Let be the first constraint strength coefficient; and let be the total loss function:
[0050] ,
[0051] Network training is conducted, among which Loss to the main task The first weighting coefficient;
[0052] Step c: During the period of rich detailed features, a second information bottleneck constraint is injected after the second intermediate layer of the network. The loss function for this second information bottleneck constraint is:
[0053] ,
[0054] in Based on the output features of the second intermediate layer The second constraint strength coefficient is used; and the first constraint strength coefficient is used. It linearly decays from an initial high value to a final low value; and the total loss function is:
[0055] ,
[0056] Network training is conducted, among which For time The coefficient after the change in decay, This is the second weighting coefficient;
[0057] Step d: During the final fine-tuning period, remove the first and second information bottleneck constraints, using the total loss function. This method trains the network by injecting differentiated information bottleneck constraints in stages and dynamically adjusting the constraint strength, guiding the model to learn progressively from macro to micro levels. This significantly reduces redundant feature computation in the early stages of training, accelerates the overall convergence process, and improves the model's feature learning efficiency and generalization ability. By constructing a three-stage progressive training framework—from core feature learning to detailed feature enrichment and finally fine-tuning—differentiated information bottleneck constraints are applied at each stage, guiding the model to follow the feature learning pattern from macro to micro. This effectively avoids wasted computation on redundant features in the early stages of training, accelerates the overall convergence process of the model, and improves accuracy. By dynamically adjusting the strength and position of information bottleneck constraints at different training stages, a balance between feature compression and detailed learning is achieved, improving training efficiency while ensuring the final representation quality and generalization performance of the model. This method demonstrates excellent acceleration and performance improvement in multiple vision and language tasks, and is particularly suitable for large-scale model training and resource-constrained application scenarios.
[0058] Specifically: In step a, the training cycle is divided based on the proportion of the total training time. The core structure learning period accounts for 20% of the total training cycle, the detailed feature enrichment period accounts for 50% of the total training cycle, and the final fine-tuning period accounts for 30% of the total training cycle. The first constraint strength coefficient... It is set to a fixed high value, specifically 10.0, to apply a stronger prediction constraint during the core structure learning period; the second constraint strength coefficient. A fixed low value of 1.0 is set to apply a relatively weak constraint during the period of rich detail features. The initial high value for linear decay is set to 10.0, and the final low value is set to 1.0. The decay process continues continuously throughout the entire period of rich detail features, and the variational prior distribution... We adopt a standard normal distribution, which has a mean of 0 and a variance equal to the identity matrix. The first and second intermediate layers are selected from the Vision Transformer architecture. Each intermediate layer corresponds to a specific Transformer Block in the model. The first intermediate layer is specifically the 16th Transformer Block, and the second intermediate layer is specifically the 32nd Transformer Block, located deeper than the first intermediate layer. The first weight coefficient... With the second weighting coefficient The values are the same. and The specific values are all set to 0.1 to control the contribution of the information bottleneck constraint to the total loss, and the main task loss. A cross-entropy loss function is employed to calculate the difference between the model's main output and the true label. By optimizing the allocation of training cycle ratios, sufficient time is ensured for the model to establish a stable feature base, progressively learn complex features, and fully converge performance, significantly improving training efficiency while guaranteeing optimal learning results at each stage. High constraint strength significantly compresses feature information, effectively reducing redundant feature learning computations while maintaining the accuracy of the auxiliary classifier. This ensures the model can quickly learn the most discriminative core features in the early stages of training. Moderate constraint strength allows for increased feature information, increasing feature diversity and enabling the model to learn finer-grained features without overfitting. This ensures the model can fully explore the detailed information of the data during the detail learning stage. A smooth linear decay strategy avoids training instability caused by abrupt changes in constraint strength, achieving a natural transition from core features to detailed features. This ensures the model can smoothly transition from macro-level features to micro-level features during the learning process. A standard Gaussian prior is used to ensure... Divergence has an analytical solution, significantly improving computational efficiency and reducing computational overhead, while ensuring rapid model convergence during training. It is injected after the Transformer Block in the ViT architecture. Constraints directly affect the feature representation output by the self-attention mechanism, improving the effectiveness of feature selection and ensuring that the model can better capture the key features of the input data. The hierarchical constraint system forms a good feature abstraction hierarchy, which gradually reduces the feature compression rate from shallow to deep layers, ensuring that the model can learn appropriate feature representations at different levels. Reasonable weight settings not only ensure the constraint effect but also avoid dominating the optimization direction, achieving the best constraint effect. This ensures that the model can balance the learning objectives of the main task and auxiliary task during training. The use of cross-entropy loss as the main loss and the auxiliary cross-entropy loss in the IB constraint form a synergistic optimization, avoiding gradient conflict problems in multi-objective optimization, making the training process more stable and smooth, and ensuring that the model can quickly converge to the optimal solution.
[0059] Example 2
[0060] Please see Figures 1-4 Building upon Example 1, this method is applied to Vision Transformer (ViT) class models, specifically for image classification tasks, using ImageNet-21K as the training dataset. During the core structure learning phase, a first information bottleneck constraint is injected after the 16th Transformer Block, setting... This method forces the model to learn highly compressed semantic features (such as object outlines and component structures) in the first 20% of training epochs. The amount of feature information is compressed to one-third of that of traditional methods, while the accuracy of the auxiliary classifier remains above 85%, significantly reducing the computational overhead of redundant features. During the period of rich detail features, a second constraint is injected after the 32nd block. Simultaneously, β1 is linearly decayed from 10.0 to 1.0, allowing the model to gradually release feature capacity over 50% of the training epochs, improving feature diversity by 45% and enhancing fine-grained texture information. In the final fine-tuning phase, all constraints are removed, enabling ViT-Giant to achieve the accuracy of a baseline of 500 epochs within 400 epochs, a 25% improvement in convergence speed, making it particularly suitable for high-resolution image classification and semantic segmentation tasks.
[0061] Example 3
[0062] Please see Figures 1-5 Based on Example 1, this method, applied to a hybrid architecture of Convolutional Neural Network (CNN) and Transformer, is suitable for multimodal learning scenarios (such as cross-modal text-image retrieval). During the core structure learning phase, a first information bottleneck constraint (β1=10.0) is injected after the 8th convolutional layer of the CNN backbone network to strongly compress visual features. Simultaneously, a second constraint is injected at the 12th layer of the Transformer encoder. Appropriate constraints are applied to text features. During the period of rich detail features, It linearly decays from 10.0 to 1.0 while maintaining... The visual features remain unchanged, allowing them to gradually enrich in detail while maintaining the stability of the text features. Finally, the constraints are removed during the fine-tuning phase. The model achieves a 3.2% improvement in mAP on the COCO and Flickr30K datasets, and a 20% reduction in training time, making it particularly suitable for joint vision and language modeling tasks.
[0063] Example 4
[0064] Please see Figures 1-5 Building upon Example 1, this method is applied to lightweight model training in resource-constrained environments, using MobileViT as the infrastructure, specifically for mobile image recognition tasks. During the core structure learning phase, the first constraint is injected after the 6th lightweight Transformer Block. Significantly compressing the feature dimension reduces the number of parameters by 40% and computational cost by 35% in the first 20% of training cycles. The second constraint is injected during the detailed feature enrichment period in the 12th block. ,at the same time The model's performance gradually decreases from 10.0 to 1.0, restoring its detailed representation capabilities. During the final fine-tuning phase, constraints are removed, achieving a top-1 accuracy of 78.4% on ImageNet-1K while maintaining a model size of no more than 50MB. This is 18% faster than standard training methods and is suitable for edge device deployment scenarios.
[0065] Example 5
[0066] Please see Figures 1-3 Building upon Example 1, this method is applied to few-shot learning tasks, using ProtoNet as the meta-learning framework and trained on the CUB-200 and MiniImageNet datasets. During the core structure learning phase, a first information bottleneck constraint is injected into the intermediate layer of the feature extractor. This forces the model to learn the most discriminative prototype features with a small number of samples, improving feature discrimination by 30% in the 5-way 1-shot task. A second constraint is injected during the period of rich detail features. and attenuation The method gradually enriches detailed features without overfitting. Finally, the constraints are removed during the fine-tuning phase, achieving an accuracy of 72.1% on the 5-way 5-shot task, 4.7% higher than the baseline method, significantly improving generalization ability with small sample sizes.
[0067] Example 6
[0068] Please see Figures 1-4 Building upon Example 1, this method is applied to high-speed continuous learning systems, specifically for scenarios involving dynamically added categories (such as online product recognition systems). During the core structure learning period, a first constraint is injected into the feature extraction layer each time a new category batch is added. Strengthen the stability of core features and avoid catastrophic forgetting. Inject a second constraint during the period of rich detailed features. It also decays β1 and gradually merges detailed features from both old and new categories. Finally, the constraints are removed during the fine-tuning period. In the CIFAR-100 incremental learning task, the average accuracy is improved by 5.2%, and the forgetting rate is reduced to 8.7%, making it suitable for real-world industrial scenarios that require frequent model updates.
[0069] Example 7
[0070] Please see Figures 1-5 Building upon Example 1, this method is applied to Transformer-based models (such as the Conformer architecture) in the field of speech recognition. For large-scale end-to-end speech recognition tasks, it is trained using the LibriSpeech and MLS datasets. During the core structure learning phase, the first information bottleneck constraint is injected after the 12th layer of the Conformer encoder. Strong compression of acoustic features forces the model to focus on core acoustic patterns at the phoneme level during the first 20% of training epochs, compressing the feature dimension to 40% of the baseline method while maintaining frame-level classification accuracy above 82%, significantly reducing over-modeling of silent segments and background noise. During the period of rich detail features, a second constraint is injected at layer 24. Simultaneously, β1 was linearly decayed from 10.0 to 1.0, allowing the model to progressively learn fine-grained information such as prosody, pitch, and speaker features over 50% of the training epochs. This resulted in a 50% increase in acoustic feature diversity and a 0.8% decrease in word error rate (WER) on the dev-clean set. In the final fine-tuning phase, all constraints were removed, and the model achieved the recognition performance of a baseline of 400 epochs within 300 epochs, improving training efficiency by 25%. This makes it particularly suitable for robust recognition tasks in low-resource and noisy environments.
[0071] Example 8
[0072] Please see Figures 1-4 Building upon Example 1, this method is applied to feature alignment and stable training of generative models (such as diffusion models or GANs), targeting high-resolution image generation tasks, using FFHQ and ImageNet as benchmarks. During the core structure learning phase, a first information bottleneck constraint is injected after the fourth residual block of the downsampling path in the UNet architecture. This forces the model to learn a structured representation of the latent space during the early stages of training, reducing the KL divergence of the latent feature distribution by 35% and significantly improving the structural consistency of the generated images. During the period of rich detail features, a second constraint is injected after the 8th block of the upsampling path. Meanwhile, β1 linearly decays from 10.0 to 1.0, allowing the model to gradually unleash its detail generation capabilities, improving texture richness and color saturation by 40% and avoiding mode collapse and detail blurring issues. Finally, the constraints are removed during the fine-tuning phase, resulting in a 2.3 improvement in FID and a 0.8 improvement in IS for the generated images, along with a 30% increase in training stability. This makes the model suitable for scenarios with extremely high detail fidelity requirements, such as high-fidelity face generation, artistic creation, and medical image synthesis.
[0073] Example 9
[0074] Please see Figures 1-5 Based on Example 1, this method is applied to multi-sensor fusion perception tasks in autonomous driving. Specifically, it is used for training on nuScenes and Waymo OpenDataset for 3D target detection scenarios involving the fusion of LiDAR and cameras. During the core structure learning phase, first information bottleneck constraints are injected after the PointNet++ module of the point cloud feature extraction network and after the third stage of the ResNet-50 image feature extraction network. The model is forced to learn core geometric features across modalities within the first 20% of training epochs, reducing the spatial alignment error between point cloud and image feature embeddings by 45%, while maintaining the resolution of the BEV feature map at 1 / 8 of the original image and ensuring the integrity of key object contours. During the period of rich detail features, a second constraint is injected after the Transformer cross-attention layer of the fusion network. Simultaneously, β1 is linearly decayed from 10.0 to 1.0, allowing the model to progressively learn fine-grained texture, material, and motion features within 50% of the training epochs. This improves multimodal feature complementarity by 60% and increases the recall rate for occluded targets by 12%. Finally, all constraints are removed during the fine-tuning phase, achieving a mAP of 42.3% on the nuScenes dataset, a 3.7% improvement over the baseline method, and reducing the training epochs by 180 epochs. This approach is particularly suitable for autonomous driving perception systems with extremely high real-time and accuracy requirements.
[0075] Example 10
[0076] Please see Figures 1-3 Building upon Example 1, this method is applied to small anomaly sample learning scenarios in industrial defect detection. Specifically, it is used for training on PCB board defect detection and metal surface crack identification tasks, employing MVTec AD and a self-built industrial dataset. During the core structure learning phase, a first information bottleneck constraint is injected after the third layer of the Feature Pyramid Network (FPN). The model is forced to learn the core feature patterns of normal samples even under conditions of extreme imbalance between positive and negative samples (abnormal samples account for only 1%), reducing the feature reconstruction error of normal samples to below 0.05, while simultaneously increasing the significance of abnormal samples by 85%. During the period of rich detail features, a second constraint is injected after the second layer of the decoder. ,at the same time The coefficient of performance (COP) is linearly reduced from 10.0 to 1.0, allowing the model to gradually learn the detailed features of minute defects, such as micron-level anomalies like weld cracks and scratches. This improves the F1-score for defect detection to 0.92 and reduces the false alarm rate to 0.5%. Finally, the constraints are removed during the fine-tuning phase, achieving an AUROC of 98.2% on the MVTEC AD dataset, a 4.3% improvement over existing methods. Training data requirements are reduced by 50%, making it suitable for industrial vision scenarios with high defect sensitivity and scarce labeled samples, such as product quality control and precision manufacturing.
[0077] The preferred embodiments of the present invention disclosed above are only for the purpose of illustrating the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation described herein. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can better understand and utilize the present invention.
Claims
1. A method for accelerating the training of large models based on staged learning, characterized in that: Includes the following steps: Step a: Divide the model training process into a continuous core structure learning period, a detailed feature enrichment period, and a final fine-tuning period; Step b: During the learning period of the core structure, a first information bottleneck constraint is injected after the first intermediate layer of the network. The loss function of the first information bottleneck constraint is: , in To output features based on the first intermediate layer With variational prior distribution Calculated Divergence, calculated as follows: , For feature-based The cross-entropy loss of the auxiliary classifier, Let be the first constraint strength coefficient; and let be the total loss function: , Network training is conducted, among which Loss to the main task The first weighting coefficient; Step c: During the detailed feature enrichment period, a second information bottleneck constraint is injected after the second intermediate layer of the network. The loss function for this second information bottleneck constraint is: , in Based on the output features of the second intermediate layer The second constraint strength coefficient is used; and the first constraint strength coefficient is used. The loss linearly decays from an initial high value to a final low value; and the total loss function is: , Network training is conducted, among which For time The coefficient after the change in decay, This is the second weighting coefficient; Step d: During the final fine-tuning period, remove the first information bottleneck constraint and the second information bottleneck constraint, using the total loss function. Train the network.
2. The method for accelerating large model training based on staged learning according to claim 1, characterized in that: In step a, the division of training cycles is based on the proportion of total training time. The core structure learning period accounts for 20% of the total training cycle, the detailed feature enrichment period accounts for 50% of the total training cycle, and the final fine-tuning period accounts for 30% of the total training cycle.
3. The method for accelerating large model training based on staged learning according to claim 1, characterized in that: First constraint strength coefficient It is set to a fixed high value, specifically 10.0, to apply a strong prediction constraint during the core structure learning period.
4. The method for accelerating large model training based on staged learning according to claim 1, characterized in that: Second constraint strength coefficient It is set to a fixed low value, specifically 1.0, to impose relatively weak constraints during periods of rich detail features.
5. The method for accelerating large model training based on staged learning according to claim 1, characterized in that: The initial high value of the linear decay is set to 10.0, and the final low value is set to 1.
0. The decay process is carried out continuously throughout the entire period of rich detail features.
6. The method for accelerating large model training based on staged learning according to claim 1, characterized in that: The variational prior distribution We adopt a standard normal distribution, which has a mean of 0 and a variance equal to the identity matrix. .
7. The method for accelerating large model training based on staged learning according to claim 1, characterized in that: The first intermediate layer and the second intermediate layer are selected from the Vision Transformer architecture, and the intermediate layer corresponds to a specific Transformer Block in the model.
8. The method for accelerating large model training based on staged learning according to claim 7, characterized in that: The first intermediate layer is specifically the 16th Transformer Block, and the second intermediate layer is specifically the 32nd Transformer Block, which is located deeper than the first intermediate layer.
9. The method for accelerating large model training based on staged learning according to claim 1, characterized in that: The first weighting coefficient With the second weighting coefficient The values are the same, the and The specific values are all set to 0.1 to control the contribution of the information bottleneck constraint to the total loss.
10. The method for accelerating large model training based on staged learning according to claim 1, characterized in that: The main task loss The cross-entropy loss function is used to calculate the difference between the model's main output and the true label.