An extended and data difficulty driven large model pre-training acceleration method

Abstract

The application provides an extended and data difficulty driven large model pre-training acceleration method, including the following steps: obtaining a pre-trained base model, and performing difficulty evaluation on samples in a pre-training corpus, and dividing the samples into data subsets of multiple difficulty levels according to the evaluation results; performing structural expansion on the base model to obtain expansion parameters; the structural expansion includes depth expansion and / or width expansion; the depth expansion is realized by copying a Transformer block in an intermediate layer region of the model and introducing a gated residual mechanism, and the width expansion is realized by adopting a combination of direct copying and near-zero initialization on a weight matrix; dividing a training process into multiple training stages, and sampling training samples from data subsets of different difficulty levels according to the expansion state of the model in different training stages. The application realizes matching of data complexity and model capacity, reduces training time and algorithmic power consumption, and guarantees model performance.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model training optimization technology, and in particular to a method for accelerating large model pre-training based on expansion and data difficulty. Background Technology

[0002] In recent years, large language models have demonstrated outstanding performance in tasks such as natural language understanding, generation, and reasoning, becoming a crucial infrastructure for general artificial intelligence. However, the improvement of large models' capabilities heavily relies on the "law of scale," namely the coordinated growth of model parameters, training data, and computational cost. In practical training, scaling from hundreds of millions of parameters to billions or more typically involves de novo pre-training or direct, full-parameter intensive updates on the large structure. This approach presents two prominent problems: firstly, the computational cost is exorbitant. As the parameter scale increases, the single-step floating-point operations and distributed communication overhead rise significantly, with training cycles often lasting weeks to months, resulting in enormous resource and energy consumption; secondly, the training path is inefficient. De novo training wastes a significant amount of computing power on "relearning" the basic language rules already mastered by smaller models, leading to significant redundancy.

[0003] Furthermore, directly performing intensive updates on the expanded model can lead to optimization instability. The newly added parameter subspace differs significantly from the original model's parameter distribution. If random initialization or uncontrolled training is used, it can easily cause loss oscillations, gradient anomalies, slower convergence, and even performance degradation in the early stages of training. Especially during depth expansion, increasing network depth lengthens the gradient propagation path, making the model more sensitive to learning rate, initialization, and data distribution. During width expansion, the original representation space needs to be embedded into a higher-dimensional space; if the strategy is inappropriate, it may disrupt the attention patterns and feature decomposition structures already formed in the original model.

[0004] From a data perspective, pre-training corpora exhibit a significant long-tail distribution: a large number of "easy samples" are highly repetitive and have simple patterns, while a small number of "difficult samples" contain complex reasoning or scarce knowledge. Traditional pre-training often organizes data using fixed proportions or random sampling, failing to fully utilize the "from easy to difficult" learning pattern. This problem is even more pronounced in extended training scenarios: in the early stages of extension, the model should prioritize alignment on low-noise samples with a distribution consistent with the source model's capabilities to avoid interference from the noise gradients of high-difficulty samples with newly added parameters; in the later stages of extension, the proportion of high-difficulty samples should be increased to encourage the new data to learn complex structures. A lack of difficulty-driven data scheduling leads to unstable training, low sample efficiency, and consequently affects the overall acceleration effect.

[0005] Therefore, there is an urgent need for a pre-training acceleration method that can smoothly scale from small trained models to large models, and can significantly reduce training time and computing power consumption without sacrificing performance as much as possible. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a method for accelerating large model pre-training based on expansion and data difficulty-driven approaches. This method utilizes a pre-trained base model for structural expansion and combines it with multi-stage training driven by data difficulty to reduce training time and computational power consumption while ensuring model performance.

[0007] To achieve the above objectives, this invention provides a method for accelerating large model pre-training based on expansion and data difficulty, comprising the following steps: Obtain the pre-trained base model; The base model is used to perform word-by-word prediction for each sample in the pre-training corpus. Based on the prediction probability distribution of each word, multiple basic difficulty indicators of the sample are calculated and the multiple basic difficulty indicators are merged into a scalar difficulty score. Based on the scalar difficulty score, the sample is divided into data subsets of multiple difficulty levels. The base model is structurally expanded to obtain expansion parameters; the structural expansion includes depth expansion and / or width expansion; the depth expansion is achieved by copying the Transformer block in the middle layer region of the model and introducing a gated residual mechanism, and the width expansion is achieved by combining direct copying and near-zero initialization of the weight matrix; The training process is divided into multiple training stages, and training samples are sampled from data subsets of different difficulty levels in different training stages according to the expansion state of the model.

[0008] Furthermore, the aforementioned basic difficulty metrics include Shannon entropy, Rényi entropy, and negative log-likelihood; The step of integrating the multiple basic difficulty indicators into a scalar difficulty score includes: combining the Shannon entropy, Rényi entropy, and negative log-likelihood into an indicator vector; normalizing the indicator vector using the Softmax function to obtain adaptive weights; and summing the element-wise multiplication of the adaptive weights with the indicator vector to obtain the scalar difficulty score.

[0009] Furthermore, the step of dividing the sample into multiple difficulty levels based on the scalar difficulty score further includes: All samples are globally sorted according to the scalar difficulty score of each sample, and the sorted samples are divided into at least three difficulty levels according to the preset quantile interval. The at least three difficulty levels include easy sample level, medium difficulty sample level and hard sample level. The easy sample level corresponds to the preset quantile interval with the lowest scalar difficulty score, the difficult sample level corresponds to the quantile interval with the highest scalar difficulty score, and the medium difficulty sample level corresponds to the quantile interval with the scalar difficulty score between the lowest and the highest.

[0010] Furthermore, the gated residual mechanism introduced in the depth expansion is as follows: Introduce a schedulable scalar gating factor for each replicated block. Output of the copy block Represented as: ,in This represents the current number of training steps. For input features, The residual term generated for the copy block; the gating factor gradually increases from close to 0 to the maximum value according to the cosine annealing curve and then decays, so that the copy block smoothly transitions from the bypass state to fully participating in the representation modeling.

[0011] Furthermore, the initialization of the copied block adopts the following strategy: The structural parameters of the attention sublayer and feedforward network sublayer inside the copy block are directly copied from the corresponding original Transformer block. For the key weights in the replication block responsible for writing the sublayer output back to the backbone residual path, a very small-scale orthogonal initialization is used to ensure that the residual output of the replication block in the initial state is optimal. satisfy ; The key weights include attention output projection weights and / or feedforward network final layer projection weights.

[0012] Furthermore, the width expansion is achieved by combining direct copying with near-zero initialization of the weight matrix, including the following steps: For the linear layer weight matrix and word embedding matrix, the weight values ​​corresponding to the original input and output dimensions in the source weight matrix are directly copied to the same positions in the expanded weight matrix. Both the newly added input dimension and the newly added output dimension are initialized with values ​​close to zero.

[0013] Furthermore, in the width expansion, the projection matrix of the self-attention module is split, expanded, and recombined according to attention heads or query groups. Specifically, this includes: obtaining the original projection weight matrix of the self-attention module in the base model, and splitting the original projection weight matrix into multiple sub-weight matrices of attention heads or query groups according to the output dimension. Each sub-weight matrix is ​​independently widened. The expanded sub-weight matrices retain the weight values ​​of the original dimensions, and the newly added dimensions are initialized with values ​​close to zero. All the expanded sub-weight matrices are reassembled along the output dimension to form the projected weight matrix of the expanded self-attention module.

[0014] Furthermore, the width expansion also includes an expansion of the normalized layer parameters: Obtain the one-dimensional scaling parameters of each normalized layer in the base model, and directly inherit the parameter values ​​corresponding to the original channels to the corresponding channel positions of the expanded normalized layers; For the newly added channels after expansion, their parameter values ​​are initialized to 1 to maintain the numerical stability and behavioral consistency of the normalization layer. The normalization layer includes layer normalization and / or root mean square normalization.

[0015] Furthermore, it also includes: training the widened model using a sparse update strategy and dynamically adjusting the sparsity rate of the sparse update strategy; the sparse update strategy selectively updates the expanded parameters, and there are expanded parameters that are frozen in at least one training phase.

[0016] Furthermore, the step of training the structurally expanded model using a sparse update strategy and dynamically adjusting the sparsity rate of the sparse update strategy further includes: During the width expansion process, the training is divided into multiple sparse stages, and a target sparsity rate is set for each sparse stage. The sparsity rate represents the proportion of frozen expansion parameters to all expansion parameters. At the beginning of each sparse phase, the importance score of each expansion parameter is calculated based on the absolute value of the weight and the magnitude of the gradient. The expansion parameters with the highest importance scores are selected to participate in the gradient update, while the remaining expansion parameters are frozen. As training progresses, the sparsity of each sparse stage is gradually reduced to make the parameter update process smooth.

[0017] Furthermore, in the self-attention module, the importance score calculation and mask generation are performed independently for each attention head or query group: The weight matrix of the self-attention module is split into sub-weight matrices of multiple attention heads or query groups according to the output dimension; For each sub-weight matrix, calculate the importance score of each extended parameter and independently rank them by importance within each sub-weight matrix. Based on the set target sparsity, the parameters with the highest importance scores in each sub-weight matrix are selected to participate in gradient updates, while the remaining parameters are masked in the current sparsity stage.

[0018] Furthermore, the step of dividing the training process into multiple training stages, and sampling training samples from data subsets of different difficulty levels according to the expansion state of the model in different training stages, includes: During the deep expansion phase, the sum of the sampling weights for easy and medium-difficulty samples is higher than the sampling weight for difficult samples. During the width expansion phase, the sampling weight of difficult samples is higher than that of easy samples and medium-difficulty samples. The sampling ratio of samples with different difficulties is adjusted by a progressively overlapping scheduling method between each stage.

[0019] Furthermore, it also includes quantifying the collaborative steps: In the final stage of width expansion training, quantization-aware training is introduced into the linear layers of the feedforward network and / or the linear layers in the self-attention module. The quantization-aware training employs low-bit asymmetric quantization and uses a pass-through estimator for backpropagation of quantization factors and gradient updates.

[0020] To achieve the above objectives, the present invention also provides a large model pre-training acceleration system based on expansion and data difficulty driven by the present invention, employing the large model pre-training acceleration method based on expansion and data difficulty driven by the present invention as described above, including: The data difficulty pre-assessment module is used to obtain the pre-trained base model, assess the difficulty of samples in the pre-trained corpus, and divide the samples into data subsets with multiple difficulty levels based on the assessment results. The structural expansion module includes a depth expansion unit and / or a width expansion unit; the depth expansion unit is used to achieve depth expansion by copying the Transformer block in the middle layer region of the model and introducing a gated residual mechanism; the width expansion unit is used to achieve width expansion by combining direct copying and near-zero initialization of the weight matrix. The multi-stage training scheduling module is used to divide the training process into multiple stages, and to sample training samples from data subsets of different difficulty levels according to the expansion state of the model in different stages.

[0021] To achieve the above objectives, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the computer program stored in the memory to implement the large model pre-training acceleration method based on expansion and data difficulty as described above.

[0022] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement the large model pre-training acceleration method based on expansion and data difficulty as described above.

[0023] The large model pre-training acceleration method provided by this invention, based on expansion and data difficulty-driven approach, reduces the total floating-point computation of expansion training through the synergistic effect of structural expansion knowledge inheritance, multi-stage scheduling to match difficulty, and data-driven optimization of sample efficiency, while ensuring the performance of the final model, thus achieving an effective balance between computational efficiency and model performance.

[0024] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0025] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a large model pre-training acceleration method based on expansion and data difficulty driven according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the training process according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the depth expansion and gating residual mechanism according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the width expansion process according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the architecture of a large model pre-training acceleration system based on expansion and data difficulty driven according to an embodiment of the present invention; Figure 6 This is a schematic diagram of an electronic device structure according to an embodiment of the present invention. Detailed Implementation

[0026] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0027] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0028] The term "comprising" and its variations as used in this invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments".

[0029] It should be noted that the concepts of "first" and "second" may be mentioned in this invention only to distinguish different devices, components or parts, and are not used to limit the order of the functions performed by these devices, components or parts or their interdependence.

[0030] It should be noted that the terms "one" and "multiple" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "Multiple" should be understood as two or more.

[0031] In this invention, the step numbers (such as steps 101, 102, 103, etc.) are only used to refer to different operational steps and do not limit the execution order of each step. Unless the context explicitly requires a specific order, the steps can be executed in any reasonable order according to actual training needs, or they can be executed in parallel or partially overlapped. For example, the data difficulty pre-assessment step can be completed all at once before model expansion, or it can be updated periodically during training; the order of depth expansion and width expansion can be interchanged, or only one of them can be executed according to the model structure design. Any reasonable adjustments to the step order made under the concept of this invention fall within the protection scope of this invention.

[0032] In this invention, the term "Transformer" refers to a transformer, a neural network architecture based on a self-attention mechanism; the term "Transformer block" refers to a transformer block, which is the basic processing unit constituting the Transformer model. Each transformer block contains a multi-head self-attention sublayer and a feedforward network sublayer. Definitions of other terms will be given in the following description.

[0033] This invention follows a technical path of "structural expansion - sparse optimization - data scheduling - approximate computation - system verification," systematically constructing a highly efficient acceleration method for pre-training large language models. Regarding structural expansion, it expands the inheritable model from both width and depth dimensions. Width expansion is based on a pre-trained, fully-fledged small-to-medium-scale model, establishing a unified width mapping mechanism for core linear modules such as the embedding layer, query / key / value projection matrix in the self-attention mechanism, output projection layer, and feedforward network. This mechanism precisely replicates the original weight sub-blocks and initializes the new dimensions with zero or minimal perturbation. Simultaneously, in the self-attention module, the weights of each head are split, expanded, and recombined based on the multi-head and query group structure information, ensuring that the expanded model is highly consistent with the source model in the function space in its initial state, achieving smooth embedding at the structural level. Furthermore, an initialization strategy of "full inheritance of the original channel and unit scaling of the new channel" is adopted for one-dimensional parameters such as the normalization layer to maintain numerical stability and provide structural support for subsequent training acceleration and quantization compression.

[0034] In terms of depth expansion, an expansion strategy combining intermediate layer replication and gated residual fusion is adopted. Specifically, the original Transformer layer sequence is divided into relatively stable head and tail intervals, and an expandable middle interval. The modules in the middle are structurally replicated and inserted into the original layers alternately or in series. To mitigate optimization oscillations that may be caused by network deepening, the replicated layer inherits only the internal structural parameters of the original layer during the initialization phase, and a small-amplitude orthogonal initialization is applied at the residual exit projection to minimize the impact on the backbone representation. Furthermore, by introducing residual modules with time-gated factors outside the replicated layer, and using a scheduling mechanism of annealing by training steps, the newly added layer gradually transitions from an initial near-closed state to fully participating in representation modeling, thereby achieving a natural expansion of network depth from N layers to nearly 2N layers. In addition, this invention also extracts stability control criteria suitable for large model depth expansion based on comparing the impact of different gating curves (such as linear, cosine, and sigmoid) on training stability and combining gradient statistics analysis.

[0035] Regarding optimization and parameter update mechanisms, this invention focuses on exploring sparse training strategies for extended scenarios and deeply integrating them with width-expanding structures. Based on the dynamic characteristics of the expansion phase, this invention abandons the traditional uniform update method and instead constructs a sparse mask generation method based on joint parameter-gradient importance evaluation. Within each sparse phase, the weights in the extended region are scored for importance based on their absolute weight values ​​and recent gradient magnitudes. For the self-attention module, this is further refined to the head and query group granularity, thereby selecting the subset of parameters with the highest importance scores to participate in gradient updates, while the remaining parameters are approximately frozen during this phase. During training, a staged sparsity rate scheduling is employed, coupled with independent learning rate preheating and cosine annealing strategies, gradually transitioning the training mode from "high-frequency fine-grained updates of a few key parameters" to a denser state of "coordinated optimization of more parameters." This mechanism can prioritize activating the parameter subspace that contributes the most to performance improvement within a given computational and storage budget, thereby significantly improving the utilization efficiency of training resources.

[0036] In terms of data organization and scheduling, this invention constructs a multi-stage pre-training mechanism centered on data difficulty assessment, achieving deep coupling with the structure expansion stage. By using indicators such as perplexity, prediction entropy, confidence, text length, and semantic diversity calculated by the basic language model, a difficulty score is assigned to each training sample from an information theory and statistical perspective, and data subsets of easy, medium, and difficult levels are constructed accordingly. During training, low-difficulty samples are prioritized in the basic stage, enabling the source model to converge quickly in simple contexts. In the deep expansion stage, medium-difficulty data is gradually introduced to incentivize new layers to better model long-range dependencies and complex semantic structures. In the breadth expansion and sparse training stages, the proportion of high-difficulty and long-tail samples is further increased, allowing the expanded wide representation space and larger model capacity to be fully utilized for high-information-density samples. Through the synergistic design between difficulty grading, structure expansion, sparse training, and quantization intensity, gradient noise can be effectively suppressed, optimization oscillations can be mitigated, and the sample efficiency and convergence quality of the expanded training can be comprehensively improved.

[0037] Symbol Explanation: In various embodiments of this invention, some formulas share the letter symbol 't', but its specific meaning depends on the context. In the data difficulty assessment stage, 't' represents the time step in the word sequence; in the depth-expanded gated residual mechanism, 't' represents the training step number. Those skilled in the art can clearly determine its meaning based on the formula and context, and it will not be elaborated further below.

[0038] Example 1 Based on the above concept, this embodiment provides a method for accelerating large model pre-training driven by expansion and data difficulty, including the following steps: obtaining a pre-trained base model and evaluating the difficulty of samples in the pre-training corpus, and dividing the samples into multiple data subsets with different difficulty levels according to the evaluation results; expanding the structure of the base model to obtain expansion parameters; the structural expansion includes depth expansion and / or width expansion; the depth expansion is achieved by copying Transformer blocks in the middle layer region of the model and introducing a gated residual mechanism, and the width expansion is achieved by combining direct copying and near-zero initialization of the weight matrix; dividing the training process into multiple training stages, and sampling training samples from data subsets with different difficulty levels according to the expansion state of the model in different training stages.

[0039] Figure 1 This is a flowchart of a large model pre-training acceleration method based on expansion and data difficulty driven according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the training process according to an embodiment of the present invention. The following will be combined with... Figure 1 and Figure 2 The specific embodiments of the present invention will be described in further detail to make the objectives, technical solutions and advantages of the present invention clearer.

[0040] In step 101, the difficulty of the pre-training corpus is assessed and a multi-level data subset is constructed.

[0041] Before initiating any model expansion operations, this invention first assesses and grades the difficulty of the pre-training corpus. To construct a data scheduling strategy that matches the model expansion process, this invention proposes a comprehensive sample difficulty measurement system based on multi-order entropy and negative log-likelihood. Its core idea is to utilize the uncertainty patterns and error types reflected in the prediction distribution of a fully trained base model, and to construct a unified scalar difficulty score by integrating three complementary dimensions: Shannon entropy, Rényi entropy, and negative log-likelihood, thereby driving the dataset partitioning and phased training process. Specific steps include: Calculate the basic metrics: Using the fully trained base model, perform calculations on each training sample in the corpus. Predicting the next word on a token-by-token basis yields the conditional probability distribution. (Where t is the time step, and v is the token in vocabulary V). Based on this distribution, for each time step... Calculate the following three complementary statistics: Indicator 1, Shannon Entropy: For each time step t, the Shannon entropy for that step is defined as... And the entropy over a sample length is averaged, that is, the sample-level average is calculated to obtain the sample-level index. ,in This represents the number of tokens in this sample. This is a sample-level metric. This reflects the overall uncertainty of the model's predictions: a larger value indicates a more stable distribution of the model's predictions for the current token, suggesting that many words may appear, indicating extreme uncertainty; a smaller value indicates a sharper distribution, suggesting that the model is certain about one or a few candidates. For a sentence, a high value indicates... This indicates that the context provides less information, or the sentence itself is rare or creative, making it more difficult for the model to learn.

[0042] Indicator 2, Rényi entropy: To compensate for the shortcomings of Shannon entropy in distinguishing distribution patterns, Rényi entropy (in terms of...) is further introduced. (For example) in, The Rényi index is used to adjust the sensitivity to different probability values ​​in a probability distribution.

[0043] Rényi entropy is more sensitive to the presence of a few high-probability candidates in a probability distribution. When the distribution exhibits a multimodal shape (e.g., the presence of several credible candidates), The value will be significantly lower than that of a perfectly uniform distribution, thus effectively distinguishing between two different difficulty modes: "the model hesitates among multiple reasonable options" and "the model is completely uncertain".

[0044] Indicator 3, Negative Log-Likelihood (NLL): For each time step t, the negative log-likelihood is defined as... ,in The next token is the real one. The sample-level negative log-likelihood is... This metric directly measures the confidence of the model's predictions on true words. Even if the model's prediction distribution is sharp (low entropy), if it assigns a high probability to incorrect words, the NLL value will still be high. Such samples usually constitute "trap samples" in model learning.

[0045] Each of the three indicators mentioned above has its own emphasis, and each has limitations when used alone. Therefore, this invention combines the three into an indicator vector. It is then learned and integrated into a comprehensive index through a softmax (activation function) mechanism, which is expressed as a scalar difficulty score. : in, For adaptive weights, the calculation method is as follows: Here, i and j are both index numbers, i=1,2,3, j=1,2,3. , , The three indicators mentioned above were weighted and averaged, with the weights dynamically generated by Softmax. This fusion method allows the model to dynamically adjust the contribution weights of each difficulty dimension based on the characteristics of the samples themselves (quantified as indicator values). For example, for "trap samples" with abnormally high NLL, softmax will give NLL a higher weight; while for samples with high uncertainty but basically correct predictions, more emphasis will be placed on the difficulty reflected by the entropy value.

[0046] Based on this comprehensive metric, the pre-training corpus is globally ranked and divided according to preset quantile intervals, classifying the data into multiple difficulty levels, such as easy, medium, and hard. Easy Sample: Located in In the lower quantile interval of the score (e.g., 0–40%), the model prediction has high certainty, low NLL value, and the distribution shows a clear unimodal pattern. Medium difficulty samples: located in the middle quantile (e.g., 40–80%), may have high prediction uncertainty or a moderate degree of prediction error; Difficult samples: located in the upper quantile range (e.g., 80–100%), including high NLL trap samples, high entropy uncertain samples, and long-tail domain texts, etc.

[0047] Based on this, the data difficulty partitioning is deeply coupled with the model structure expansion stage to form a multi-stage training strategy, such as... Figure 2 As shown, in Figure 2 In the diagram: the horizontal axis 'training steps t→T' represents the progression of training steps from the initial number of steps to the total number of steps T; Layer 1…Layer Ls represent the network layers of the model, with the left side illustrating the initial small model structure, the middle section illustrating width expansion (increased hidden dimensions), and the right side illustrating depth expansion (increased number of layers). This diagram is merely an illustrative representation of the correspondence between multi-stage training and data difficulty scheduling in this invention, and does not limit the actual duration of each stage or the precise number of steps for sample switching. Specifically: In the basic training stage of the small model: easy sample subsets are mainly used to enable the model to quickly master basic language patterns in a low-noise environment and lay the foundation for high-quality parameter initialization; In the deep expansion stage: with the insertion of the copy layer and the gradual activation of the gating mechanism, the proportion of medium-difficulty samples in the training data increases accordingly, guiding the new network to process more complex structures and rely on longer semantic information. In the width expansion and sparse update phase: After expanding the model width and introducing a sparse mask, the training focus shifts to difficult samples. By increasing the sampling weights of high-difficulty samples, the newly added attention head and feedforward network capacity are prioritized for modeling long-tail knowledge, complex reasoning, and trap patterns, thereby maximizing the utility of the expanded parameters.

[0048] It's important to note that the data difficulty at each stage is not strictly separated, but rather a progressively overlapping scheduling approach is adopted. For example, some easy and medium-difficulty samples are retained in the later stages of width expansion to maintain the model's general performance; when a large number of difficult samples are introduced, the sparsity rate, learning rate, and quantization intensity are adjusted simultaneously to maintain a dynamic balance between sample difficulty, model capacity, and numerical accuracy. Through this multi-stage training strategy centered on data difficulty, an organic synergy between the data level and structure expansion and sparsity optimization is achieved, significantly improving the overall efficiency and final performance of extended pre-training while ensuring training stability.

[0049] As some other embodiments of the present invention, in the data difficulty assessment module (or stage), Shannon entropy ( ), Rényi entropy ( The negative log-likelihood (NLL) and negative log-likelihood (NLL) are combined into a single fraction. In addition to using learnable neural network fusion mechanisms, linear combinations with fixed weights can also be employed (such as...). , , and The weights of the three indicators are respectively. D is the fusion score; D and (The meanings are the same, used only to distinguish different fusion methods), determining the optimal weights through grid search; or using rule-based nonlinear functions, such as... This emphasizes samples with high prediction errors and high uncertainty; unsupervised clustering methods can even be used to group samples into (…). The position in the NLL (Non-Limited Least) three-dimensional space is directly used as a difficulty representation for phased sampling.

[0050] In step 102, the model is extended in depth by introducing a gated residual mechanism.

[0051] In the embodiments of this invention, the goal of depth extension is to enhance the representational power and inference depth of the existing small model by increasing the number of network layers, while retaining its functionality, and effectively avoiding training instability and performance degradation that may be caused by directly deepening the network. Its core idea is to perform structured replication in the intermediate layers of the model and, with the help of near-zero residual initialization and gated annealing mechanisms, achieve a depth extension strategy with minimal perturbation to the behavior of the original model.

[0052] Specifically, this invention divides the Transformer layer sequence into three sections: the top layer near the input side, which is mainly responsible for bottom layer feature extraction and whose structure remains unchanged; the bottom layer near the output side, which undertakes high-level semantic integration and output decoding functions and is also retained; and the scalable region in the middle, which serves as the main operating area for depth expansion. Figure 3 This is a schematic diagram of the depth expansion and gating residual mechanism according to an embodiment of the present invention. Figure 3 The curve in the figure represents the gating factor. A schematic diagram of the cosine annealing curve as a function of the normalized training steps t / T. The horizontal axis 'Normalizedstep' represents the normalized training steps, and the vertical axis 'Value' represents the gating factor. The value of is such that the curve gradually rises from near 0 to a maximum value of 1 and then decays back to 0, so that the replicated block smoothly transitions from a bypass state to fully participating in representation modeling. For example... Figure 3 As shown, within the scalable region (excluding the bottom layer (Layer 1) and top layer (Layer 4)), each original Transformer block is copied and inserted. This results in an alternating structure of original and copied blocks in the middle portion of the network of length L. Specifically, new Layers 3 and 5 are copied blocks, while new Layers 2 and 4 are original blocks, thus expanding the overall depth to nearly twice its original size. This depth expansion method avoids modifications to structurally sensitive areas such as model inputs and outputs, focusing the expansion on the intermediate semantic abstraction stage, which helps maintain the overall stability of the model.

[0053] In the intermediate scalable region, for each original Transformer block, a copy block with an identical structure is constructed and inserted after the original block, forming a cross-over of local sequences. To control the perturbation of the overall model behavior during the initial expansion, the initialization of the copy block follows these principles: The structural parameters, such as the attention sublayer and feedforward network sublayer, inside the copy block are directly copied from the original block to ensure that it has a potential representation capability similar to that of the original block. For key weights in the copy block responsible for writing the sublayer output back to the backbone residual path (such as attention output projection, feedforward network final layer projection, etc.), instead of directly copying the original parameters, orthogonal initialization at a very small scale or other stable near-zero initialization methods are used to ensure that the residual output of the copy block in the initial state is... satisfy This ensures that the overall model output still approximates the behavior of the original model. In effect, this design is similar to a "lossless expansion," meaning that in the initial stages of expansion, adding depth hardly changes the original functionality of the network. The top and bottom layers do not participate in the replication expansion to maintain the structural consistency between the input feature extraction and output decoding stages, reduce the uncertainty of the training process, and retain the good performance that the small model already possesses in downstream tasks.

[0054] Near-zero initialization alone is insufficient to fully unleash the representational potential of newly added layers. Therefore, this invention introduces a gated residual mechanism outside the replication block, treating the replication block as a residual subpath and using a learnable or schedulable scalar gating factor. 0, Control its contribution to the trunk representation. This represents the maximum gating factor, which is 1. In the local structure of each layer, the output of the copy block can be represented as: in For input features, The residual term generated for the copied block, where t represents the current training step number. In terms of training strategy, a stepwise annealing gating scheduling mechanism is adopted: at the beginning of the depth expansion phase, [the gating mechanism is implemented]. Near-zero initialization: setting initial values ​​close to 0 to put the copied blocks in a near-bypass state; as training progresses, gradually adjusting according to a pre-set cosine curve... Increase to the maximum The parameters are then gradually reduced to 0, allowing the replicated block to integrate and participate in the representation modeling process. During this process, the parameters inside the replicated block continuously adapt through gradient updates, thereby learning feature representations that are complementary to the original network.

[0055] By gradually annealing the gating factor, the model can smoothly transition from "approximate small model behavior" to "full-depth large model behavior," effectively avoiding gradient anomalies or convergence difficulties that may be caused by a sudden increase in network depth. Preferably, the total number of gating steps is set to a certain proportion of the total training steps in the depth expansion phase to ensure that the replication layer has sufficient time to gradually integrate into the network.

[0056] During the depth expansion phase, considering the increased complexity of the model structure vertically, this invention employs a full-parameter update strategy to provide ample optimization space for the newly added layers. Simultaneously, a separate learning rate scheduling strategy is designed for this phase, such as resetting the warm-up phase and cosine annealing interval, to avoid instability that might result from using the learning rate from the previous phase. After the depth expansion has essentially converged, quantization-aware training can be further introduced for some layers to enhance their numerical stability under low-precision representation; alternatively, quantization can be implemented uniformly after all expansions are complete to simplify the training process. Through these designs, the depth expansion phase effectively utilizes the increased representational power brought by the new layers while controlling training risks through gating and initialization mechanisms, laying a solid foundation for subsequent width expansion and sparse optimization.

[0057] In other embodiments of the present invention, in deep expansion, in addition to using schedulable scalar gating factors... Furthermore, learnable vector gating (independent gating for each feature dimension) or dynamically computed gating based on input content (such as generating gating values ​​based on input features through a lightweight network) can be employed. In addition to cosine annealing, the gating scheduling curve can also use a linear growth-hold-decay pattern or a reinforcement learning strategy that adaptively adjusts based on validation set performance.

[0058] In step 103, the model width is expanded and coordinated with a sparse update mechanism.

[0059] In embodiments of the present invention, after depth expansion converges, the width expansion stage begins. The goal of width expansion is to expand the lateral structural parameters of a given trained small model, such as the hidden dimension of each layer, the number of attention heads, or the intermediate dimension of the feedforward network, to a larger scale, while maximizing the inheritance of the existing representational capabilities of the small model and controlling the perturbation of model behavior in the early stages of expansion.

[0060] Formalistically, taking a linear transformation of a certain layer as an example, the weight matrix of that layer in the small model is: in The dimension is The real space, , These represent the output dimension and the input dimension, respectively.

[0061] The weight matrix in the expanded large model is as follows Among them, the expanded output dimension Expanded input dimensions The core of width expansion lies in building a... arrive The mapping mechanism enables the original Dimensional input and Within the dimensional output range, the mapping behavior of the larger model should be as consistent as possible with that of the smaller model; the contribution of newly added input and output dimensions to the overall model behavior during the initialization phase should be as small as possible, so as to gradually learn new functions in subsequent training. In the self-attention module, the expansion process needs to fully consider the structural characteristics of multi-head and query groups to avoid disrupting the existing attention pattern due to disordered mixing.

[0062] Figure 4 This is a schematic diagram of the width expansion process according to an embodiment of the present invention. Figure 4In the middle section, the matrix section illustrates the mapping strategy of the weight matrix in the width expansion: S, p, q, and r are the original parameters directly copied from the source model, and ε is the minimum value, indicating that the new dimension is initialized with a value close to zero; the curves and formulas in the middle section illustrate the importance score-based approach. The progressive sparse training process; Training and freeze represent the parameters involved in gradient update and frozen in the current sparse stage, respectively; independent learning rates r1, r2, ..., rn are configured for different stages. Figure 4 As shown, for the portion where both the input dimension and output channels are within the range of the small model, the corresponding submatrix of the small model is directly copied to the same position in the large model. This maintains the consistency of linear transformation within the original dimensions.

[0063] For the portion of the large model that exceeds the input dimension range of the small model, i.e., the column index... : Initialize with extremely small values ​​(close to zero): This strategy ensures that the impact of the newly added feature dimensions on the layer output is extremely low in the early stages of expansion, thus avoiding significant perturbations to the network behavior.

[0064] For the newly added output channel (row index) : Instead of using completely random initialization, all values ​​are set to minimum values ​​(close to zero), which facilitates the gradual differentiation of new representation capabilities through gradient updates during training.

[0065] For one-dimensional scaling parameters such as LayerNorm (layer normalization) and RMSNorm (root mean square normalization), during expansion, the coefficients of the original channels are reused from the small model, while the newly added channels are initialized to 1 to maintain numerical stability and consistency of normalization behavior. The expansion strategy for the word embedding matrix is ​​similar: the original vocabulary or original dimensions are directly copied, and the new dimensions are initialized with values ​​close to zero, thereby reserving representational freedom for subsequent learning without interfering with the original semantic structure.

[0066] The projection matrix in the self-attention module has a clearly defined multi-head and query group structure. To avoid disrupting this structure during expansion, this invention first splits the weight matrix by head or group, independently executes the above expansion strategy on each sub-block, and then reassembles it into a complete matrix. Specifically, for a given attention projection weight, it is first rearranged into multiple Q heads (attention heads used to calculate the query in the multi-head attention mechanism, with a shape of [...]) according to its output dimension. ]) and K / V (key or value) groups (shaped [ Subsequently, width expansion is performed on each Q-head and K / V group: the original dimensions are retained, the new input dimensions are initialized with values ​​close to zero, and the new output channels are initialized by copying existing rows. After expansion, each head and group is reassembled along the output dimension to form the attention projection matrix of the large model. This method maintains structural integrity while ensuring that the new head or group functions similarly to the existing head or group during the initialization phase, laying the foundation for forming diverse attention patterns in subsequent training.

[0067] Relying solely on structural expansion cannot reduce computational costs; in fact, if all parameters are updated intensively immediately, FLOPs (Floating Point Operations) actually increase. Therefore, this invention introduces a sparse update mechanism based on parameter-gradient importance evaluation after width expansion, combined with multi-stage learning rate scheduling and quantization techniques, to achieve efficient training.

[0068] Specifically, during the width expansion process, a binary mask is constructed for each weight matrix to identify which parameters belong to the new degrees of freedom introduced during the expansion and which parameters are inherited from the smaller model. Subsequently, the width expansion training is divided into several sparse stages, each with a different target sparsity rate (e.g., gradually decreasing from 40% to 0%). At the beginning of each stage, based on the parameter values ​​and gradient information from the most recent steps, an importance score is calculated for each parameter in the expanded region. Specifically, for the i-th expanded parameter, its importance score is defined as... : in, This is the current weight value of this parameter. This is the average gradient magnitude of the parameter over the most recent training steps in the current stage, and σ is a control factor that controls the degree of influence of the gradient magnitude (σ=1 in this embodiment). It is a minimum value (taken in this embodiment) =1×10 -10 To prevent the denominator from being zero, the score comprehensively considers the current importance of the parameter (absolute weight value) and its sensitivity to the loss function (gradient magnitude). A higher score indicates that the parameter is more critical to model performance and should be given priority in gradient updates. Based on the score ranking, only the parameters with the highest importance (e.g., the top 60%) are allowed to have their gradients updated, while the remaining parameters are masked (gradient is zero) at the current stage. In the self-attention module, importance evaluation and mask generation are performed separately for each head or group to maintain structural consistency.

[0069] Each sparse stage is equipped with an independent learning rate warm-up and cosine annealing strategy to ensure a smooth parameter update process. As training progresses, the sparsity gradually decreases, and more parameters are incorporated into the update, achieving a transition from high sparsity to fewer parameters to low sparsity to more parameters, ultimately approaching full parameter updates. Simultaneously, at appropriate training points (such as the last 10% of the total training steps), low-bit quantization or quantization-aware training is introduced into some layers or modules to further compress the precision of parameters and activation values, reducing memory and computational overhead. Through this design, training after width expansion can prioritize the optimization of key expanded parameters and gradually refine the entire expanded representation space while controlling the total FLOPs. As a specific implementation, only the linear layers of the feedforward network (FFN) and the linear layers in the self-attention module are quantized, while the rest remain unchanged. Low-bit quantization uses 8 bits and employs asymmetric quantization; quantization-aware training uses a pass-through estimator to train the quantization factor.

[0070] In other embodiments of the present invention, the sparse update module uses, in addition to, a joint importance score based on the absolute value of the weights and the magnitude of the gradient. Other importance metrics can also be considered. For example, a sensitivity assessment based on Hessian information (or its approximation, such as the diagonal of the Fisher information matrix) can be used to take into account the second-order effect of parameters on the loss function; or a criterion based on the magnitude of parameter changes during training can be used, considering parameters that change drastically to be more important; or a head-level importance assessment based on the attention output distribution entropy can be introduced to determine which new attention heads should be updated first. Here, Hessian information is the matrix of second-order partial derivatives of the loss function with respect to the model parameters, measuring the curvature of the loss surface; Fisher information is the expectation of the squared log-likelihood gradient, measuring the amount of information carried by the parameters; both can be used to assess parameter importance and are equivalent under certain conditions.

[0071] In other embodiments of the present invention, the specific implementation of structural expansion does not necessarily require both depth expansion and width expansion. For scenarios where only a moderate increase in model capacity is needed or where computational resources are severely limited, those skilled in the art can use only the aforementioned depth expansion scheme or only the aforementioned width expansion scheme as simplified variations. In such simplified variations, although the training cost can still be reduced to some extent and some acceleration effect can be obtained by combining it with the aforementioned data difficulty-driven multi-stage training scheduling mechanism, the improvement in model representation capability (the synergistic gain of depth and width) is limited. To maximize the synergistic advantages of structural expansion and data scheduling and achieve the optimal balance between model performance and training efficiency, the embodiments of the present invention preferably include a complete scheme that simultaneously includes depth expansion and width expansion. In other words, schemes that use depth expansion alone or width expansion alone should be regarded as simplified or degenerate implementations of the complete technical solution of the present invention under specific constraints, and they do not depart from the core concept of the present invention of accelerating training through the synergistic effect of "structural expansion-data scheduling".

[0072] In step 104, the overall coordination and convergence of multi-stage training.

[0073] In embodiments of the present invention, the various stages in the entire extended pre-training process do not run in isolation, but are organically coordinated through data difficulty scheduling. Specifically: In the deep expansion phase, the model capacity increases vertically, and the data mainly consists of easy and medium-sized samples, guiding the new layers to learn robust deep representations. During the width expansion and sparse update phase, the model capacity increases horizontally, and the data consists mainly of medium and difficult samples, which incentivizes new heads and channels to learn high information density patterns. The data difficulty across different stages is scheduled using a progressively overlapping approach. For example, a small number of easy and medium-difficulty samples are retained in the later stages of width expansion to maintain general performance. The sparsity rate, learning rate, and quantization intensity are adjusted synchronously to maintain a dynamic balance between sample difficulty, model capacity, and numerical accuracy.

[0074] Through the above steps, this invention achieves a deep coupling of model structure expansion, sparse optimization, and data difficulty-driven approaches. In a typical large-scale model pre-training task, compared to a de novo training baseline, the method of this invention can reduce the total number of floating-point operations (FLOPs) during expanded training by approximately 40-50%, while maintaining or even improving performance on authoritative benchmarks such as MMLU and C-Eval (MMLU and C-Eval are two authoritative benchmarks for evaluating large language models in general knowledge reasoning and comprehensive Chinese comprehension capabilities). The loss curve during training is smooth, avoiding performance crashes and gradient oscillations in the early stages of expansion, and significantly improving training success rate and sample efficiency.

[0075] The large model pre-training acceleration method based on expansion and data difficulty driven by this invention has the following advantages: (1) Achieving an effective balance between computational efficiency and model performance: This invention optimizes the resource consumption pattern of large model pre-training through a three-pronged collaborative design of structural extension inheriting knowledge, sparse updates focusing computational power, and data scheduling matching difficulty. Experimental results show that the method of this invention can reduce the total FLOPs of extended training by about 40-50%, while maintaining the performance of the final model on authoritative benchmarks such as MMLU and C-Eval at a level comparable to or even better than the original model, which helps to alleviate the trade-off between efficiency and effectiveness in efficient training.

[0076] (2) Ensures the smoothness and stability of the expansion process: Through the strategy of weight reuse and near-zero initialization for width expansion and copying and gated residuals for depth expansion, the newly added model structure and parameters can be highly aligned with the original model function during the initialization stage, which helps to avoid performance degradation and training oscillations in the early stage of expansion. This smooth embedding mechanism lays a good foundation for subsequent optimization, can improve the training success rate, and is especially suitable for large-scale expansion under limited computing power budget.

[0077] (3) Achieving organic coupling between data-driven approaches and architectural evolution: The data difficulty assessment system based on multi-dimensional information theory indicators proposed in this invention, along with the scheduling strategy corresponding to difficulty grading and multi-stage training, enables training data to be dynamically and adaptively adjusted according to the development of model capabilities. This mechanism helps improve sample learning efficiency and accelerates the convergence speed of the model on complex tasks.

[0078] (4) A systematic and reusable engineering framework is provided: This invention covers multiple aspects such as model structure design, optimization algorithm, data management and system scheduling, forming a complete solution. The framework has a high degree of modularity, and each technical component can be independently verified or work together. It has good versatility and scalability, and can be applied to the pre-training task of expanding from different scale base models to different target scale models, thus having high industrial application value.

[0079] Example 2 This embodiment provides a large model pre-training acceleration system based on expansion and data difficulty driven, which is configured to execute the steps of the large model pre-training acceleration method based on expansion and data difficulty driven as described in Embodiment 1.

[0080] Figure 5 This is a schematic diagram of the architecture of a large model pre-training acceleration system based on expansion and data difficulty driven according to an embodiment of the present invention, such as... Figure 5As shown, the system includes a data difficulty pre-evaluation module 501, a depth expansion module 502, a width expansion module 503, a sparse update module 504, a quantization collaboration module 505, and a multi-stage training scheduling module 506.

[0081] The data difficulty pre-evaluation module 501 is configured as follows: Before model expansion, it uses the pre-trained base model to perform word-by-word meta-prediction on each sample in the pre-training corpus, calculating Shannon entropy, Rényi entropy, and negative log-likelihood respectively, and obtaining a unified scalar difficulty score through a softmax fusion mechanism; based on this difficulty score, it performs global sorting and quantile partitioning of the corpus, constructing easy sample subsets, medium-difficulty sample subsets, and difficult sample subsets; and outputs the subset index and sampling weights to the multi-stage training scheduling module 506. The data difficulty pre-evaluation module 501 can be run offline once or re-evaluated periodically during training.

[0082] The deep expansion module 502 is configured to: receive the start command and easy and medium difficulty samples from the multi-stage training scheduling module 506; divide the Transformer layer sequence of the base model into a top layer, an expandable region, and a bottom layer; within the expandable region, structurally copy each original Transformer block and insert it after it (from the bottom layer to the top layer), forming an alternating structure of original and copied blocks; initialize the key weights responsible for residual write-back in the copied blocks with near-zero values; introduce a schedulable scalar gating factor for each copied block and gradually adjust the value of the gating factor according to the cosine annealing curve (gradually increasing from near 0 to the maximum value and then decaying), so that the copied block smoothly transitions from an approximately bypassed state to fully participating in representation modeling; adopt a full parameter update strategy during the deep expansion stage and independently configure learning rate preheating and cosine annealing scheduling; after the expansion is completed, return the deep-expanded model parameters to the multi-stage training scheduling module 506 and trigger a stage switching signal.

[0083] The width expansion module 503 is configured as follows: after receiving the width expansion start command from the multi-stage training scheduling module 506, it reads medium and hard difficulty samples from the multi-stage training scheduling module 506 and receives the deeply expanded model as input; it expands the width of each linear layer (including the attention projection matrix and the feedforward network projection matrix), normalized layer parameters, and word embedding matrix of the model; for the linear layer weight matrix, it adopts a mapping strategy of directly copying the original submatrix, initializing the new input dimension to near zero, and initializing the new output dimension to near zero; for the normalized layer parameters, the original channels are inherited, and the new channels are initialized to 1; for the word embedding matrix, the original dimensions are directly copied, and the new dimensions are initialized to near zero; for the attention projection matrix, it is split, expanded, and reorganized according to the attention head or query group; after the expansion is completed, it generates an expanded parameter mask for each weight matrix (identifying which parameters are new and which parameters are inherited), and passes the mask to the sparse update module 504.

[0084] The sparse update module 504 is configured to: receive the expanded parameter mask from the width expansion module 503; divide the training into multiple sparse phases during the width expansion process, setting a target sparsity for each phase; at the beginning of each phase, calculate the importance score of each parameter in the expanded region based on the absolute value of the weights and the gradient magnitude, sort the parameters by score, and only allow the parameters with the highest importance to participate in gradient updates, while the remaining parameters are masked (gradients set to zero) in the current phase; gradually reduce the sparsity as training progresses, eventually transitioning to full parameter updates; in the self-attention module, importance evaluation and mask generation are performed head-by-head or group-by-group. The sparse update module 504 is tightly coupled with the width expansion module 503, jointly completing the training of the width expansion phase.

[0085] The quantization coordination module 505 is configured to be triggered by the multi-stage training scheduling module 506 after the depth expansion has basically converged or during the width expansion training; to introduce low-bit quantization or quantization-aware training to specified layers or modules (such as linear layers of a feedforward network) to compress the precision of parameters and activation values ​​and reduce memory and computational overhead; and to return the quantized model parameters to the multi-stage training scheduling module 506, which can choose to continue training or stop training.

[0086] The multi-stage training scheduling module 506, as the core coordinator of the system, is configured to: hold global training steps, current stage identifier, current sparsity rate, gating factor annealing progress, and other statuses; at the start of training, it reads easy, medium, and difficult sample subsets and sampling weights from the data difficulty pre-assessment module 501; in the depth expansion stage, it starts the depth expansion module 502 and provides easy and medium samples, gradually annealing the gating factor; when the end conditions of the depth expansion stage are met (such as reaching the preset number of steps or verifying stable performance), it switches to the width expansion stage, starts the width expansion module 503 and the sparse update module 504, and notifies the data difficulty pre-assessment module 501 to switch to data sampling mainly based on medium and difficult samples; within the width expansion stage, it gradually decays the sparsity rate and adjusts the learning rate; at an appropriate time, it triggers the quantization coordination module 505; the scheduling method of progressive overlap between stages is adopted, and the sparsity rate, learning rate, and quantization intensity are adjusted synchronously to achieve deep coupling of model structure expansion, sparsity optimization, and data difficulty-driven processes.

[0087] In this embodiment, the specific implementation details of each module in the system are consistent with the corresponding method steps in Embodiment 1, and will not be repeated here. Through the collaborative work of the above modules, the system can achieve efficient and stable expansion from small trained models to large models, significantly reducing the computational cost and time overhead of pre-training.

[0088] Example 3 In embodiments of the present invention, an electronic device is also provided. Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, such as... Figure 6 As shown, the electronic device of the present invention includes a processor 601 and a memory 602, wherein, The memory 602 stores a computer program, which, when read and executed by the processor 601, performs the steps described above in the embodiment of the large model pre-training acceleration method based on expansion and data difficulty.

[0089] Example 4 In embodiments of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the steps described above in the embodiments of the large model pre-training acceleration method based on expansion and data difficulty driven.

[0090] In this embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0091] It will be understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A large model pre-training acceleration method based on extension and data difficulty driving, characterized in that, Includes the following steps: Obtain the pre-trained base model; The base model is used to perform word-by-word prediction for each sample in the pre-training corpus. Based on the prediction probability distribution of each word, multiple basic difficulty indicators of the sample are calculated and the multiple basic difficulty indicators are merged into a scalar difficulty score. Based on the scalar difficulty score, the sample is divided into data subsets of multiple difficulty levels. The base model is structurally expanded to obtain expansion parameters; the structural expansion includes depth expansion and / or width expansion; the depth expansion is achieved by copying the Transformer block in the middle layer region of the model and introducing a gated residual mechanism, and the width expansion is achieved by combining direct copying and near-zero initialization of the weight matrix; The training process is divided into multiple training stages, and training samples are sampled from data subsets of different difficulty levels according to the expansion state of the model in different training stages. The key difficulty metrics include Shannon entropy, Rényi entropy, and negative log-likelihood. The gated residual mechanism introduced in the depth expansion is as follows: Introducing a tunable scalar gating factor for each replication block , the output of the replication block is represented as: , where is the current training step, is the input feature, is the residual term generated by the replication block; the gating factor is gradually raised from close to 0 to a maximum value and then decayed according to a cosine annealing curve, so that the replication block is smoothly transitioned from bypass state to full participation in representation modeling. The width expansion is achieved by combining direct copying with near-zero initialization of the weight matrix, including the following steps: For the linear layer weight matrix and word embedding matrix, the weight values ​​corresponding to the original input and output dimensions in the source weight matrix are directly copied to the same positions in the expanded weight matrix. Both the newly added input dimension and the newly added output dimension are initialized with values ​​close to zero.

2. The method for accelerating large model pre-training based on expansion and data difficulty as described in claim 1, characterized in that, The step of integrating the multiple basic difficulty indicators into a scalar difficulty score includes: combining the Shannon entropy, Rényi entropy, and negative log-likelihood into an indicator vector; normalizing the indicator vector using the Softmax function to obtain adaptive weights; and summing the element-wise multiplication of the adaptive weights with the indicator vector to obtain the scalar difficulty score.

3. The method for accelerating large model pre-training based on expansion and data difficulty as described in claim 1, characterized in that, The step of dividing the sample into multiple difficulty level subsets based on the scalar difficulty score further includes: All samples are globally sorted according to the scalar difficulty score of each sample, and the sorted samples are divided into at least three difficulty levels according to the preset quantile interval. The at least three difficulty levels include easy sample level, medium difficulty sample level and hard sample level. The easy sample level corresponds to the preset quantile interval with the lowest scalar difficulty score, the difficult sample level corresponds to the quantile interval with the highest scalar difficulty score, and the medium difficulty sample level corresponds to the quantile interval with the scalar difficulty score between the lowest and the highest.

4. The method for accelerating large model pre-training based on expansion and data difficulty as described in claim 1, characterized in that, The initialization of the copy block adopts the following strategy: The structural parameters of the attention sublayer and feedforward network sublayer inside the copy block are directly copied from the corresponding original Transformer block. For the key weights in the replication block responsible for writing the sublayer output back to the backbone residual path, a very small-scale orthogonal initialization is used to ensure that the residual output of the replication block in the initial state is optimal. satisfy ; The key weights include attention output projection weights and / or feedforward network final layer projection weights.

5. The method for accelerating large model pre-training based on expansion and data difficulty as described in claim 1, characterized in that, In the width expansion, the projection matrix of the self-attention module is split, expanded and recombined according to attention heads or query groups. Specifically, this includes: obtaining the original projection weight matrix of the self-attention module in the base model, and splitting the original projection weight matrix into multiple sub-weight matrices of attention heads or query groups according to the output dimension. Each sub-weight matrix is ​​independently widened. The expanded sub-weight matrices retain the weight values ​​of the original dimensions, and the newly added dimensions are initialized with values ​​close to zero. All the expanded sub-weight matrices are reassembled along the output dimension to form the projected weight matrix of the expanded self-attention module.

6. The method for accelerating large model pre-training based on expansion and data difficulty as described in claim 1, characterized in that, The width expansion also includes expansion of the normalized layer parameters: Obtain the one-dimensional scaling parameters of each normalized layer in the base model, and directly inherit the parameter values ​​corresponding to the original channels to the corresponding channel positions of the expanded normalized layers; For the newly added channels after expansion, their parameter values ​​are initialized to 1 to maintain the numerical stability and behavioral consistency of the normalization layer. The normalization layer includes layer normalization and / or root mean square normalization.

7. The method for accelerating large model pre-training based on expansion and data difficulty as described in claim 1, characterized in that, It also includes: training the model with the width expanded using a sparse update strategy, and dynamically adjusting the sparsity rate of the sparse update strategy; the sparse update strategy selectively updates the expanded parameters, and there are expanded parameters that are frozen in at least one training phase.

8. The method for accelerating large model pre-training based on expansion and data difficulty as described in claim 7, characterized in that, The step of training the structurally expanded model using a sparse update strategy and dynamically adjusting the sparsity rate of the sparse update strategy further includes: During the width expansion process, the training is divided into multiple sparse stages, and a target sparsity rate is set for each sparse stage. The sparsity rate represents the proportion of frozen expansion parameters to all expansion parameters. At the beginning of each sparse phase, the importance score of each expansion parameter is calculated based on the absolute value of the weight and the magnitude of the gradient. The expansion parameters with the highest importance scores are selected to participate in the gradient update, while the remaining expansion parameters are frozen. As training progresses, the sparsity of each sparse stage is gradually reduced to make the parameter update process smooth.

9. The method for accelerating large model pre-training based on expansion and data difficulty as described in claim 8, characterized in that, In the self-attention module, the importance score calculation and mask generation are performed independently for each attention head or query group: The weight matrix of the self-attention module is split into sub-weight matrices of multiple attention heads or query groups according to the output dimension; For each sub-weight matrix, calculate the importance score of each extended parameter and independently rank the importance of the extended parameters within each sub-weight matrix. Based on the set target sparsity, the parameters with the highest importance scores in each sub-weight matrix are selected to participate in gradient updates, while the remaining parameters are masked in the current sparsity stage.

10. The method for accelerating large model pre-training based on expansion and data difficulty as described in claim 1, characterized in that, The step of dividing the training process into multiple training stages and sampling training samples from data subsets of different difficulty levels according to the expansion state of the model in different training stages includes: During the deep expansion phase, the sum of the sampling weights for easy and medium-difficulty samples is higher than the sampling weight for difficult samples. During the width expansion phase, the sampling weight of difficult samples is higher than that of easy samples and medium-difficulty samples. The sampling ratio of samples with different difficulties is adjusted by a progressively overlapping scheduling method between each stage.

11. The method for accelerating large model pre-training based on expansion and data difficulty as described in claim 1, characterized in that, It also includes quantifying the collaboration steps: In the final stage of width expansion training, quantization-aware training is introduced into the linear layers of the feedforward network and / or the linear layers in the self-attention module. The quantization-aware training employs low-bit asymmetric quantization and uses a pass-through estimator for backpropagation of quantization factors and gradient updates.

12. A large model pre-training acceleration system based on expansion and data difficulty-driven methods, characterized in that, The method for accelerating large model pre-training based on expansion and data difficulty as described in any one of claims 1 to 11 includes: The data difficulty pre-assessment module is used to obtain the pre-trained base model, assess the difficulty of samples in the pre-trained corpus, and divide the samples into data subsets with multiple difficulty levels based on the assessment results. The structural expansion module includes a depth expansion unit and / or a width expansion unit; the depth expansion unit is used to achieve depth expansion by copying the Transformer block in the middle layer region of the model and introducing a gated residual mechanism; the width expansion unit is used to achieve width expansion by combining direct copying and near-zero initialization of the weight matrix. The multi-stage training scheduling module is used to divide the training process into multiple stages, and to sample training samples from data subsets of different difficulty levels according to the expansion state of the model in different stages.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor is configured to execute the computer program stored in the memory to implement the large model pre-training acceleration method based on expansion and data difficulty driven by any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is loaded and executed by a processor to implement the large model pre-training acceleration method based on expansion and data difficulty driven by any one of claims 1-11.

Images