Large language model multi-stage simulation training method, system, device and medium

Abstract

The application discloses a large language model multi-stage simulation training method, system, device and medium. The method comprises the following steps: performing multi-dimensional difficulty feature extraction on training samples, dividing difficulty levels to construct a sample difficulty label system; setting a plurality of sequentially progressive training stages, configuring a matching preset sample difficulty distribution for each stage, and sampling to construct training data sets of each stage; monitoring the accuracy of the model on the verification set of each difficulty level in real time during training; dynamically adjusting the sampling weight based on the accuracy; reducing the weight of the level and assigning it to a higher level when the accuracy continuously exceeds a first threshold value; increasing the weight of the level when the accuracy continuously falls below a second threshold value, and then normalizing; extracting low difficulty samples after the stage ends to add to the data set of the next stage, and performing parameter freezing and fine tuning; and adjusting the difficulty feature weight, stage division threshold and control judgment threshold by using the training process data. The application realizes self-adaptive control and multi-stage progressive learning of training difficulty.

Description

Technical Field

[0001] This invention relates to the field of large language model training technology, specifically to a multi-stage simulation training method, system, device, and medium for large language models. Background Technology

[0002] Currently, large language models are typically trained using large-scale, high-quality general corpora for pre-training, followed by task adaptation through instruction fine-tuning or reinforcement learning. However, existing training methods have significant shortcomings in areas such as sample difficulty control and training pace.

[0003] On the one hand, the uneven distribution of training data difficulty makes it easy for models to become unstable in convergence when exposed to complex samples in the early stages of training. On the other hand, the lack of effective connections between training stages makes it difficult to achieve gradual accumulation of knowledge and gradual improvement of capabilities. In addition, existing methods do not make sufficient use of the model's performance feedback during training and cannot dynamically adjust subsequent training strategies according to the model's current learning state, resulting in low training efficiency and limited model generalization ability.

[0004] Therefore, there is an urgent need in this field for a multi-stage large language model training method that can adaptively adjust the training difficulty and training pace according to the model's learning state, in order to solve the technical problems of uneven distribution of training difficulty, insufficient stage connection, and insufficient utilization of feedback in the existing technology. Summary of the Invention

[0005] To address the problems of uneven training difficulty distribution, insufficient stage transitions, and inadequate feedback utilization in existing technologies, this invention provides a multi-stage simulation training method, system, device, and medium for large language models, thereby resolving the aforementioned technical deficiencies.

[0006] This invention proposes a multi-stage simulation training method for large language models, comprising the following steps:

[0007] S1. Extract multi-dimensional difficulty features from the training samples, calculate the comprehensive difficulty score of each training sample based on the extracted difficulty features, and divide the training samples into multiple difficulty levels based on the comprehensive difficulty score to construct a sample difficulty label system.

[0008] S2. Based on the sample difficulty labeling system, set up multiple progressive training stages, where the training objective of each stage matches the preset sample difficulty distribution, and sample according to the preset sample difficulty distribution to construct the training dataset for each stage.

[0009] S3. During the training of the large language model based on multiple training stages, monitor the performance of the large language model on the validation set containing training samples of various difficulty levels in real time, and obtain the accuracy monitoring data of each difficulty level.

[0010] S4. Based on accuracy monitoring data, dynamically adjust the sampling weights of training samples at each difficulty level during subsequent training. The adjustments include: for difficulty levels where the accuracy meets the first condition, reduce their sampling weights and allocate the reduced weights to higher difficulty levels; for difficulty levels where the accuracy meets the second condition, increase their sampling weights; and normalize the adjusted sampling weights of each difficulty level to adaptively regulate the distribution of training difficulty.

[0011] S5. After the current training phase ends, extract a preset proportion of low-difficulty training samples from the training dataset of the current training phase, add the low-difficulty training samples to the training dataset of the next training phase, and freeze and fine-tune the parameters of the large language model when starting the next training phase.

[0012] S6. Use the model performance data and adjustment records collected throughout the training process as feedback information to adjust at least one of the following: the weight coefficient used for difficulty feature extraction in step S1, the division threshold of the training phase in step S2, and the preset threshold used to judge the adjustment conditions in step S4.

[0013] Preferably, step S1 includes the following sub-steps:

[0014] S11. Extract difficulty features for each training sample in five dimensions: text length, syntactic complexity, vocabulary difficulty, reasoning steps, and knowledge breadth.

[0015] S12. Calculate the overall difficulty score for each training sample using a weighted summation method. The calculation expression is as follows:

[0016]

[0017] ;

[0018] in Indicates training samples, Indicates the sample number. This indicates the overall difficulty score. Indicates text length. Indicate syntactic complexity, Indicates vocabulary difficulty. Indicates the number of reasoning steps. Indicates breadth of knowledge; The weighting coefficients are adjustable and , This represents the normalization function that normalizes the difficulty feature values ​​of each dimension to the interval [0,1].

[0019] S13. Based on the numerical range of the comprehensive difficulty score, the training samples are divided into five difficulty levels, specifically: When ∈[0,0.2), it is classified as L1 level. When ∈ [0.2, 0.4), it is classified as L2 level. When ∈ [0.4, 0.6), it is classified as level L3. When ∈ [0.6, 0.8), it is classified as level L4. When ∈[0.8,1], it is classified as level L5.

[0020] More preferably, in step S2, a training dataset for each stage is constructed by setting up multiple progressively advancing training stages and sampling according to a preset sample difficulty distribution, including the following sub-steps:

[0021] S21. Set the first training stage as the basic capability building stage, configure the preset sample difficulty distribution of the basic capability building stage as L1 level to L2 level, sample training samples of L1 level to L2 level from the sample difficulty label system, and construct the training dataset of the first training stage.

[0022] S22. Set the second training stage as the capability expansion and fusion stage, configure the preset sample difficulty distribution of the capability expansion and fusion stage as L2 level to L4 level, sample training samples of L2 level to L4 level from the sample difficulty label system, and construct the training dataset for the second training stage.

[0023] S23. Set the third training stage as the advanced ability enhancement stage, configure the preset sample difficulty distribution of the advanced ability enhancement stage as L3 level to L5 level, sample training samples of L3 level to L5 level from the sample difficulty label system, and construct the training dataset of the third training stage.

[0024] Preferably, step S3 includes the following sub-steps:

[0025] S31. In each preset verification cycle, calculate the accuracy of the large language model on the verification set of each difficulty level, and use it as accuracy monitoring data.

[0026] S32. Record the loss function value of the large language model on the overall validation set, and generate a loss function descent curve based on the loss function value;

[0027] S33. Record the gradient norm of the large language model during the training process, and generate the gradient norm change trend based on the gradient norm.

[0028] S34. Accuracy monitoring data, loss function values, gradient norms, and performance differences of large language models on different task types are stored in a sliding window for use in step S4.

[0029] Preferably, in step S4, dynamically adjusting the sampling weights of training samples at each difficulty level during subsequent training specifically includes:

[0030] Based on the accuracy monitoring data, determine the difficulty level of an accuracy exceeding the first threshold for multiple consecutive monitoring periods and the difficulty level of an accuracy below the second threshold for multiple consecutive monitoring periods.

[0031] For difficulty levels where the accuracy is below the second threshold for multiple consecutive monitoring periods, increase its sampling weight; for difficulty levels where the accuracy exceeds the first threshold for multiple consecutive monitoring periods, decrease its sampling weight and allocate the decreased weight to the next higher difficulty level.

[0032] When there are difficulty levels with an accuracy rate below the second threshold for multiple consecutive monitoring periods and difficulty levels with an accuracy rate above the first threshold for multiple consecutive monitoring periods, the difficulty levels with an accuracy rate below the second threshold are processed first, followed by the difficulty levels with an accuracy rate above the first threshold. For cases where there are multiple difficulty levels within the same priority, the weight adjustment amount of each difficulty level is calculated in order from low to high difficulty level.

[0033] After adjusting the sampling weights of the corresponding difficulty levels according to the weight adjustment amount, the sampling weights of all difficulty levels are normalized so that the sum of the sampling weights of each difficulty level is 1.

[0034] More preferably, step S5 includes the following sub-steps:

[0035] S51. Randomly select a preset proportion of low-difficulty training samples from the training dataset of the current training stage. The low-difficulty training samples are training samples of L1 level to L2 level.

[0036] S52. Add the extracted low-difficulty training samples to the training dataset for the next training stage.

[0037] S53. When starting the next training phase, freeze the parameters of the lower-level networks in the large language model at a preset number of layers, and only fine-tune the parameters of the higher-level networks that are not frozen.

[0038] Preferably, in step S6, the preset threshold used to determine the adjustment condition in step S4 is adjusted using a reinforcement learning method, including the following sub-steps:

[0039] S61. Construct the state space, which includes the accuracy of each difficulty level on the validation set obtained in step S3, the overall loss value of the current validation set, the loss descent slope of multiple validation cycles, the mean gradient norm, the sampling weights of each difficulty level at the current time, and the identifier of the current training stage.

[0040] S62. Construct the action space, which includes the adjustment range of the sampling weights for each difficulty level, the scaling factor of the learning rate, the adjustment coefficient of the regularization strength, and the adjustment instructions for the first threshold and the second threshold.

[0041] S63. Define the reward function as follows:

[0042] ;

[0043] in, Indicates the reward value. This indicates the improvement in the overall accuracy of the validation set after executing the adjustment instructions in S62. This indicates the increase in the validation set loss value after the adjustment instruction is executed. For balance coefficient, As a variety of reward items, The weighting coefficients for the diversity of reward items;

[0044] S64. Using the state space as input, the policy network outputs adjustment instructions in the action space. After executing the adjustment instructions, the reward value is calculated according to the reward function. State transition experience is collected and stored in the experience pool. The policy network parameters are updated using reinforcement learning algorithms until the large language model is trained or the policy converges. The optimal policy for adjusting the first and second thresholds is output.

[0045] This invention also proposes a multi-stage simulation training system for large language models to implement the method described above. The system includes:

[0046] The difficulty grading module is configured to extract multi-dimensional difficulty features from training samples, calculate the comprehensive difficulty score of each training sample based on the extracted difficulty features, and divide the training samples into multiple difficulty levels based on the comprehensive difficulty score to construct a sample difficulty labeling system.

[0047] The multi-stage training module is configured to include multiple progressively advancing training stages based on a sample difficulty labeling system. Each training stage is configured with a preset sample difficulty distribution that matches its training objective, and sampling is performed based on the preset sample difficulty distribution to construct the training dataset for each training stage.

[0048] The real-time monitoring module is configured to monitor the performance of the large language model on the validation set containing training samples of various difficulty levels in real time during the training of the large language model based on multiple training stages, and obtain accuracy monitoring data for each difficulty level.

[0049] The adaptive adjustment module is configured to dynamically adjust the sampling weights of training samples at each difficulty level during subsequent training based on accuracy monitoring data. The adjustment includes: reducing the sampling weight of difficulty levels whose accuracy meets the first condition and allocating the reduced weights to higher difficulty levels; increasing the sampling weight of difficulty levels whose accuracy meets the second condition; and normalizing the adjusted sampling weights of each difficulty level.

[0050] The knowledge transfer module is configured to extract a preset proportion of low-difficulty training samples from the training dataset of the current training phase after the current training phase ends, add the low-difficulty training samples to the training dataset of the next training phase, and freeze and fine-tune the parameters of the large language model when starting the next training phase.

[0051] The strategy optimization module is configured to use the model performance data and control records collected throughout the training process as feedback information to adjust at least one of the following: the weight coefficient used for difficulty feature extraction in the difficulty grading module, the threshold for dividing the training stage in the multi-stage training module, and the preset threshold used to determine the adjustment conditions in the adaptive control module.

[0052] The present invention also proposes a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the steps of the large language model multi-stage simulation training method as described above.

[0053] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multi-stage simulation training method for large language models as described above.

[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0055] (1) Based on the accuracy monitoring data, the sampling weight of training samples of each difficulty level is dynamically adjusted. The sampling weight of difficulty levels with good model mastery is reduced, and the sampling weight of difficulty levels with insufficient model mastery is increased. This solves the problem of fixed training difficulty distribution and inability to adapt to the model learning state in the existing technology, so that the model is always in a suitable learning difficulty range and the training stability is improved.

[0056] (2) By setting multiple progressive training stages and configuring a preset sample difficulty distribution that matches the training objective for each stage, a progressive training path from low difficulty to high difficulty is formed, which solves the problem of lack of effective connection between training stages and difficulty in achieving progressive knowledge accumulation in the existing technology, and enables the model capability to be gradually improved.

[0057] (3) By extracting low-difficulty samples after the current training phase ends and adding them to the training dataset of the next training phase, and freezing and fine-tuning the model parameters, the learned general knowledge is preserved and catastrophic forgetting is prevented. This solves the problem of knowledge gaps during phase switching in existing technologies and achieves smooth transition and accumulation of knowledge.

[0058] (4) By using the model performance data and regulation records collected throughout the training process as feedback information, the difficulty feature weights, stage division thresholds and regulation judgment thresholds are adjusted to form a closed-loop optimization mechanism, which solves the problem that the training strategy is fixed and cannot be adaptively optimized according to the training effect in the existing technology, and realizes the continuous evolution of the training strategy.

[0059] (5) By extracting and comprehensively scoring the difficulty features of the training samples in five dimensions—text length, syntactic complexity, vocabulary difficulty, reasoning steps, and knowledge breadth—a scientific and comprehensive sample difficulty labeling system was constructed, providing an accurate data foundation for subsequent adaptive difficulty control. Attached Figure Description

[0060] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments, taken with reference to the accompanying drawings:

[0061] Figure 1 This is a flowchart of the multi-stage simulation training method for the large language model of the present invention;

[0062] Figure 2 This is a structural diagram of the multi-stage simulation training system for the large language model of this invention;

[0063] Figure 3 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention. Detailed Implementation

[0064] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. The specific embodiments described herein are merely illustrative of the invention and not intended to limit it. It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the drawings. Unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0065] Figure 1 The flowchart of the multi-stage simulation training method for the large language model of the present invention is shown, as follows: Figure 1 As shown, the method includes the following steps:

[0066] S1. Extract multi-dimensional difficulty features from the training samples, calculate the comprehensive difficulty score for each training sample based on the extracted features, and classify the training samples into multiple difficulty levels based on the comprehensive difficulty score, thus constructing a sample difficulty labeling system. This specifically includes the following sub-steps:

[0067] S11. Input training sample set: Given a training sample set ,in Indicates the first There are 10 training samples, each sample being a question-answer pair or an instruction-response pair.

[0068] S12. Difficulty Feature Extraction: For each training sample Extract the following five dimensions of difficulty features:

[0069] Text length : The number of characters or words in the sample.

[0070] Syntactic complexity The depth of the syntax tree, which is the path length from the root node to the farthest leaf node.

[0071] vocabulary difficulty Rare word percentage is defined as the proportion of words in the sample whose frequency is below a preset threshold to the total number of words.

[0072] Number of reasoning steps The minimum number of logical reasoning steps required for a sample is obtained through parsing a predefined reasoning chain.

[0073] Breadth of knowledge The number of knowledge domains involved in the sample, estimated based on entity linking or topic classification models.

[0074] S13. Difficulty Score Calculation: The overall difficulty score for each sample is calculated using a weighted summation method. First, the features of each dimension are normalized to map them to... The interval, the normalization function is denoted as The formula for calculating the overall difficulty score is as follows:

[0075]

[0076] ;

[0077] in The weighting coefficients are adjustable and satisfy the following conditions: These weighting coefficients can be dynamically adjusted based on subsequent feedback.

[0078] S14. Difficulty Level: Based on The numerical values ​​divide the samples into five difficulty levels:

[0079] L1 (Extremely Easy): ;

[0080] L2 (Easier): ;

[0081] L3 (Intermediate): ;

[0082] L4 (Difficult): ;

[0083] L5 (Extremely Difficult): ;

[0084] Through the above steps, each training sample is assigned a difficulty level label, forming a sample difficulty label system.

[0085] S2. Based on the sample difficulty labeling system, set up multiple progressive training stages, where the training objective of each stage matches the preset sample difficulty distribution, and sample according to the preset sample difficulty distribution to construct the training dataset for each stage.

[0086] This embodiment sets up three progressively advancing training phases:

[0087] Phase 1: Foundation Building Phase. The training objectives of this phase are basic language modeling skills, instruction comprehension, and basic task execution. The preset sample difficulty distribution is mainly L1 to L2 levels. Training samples of L1 and L2 levels are sampled from the sample difficulty labeling system to construct the training dataset for the first training phase.

[0088] Phase Two: Capability Expansion and Integration. The training objectives for this phase are multi-task learning, cross-task transfer, and preliminary reasoning abilities. A pre-defined sample difficulty distribution is introduced, ranging from L2 to L4 levels. Training samples at levels L2, L3, and L4 are sampled from the sample difficulty labeling system to construct the training dataset for the second training phase.

[0089] Phase Three: Advanced Capability Enhancement Phase. The training objectives for this phase are complex reasoning, long text comprehension, and multi-step task execution. The preset sample difficulty distribution is mainly at levels L3 to L5. Training samples at levels L3, L4, and L5 are sampled from the sample difficulty labeling system to construct the training dataset for the third training phase.

[0090] Training in each stage lasts for several rounds. The stage switching condition is determined by the model's performance on the validation set. For example, when the model's validation accuracy in the current training stage reaches a preset threshold or the loss no longer decreases, it automatically switches to the next stage.

[0091] S3. During the training of the large language model across multiple training phases, monitor the performance of the large language model in real time on the validation set containing training samples of various difficulty levels, and obtain accuracy monitoring data for each difficulty level. This includes the following sub-steps:

[0092] S31: In each preset verification cycle (e.g., every 500 steps), calculate the accuracy of the large language model on the verification set at each difficulty level, denoted as... ( ), which serves as accuracy monitoring data.

[0093] S32: Record the loss function value of the large language model on the overall validation set. The model generates a loss function decline curve based on the loss function value to observe the model convergence trend.

[0094] S33: Records the gradient norm of the large language model during training. It generates gradient norm change trends based on the gradient norm, which are used to monitor gradient vanishing or exploding.

[0095] S34: Accuracy monitoring data, loss function values, gradient norms, and performance differences of the large language model across different task types (such as question answering, summarizing, and inference) are stored in a sliding window for use in subsequent steps (S4). The size of the sliding window can be set according to actual needs, for example, storing the most recent data. Data from one period.

[0096] S4. Based on accuracy monitoring data, dynamically adjust the sampling weights of training samples at each difficulty level during subsequent training. The adjustments include: for difficulty levels where accuracy meets the first condition, reducing their sampling weights and allocating the reduced weights to higher difficulty levels; for difficulty levels where accuracy meets the second condition, increasing their sampling weights; and normalizing the adjusted sampling weights for each difficulty level to adaptively regulate the training difficulty distribution. The specific regulation strategy is as follows:

[0097] First, based on the accuracy monitoring data, determine multiple consecutive monitoring periods (e.g., consecutive monitoring periods). The accuracy (in cycles) exceeds the first threshold. A difficulty level of 0.8 (e.g., and an accuracy rate below the second threshold for multiple consecutive monitoring periods). Difficulty level (e.g., 0.4).

[0098] For difficulty levels where the accuracy is below the second threshold for multiple consecutive monitoring periods, increasing the sampling weight (e.g., increasing the relative weight by 10%) indicates that the training of samples at that difficulty level needs to be strengthened.

[0099] For difficulty levels where the accuracy exceeds the first threshold for multiple consecutive monitoring periods, reduce its sampling weight (e.g., reduce the relative weight by 10%), and allocate the reduced weight to the next higher difficulty level. If the current level is the highest, L5, the weight can be allocated to L5 itself or not allocated at all, depending on actual needs.

[0100] When multiple difficulty levels meet the adjustment criteria simultaneously, the following coordination strategy is adopted:

[0101] Prioritize processing difficulty levels with accuracy below the second threshold (weak links), and then process difficulty levels with accuracy above the first threshold (oversaturated links).

[0102] For multiple difficulty levels within the same priority, the weight adjustment amount for each difficulty level is calculated sequentially from low to high.

[0103] For example, suppose that during a certain validation period, the accuracy of L2 level continuously falls below the second threshold, then its weight needs to be increased; while the accuracy of L4 level continuously exceeds the first threshold, then its weight needs to be decreased. Following priority, L2 level is processed first, with its weight increased by 10%; then L4 level is processed, with its weight decreased by 10%. If directly adding the weights might cause the total weight to deviate from 1, then normalization is performed last.

[0104] After all adjustment calculations are completed, the sampling weights for all five difficulty levels are normalized to ensure that the sum of the adjusted weights is 1. Specifically, let the weights of each level before adjustment be... The weight increment calculated according to the adjustment rules is (This weight increment can be positive or negative), then the adjusted weight is Finally, it is normalized to .

[0105] As an optional embodiment, this step may also include training objective adjustment and hyperparameter adjustment: dynamically adjust the weights of the task loss function according to the model's performance on various tasks, giving more attention to tasks with poor performance; if the overall convergence speed of the model slows down (e.g., the loss decrease is less than [a certain value] for several consecutive periods); If the learning rate is adjusted dynamically or a regularization strategy is introduced, such as weight decay or label smoothing, then the learning rate can be adjusted dynamically.

[0106] S5. After the current training phase ends, extract a preset proportion of low-difficulty training samples from the training dataset of the current training phase, add the low-difficulty training samples to the training dataset of the next training phase, and freeze and fine-tune the parameters of the large language model when starting the next training phase.

[0107] Step S5 includes the following sub-steps:

[0108] S51: Randomly select a predetermined proportion (e.g., 10%) of low-difficulty training samples from the training dataset of the current training phase. These low-difficulty training samples are L1-L2 level training samples. These samples are used to maintain the basic capabilities of the model and prevent catastrophic forgetting.

[0109] S52: Add the extracted low-difficulty training samples to the training dataset of the next training stage, and merge them with the new training samples sampled in the next stage according to its preset sample difficulty distribution to form the enhanced training dataset of the next stage.

[0110] S53: When starting the next training phase, freeze the low-level network parameters (e.g., the first few layers) representing general knowledge in the large language model, and only fine-tune the high-level network parameters responsible for adapting to high-level tasks. This allows for adaptation to new training tasks while retaining the learned general knowledge.

[0111] S6. Using the model performance data and adjustment records collected throughout the training process as feedback information, adjust at least one of the following: the weight coefficient used for difficulty feature extraction in step S1, the partitioning threshold during the training phase in step S2, and the preset threshold used to determine the adjustment conditions in step S4. This can be achieved in the following ways:

[0112] (1) Adjusting the weight coefficients of difficulty features: Based on the actual performance of each difficulty level sample on the validation set, i.e., the accuracy of each difficulty level in the model performance data, adjust the weight coefficients α, β, γ, δ, ε in the difficulty scoring formula. Specifically, if the accuracy of a certain difficulty level sample is consistently low, the weight of the corresponding difficulty feature can be appropriately increased to make the difficulty assessment of that level sample more accurate; conversely, if the accuracy is consistently high, the corresponding weight can be appropriately decreased.

[0113] (2) Adjust the threshold for dividing the training phase: Optimize the threshold conditions for dividing the phase based on the model state at the time of phase switching, i.e., the validation set accuracy, loss value, etc. in the model performance data. For example, if the model's validation accuracy in the current phase reaches the preset threshold in advance and tends to stabilize, it can switch to the next phase in advance; if the model still does not achieve the expected performance after the preset number of training rounds, the training of the current phase can be appropriately extended.

[0114] (3) Adjusting the preset threshold: Using reinforcement learning or Bayesian optimization methods, the optimal control strategy parameters are automatically searched using the model performance data (such as accuracy, loss value, gradient norm, etc.) collected throughout the training process and the control records as input, including the first and second thresholds used to determine the adjustment conditions in step S4, the number of continuous monitoring periods k, etc. For the specific implementation method, please refer to the detailed description of S61 to S64 below.

[0115] Through the above method, the present invention achieves adaptive evolution of the training strategy, ensuring that the training difficulty distribution always matches the current learning state of the model. As a preferred embodiment, this step utilizes reinforcement learning methods to adjust the preset threshold used to determine the adjustment conditions in step S4, specifically including the following sub-steps:

[0116] S61. Constructing the state space: At each adjustment decision moment, such as each validation cycle, the current model state and training progress are encoded into a state vector. The state space consists of the following 16 components:

[0117] The accuracy of each difficulty level on the validation set has 5 components, corresponding to the five difficulty levels from L1 to L5; the overall loss value of the current validation set has 1 component; the loss descent slope of the most recent multiple validation periods has 1 component; the mean gradient norm has 1 component; the sampling weights of each difficulty level have 5 components, corresponding to the five difficulty levels from L1 to L5; and the current training stage identifier uses one-hot encoding and has 3 components, corresponding to the three training stages.

[0118] The above 16 components together constitute the state vector. , is used to characterize the model training state at the current moment.

[0119] S62. Constructing the Action Space: The agent outputs continuous action vectors. This determines the amount of strategy adjustment in the next regulatory cycle. The action space includes the following 7 continuous variables:

[0120] The adjustment range of the sampling weights for each difficulty level consists of 5 components, corresponding to the five difficulty levels from L1 to L5. The value range of each component is as follows: After execution, the sampling weights of each level need to be normalized.

[0121] The learning rate scaling factor has one component and its value ranges from [value range missing]. ;

[0122] The regularization intensity adjustment coefficient is a single component with a value range of [value range missing]. .

[0123] Furthermore, the adjustment instructions for the first and second thresholds can be used as an extended dimension of the aforementioned action vector, or implemented in other ways. In this embodiment, they are used as part of the action space, and can be designed as needed for specific implementation.

[0124] S63. Define the reward function: when performing an action... Afterward, continue training for a fixed number of steps (e.g., 500 steps), then evaluate the performance change on the validation set and define the reward. for:

[0125] ;

[0126] in This indicates the improvement in the overall accuracy of the validation set after the action was performed. This is the amount of increase in the loss value (penalty). This is the balance coefficient; Diverse reward items are provided to encourage exploration, such as reducing the variance of accuracy at each level. This represents the weighting coefficient for the diversity reward items.

[0127] S64. Optimization Iteration: Initialize the policy network (such as the Actor-Critic network in the PPO algorithm) and the value network. In each training phase, when the regulatory trigger condition is met, collect the current state. Actions are output by the policy network. The actions are mapped to actual control parameters (adjusting sampling weights, learning rate, regularization strength, threshold, etc.), and training continues for several more steps. The next state is then collected. And calculate the reward , experience The experience is stored in the experience pool. After accumulating a certain amount of experience, the policy network is updated using the PPO algorithm to maximize the accumulated expected reward. This process is repeated until the model training is complete or the policy converges, ultimately resulting in a policy network that can dynamically output the optimal control parameters based on the real-time state, achieving adaptive evolution of the training policy.

[0128] To illustrate the invention more clearly, examples are given below in conjunction with specific tasks.

[0129] Example 1: Mathematical Reasoning Task

[0130] First, following step S1, the training samples are divided into five difficulty levels, L1 to L5, with an initial sampling weight of 20% for each level. During training, every 500 steps (one validation cycle), the model's accuracy on the validation set for each difficulty level is calculated. A threshold is then set. , Continuous monitoring cycle .

[0131] In the later stages of building the basic capabilities in Phase 1, the model's accuracy at L1 and L2 levels exceeded 0.85 for three consecutive cycles, while its accuracy at L3 level was only 0.3. According to the adjustment strategy in step S4, the sampling weights for L1 and L2 levels were each reduced by 10%, and the total reduced weight (20%) was allocated to L3 level, increasing the sampling weight of L3 level from 20% to 40%. After this adjustment, the proportion of L3 level samples in the training data increased significantly, allowing the model to encounter more moderately difficult samples in subsequent training, and its L3 level accuracy gradually improved to above 0.7. Simultaneously, due to the reduced sample weights at L1 and L2 levels, the model avoided overfitting on simple samples, and its basic capabilities remained stable. After multiple rounds of dynamic adjustments, the model's accuracy at L4 and L5 levels also significantly improved.

[0132] Example 2: Multi-agent game AI training task

[0133] In the multi-agent game AI training task, the difficulty features in step S1 are reconstructed in a scenario-based manner:

[0134] The number of reasoning steps is specifically defined as the number of game deduction steps, which is the number of key decision-making steps required to go from the current game state to achieving the game objective, such as a multi-step decision chain of "resource gathering → unit production → path planning → combat decision → battle result evaluation".

[0135] The breadth of knowledge is concretized into the types of units involved and the number of game dimensions. For example, a real-time strategy game may involve multiple unit types such as infantry, armor, air force, and logistics, as well as multiple game dimensions such as resources, terrain, technology, and tactics.

[0136] Syntactic complexity corresponds to the degree of structuring of game command text. For example, a complex tactical command may contain multiple clauses and conditional branches.

[0137] The vocabulary difficulty corresponds to the rarity of game terms, such as the frequency of occurrence of professional terms like fast attack tactics, economic transformation, and micro-management.

[0138] Phase 1: Focusing on simple cases involving a single unit type and typical tactical scenarios, the training model aims to understand basic game terminology, master single-dimensional state descriptions, and execute simple commands. The sample difficulty is primarily L1 to L2 level. For example, "Identify and report the location of enemy units."

[0139] Phase Two: Introducing multi-unit collaborative, medium-complexity campaign-level cases to train the model to master cross-unit collaboration rules, multi-source information fusion, and preliminary game simulation capabilities. The sample difficulty is mainly L2 to L4 level. For example, "Based on reconnaissance information, coordinate ground and air units to launch a joint attack on the enemy base."

[0140] Phase Three: Focusing on advanced cases involving multi-agent collaboration and complex dynamic environments, the model is trained to handle multi-dimensional threats, dynamic game changes, and multi-step tactical simulations. The sample difficulty is mainly L3 to L5 level. For example, "Faced with enemy multi-line operations and tactical interference, formulate a joint attack plan and dynamically adjust troop deployment."

[0141] The method of this invention significantly improves the model's understanding of game situation, accuracy of tactical deduction, rationality of decision-making schemes, and understanding of game terminology.

[0142] Further reference Figure 2 As an implementation of the above method, this invention also proposes a multi-stage simulation training system 200 for large language models, which can be applied to various electronic devices. The multi-stage simulation training system 200 for large language models includes the following modules:

[0143] The difficulty grading module 210 is configured to extract multi-dimensional difficulty features from training samples, calculate the comprehensive difficulty score of each training sample based on the extracted difficulty features, and divide the training samples into multiple difficulty levels based on the comprehensive difficulty score to construct a sample difficulty label system.

[0144] The multi-stage training module 220 is configured to include multiple progressively advancing training stages based on a sample difficulty labeling system. Each training stage is configured with a preset sample difficulty distribution that matches its training objective, and sampling is performed based on the preset sample difficulty distribution to construct the training dataset for each training stage.

[0145] The real-time monitoring module 230 is configured to monitor the performance of the large language model on the validation set containing training samples of various difficulty levels in real time during the training of the large language model according to multiple training stages, and obtain the accuracy monitoring data of each difficulty level.

[0146] The adaptive adjustment module 240 is configured to dynamically adjust the sampling weights of training samples at each difficulty level during subsequent training based on accuracy monitoring data. The adjustment includes: reducing the sampling weight of difficulty levels whose accuracy meets the first condition and allocating the reduced weights to higher difficulty levels; increasing the sampling weight of difficulty levels whose accuracy meets the second condition; and normalizing the adjusted sampling weights of each difficulty level.

[0147] The knowledge transfer module 250 is configured to extract a preset proportion of low-difficulty training samples from the training dataset of the current training phase after the current training phase ends, add the low-difficulty training samples to the training dataset of the next training phase, and freeze and fine-tune the parameters of the large language model when starting the next training phase.

[0148] The strategy optimization module 260 is configured to use the model performance data and control records collected throughout the training process as feedback information to adjust at least one of the following: the weight coefficient used for difficulty feature extraction in the difficulty grading module, the threshold for dividing the training stage in the multi-stage training module, and the preset threshold used to determine the adjustment conditions in the adaptive control module.

[0149] The present invention also proposes a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the steps of the large language model multi-stage simulation training method as described above.

[0150] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multi-stage simulation training method for large language models as described above.

[0151] The following is for reference. Figure 3 It shows a schematic diagram of the structure of a computer system 300 suitable for implementing terminal devices or servers in the embodiments of this application. Figure 3 The terminal device or server shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0152] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 302 or programs loaded from storage section 308 into random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the computer system 300. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0153] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a liquid crystal display (LCD) and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card and a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 310 as needed so that computer programs read from it can be installed into storage section 308 as needed.

[0154] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable medium or any combination thereof. The computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0155] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A multi-stage simulation training method for a large language model, characterized in that, Includes the following steps: S1. Extract difficulty features for each training sample across five dimensions: text length, syntactic complexity, lexical difficulty, inference steps, and knowledge breadth. Text length is the number of characters or words in the sample; syntactic complexity is the depth of the syntactic tree, i.e., the path length from the root node to the farthest leaf node; lexical difficulty is the proportion of rare words, defined as the percentage of words in the sample whose frequency is below a preset threshold; inference steps are the minimum number of logical inference steps required for the sample, obtained through parsing a predefined inference chain; and knowledge breadth is the number of knowledge domains involved in the sample, estimated based on entity linking or topic classification models. The comprehensive difficulty score of each training sample is calculated based on the extracted difficulty features, and the training samples are divided into multiple difficulty levels based on the comprehensive difficulty score to construct a sample difficulty labeling system. S2. Based on the sample difficulty labeling system, a training phase consisting of multiple progressively advancing stages is set, wherein the training objective of each stage matches the preset sample difficulty distribution, and sampling is performed according to the preset sample difficulty distribution to construct the training dataset for each stage. S3. During the training of the large language model based on the multiple training stages, the performance of the large language model on the validation set containing training samples of various difficulty levels is monitored in real time to obtain accuracy monitoring data for each difficulty level, including the following sub-steps: S31. In each preset verification cycle, calculate the accuracy of the large language model on each difficulty level verification set, and use it as the accuracy monitoring data. S32. Record the loss function value of the large language model on the overall validation set, and generate a loss function descent curve based on the loss function value; S33. Record the gradient norm of the large language model during the training process, and generate a gradient norm change trend based on the gradient norm; S34. Store the accuracy monitoring data, the loss function value, the gradient norm, and the performance difference data of the large language model on different task types in the form of a sliding window for use in step S4. S4. Based on the accuracy monitoring data, dynamically adjust the sampling weights of training samples at each difficulty level during subsequent training; the adjustment includes: for difficulty levels where the accuracy meets the first condition, reduce their sampling weights and allocate the reduced weights to higher difficulty levels; for difficulty levels where the accuracy meets the second condition, increase their sampling weights; and normalize the adjusted sampling weights of each difficulty level to adaptively regulate the distribution of training difficulty. S5. After the current training phase ends, a preset proportion of low-difficulty training samples are extracted from the training dataset of the current training phase, and the low-difficulty training samples are added to the training dataset of the next training phase. When the next training phase starts, the parameters of the large language model are frozen and fine-tuned. S6. Use the model performance data and adjustment records collected throughout the training process as feedback information to adjust at least one of the following: the weight coefficient used for difficulty feature extraction in step S1, the division threshold of the training phase in step S2, and the preset threshold used to judge the adjustment conditions in step S4.

2. The multi-stage simulation training method for large language models according to claim 1, characterized in that, Step S1 includes the following sub-steps: The overall difficulty score for each training sample is calculated using a weighted summation method, expressed as follows: ； in Indicates training samples, Indicates the sample number. This indicates the overall difficulty score. Indicates text length. Indicate syntactic complexity, Indicates vocabulary difficulty. Indicates the number of reasoning steps. Indicates breadth of knowledge; These are adjustable weighting coefficients, and , This represents the normalization function that normalizes the difficulty feature values ​​of each dimension to the interval [0,1]. The training samples are divided into five difficulty levels based on the numerical range of the comprehensive difficulty score, as follows: When ∈[0,0.2), it is classified as L1 level. When ∈ [0.2, 0.4), it is classified as L2 level. When ∈ [0.4, 0.6), it is classified as level L3. When ∈ [0.6, 0.8), it is classified as level L4. When ∈[0.8,1], it is classified as level L5.

3. The multi-stage simulation training method for large language models according to claim 2, characterized in that, In step S2, multiple progressively advancing training stages are defined, and sampling is performed according to the preset sample difficulty distribution to construct training datasets for each stage. This includes the following sub-steps: S21. Set the first training stage as the basic capability building stage, configure the preset sample difficulty distribution of the basic capability building stage as L1 level to L2 level, sample training samples of L1 level to L2 level from the sample difficulty label system, and construct the training dataset of the first training stage. S22. Set the second training stage as the capability expansion and fusion stage, configure the preset sample difficulty distribution of the capability expansion and fusion stage as L2 level to L4 level, sample training samples of L2 level to L4 level from the sample difficulty label system, and construct the training dataset of the second training stage. S23. Set the third training stage as the advanced ability enhancement stage, configure the preset sample difficulty distribution of the advanced ability enhancement stage as L3 level to L5 level, sample training samples of L3 level to L5 level from the sample difficulty label system, and construct the training dataset of the third training stage.

4. The multi-stage simulation training method for large language models according to claim 1, characterized in that, In step S4, the sampling weights of training samples at each difficulty level are dynamically adjusted during subsequent training, specifically including: Based on the accuracy monitoring data, determine the difficulty level of an accuracy exceeding a first threshold for multiple consecutive monitoring periods and the difficulty level of an accuracy below a second threshold for multiple consecutive monitoring periods. For difficulty levels where the accuracy is below the second threshold for multiple consecutive monitoring periods, increase its sampling weight; for difficulty levels where the accuracy exceeds the first threshold for multiple consecutive monitoring periods, decrease its sampling weight and allocate the decreased weight to the next higher difficulty level. When there are difficulty levels with an accuracy rate below the second threshold for multiple consecutive monitoring periods and difficulty levels with an accuracy rate above the first threshold for multiple consecutive monitoring periods, the difficulty levels with an accuracy rate below the second threshold are processed first, followed by the difficulty levels with an accuracy rate above the first threshold. For cases where there are multiple difficulty levels within the same priority, the weight adjustment amount of each difficulty level is calculated in order from low to high difficulty level. After adjusting the sampling weights of the corresponding difficulty levels according to the weight adjustment amount, the sampling weights of all difficulty levels are normalized so that the sum of the sampling weights of each difficulty level is 1.

5. The multi-stage simulation training method for large language models according to claim 2, characterized in that, Step S5 includes the following sub-steps: S51. Randomly select a preset proportion of low-difficulty training samples from the training dataset of the current training stage. The low-difficulty training samples are training samples of L1 level to L2 level. S52. Add the extracted low-difficulty training samples to the training dataset for the next training stage. S53. When starting the next training phase, freeze the parameters of the lower-level network of the preset number of layers in the large language model, and only fine-tune the parameters of the upper-level network that are not frozen.

6. The multi-stage simulation training method for large language models according to claim 4, characterized in that, In step S6, the preset threshold used to determine the adjustment condition in step S4 is adjusted using a reinforcement learning method, including the following sub-steps: S61. Construct a state space, which includes the accuracy of each difficulty level on the validation set obtained in step S3, the overall loss value of the current validation set, the loss descent slope of multiple validation cycles, the average gradient norm, the sampling weight of each difficulty level at the current time, and the identifier of the current training stage. S62. Construct an action space, which includes the adjustment range of the sampling weights for each difficulty level, the scaling factor of the learning rate, the adjustment coefficient of the regularization intensity, and the adjustment instructions for the first threshold and the second threshold. S63. Define a reward function, wherein the reward function is: ； in, Indicates the reward value. This indicates the improvement in the overall accuracy of the validation set after executing the adjustment instructions described in S62. This indicates the increase in the validation set loss value after executing the adjustment instruction. For balance coefficient, As a variety of reward items, The weighting coefficients for the diversity of reward items; S64. Using the state space as input, the policy network outputs adjustment instructions in the action space. After executing the adjustment instructions, the reward value is calculated according to the reward function. State transition experience is collected and stored in the experience pool. The policy network parameters are updated using a reinforcement learning algorithm until the large language model is trained or the policy converges. The optimal policy for adjusting the first threshold and the second threshold is output.

7. A multi-stage simulation training system for a large language model, used to implement the multi-stage simulation training method for a large language model as described in any one of claims 1 to 6, characterized in that, The system includes: The difficulty grading module is configured to extract multi-dimensional difficulty features from training samples, calculate the comprehensive difficulty score of each training sample based on the extracted difficulty features, and divide the training samples into multiple difficulty levels based on the comprehensive difficulty score to construct a sample difficulty labeling system. The multi-stage training module is configured to set multiple progressively advancing training stages according to the sample difficulty labeling system, configure a preset sample difficulty distribution that matches its training objective for each training stage, and sample according to the preset sample difficulty distribution to construct the training dataset for each training stage. The real-time monitoring module is configured to monitor the performance of the large language model on a validation set containing training samples of various difficulty levels in real time during the training of the large language model according to multiple training stages, and obtain accuracy monitoring data for each difficulty level. The adaptive adjustment module is configured to dynamically adjust the sampling weights of training samples at each difficulty level during subsequent training based on the accuracy monitoring data. The adjustment includes: reducing the sampling weight of a difficulty level whose accuracy meets a first condition and allocating the reduced weight to a higher difficulty level; increasing the sampling weight of a difficulty level whose accuracy meets a second condition; and normalizing the adjusted sampling weights of each difficulty level. The knowledge transfer module is configured to extract a preset proportion of low-difficulty training samples from the training dataset of the current training phase after the current training phase ends, add the low-difficulty training samples to the training dataset of the next training phase, and freeze and fine-tune the parameters of the large language model when starting the next training phase. The strategy optimization module is configured to use the model performance data and control records collected throughout the training process as feedback information to adjust at least one of the following: the weight coefficient used for difficulty feature extraction in the difficulty grading module, the threshold for dividing the training stage in the multi-stage training module, and the preset threshold used to determine the adjustment conditions in the adaptive control module.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multi-stage simulation training method for large language models as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-stage simulation training method for large language models as described in any one of claims 1 to 6.

Images