Large model training system and training method

Abstract

The present disclosure provides a large model training system and a training method, and relates to the technical field of artificial intelligence. The system comprises: a data construction module configured to maintain a training data set and extract an evaluation sample set from each of a plurality of data subsets in the training data set; a training execution module configured to perform a first training operation on a to-be-trained model based on the evaluation sample set corresponding to each data subset, and obtain performance change information corresponding to each data subset; and a decision module configured to determine a target data subset from the plurality of data subsets according to the performance change information, determine a target training mode corresponding to the target data subset, and generate a training control instruction. The present disclosure quantifies the learning state of the perception model through low-cost detection evaluation, and dynamically schedules the training mode based on the model state, thereby achieving adaptive control of the large model training process and improving the training efficiency and the utilization rate of computing resources.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a large model training system and training method. Background Technology

[0002] The evolutionary training of large models typically involves the synergy of multiple training methods, such as supervised fine-tuning and reinforcement learning. In related technologies, the training data and methods for large models usually rely on human experience for configuration. This experience-driven approach cannot accurately configure training strategies based on the actual learning state of the model, resulting in low training efficiency for large models and a waste of computational resources. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a large model training system and training method.

[0004] According to a first aspect of the present disclosure, a large model training system is provided, comprising: A data construction module is used to maintain a training data set and extract evaluation sample sets from multiple data subsets of the training data set. The training execution module is used to perform a first training operation on the model to be trained based on the evaluation sample set corresponding to each data subset, and obtain the performance change information corresponding to each data subset; wherein, the performance change information is used to characterize the predicted performance change trend of the model to be trained after training on the corresponding data subset. The decision module is used to determine the target data subset from multiple data subsets based on the performance change information, determine the target training mode corresponding to the target data subset, and generate training control instructions. The training control instruction is used to instruct the training execution module to perform a second training operation on the model to be trained based on the target training mode and the training samples in the target data subset; the training scale of the second training operation is greater than the training scale of the first training operation.

[0005] In some embodiments, the data construction module is specifically used for: A stratified sampling method is used to extract an evaluation sample set from each of the data subsets; wherein the sample size of the evaluation sample set is smaller than the sample size of the data subset.

[0006] In some embodiments, the training execution module is specifically used for: Perform the following steps for each evaluation sample set corresponding to each subset of data: Based on the current model parameters of the model to be trained and the evaluation sample set, the first training operation is performed on the model to be trained; wherein the first training operation includes a preset number of parameter update operations; Obtain the first performance index of the model to be trained on the evaluation sample set before the first training operation is executed, and the second performance index of the model to be trained on the evaluation sample set after the first training operation is executed. Based on the first performance index and the second performance index, determine the performance change information corresponding to the data subset.

[0007] In some embodiments, the performance change information includes performance evaluation parameters and learning rate parameters; The performance evaluation parameters are determined based on the second performance index and are used to characterize the current processing capability of the model to be trained on the training samples in the corresponding data subset. The learning rate parameter is determined based on the change between the first performance index and the second performance index, and is used to characterize the performance improvement speed of the model to be trained after training based on the corresponding evaluation sample set. The first performance metric and the second performance metric include the prediction accuracy and / or reward function value of the model to be trained on the corresponding evaluation sample set.

[0008] In some embodiments, the decision module is specifically used for: For each subset of data, perform the following processing: Based on the performance evaluation parameters corresponding to the data subset, a capability gap index for the data subset is determined; wherein, the capability gap index is negatively correlated with the performance evaluation parameters and is used to characterize the degree of capability deficiency of the model to be trained on the corresponding data subset; Based on the learning rate parameter corresponding to the data subset, a training benefit index for the data subset is determined; wherein, the training benefit index is positively correlated with the learning rate parameter and is used to characterize the performance improvement benefit that can be brought about by training the model to be trained based on the corresponding data subset; Based on the capability gap index and the training benefit index, determine the priority score of the data subset, and determine the priority score as the training priority of the data subset; The subset of data whose training priority meets the preset selection criteria is determined as the target data subset; The preset selection conditions include at least one of the following: the training priority is higher than a preset priority threshold; the training priority is a preset number of the highest-ranking subsets of the multiple data sets.

[0009] In some embodiments, the decision module is specifically used for: Based on the performance evaluation parameters and learning rate parameters corresponding to the target data subset, the learning state type of the target data subset is determined; the learning state type includes initial learning state, intermediate learning state, autonomous learning state, and stable learning state. Based on the learning state type, the target training mode and training strategy parameters are determined; wherein, the target training mode includes supervised training mode and reinforcement training mode; the training strategy parameters are used to indicate the computational resource allocation ratio and training frequency of the target training mode.

[0010] In some embodiments, the decision module is specifically used for: If the performance evaluation parameter is lower than the first capability threshold, the learning state type is determined as the initial learning state; If the performance evaluation parameter is not lower than the first capability threshold and is lower than the second capability threshold, and the learning rate parameter is lower than the first rate threshold, the learning state type is determined as the intermediate learning state. If the performance evaluation parameter is not lower than the first capability threshold and the learning rate parameter is not lower than the first rate threshold, the learning state type is determined as the autonomous learning state. When the performance evaluation parameter is not lower than the second capability threshold and the learning rate parameter is lower than the first rate threshold, the learning state type is determined as the stable learning state; wherein, the second capability threshold is higher than the first capability threshold.

[0011] In some embodiments, the decision module is specifically used for: When the learning state type is the initial learning state or the intermediate learning state, the target training mode is determined as the supervised training mode, and the computing resource allocation ratio corresponding to the supervised training mode is increased. When the learning state type is the autonomous learning state, the target training mode is determined as the reinforcement training mode, and the computing resource allocation ratio corresponding to the reinforcement training mode is increased. When the learning state type is the stable learning state, the training frequency corresponding to the target data subset is reduced.

[0012] In some embodiments, the training execution module is specifically used for: When the target training mode is the supervised training mode, the parameters of the model to be trained are updated based on the training samples in the target data subset, so that the output of the model to be trained converges to the target result corresponding to the training samples; When the target training mode is the reinforcement training mode, the output result of the model to be trained is generated based on the online sampling method, the output result is evaluated according to the preset reward function, and the policy gradient update is performed according to the evaluation result.

[0013] In some embodiments, the large model training system further includes a candidate training data generation module; The training execution module is further configured to feed back the training results corresponding to the second training operation to the data construction module; wherein, the training results include the performance evaluation results of each training sample; The data construction module is further configured to configure the sample state information of the training samples in the training dataset according to the training results; wherein the sample state information is used to characterize the training evaluation state of the training samples. The candidate training data generation module is used to determine the corresponding training sample as the target training sample when the sample state information meets the data generation conditions, and to call an expert model with higher capabilities than the model to be trained to generate candidate training data based on the target training sample, perform a verification operation on the candidate training data, and write the candidate training data that meets the preset verification conditions into the training data set.

[0014] In some embodiments, the data construction module is specifically used for: If the performance evaluation result of the training sample meets the preset low-performance sample condition, the sample state information of the training sample is updated to the low-performance state identifier. The candidate training data generation module is specifically used for: For each training sample with the low-performance state identifier, generate multiple candidate outputs and process annotation information corresponding to the candidate outputs; Perform a verification operation on multiple candidate outputs, and write the candidate outputs that meet the preset verification conditions and the corresponding process annotation information into the training data set; If none of the multiple candidate outputs meet the preset verification conditions, the corresponding training sample is determined as an unlearnable sample, and the sample status information of the unlearnable sample is fed back to the decision module.

[0015] In some embodiments, the data construction module is specifically used for: If the performance evaluation result of the training sample meets the preset training completion conditions, the sample status information of the training sample is updated to the completed status identifier, and the training sample is removed from the training data set, or the sampling weight of the training sample is reduced when performing the second training operation.

[0016] According to a second aspect of the present disclosure, a large model training method is provided, comprising: Obtain a training dataset; the training dataset includes multiple data subsets. Evaluation sample sets are extracted from the multiple data subsets respectively; Based on the evaluation sample set corresponding to each data subset, the first training operation is performed on the model to be trained to obtain the performance change information corresponding to each data subset; wherein, the performance change information is used to characterize the predicted performance change trend of the model to be trained after training on the corresponding data subset; Based on the performance change information, a target data subset is determined from the plurality of data subsets, and a target training mode corresponding to the target data subset is determined, and training control instructions are generated. According to the training control instructions, a second training operation is performed on the model to be trained based on the target training mode and the training samples in the target data subset; wherein the training scale of the second training operation is greater than the training scale of the first training operation.

[0017] The large-scale model training system provided in this disclosure maintains the training dataset and extracts an evaluation sample set through a data construction module. A training execution module performs a first training operation on the model to be trained based on the evaluation sample set to obtain performance change information. A decision module then determines the target data subset and its target training mode based on the performance change information and generates training control instructions. This allows for a second training operation with a larger training scale to perform formal training on the model to be trained. This disclosure achieves quantitative perception of the model's learning state across different capability dimensions through low-cost, fine-grained probing evaluation, and dynamically schedules the model's training method accordingly. This enables the training process to automatically adjust the training strategy based on model performance changes, reducing reliance on human experience and improving training efficiency, stability, and utilization of computing resources. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structure of a large model training system according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a large model training method according to an embodiment of this disclosure. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0020] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0021] Before describing the specific embodiments of this disclosure in detail, the core concepts, basic principles and overall system concept involved in this disclosure will be explained first, so that those skilled in the art can accurately understand the technical concept of this disclosure.

[0022] In this embodiment of the disclosure, the model to be trained refers to a large model that is in the capability evolution training stage and needs to continuously improve its task processing capabilities through continuous training. The model to be trained usually has a massive number of trainable parameters, and its training process involves the consumption of a large amount of computing resources (such as the operation time, memory usage, and communication bandwidth of computing units such as graphics processing units and tensor processors).

[0023] For example, the model to be trained can be a Vision-Language Model (VLM), which is a multimodal model capable of simultaneously understanding visual signals and natural language signals, and performing tasks such as visual question answering, image-text reasoning, and visual description accordingly. Alternatively, the model to be trained can be a Vision-Language-Action Model (VLA), which, based on visual and linguistic input, further outputs action commands to drive physical entities (such as robotic arms, mobile chassis, humanoid robots, etc.) to perform operational tasks. When the model to be trained is trained to possess visual understanding, linguistic reasoning, and action generation capabilities, it is essentially a large multimodal model oriented towards embodied intelligence scenarios. Those skilled in the art should understand that this disclosure does not limit the specific modal type or downstream tasks of the model to be trained. Any large model that requires coordinated evolution through supervised training and reinforcement training can be efficiently trained using the training system and training method provided in this disclosure.

[0024] To perceive the model's learning state without performing a full training evaluation, this disclosure introduces a low-cost probing mechanism independent of the main training loop, figuratively called the sentinel probing mechanism. The basic idea is to extract a small evaluation sample set from each data subset, allowing the model to undergo short-step simulated training on this sample set, and comparing the changes in model performance metrics before and after the simulated training. This allows for a small-scale estimation of the model's mastery and learning speed in the corresponding capability dimension of that data subset. Since the size of the evaluation sample set is much smaller than the size of the complete data subset, and the number of simulated training steps is controlled, the computational cost of the sentinel probing mechanism is significantly lower than that of a full evaluation. Therefore, it can be executed frequently at the beginning of each training iteration cycle, providing timely and fine-grained quantitative evidence for dynamic decision-making regarding training strategies.

[0025] From a functional perspective, this disclosure decouples the large-scale model training process into four collaborative functional roles: data management, training strategy planning, training data generation, and training execution. These correspond to the data construction module, decision-making module, candidate training data generation module, and training execution module, respectively, as described later. Specifically, the data management role is responsible for organizing, sampling, and maintaining the state of training data; the training strategy planning role is responsible for making decisions and scheduling based on the model's learning state; the training data generation role is responsible for generating targeted data and verifying the quality of difficult samples; and the training execution role is responsible for actually updating parameters and switching between different training modes. These functional roles work collaboratively through information interaction and feedback mechanisms to form a feedback-based iterative self-adjusting training system, replacing the traditional training process's reliance on manual experience for data allocation adjustments and training mode switching.

[0026] The large model training system provided in this disclosure will be described in detail below with reference to specific embodiments.

[0027] First, please refer to... Figure 1 , Figure 1 This is a schematic diagram of the structure of a large model training system according to an embodiment of this disclosure. Figure 1 As shown, the large model training system includes: a data construction module 100, a training execution module 200, and a decision module 300.

[0028] The data construction module 100 is used to maintain the training data set and extract evaluation sample sets from multiple data subsets in the training data set.

[0029] For example, the training dataset is the total amount of data used to train the capabilities of the model to be trained, encompassing training samples covering multiple task types and capability dimensions. Before or during training, the data construction module 100 classifies and manages the training data, dividing the original training data into multiple independent data subsets according to task type or capability dimension. Each data subset corresponds to a different capability range that the model to be trained needs to master. For example, when the model to be trained is a multimodal large model for embodied intelligence scenarios, the multiple data subsets can correspond to different capability dimensions such as visual understanding, spatial reasoning, tool invocation, action planning, and language command following. Similarly, the multiple data subsets can also correspond to different task types such as mathematical reasoning tasks, code generation tasks, image-text question answering tasks, and robot operation tasks. When maintaining the training dataset, the data construction module 100 maintains corresponding metadata information such as sample index, sample size, sample source, and sample status for each data subset, and continuously updates this information during subsequent training. After dividing the data into subsets, the data construction module 100 extracts an evaluation sample set from each subset. The evaluation sample set is a collection of a small number of samples selected from the corresponding subset to quickly evaluate the model's capability status. By configuring an evaluation sample set for each data subset, the system can conduct independent capability awareness for different capability dimensions, avoiding the problem of only obtaining overall performance and losing fine-grained information when evaluating all data together.

[0030] The training execution module 200 is used to perform the first training operation on the model to be trained based on the evaluation sample set corresponding to each data subset, and obtain the performance change information corresponding to each data subset. The performance change information is used to characterize the predicted performance change trend of the model to be trained after training on the corresponding data subset.

[0031] For example, the first training operation is a controlled, short-step simulated training operation. For each data subset, the training execution module 200 performs the first training operation on the model to be trained using an evaluation sample set extracted from that subset. It observes the learning performance of the model on this small batch of samples and obtains performance change information corresponding to that data subset. This performance change information reflects the expected performance trend of the model if it were to undergo formal training on the corresponding data subset—for example, rapid improvement, slow improvement, near-consistency, or difficulty in improvement. Since the first training operation is only performed within the small sample set of evaluation samples, its computational power consumption is far less than that of training the entire data subset. Therefore, the training execution module 200 can perform this probing evaluation on all data subsets in parallel or sequentially within each training iteration cycle, thereby obtaining a learning state profile covering all capability dimensions at a lower cost. This probing-then-decision approach allows the system to grasp the model's learning dynamics in a timely manner without relying on periodic full-scale evaluations.

[0032] The decision module 300 is used to determine the target data subset from multiple data subsets based on various performance change information, determine the target training mode corresponding to the target data subset, and generate training control instructions.

[0033] For example, the decision module 300 is the decision control center of the entire system. After obtaining the performance change information corresponding to each data subset, the decision module 300 comprehensively analyzes the learning status of each data subset and selects the data subset that is most worthy of training resources as the target data subset. "Worth investing training resources" means that the capability dimension corresponding to the data subset is in a state where the model urgently needs improvement and can gain actual benefits through training. After determining the target data subset, the decision module 300 further determines which training mode should be used to train it based on the learning status of the target data subset, thus determining the target training mode. Based on this, the decision module 300 further generates training control instructions, which instruct the training execution module 200 to perform a second training operation on the model to be trained based on the target training mode and the training samples in the target data subset. The training scale of the second training operation is larger than that of the first training operation. Through the centralized decision-making of the decision module 300, the system automates the switching between training data and training modes, avoiding the problems of frequent intervention and repeated trial and error under the manual experience-driven approach.

[0034] Understandably, the second training operation is the formal training operation performed by the system on the target data subset. Unlike the first training operation, which serves as a probing method, the second training operation targets the training samples within the target data subset (rather than just the evaluation sample set). Its training scale is larger, which can be manifested in a greater number of samples participating in training, more parameter update steps, more training epochs, or more computational resources invested. The difference in training scale between the first and second training operations reflects the division of labor between probing and formal training: the first training operation acquires learning state information at the lowest possible cost, while the second training operation invests real computing power to improve model capabilities under the guidance of decision results. After receiving the training control command, the training execution module 200 executes the second training operation on the model to be trained according to the target training mode and target data subset indicated by the command, thereby completing one round of training iteration. As the training iteration progresses, the system periodically re-performs probing evaluation and decision-making, enabling the training process to continuously iterate and evolve based on the perceived results.

[0035] The large-scale model training system provided in this embodiment constructs a complete functional chain of perception, decision-making, and execution through the collaboration of data construction, training execution, and decision-making modules. This enables adaptive scheduling and automated control of the large-scale model training process, allowing for continuous iterative evolution based on perception results. Compared to technologies that rely on human experience for data allocation adjustments and training phase switching, this embodiment uses low-cost, probing assessment to quantify the model's learning status and dynamically determine the allocation of training resources and the selection of training modes. This reduces reliance on human intervention and improves training efficiency and stability.

[0036] In some embodiments, the data construction module 100 is specifically used to: extract an evaluation sample set from each data subset using a stratified sampling method. The sample size of the evaluation sample set is smaller than the sample size of the data subset.

[0037] For example, stratified sampling is a sampling method that first divides the population into several non-overlapping strata, and then independently extracts samples from each stratum. In this embodiment, when the data construction module 100 extracts an evaluation sample set from a certain data subset, it can first divide the samples within the data subset into several strata according to a preset stratification criterion, and then extract samples from each stratum proportionally or in fixed quantities to obtain the evaluation sample set corresponding to the data subset. The stratification criterion may include sample difficulty level, sample source, task sub-category, label category, etc. For example, for the data subset corresponding to mathematical reasoning ability, the samples can be divided into three strata—easy, medium, and difficult—according to the difficulty of the questions, and then samples can be extracted from each stratum according to its proportion in the data subset, so that the final evaluation sample set is consistent with the complete data subset in terms of difficulty distribution. Through stratified sampling, the evaluation sample set can maintain a distribution characteristic similar to the complete data subset, thereby ensuring that the performance change information obtained based on the evaluation sample set is statistically representative and avoiding misjudgment of the model learning state due to sampling bias. Meanwhile, the size of the evaluation sample set is smaller than that of the data subset; that is, the evaluation sample set only contains a portion of the samples in the data subset. This small-scale sampling keeps the computational cost of the first training operation at a low level, enabling the system to perform capability awareness at a high frequency. The specific size of the evaluation sample set can be configured according to the computing power budget and evaluation accuracy requirements, for example, it can be set to a percentage between one percent and ten percent of the data subset sample size.

[0038] This embodiment constructs an evaluation sample set through stratified sampling, which significantly reduces the computational overhead of detection while ensuring the representativeness of the evaluation sample set to the data subset distribution. This enables performance change information to accurately characterize the learning state of the model in the corresponding capability dimension, providing a reliable quantitative basis for subsequent training decisions.

[0039] In some embodiments, the training execution module 200 is specifically configured to: perform the following steps for each evaluation sample set corresponding to each data subset: perform a first training operation on the model to be trained based on the current model parameters of the model to be trained and the evaluation sample set; wherein the first training operation includes a preset number of parameter update operations; obtain a first performance index of the model to be trained on the evaluation sample set before the execution of the first training operation, and a second performance index of the model to be trained on the evaluation sample set after the execution of the first training operation; and determine the performance change information corresponding to the data subset based on the first performance index and the second performance index.

[0040] Specifically, the training execution module 200 uses the evaluation sample set as training data and the current model parameters of the model to be trained as initial parameters to perform the first training operation on the model to be trained. The use of the current model parameters is to ensure that the probing evaluation of each data subset is performed from the same parameter starting point, making the performance changes of different data subsets comparable. The first training operation includes a preset number of parameter update operations; that is, the training execution module 200 uses the evaluation sample set to perform a finite number of gradient updates on the model to be trained, such as three, five, or ten parameter updates. The preset number is usually set to a small value to ensure the probing nature of the first training operation and avoid substantially changing the main parameters of the model to be trained. In actual deployment, the probing evaluation of each data subset can be performed independently based on a copy of the current model parameters. The parameter updates generated during the probing process are not written back to the main parameters of the model to be trained, thus ensuring that the probing operation does not pollute the main loop of the formal training.

[0041] Before performing the first training operation, the training execution module 200 first acquires the first performance metric of the model to be trained on the evaluation sample set. This first performance metric reflects the model's performance level before training on the evaluation sample set. After performing the first training operation, the training execution module 200 acquires the second performance metric of the model to be trained on the evaluation sample set. This second performance metric reflects the model's performance level after a short training run. By comparing the first and second performance metrics, the training execution module 200 determines the performance change information corresponding to this data subset. The performance change information characterizes the model's learning dynamics on this data subset: if the second performance metric shows a significant improvement relative to the first performance metric, it indicates that the model learns quickly and has high training gains in this capability dimension; if the improvement is limited, it indicates that the model learns slowly in this capability dimension; if the second performance metric itself is already at a high level, it indicates that the model has a strong grasp of this capability dimension.

[0042] This embodiment obtains performance change information by comparing performance indicators before and after training, and describes the mastery and learning speed of the model in each capability dimension in a quantifiable and comparable way, so that the system can obtain fine-grained learning state perception results without performing full evaluation.

[0043] In some embodiments, the performance change information includes performance evaluation parameters and learning rate parameters; the performance evaluation parameters are determined based on a second performance metric and are used to characterize the current processing capability of the model to be trained on training samples in the corresponding data subset; the learning rate parameters are determined based on the change between the first performance metric and the second performance metric and are used to characterize the performance improvement rate of the model to be trained after training on the corresponding evaluation sample set; wherein the first performance metric and the second performance metric include the prediction accuracy and / or reward function value of the model to be trained on the corresponding evaluation sample set.

[0044] For example, performance evaluation parameters are measures of the model's current capability level. Since the second performance metric reflects the model's performance on the evaluation sample set after a short training run, a higher value generally indicates a stronger current processing capability of the model on the training samples in that subset of data. Therefore, performance evaluation parameters can be determined based on the second performance metric. For example, the second performance metric can be directly used as the performance evaluation parameter. Alternatively, the second performance metric can be normalized or smoothed to obtain the performance evaluation parameter. Thus, performance evaluation parameters essentially characterize the model's mastery of the corresponding subset of data.

[0045] The learning rate parameter is a measure of how fast the model learns. It is determined based on the change between the first and second performance metrics, which can be expressed as the difference between the two or their relative change. Taking a subset of data as an example, let its first performance metric be... The second performance indicator is Then the learning rate parameter It can be represented as:

[0046] Alternatively, to eliminate the influence of performance metric dimensions between different data subsets, a relative variation can also be used:

[0047] in To prevent extremely small positive numbers with a denominator of zero, a larger learning rate parameter indicates that the model's performance improves faster after a short training step on that data subset, meaning that the data subset is in a high-learning-efficiency range for the current model; a smaller learning rate parameter, or even close to zero, indicates that the model improves slowly on that data subset.

[0048] Regarding the specific forms of the first and second performance metrics, they can include the prediction accuracy of the model under training on the corresponding evaluation sample set and / or the value of the reward function. Prediction accuracy is suitable for tasks with clear right / wrong judgments, such as mathematical reasoning, multiple-choice question answering, and instruction following; it can be obtained by statistically analyzing the proportion of samples whose output matches the standard answer. The reward function value is suitable for tasks where training is driven by reward signals, such as exploration tasks or embodied manipulation tasks in reinforcement training mode; it can be obtained by evaluating the model output using a preset reward function. In practical applications, the appropriate performance metric form can be selected based on the task type corresponding to the data subset, or multiple forms can be combined.

[0049] This embodiment refines the performance change information into two dimensions: performance evaluation parameters and learning rate parameters. These parameters characterize the model's learning state from the perspectives of current capability level and performance improvement speed, providing more refined quantitative input for determining subsequent training priorities and judging the learning state type.

[0050] In some embodiments, the decision module 300 is specifically configured to: for each data subset, perform the following processing: determine a capability gap index for the data subset based on the performance evaluation parameters corresponding to the data subset; wherein the capability gap index is negatively correlated with the performance evaluation parameters and is used to characterize the degree of capability deficiency of the model to be trained on the corresponding data subset; determine a training benefit index for the data subset based on the learning rate parameters corresponding to the data subset; wherein the training benefit index is positively correlated with the learning rate parameters and is used to characterize the performance improvement benefit that can be brought by training the model to be trained based on the corresponding data subset; determine a priority score for the data subset based on the capability gap index and the training benefit index, and determine the priority score as the training priority of the data subset; determine the data subset whose training priority meets the preset selection conditions as the target data subset; wherein the preset selection conditions include at least one of the following: the training priority is higher than a preset priority threshold; the training priority is a preset number of the highest-ranked data subsets.

[0051] Specifically, the capability gap index is used to characterize the degree of capability deficiency of the model to be trained on the corresponding subset of data. It is negatively correlated with the performance evaluation parameter; that is, the lower the performance evaluation parameter, the larger the capability gap index, indicating that the model is more deficient in that capability dimension. Let the... The performance evaluation parameters for each data subset are: The preset capability reference value is The capacity gap indicator It can be represented as:

[0052] Among them, the maximum value of the operation guarantee capability gap index is a non-negative value. When Below This indicates that the model's current capabilities have not yet reached the reference level, and there is a capability gap. The lower the value, the larger the gap; when Reached or exceeded When the capability gap indicator is zero, it means that the model has no significant deficiency in this capability dimension.

[0053] The training gain metric is used to characterize the performance improvement gained by training the model to be trained based on a corresponding subset of data. It is positively correlated with the learning rate parameter; that is, the larger the learning rate parameter, the greater the training gain metric. Let the... The learning rate parameter for each data subset is Then the training benefit metric It can be directly taken as the learning rate parameter itself, that is Alternatively, to ensure that the training gain metric is non-negative and to constrain its range, it can also be expressed as... This can be obtained through normalization. The training benefit metric reflects the expected return on investment in training resources on this subset of data: the faster the learning rate, the greater the performance improvement that can be achieved with the same training investment, and the higher the training benefit.

[0054] After obtaining the capability gap index and training return index, the decision module 300 determines the priority score of the data subset based on both. The priority score comprehensively considers both the model's deficiency and the training return, and a typical implementation is a weighted sum of the two. Let the... The priority score for each data subset is ,but:

[0055] in, and These are non-negative weighting coefficients, used to adjust the relative importance of capability gap and training gain in the priority score, respectively. Their sum can be normalized to 1. The decision module 300 determines the priority score as the training priority for that data subset. In this way, the training priority tends to select both data subsets where the model is currently weak (large capability gap) and data subsets where the model currently has high learning efficiency (high training gain), thus prioritizing training resources for those capability dimensions that the model urgently needs to improve and can actually learn, aligning with the gradual learning philosophy of a course.

[0056] After determining the training priority of each data subset, the decision module 300 identifies the data subsets whose training priorities meet preset selection criteria as the target data subsets. The preset selection criteria include at least one of the following: First, the training priority is higher than a preset priority threshold, meaning only data subsets with training priorities exceeding a certain threshold are included in the target data subset. This method ensures that all selected data subsets have sufficient training value. Second, the training priority is a preset number of the highest-ranking data subsets among multiple data subsets, meaning the top few data subsets are selected as the target data subsets after sorting them from high to low training priority. This method controls the number of data subsets covered in a single training iteration, making the allocation of training resources more focused. In practical applications, the two methods can be used individually or in combination. For example, first select the top-ranking data subsets, and then further filter out data subsets with training priorities higher than the threshold.

[0057] This embodiment determines training priorities through a two-dimensional evaluation of capability gap indicators and training benefit indicators. This ensures that the selection of target data subsets takes into account both the model's weaknesses and learning efficiency, avoiding the waste of training resources on data subsets that the model has already mastered or is temporarily unable to learn. This improves the rationality of training resource allocation and training efficiency.

[0058] In some embodiments, the decision module 300 is specifically used to: determine the learning state type of the target data subset based on the performance evaluation parameters and learning rate parameters corresponding to the target data subset; the learning state type includes initial learning state, intermediate learning state, autonomous learning state, and stable learning state; determine the target training mode and training strategy parameters based on the learning state type; wherein, the target training mode includes supervised training mode and reinforcement training mode; the training strategy parameters are used to indicate the computational resource allocation ratio and training frequency of the target training mode.

[0059] Specifically, the learning state type describes the learning stage of the model on the target data subset. This embodiment divides it into four types. The initial learning state indicates that the model's basic capabilities in the corresponding capability dimension of this data subset have not yet been formed, and it is not yet ready for direct reinforcement training; supervised training is needed to establish a foundation first. The intermediate learning state indicates that the model has acquired some capabilities, but learning progress is slow, requiring external supervision to overcome bottlenecks. The autonomous learning state indicates that the model learns quickly and has the conditions for autonomous exploration, making it suitable for parameter updates using reinforcement training. The stable learning state indicates that the model has basically mastered the corresponding capabilities, requiring only a small amount of training to maintain stable performance without excessive resource investment.

[0060] After determining the learning state type, the decision module 300 determines the target training mode and training strategy parameters based on the learning state type.

[0061] The target training modes include supervised training and reinforcement training. Supervised training corresponds to the Supervised Fine-Tuning (SFT) paradigm, its core being the use of training samples labeled with the target result to drive the model's output towards the target result, enabling the model to mimic expert demonstration behavior. Reinforcement training corresponds to the Reinforcement Learning (RL) paradigm, its core being allowing the model to autonomously generate output results through online sampling, with a pre-defined reward function evaluating the output results, and then performing policy gradient updates based on the evaluation results, allowing the model to autonomously discover better strategies through interaction with environmental reward signals. Supervised and reinforcement training modes each have their applicable conditions: when the model's basic capabilities in a certain dimension are not yet established, directly using reinforcement training often leads to training instability due to sparse reward signals; when the model already has a strong foundation in a certain capability dimension, continuing to repeat supervised training results in diminishing returns and wastes computational resources. Therefore, a decision module 300 is needed to determine the currently applicable training mode.

[0062] Training strategy parameters indicate the allocation ratio of computing resources and the training frequency for the target training mode. The allocation ratio describes the proportion of available computing resources allocated to this training mode, such as the percentage of GPU time allocated to supervised training. The training frequency describes how frequently training is conducted on this subset of data, such as the frequency of training iterations. Through these two parameters—the allocation ratio and the training frequency—the decision module 300 can precisely control the intensity and pace of training resource allocation, achieving a quantitative expression of the training strategy.

[0063] This embodiment classifies and determines the learning state, mapping the abstract model learning situation into a clear training strategy, thereby enabling automatic control of the training mode switching and resource allocation process, and avoiding the efficiency loss caused by fixed training phases and fixed training ratios.

[0064] In some embodiments, the decision module 300 is specifically configured to: determine the learning state type as an initial learning state when the performance evaluation parameter is lower than a first capability threshold; determine the learning state type as an intermediate learning state when the performance evaluation parameter is not lower than the first capability threshold and is lower than a second capability threshold, and the learning rate parameter is lower than a first rate threshold; determine the learning state type as an autonomous learning state when the performance evaluation parameter is not lower than the first capability threshold and the learning rate parameter is not lower than the first rate threshold; and determine the learning state type as a stable learning state when the performance evaluation parameter is not lower than the second capability threshold and the learning rate parameter is lower than the first rate threshold; wherein the second capability threshold is higher than the first capability threshold.

[0065] For example, this embodiment provides specific rules for determining the learning state type based on performance evaluation parameters and learning rate parameters. Let the first capability threshold be... The second capability threshold is The first rate threshold is ,and The decision logic is as follows: when the performance evaluation parameters... When the model's current ability level is low and its foundation has not yet been established, it is judged as the initial learning state; when And the learning rate parameter When the model has acquired certain basic capabilities, but its learning progress is slow, it is judged to be in an intermediate learning state; when and When the model's capabilities meet the standards and its learning speed is fast, it possesses the conditions for autonomous exploration and is judged to be in an autonomous learning state; when and When the model's capabilities have reached a relatively high level and the rate of improvement has slowed down, it is considered to be in a stable learning state.

[0066] The above-mentioned judgment rules can be summarized as follows: Performance evaluation parameters below the first capability threshold correspond to the initial learning state; performance evaluation parameters between the first and second capability thresholds and the learning rate parameter below the first rate threshold correspond to the intermediate learning state; performance evaluation parameters not lower than the first capability threshold and the learning rate parameter not lower than the first rate threshold correspond to the autonomous learning state; performance evaluation parameters not lower than the second capability threshold and the learning rate parameter lower than the first rate threshold correspond to the stable learning state. By setting the second capability threshold higher than the first capability threshold, the system distinguishes between the initial establishment of capabilities and the maturation of capabilities, making the characterization of learning states more refined. The specific values ​​of each threshold can be calibrated according to the type of the model to be trained, the characteristics of the data subset, and the value range of the performance indicators; this disclosure does not impose any limitations on this.

[0067] This embodiment achieves automatic classification of learning state types through explicit threshold determination rules, making the decision-making process reproducible and interpretable, and providing a stable basis for the automated scheduling of training strategies.

[0068] In some embodiments, the decision module 300 is specifically used to: determine the target training mode as a supervised training mode and increase the computing resource allocation ratio corresponding to the supervised training mode when the learning state type is an initial learning state or an intermediate learning state; determine the target training mode as a reinforcement training mode and increase the computing resource allocation ratio corresponding to the reinforcement training mode when the learning state type is an autonomous learning state; and reduce the training frequency corresponding to the target data subset when the learning state type is a stable learning state.

[0069] For example, this embodiment illustrates the mapping relationship between learning state types and training strategies. When the target data subset is in the initial or intermediate learning state, it indicates that the model's basic capabilities in that dimension are insufficient or its learning progress is slow. Directly conducting reinforcement training at this time can easily lead to training instability due to sparse reward signals. Therefore, the decision module 300 determines the target training mode as supervised training mode, providing the model with clear learning objectives through expert demonstration data and correspondingly increasing the proportion of computational resources allocated to the supervised training mode to accelerate the establishment of basic capabilities. When the target data subset is in the autonomous learning state, it indicates that the model already has a good foundation and learns quickly, making it suitable for further exploring its capabilities through autonomous exploration. Therefore, the decision module 300 determines the target training mode as reinforcement training mode and correspondingly increases the proportion of computational resources allocated to the reinforcement training mode, enabling the model to continuously optimize its strategy through interaction with reward signals. When the target data subset is in a stable learning state, it means that the model has basically mastered the corresponding ability, and the benefits of continuing high-intensity training are limited. Therefore, the decision module 300 reduces the training frequency of the target data subset, retaining only a small amount of training to maintain performance stability, thereby transferring the saved computing resources to other data subsets that need to be improved more.

[0070] This embodiment achieves adaptive matching between training mode and resource allocation by associating learning state type with training mode, resource allocation ratio and training frequency, so that computing resources always flow to the direction with the highest training benefit, thereby improving overall training efficiency and resource utilization.

[0071] In some embodiments, the training execution module 200 is specifically used to: when the target training mode is supervised training mode, update the parameters of the model to be trained based on the training samples in the target data subset so that the output of the model to be trained converges to the target result corresponding to the training samples; when the target training mode is reinforcement training mode, generate the output of the model to be trained based on online sampling, evaluate the output according to a preset reward function, and perform policy gradient update according to the evaluation result.

[0072] For example, when the target training mode is supervised training, the training execution module 200 uses training samples from the target data subset to update the parameters of the model to be trained. Each training sample contains an input sample and a corresponding target result. The training execution module 200 feeds the input sample into the model to be trained to obtain the model output, calculates the loss between the model output and the target result, and updates the model parameters through backpropagation, so that the model output gradually converges to the target result. The form of the loss function can be determined according to the task type. For example, cross-entropy loss can be used for language generation tasks, and mean squared error loss can be used for regression tasks. In supervised training mode, training samples can include original labeled data or high-quality training data generated by an expert model and validated, thereby enabling the model to learn from expert demonstration behavior.

[0073] When the target training mode is reinforcement training, the training execution module 200 generates the output of the model to be trained based on online sampling. This means the model autonomously samples and generates candidate outputs for a given input according to the current policy. The output is then evaluated using a preset reward function to obtain a reward value, and the policy gradient is updated based on the evaluation result. The core idea of ​​policy gradient update is to increase the generation probability of outputs that yield higher rewards and decrease the generation probability of outputs that yield lower rewards. Through repeated sampling, evaluation, and updating, the model policy gradually evolves towards higher rewards. The preset reward function can be designed according to the task objective. For example, for embodied tasks, the reward function can comprehensively evaluate factors such as task completion, action smoothness, and safety.

[0074] This embodiment distinguishes the execution logic of supervised training mode and reinforcement training mode, enabling the training execution module 200 to accurately implement the decision results of the decision module 300 and flexibly switch between different training modes, thus ensuring the effective execution of training control commands.

[0075] Please continue to refer to Figure 1 In some embodiments, the large model training system further includes a candidate training data generation module 400; the training execution module 200 is also used to feed back the training results corresponding to the second training operation to the data construction module 100; wherein, the training results include the performance evaluation results of each training sample; the data construction module 100 is also used to configure the sample state information of the training samples in the training data set according to the training results; wherein, the sample state information is used to characterize the training evaluation state of the training sample; the candidate training data generation module 400 is used to determine the corresponding training sample as the target training sample when the sample state information meets the data generation conditions, and call an expert model with higher capabilities than the model to be trained to generate candidate training data based on the target training sample, perform a verification operation on the candidate training data, and write the candidate training data that meets the preset verification conditions into the training data set.

[0076] Specifically, after completing a second round of training operations, the training execution module 200 feeds back the corresponding training results to the data construction module 100. The training results include the performance evaluation results of each training sample, i.e., the reward value or performance score obtained by the model for each training sample during training. Based on the training results, the data construction module 100 configures the sample state information of each training sample in the training dataset. The sample state information characterizes the training evaluation state of the training sample, reflecting its performance in the current training stage, whether it has been mastered by the model, and whether it needs further reinforcement. Through the dynamic maintenance of sample state information, the training dataset is no longer a static accumulation of data but possesses the ability to evolve independently as the training process progresses.

[0077] When the sample state information of certain training samples meets the data generation conditions, the candidate training data generation module 400 is triggered to execute the data generation process. The data generation conditions can be understood as the sample state information indicating that the training sample has consistently performed poorly and the model is struggling to learn; in this case, external knowledge needs to be introduced for targeted reinforcement. Specifically, the data generation conditions include: the sample state information of the training sample being a low-performance state identifier. The candidate training data generation module 400 iterates through the sample state information of each training sample in the training dataset. When it reads that the sample state information of a certain training sample is a low-performance state identifier, it determines that the training sample meets the data generation conditions and triggers the data generation process for that training sample. The low-performance state identifier is written by the data construction module 100 based on a numerical comparison between the performance evaluation result and a preset low-performance threshold; the specific determination method is detailed in subsequent embodiments.

[0078] After triggering the data generation process, the candidate training data generation module 400 first identifies the corresponding training samples as target training samples. Then, it invokes an expert model with superior capabilities to the model being trained to generate candidate training data based on the target training samples. The expert model is a more capable teacher model, such as a multimodal large model with a larger parameter scale and stronger reasoning ability. The candidate training data generation module 400 uses the expert model to generate training data containing explicit reasoning steps for the target training samples, supplementing them with high-quality demonstration answers and process information. After generating the candidate training data, the module 400 performs a verification operation to determine the correctness and learnability of the generated results. Only candidate training data that meets preset verification conditions is written into the training data set for subsequent supervised training. Through this data supplementation mechanism—generated by an expert model and then verified and filtered—the system achieves automatic generation and quality control of training data without relying on manual annotation, avoiding interference from erroneous data or unlearnable samples in model training.

[0079] This embodiment, through the coordinated efforts of training result feedback, sample state maintenance, and expert data generation, enables the system to provide targeted knowledge supplementation for the shortcomings exposed by the model, forming a virtuous cycle of discovering weaknesses, supplementing knowledge, and re-verifying, thereby improving the upper limit of the model's capabilities and the stability of the training process.

[0080] In some embodiments, the data construction module 100 is specifically used to: update the sample state information of the training sample to a low-performance state identifier when the performance evaluation result of the training sample meets the preset low-performance sample condition; the candidate training data generation module 400 is specifically used to: generate multiple candidate outputs and corresponding process annotation information for each training sample with a low-performance state identifier; perform a verification operation on the multiple candidate outputs, and write the candidate outputs that meet the preset verification conditions and the corresponding process annotation information into the training data set; when none of the multiple candidate outputs meet the preset verification conditions, determine the corresponding training sample as an unlearnable sample, and feed back the sample state information of the unlearnable sample to the decision module 300.

[0081] For example, after receiving the training results, the data construction module 100 determines, for each training sample, whether its performance evaluation result meets a preset low-performance sample condition. The low-performance sample condition is used to filter out samples where the model performance is consistently poor. For example, it could be that the performance evaluation result is lower than a preset low-performance threshold, or that the performance evaluation result is lower than the threshold for several consecutive training cycles. Specifically, the low-performance sample condition includes at least one of the following two: first, the performance evaluation result of the training sample is lower than the preset low-performance threshold; second, the performance evaluation result of the training sample is consistently lower than the threshold. Each training iteration cycle was below the preset low-performance threshold, among which It is a positive integer greater than 1. In the decision-making implementation, the data construction module 100 compares the performance evaluation result (e.g., reward value or performance score) of each training sample with a low performance threshold, and records the number of consecutive periods where the performance evaluation result is lower than the low performance threshold using a counter corresponding to that training sample; when the count reaches... When a training sample meets the low-performance sample condition, the count is reset to zero when the performance evaluation result of a certain period is not lower than the low-performance threshold. For training samples that meet the low-performance sample condition, the data construction module 100 updates their sample status information to a low-performance status identifier, thereby marking them as difficult samples that require special attention.

[0082] For each training sample with a low-performance state identifier, the candidate training data generation module 400 calls an expert model to generate multiple candidate outputs, along with corresponding process annotation information. Process annotation information refers to the reasoning process information corresponding to the candidate outputs, such as thought chains, step-by-step derivations, and intermediate decisions. Its purpose is to provide process-level learning signals for subsequent supervised training, enabling the model to learn not only the final answer but also the reasoning path leading to that answer. By generating multiple candidate outputs independently multiple times, the system can examine the stability of the expert model in solving the training sample. Subsequently, the candidate training data generation module 400 performs a verification operation on the multiple candidate outputs to determine whether each candidate output meets preset verification conditions. Specifically, the preset verification conditions include at least one of the following determination methods: The first is answer matching determination, applicable to tasks where the output result has a unique standard answer (e.g., math problems, multiple-choice questions). The candidate training data generation module 400 extracts the answer field from the candidate outputs and compares it with the standard answer corresponding to the target training sample. If the comparison result is consistent, the candidate output is determined to meet the preset verification conditions. The second is model evaluation verification. The candidate training data generation module 400 inputs candidate outputs and target training samples into a pre-trained validation model to obtain a correctness score for the validation model output. When the correctness score is not lower than a preset validation threshold, the candidate output is determined to meet the preset validation conditions. The third method is majority voting. The candidate training data generation module 400 generates data for the same target training sample. candidate outputs ( After determining the positive integers (greater than 1), cluster the candidate outputs according to the equivalence relationships between the answers contained in each candidate output; if the number of answers in a candidate output belongs to a category greater than 1... If the candidate output meets the preset verification conditions, then the candidate output is determined to satisfy the preset verification conditions. The above three determination methods can be used individually or in combination. For candidate outputs that meet the preset verification conditions, the candidate training data generation module 400 writes them together with the corresponding process annotation information into the training data set as data for subsequent supervised training.

[0083] For training samples where multiple candidate outputs fail to meet the preset verification conditions, it indicates that even a more capable expert model would struggle to stably solve the sample at this stage. The sample may exceed the model's current learnable limit or contain inherent noise. In this case, the candidate training data generation module 400 identifies the corresponding training sample as unlearnable and feeds back the sample status information of the unlearnable sample to the decision module 300. This allows the decision module 300 to be aware of the sample's status and avoids forcibly including it in training, which could lead to model training deviation. Through this solvability screening mechanism, the system effectively eliminates temporarily unlearnable, overly difficult samples or noisy data, ensuring the quality of the injected training data.

[0084] This embodiment achieves refined processing of difficult samples by identifying low-performance samples, generating multiple candidates, labeling processes, and verifying solvability. It not only supplements the model with high-quality process demonstration data, but also filters out unlearnable samples through the verification mechanism, thereby improving the reliability of training data and the stability of the training process.

[0085] In some embodiments, the data construction module 100 is specifically used to: update the sample state information of the training sample to a completed state identifier when the performance evaluation result of the training sample meets the preset training completion conditions, and remove the training sample from the training data set, or reduce the sampling weight of the training sample when performing the second training operation.

[0086] For example, after receiving the training results, the data construction module 100 determines, for each training sample, whether its performance evaluation result meets the preset training completion conditions. The training completion conditions are used to filter out samples that the model has fully mastered. For example, the performance evaluation result may consistently be higher than the preset completion threshold, or the performance evaluation result may remain at a high level for several consecutive training cycles. Specifically, the training completion conditions include at least one of the following two: first, the performance evaluation result of the training sample is not lower than the preset completion threshold; second, the performance evaluation result of the training sample continuously... Each training iteration cycle is no less than the preset completion threshold, among which It is a positive integer greater than 1; its determination method is the same as the counting determination method for low-performance sample conditions, and will not be repeated here. For training samples that meet the training completion conditions, the data construction module 100 updates its sample state information to the completed state identifier, indicating that the model has learned the ability corresponding to the sample.

[0087] For training samples marked as completed, the data construction module 100 can adopt two processing methods: First, remove the training sample from the training dataset so that it no longer participates in subsequent training, thereby reducing the computational and data storage overhead caused by repeated training; second, instead of removing it directly, reduce the sampling weight of the training sample when performing the second training operation, significantly reducing the probability of it being sampled in subsequent training, but still retaining it in the training dataset for later use. These two methods can be used individually or flexibly selected according to the storage status of the training dataset and task requirements. For example, removal is preferred when storage space is limited, while reducing sampling weight is preferred when data needs to be retained for review or to prevent forgetting. By timely removing or reducing the weight of already mastered samples, the system avoids repeatedly consuming computational power on samples that the model has already learned, allowing limited computational resources to be concentrated on unmastered capability dimensions.

[0088] This embodiment achieves self-simplification of the training dataset by identifying and dynamically processing completed samples, reducing redundant calculations and storage overhead, and further improving training efficiency and resource utilization.

[0089] To enable those skilled in the art to more clearly understand the overall technical concept of this disclosure, the following is combined with... Figure 1 The complete architecture of the large model training system in this disclosure embodiment will be described again. Please refer to... Figure 1 The large model training system is composed of four functional roles: data management, training strategy planning, training data generation, and training execution. These roles are respectively carried out by the data construction module 100, the decision-making module 300, the candidate training data generation module 400, and the training execution module 200.

[0090] At the beginning of each training iteration, the data construction module 100 extracts evaluation sample sets from each data subset. The training execution module 200 performs the first training operation on the model to be trained based on the evaluation sample sets, obtaining performance change information for each data subset, thereby completing the quantitative perception of the model's capability status. The decision module 300 receives the performance change information, determines the learning status and training priority of each data subset based on performance evaluation parameters and learning rate parameters, identifies the target data subset and target training mode, and generates training control instructions. According to the training control instructions, the training execution module 200 performs formal training on the model to be trained with a second training operation of a larger training scale, and switches between supervised training mode and reinforcement training mode. After completing one round of training, the training execution module 200 feeds back the training results to the data construction module 100, which updates the sample status information of each training sample: samples with consistently high performance are marked as completed and removed or downweighted, while samples with consistently low performance are marked as low-performance samples. For low-performance samples, the candidate training data generation module 400 calls the expert model to generate candidate training data in a targeted manner, and writes it into the training data set after verification, providing data support for the next round of supervised training. For samples that still cannot be learned after multiple attempts, their status information is fed back to the decision module 300. In this way, the system forms a complete processing link of capability perception, strategy decision, training execution, sample update, and data replenishment, and the output of each link serves as the input for the next round of perception and decision, enabling the training process to automatically adjust the training strategy according to changes in model performance and continuously iterate and evolve, achieving adaptive scheduling and automated control.

[0091] Without departing from the overall inventive concept of this disclosure, various alternative technical solutions or equivalent implementations are also possible. Regarding model capability perception, in addition to the sentinel detection mechanism, other lightweight capability assessment methods can be used, such as statistical analysis based on historical gradient changes, trend prediction based on reward distribution changes, or rapid evaluation based on a small-scale online validation set, as long as the method can reflect the model's learning state across different capability dimensions with low computational cost. Regarding training mode decision-making and scheduling, in addition to centralized decision-making by the decision module, distributed agent negotiation mechanisms, multi-strategy voting mechanisms, or meta-policy controllers based on reinforcement learning can also be used, as long as they can adaptively determine the training path based on the model's state. Regarding knowledge injection and data generation, in addition to calling a single expert model, multi-teacher model cross-validation, rule constraint generation, or a model self-generation plus self-criticism mechanism can be used to complete knowledge replenishment and quality control. Regarding the utilization of failure samples, in addition to converting low-reward samples into supervised training data, they can also be used to construct adversarial training sets, hard sample replay caches, or a priority-based experience replay mechanism can be introduced. From a system architecture perspective, this disclosure can adopt either a centralized system or a distributed deployment approach. For example, data management, strategy decision-making, expert generation, and training execution can be deployed as independent services, communicating through message queues or scheduling systems. All equivalent substitutions or combinations described above should be understood as reasonable variations of this disclosure.

[0092] Based on the same inventive concept, this disclosure also provides a method for training large models. Please refer to... Figure 2 , Figure 2 This is a flowchart illustrating a large model training method according to an embodiment of this disclosure. Figure 2 As shown, the method includes the following steps S201 to S205.

[0093] S201, Obtain the training data set.

[0094] The training dataset includes multiple subsets of data.

[0095] In some embodiments, the training dataset is the total amount of data used to train the model to evolve its capabilities. Before training begins, the system acquires the training dataset and divides it into multiple independent subsets based on task type or capability dimension. Each subset corresponds to a different capability range that the model to be trained needs to master. When the model to be trained is a large multimodal model oriented towards embodied intelligence, the multiple subsets can correspond to capability dimensions such as visual understanding, spatial reasoning, action planning, and language command following.

[0096] S202, extract evaluation sample sets from multiple data subsets respectively.

[0097] In some embodiments, for each data subset, an evaluation sample set is drawn from it using a sampling method. The evaluation sample set is a small collection of samples selected from the corresponding data subset to quickly assess the model's capability status; its sample size is smaller than the sample size of the data subset. In one example, a stratified sampling method can be used, first dividing the data subset into several strata according to sample difficulty, source, or sub-category, and then drawing samples proportionally from each stratum, so that the evaluation sample set maintains the representativeness of the data subset in terms of distribution.

[0098] S203, based on the evaluation sample set corresponding to each data subset, perform the first training operation on the model to be trained to obtain the performance change information corresponding to each data subset.

[0099] Among them, the performance change information is used to characterize the predicted performance change trend of the model to be trained after being trained on the corresponding data subset.

[0100] In some embodiments, for each data subset, starting with the current model parameters of the model to be trained and using the corresponding evaluation sample set as training data, a controlled short-step first training operation is performed. Performance changes for that data subset are obtained by comparing the model's performance metrics on the evaluation sample set before and after training. These performance changes reflect the model's mastery and learning speed on that data subset and are used to predict the performance trend of the model after training on the corresponding data subset.

[0101] S204: Based on the performance change information, determine the target data subset from multiple data subsets, determine the target training mode corresponding to the target data subset, and generate training control instructions.

[0102] In some embodiments, based on the performance change information of each data subset, the capability gap and training gains of each data subset are comprehensively evaluated to determine the training priority, and the data subset whose training priority meets the preset selection conditions is determined as the target data subset; then, the target training mode and training strategy parameters are determined according to the learning state type of the target data subset, and training control instructions are generated.

[0103] S205, according to the training control instructions, based on the training samples in the target training mode and the target data subset, perform the second training operation on the model to be trained.

[0104] The training scale of the second training operation is larger than that of the first training operation.

[0105] In some embodiments, a second training operation is performed on the model to be trained using training samples from the target data subset, according to the target training mode indicated by the training control instructions. In supervised training mode, the model output is made to converge to the target result corresponding to the training samples through parameter updates; in reinforcement training mode, the model policy is optimized through online sampling, reward evaluation, and policy gradient updates. Since the second training operation is oriented towards the complete training samples and invests more computing power, its training scale is larger than that of the first training operation, which serves as a probing method.

[0106] Through the above steps S201 to S205, the embodiments of this disclosure realize adaptive scheduling and automated control of the large model training process, which improves training efficiency and computing resource utilization while reducing reliance on manual intervention.

[0107] To more intuitively illustrate the specific implementation process of this disclosed technical solution, a complete application example is provided below. Assume the model to be trained is a multimodal large model for embodied intelligence scenarios, and the training dataset is divided into four data subsets: visual understanding, spatial reasoning, tool invocation, and action planning. At the beginning of a certain training iteration cycle, the data construction module 100 uses stratified sampling to extract evaluation sample sets from each of the four data subsets. The training execution module 200 performs a preset number of first training operations on each of the four evaluation sample sets based on the current model parameters, obtaining performance change information for the four data subsets. The decision module 300 calculates the capability gap index and training benefit index for each data subset based on the performance change information, determining the training priority. Assuming the action planning data subset has the largest capability gap and a high learning rate, its training priority is the highest, and it is determined as the target data subset. The decision module 300 further determines that the performance evaluation parameters of the action planning data subset are below a first capability threshold, classifying its learning state as the initial learning state. Therefore, the target training mode is determined as the supervised training mode, and the computational resource allocation ratio for the supervised training mode is increased. The training execution module 200 performs a second training operation on the action planning data subset in supervised training mode. After several iterations, the model's performance evaluation parameters in the action planning capability dimension are improved to no less than the first capability threshold and the learning rate parameter is no less than the first rate threshold. The learning state type changes to autonomous learning state, and the decision module 300 then switches the target training mode to reinforcement training mode. During reinforcement training, the model's performance evaluation results on some difficult samples remain low. The data construction module 100 marks these samples as low-performance state identifiers. The candidate training data generation module 400 calls the expert model to generate multiple candidate outputs and process annotation information for them. After verification, qualified data is written into the training dataset for targeted supervised training. For samples that cannot be solved after multiple attempts, they are marked as unlearnable samples and fed back to the decision module 300. At the same time, for some samples in the visual understanding data subset that the model has fully mastered, the data construction module 100 marks them as completed state identifiers and reduces their sampling weights. This process is repeated, and the model's various capabilities evolve collaboratively under the system's adaptive scheduling, with training resources always flowing to the direction that needs the most improvement.

[0108] As demonstrated by the examples above, the large-scale model training system and method provided in this disclosure can quantitatively perceive the learning state of the model in a low-cost manner, dynamically schedule training modes and computing resources, and differentiate between difficult samples and well-known samples, thus achieving adaptive scheduling and automated control of the large-scale model training process. Applied to the specific technical field of large-scale model training, this disclosure solves the technical problems of training relying on human experience and low resource utilization through data sampling, parameter updating, performance evaluation, and resource allocation, achieving improved training efficiency and stability.

[0109] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A large model training system, characterized in that, include: A data construction module is used to maintain a training data set and extract evaluation sample sets from multiple data subsets of the training data set. The training execution module is used to perform a first training operation on the model to be trained based on the evaluation sample set corresponding to each data subset, and obtain the performance change information corresponding to each data subset; wherein, the performance change information is used to characterize the predicted performance change trend of the model to be trained after training on the corresponding data subset. The decision module is used to determine the target data subset from multiple data subsets based on the performance change information, determine the target training mode corresponding to the target data subset, and generate training control instructions. The training control instruction is used to instruct the training execution module to perform a second training operation on the model to be trained based on the target training mode and the training samples in the target data subset; the training scale of the second training operation is greater than the training scale of the first training operation.

2. The large model training system according to claim 1, characterized in that, The data construction module is specifically used for: A stratified sampling method is used to extract an evaluation sample set from each of the data subsets; wherein the sample size of the evaluation sample set is smaller than the sample size of the data subset.

3. The large model training system according to claim 1, characterized in that, The training execution module is specifically used for: Perform the following steps for each evaluation sample set corresponding to each subset of data: Based on the current model parameters of the model to be trained and the evaluation sample set, the first training operation is performed on the model to be trained; wherein the first training operation includes a preset number of parameter update operations; Obtain the first performance index of the model to be trained on the evaluation sample set before the first training operation is executed, and the second performance index of the model to be trained on the evaluation sample set after the first training operation is executed. Based on the first performance indicator and the second performance indicator, determine the performance change information corresponding to the data subset.

4. The large model training system according to claim 3, characterized in that, The performance change information includes performance evaluation parameters and learning rate parameters; The performance evaluation parameters are determined based on the second performance index and are used to characterize the current processing capability of the model to be trained on the training samples in the corresponding data subset. The learning rate parameter is determined based on the change between the first performance index and the second performance index, and is used to characterize the performance improvement speed of the model to be trained after training based on the corresponding evaluation sample set. The first performance metric and the second performance metric include the prediction accuracy and / or reward function value of the model to be trained on the corresponding evaluation sample set.

5. The large model training system according to claim 4, characterized in that, The decision-making module is specifically used for: For each subset of data, perform the following processing: Based on the performance evaluation parameters corresponding to the data subset, a capability gap index for the data subset is determined; wherein, the capability gap index is negatively correlated with the performance evaluation parameters and is used to characterize the degree of capability deficiency of the model to be trained on the corresponding data subset; Based on the learning rate parameter corresponding to the data subset, a training benefit index for the data subset is determined; wherein, the training benefit index is positively correlated with the learning rate parameter and is used to characterize the performance improvement benefit that can be brought about by training the model to be trained based on the corresponding data subset; Based on the capability gap index and the training benefit index, determine the priority score of the data subset, and determine the priority score as the training priority of the data subset; The subset of data whose training priority meets the preset selection criteria is determined as the target data subset; The preset selection conditions include at least one of the following: the training priority is higher than a preset priority threshold; the training priority is a preset number of the highest-ranking subsets of the multiple data sets.

6. The large model training system according to claim 5, characterized in that, The decision-making module is specifically used for: Based on the performance evaluation parameters and learning rate parameters corresponding to the target data subset, the learning state type of the target data subset is determined; the learning state type includes initial learning state, intermediate learning state, autonomous learning state, and stable learning state. Based on the learning state type, the target training mode and training strategy parameters are determined; wherein, the target training mode includes supervised training mode and reinforcement training mode; the training strategy parameters are used to indicate the computational resource allocation ratio and training frequency of the target training mode.

7. The large model training system according to claim 6, characterized in that, The decision-making module is specifically used for: If the performance evaluation parameter is lower than the first capability threshold, the learning state type is determined as the initial learning state; If the performance evaluation parameter is not lower than the first capability threshold and is lower than the second capability threshold, and the learning rate parameter is lower than the first rate threshold, the learning state type is determined as the intermediate learning state. If the performance evaluation parameter is not lower than the first capability threshold and the learning rate parameter is not lower than the first rate threshold, the learning state type is determined as the autonomous learning state. When the performance evaluation parameter is not lower than the second capability threshold and the learning rate parameter is lower than the first rate threshold, the learning state type is determined as the stable learning state; wherein, the second capability threshold is higher than the first capability threshold.

8. The large model training system according to claim 7, characterized in that, The decision-making module is specifically used for: When the learning state type is the initial learning state or the intermediate learning state, the target training mode is determined as the supervised training mode, and the computing resource allocation ratio corresponding to the supervised training mode is increased. When the learning state type is the autonomous learning state, the target training mode is determined as the reinforcement training mode, and the computing resource allocation ratio corresponding to the reinforcement training mode is increased. When the learning state type is the stable learning state, the training frequency corresponding to the target data subset is reduced.

9. The large model training system according to claim 6, characterized in that, The training execution module is specifically used for: When the target training mode is the supervised training mode, the parameters of the model to be trained are updated based on the training samples in the target data subset, so that the output of the model to be trained converges to the target result corresponding to the training samples; When the target training mode is the reinforcement training mode, the output result of the model to be trained is generated based on the online sampling method, the output result is evaluated according to the preset reward function, and the policy gradient update is performed according to the evaluation result.

10. The large model training system according to claim 1, characterized in that, The large model training system also includes a candidate training data generation module; The training execution module is further configured to feed back the training results corresponding to the second training operation to the data construction module; wherein, the training results include the performance evaluation results of each training sample; The data construction module is further configured to configure the sample state information of the training samples in the training dataset according to the training results; wherein the sample state information is used to characterize the training evaluation state of the training samples. The candidate training data generation module is used to determine the corresponding training sample as the target training sample when the sample state information meets the data generation conditions, and to call an expert model with higher capabilities than the model to be trained to generate candidate training data based on the target training sample, perform a verification operation on the candidate training data, and write the candidate training data that meets the preset verification conditions into the training data set.

11. The large model training system according to claim 10, characterized in that, The data construction module is specifically used for: If the performance evaluation result of the training sample meets the preset low-performance sample condition, the sample state information of the training sample is updated to the low-performance state identifier. The candidate training data generation module is specifically used for: For each training sample with the low-performance state identifier, generate multiple candidate outputs and process annotation information corresponding to the candidate outputs; Perform a verification operation on multiple candidate outputs, and write the candidate outputs that meet the preset verification conditions and the corresponding process annotation information into the training data set; If none of the multiple candidate outputs meet the preset verification conditions, the corresponding training sample is determined as an unlearnable sample, and the sample status information of the unlearnable sample is fed back to the decision module.

12. The large model training system according to claim 11, characterized in that, The data construction module is specifically used for: If the performance evaluation result of the training sample meets the preset training completion conditions, the sample status information of the training sample is updated to the completed status identifier, and the training sample is removed from the training data set, or the sampling weight of the training sample is reduced when performing the second training operation.

13. A method for training large models, characterized in that, include: Obtain the training dataset; The training dataset includes multiple subsets of data; Evaluation sample sets are extracted from the multiple data subsets respectively; Based on the evaluation sample set corresponding to each data subset, the first training operation is performed on the model to be trained to obtain the performance change information corresponding to each data subset; wherein, the performance change information is used to characterize the predicted performance change trend of the model to be trained after training on the corresponding data subset; Based on the performance change information, a target data subset is determined from the plurality of data subsets, and a target training mode corresponding to the target data subset is determined, and training control instructions are generated. According to the training control instructions, a second training operation is performed on the model to be trained based on the target training mode and the training samples in the target data subset; wherein the training scale of the second training operation is greater than the training scale of the first training operation.

Images