Model training data construction method and system based on minimum training step coverage frequency
By using the minimum training step coverage frequency method to reversely calculate the number and proportion of samples, the problem of unscientific training data construction is solved, the controllability and effect of model training are achieved, and it is suitable for multi-task and multimodal training scenarios.
Patent Information
- Application Number
- CN202510831529.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-20
- Publication Date
- 2025-10-17
AI Technical Summary
In the process of fine-tuning large models, existing technologies rely on manual experience to construct training data, resulting in unscientific sample ratios, uneven data distribution, overfitting of main task samples, insufficient training of marginal samples, and the inability to link training configurations, which cannot meet the needs of efficient and controllable data training.
By reversely calculating the number and proportion of samples in each category based on the minimum training step coverage frequency, combined with training configuration parameters, we ensure that each category of samples has sufficient learning opportunities during training. Mathematical formulas are used for quantitative deduction, and the sample frequency is monitored and adjusted in real time, allowing for dynamic replenishment or rescheduling.
It achieves the quantitative and controllable construction of training data, ensures that long-tail and key class samples have sufficient learning opportunities during training, improves the robustness and generalization ability of the model, and is suitable for multi-task and multimodal training scenarios.
Smart Images
Figure CN120806052A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of model data analysis processing, in particular to a model training data construction method and system based on minimum training step coverage frequency. BACKGROUND
[0002] In the current fine-tuning process of large models (such as large language models, dialogue models, etc.), the construction of training data usually relies on artificial experience or preset proportion configuration. For example, when fine-tuning a dialogue language model, the developer only configures 3% of the adversarial samples to enhance the robustness of the model to abnormal input according to experience. However, after completing the training and evaluation, it is found that the model has very poor processing ability for non-normal input. Further analysis of the training log shows that this type of sample has been randomly selected less than 5 times in the entire training process, and has not formed effective parameter learning. This problem reflects that: although the sample exists in the training set, due to the low proportion and the fact that the training configuration is not adapted, it is almost not "seen" by the model in the training process.
[0003] This problem is particularly common in multi-task and multi-modal training. Since the frequency of using different sample types in training is highly dependent on their proportion in the training set, and existing construction methods cannot ensure that different sample types obtain reasonable training opportunities, the following technical defects are caused:
[0004] 1. Unscientific data distribution: The proportion of training set samples is usually based on experience estimation, lacks quantitative basis, and is out of touch with the actual training effect, which can easily cause overfitting of main task samples and insufficient training of edge samples;
[0005] 2. Small class sample learning failure: Key sample types with extremely low proportion (such as long-tail samples, abnormal samples) have too low frequency in the training process, which cannot form effective learning, ultimately leading to model performance bottleneck;
[0006] 3. Training configuration cannot be linked: Traditional data construction methods cannot form a synergistic optimization with training parameters (such as batch size, epoch), so that even if the training configuration is reasonable, the data itself cannot support the target performance.
[0007] Existing solutions are mostly manual parameter tuning and incremental debugging, lacking systematic theoretical basis and tool support. Data construction is still a repetitive work that relies on experience and trial and error, and cannot meet the efficient and controllable data training needs. SUMMARY
[0008] One of the purposes of the present application is to provide a model training data construction method based on minimum training step coverage frequency, which solves the technical problems of unscientific sample configuration, uncontrollable training effect, and insufficient frequency of small class samples in existing fine-tuning tasks.
[0009] In order to achieve the above purpose, a model training data construction method based on minimum training step coverage frequency is provided, comprising the following steps:
[0010] Training configuration input step: receiving basic parameters of training task; the basic parameters include total training sample number, sample number in each training step, training round, and target frequency T of each type of sample expected to appear in training class ;
[0011] Sample number backstepping step: according to the input basic parameters, calling a preset formula to reversely calculate the minimum sample number N class and minimum sample proportion P class of each type of sample; the preset formula includes sample number backstepping formula and sample proportion backstepping formula
[0012] Sample configuration output step: writing the minimum sample number N class and minimum sample proportion P class into a sample construction configuration file to guide the selection and generation of subsequent training sample data
[0013] Monitoring and verification step: after the sample preparation is completed, in the simulation training process, the actual step coverage frequency E step of each type of sample is monitored in real time; then E step is compared with the set target frequency T class to determine whether the type of sample meets the training target frequency T class ; if not, output prompt information, the prompt information including increasing the number of this type of sample in the basic parameters, adjusting the training parameters or increasing the sampling weight of this type of sample.
[0014] Further, the sample number backstepping formula is as follows:
[0015]
[0016] N class : the number of samples of a certain type that should be prepared in the training set at least;
[0017] T class : the minimum number or target frequency of the type of sample expected to appear in training;
[0018] batch_size: the number of samples contained in each training step;
[0019] epoch: the number of rounds of data set in training.
[0020] Further, the sample proportion backstepping formula is as follows:
[0021]
[0022] P class : the minimum proportion of this type of sample in the training set;
[0023] N total : the total number of samples in the training set.
[0024] Further, the calculation formula of the coverage frequency E step in the monitoring and verification step is as follows:
[0025] E step = step total ×[1-(1-P class ) B ]
[0026] Wherein, E step : the number of steps in which this type of sample is expected to appear in the training process;
[0027] step total : the total number of steps, step total =(N total / batch_size)×epoch;
[0028] B: the number of samples in each step, i.e. batch_size.
[0029] Further, the acquisition of the basic parameters includes: acquisition from the front-end visual form interface; automatic reading from the training platform configuration file; calling through the REST API interface.
[0030] Further, the following steps are included:
[0031] Sample structure and function classification step: divide the original data set into structured sample categories; including but not limited to: main task samples: core task target related samples; expression variant samples: synonymous expression, style difference, word order change, etc.; edge / long tail samples: low-frequency appearing, complex structure task samples; adversarial samples: abnormal input, noise samples, misleading input; prompt style samples: instruction templates, samples with diversified expression formats.
[0032] Sample classification methods include: based on preset label fields; combined with metadata; automatically determined by internal semantic classification model; optionally: set sample identification rules by rule engine;
[0033] After classification, each type of sample is bound to a corresponding T class value, which participates in subsequent sample construction calculation.
[0034] Further, the following steps are included:
[0035] Sample screening and supplementing step: ensure that the actual number of each type of sample meets the N obtained by backstepping class Requirements; the specific process is as follows:
[0036] Connect the data warehouse, and screen according to the sample categories output by the sample structure and function classification step;
[0037] Analyze whether the number of a certain type of sample is less than N class If it is less than, trigger the sample supplement strategy, which includes:
[0038] Text samples: call a pre-trained language model for semantic equivalent transformation and style diversification generation;
[0039] Image samples: use image enhancement algorithms and GAN models to generate samples;
[0040] Multimodal samples: supplement samples by intra-modal data enhancement or cross-modal generation;
[0041] All generated samples record the following metadata:
[0042] Original sample ID; generation method; generation model version; similarity score or data quality score.
[0043] The second object of the present application is to provide a model training data construction system based on minimum training step coverage frequency, which uses the method of any one of claims 1-7, comprising the following modules:
[0044] Training configuration input module: used for receiving basic parameters of training tasks; the basic parameters include total training sample number, sample number in each training step, training round, and target frequency T of each type of sample expected to appear in training class ;
[0045] Sample number backstepping module: used for backstepping the minimum sample number N class and the minimum sample proportion P class of each type of sample according to the input basic parameters, by calling a preset formula; the preset formula includes a sample number backstepping formula and a sample proportion backstepping formula;
[0046] Sample configuration output module: used for writing the minimum sample number N class and the minimum sample proportion P class into a sample construction configuration file to guide the selection and generation of subsequent training sample data;
[0047] Monitoring and verification module: used for monitoring the actual step coverage frequency E step of each type of sample in real time during the simulation training process after the sample preparation is completed; and comparing E step with the set target frequency Tclass , judge whether the sample meets the training target frequency T class ; if not, output prompt information, the prompt information includes: increase the number of this type of sample in the basic parameters, adjust the training parameters or increase the sampling weight of this type of sample.
[0048] Further, it further includes the following modules:
[0049] Sample structure and function classification module: used to divide the original data set into structured sample categories; including but not limited to: main task sample: core task target related sample; expression variant sample: synonymous expression, style difference, word order change, etc. Sample; edge / long tail sample: low frequency, complex structure task sample; adversarial sample: abnormal input, noise sample, misleading input; prompt style sample: instruction template, sample with diversified expression format.
[0050] Sample classification methods include: based on preset label field; combined with metadata; automatically determined by internal semantic classification model; optional: rule engine sets sample identification rules;
[0051] After classification, each type of sample is bound to the corresponding Tclass value, and participates in subsequent sample construction calculation.
[0052] Further, it further includes the following modules:
[0053] Sample screening and supplement module: used to ensure that the actual number of each type of sample meets the N class requirement obtained by back calculation; the specific process is as follows:
[0054] Connect the data warehouse, and filter according to the sample categories output by the sample structure and function classification module;
[0055] Analyze whether the number of a certain type of sample is less than N class , if less than, trigger the sample supplement strategy, which includes:
[0056] Text sample: call the pre-trained language model to perform semantic equivalent transformation and style diversification generation;
[0057] Image sample: use image enhancement algorithm, GAN model to generate sample;
[0058] Multimodal sample: supplement samples by intra-modal data enhancement or cross-modal generation;
[0059] All generated samples record the following metadata:
[0060] Original sample ID; generation method; generation model version; similarity score or data quality score.
[0061] Principle and advantages:
[0062] 1, This scheme is around the minimum target frequency T of the sample effectively "seen" in training class , combined with batch_size and epoch training configuration parameters, the minimum number or proportion of various samples required is deduced reversely, and then a quantitative and controllable data construction mechanism is realized. Technical highlights include: data construction quantitative modeling: mathematical formula is used to deduce the linkage of sample distribution and training configuration, avoiding the dependence on subjective experience configuration; Small class sample protection mechanism: through the minimum step coverage frequency T, ensure that long tail, edge or key class samples get sufficient learning opportunities in the training process; High universality design: this method does not depend on specific model structure or task type, only based on training iteration logic (batch sampling and epoch round), can adapt to any deep learning platform; Support dynamic feedback and scheduling in training: the system can monitor the actual step coverage of the sample in real time, compare with the preset T value, trigger the sample supplement or rescheduling mechanism, automatically supplement or reschedule when insufficient, form a data closed loop; Adapt to complex tasks and multi-modal applications: especially suitable for unbalanced training scenarios with diverse task structures and complex sample distributions, such as dialogue, recommendation, visual question answering, multi-label classification, etc. Enhance model robustness and generalization ability: by ensuring the linkage of sample distribution and training frequency, more comprehensive and stable model performance improvement is realized.
[0063] 2, This scheme is suitable for automatic construction of training data set in deep learning training scene, especially suitable for pre-training or fine-tuning process of multi-task and multi-sample type. It can be deployed as an independent module of the training platform, or it can be embedded in the data preprocessing pipeline before training to ensure that different types of samples get enough step coverage frequency in the training process, so as to improve the training effect and avoid learning failure of key sample types. BRIEF DESCRIPTION OF DRAWINGS
[0064] Figure 1 The flow chart of a model training data construction method based on the minimum training step coverage frequency according to an embodiment of the application is shown in the figure;
[0065] Figure 2 The logic diagram of a model training data construction system based on the minimum training step coverage frequency according to an embodiment of the application is shown in the figure. DETAILED DESCRIPTION
[0066] The following will be further described in detail through specific embodiments:
[0067] EMBODIMENT
[0068] A model training data construction method based on the minimum training step coverage frequency, as shown in the figure, includes the following steps: Figure 1
[0069] Training configuration input step: receiving basic parameters of training task; the basic parameters include total training sample number N class , sample number batch_size in each training step, training round epoch, and target frequency T of each type of sample expected to appear in training class ; the acquisition of the basic parameters includes: obtaining from a front-end visual form interface; automatically reading from a training platform configuration file; calling through a REST API interface. Implement a small class sample guarantee mechanism: ensure that long-tail, edge or key class samples get sufficient learning opportunities in the training process through the minimum step coverage frequency T class ; users can individually set the T class value of each class or adopt the default value recommended by the system (for example, T class = 50).
[0070] Sample structure and function classification step: divide the original data set into structured sample categories; including but not limited to: main task samples: core task target related samples; expression variant samples: samples with synonymous expressions, style differences, and word order changes; edge / long-tail samples: low-frequency appearing, complex structure task samples; adversarial samples: abnormal input, noise samples, misleading input; prompt style samples: instruction templates, samples with diversified expression formats.
[0071] Sample classification methods include: based on preset label fields (such as task_type, difficulty); combined with metadata (such as prompt template ID, user intent label); automatically determined by internal semantic classification model (such as text classifier);
[0072] Optional: rule engine sets sample recognition rules (such as including keywords in the instruction, then classified as adversarial samples);
[0073] After classification, each type of sample is bound to a corresponding T class value, which participates in subsequent sample construction calculation.
[0074] Sample number reverse step: according to the input basic parameters, call the preset formula to reversely calculate the minimum sample number N class and the minimum sample proportion P class of each type of sample; the preset formula includes a sample number reverse formula and a sample proportion reverse formula; the sample number reverse formula is as follows:
[0075]
[0076] N class : the number of a certain type of sample that should be prepared in the training set at least;
[0077] T class: the minimum number of occurrences or target frequency of this type of sample in the training;
[0078] batch_size: the number of samples included in each training step;
[0079] epoch: the number of rounds of the dataset in training.
[0080] The sample proportion back calculation formula is as follows:
[0081]
[0082] P class : the minimum proportion of this type of sample in the training set;
[0083] N total : the total number of samples in the training set.
[0084] Sample configuration output step: write the minimum sample number N class and the minimum sample proportion P class into the sample construction configuration file to guide the selection and generation of subsequent training sample data;
[0085] Sample screening and supplement step: ensure that the actual number of each type of sample meets the requirement of N class obtained by back calculation; the specific process is as follows:
[0086] Connect data warehouses (such as object storage, annotation platforms, and data management systems) and perform screening according to the sample categories output by the sample structure and functional classification step;
[0087] Analyze whether the number of a certain type of sample is less than N class , if it is less than N , trigger the sample supplement strategy, which includes:
[0088] Text samples: call pre-trained language models (such as GPT) for semantic equivalent transformation and style diversification generation;
[0089] Image samples: use image enhancement algorithms and GAN models to generate samples;
[0090] Multi-modal samples: supplement samples through intra-modal data enhancement or cross-modal generation (such as image-text conversion);
[0091] All generated samples record the following metadata:
[0092] Original sample ID; generation method (enhancement / model generation / external acquisition); generation model version; similarity score or data quality score.
[0093] The sample screening and supplement step is used to ensure that the generated samples are traceable, evaluable, and meet the training task requirements.
[0094] Monitoring and verification steps: After the sample preparation is completed, the simulation training process is carried out to monitor the actual step coverage frequency E of various samples in real time. step ; Compare with E step and the target frequency T class , determine whether the sample meets the training target frequency T class If not satisfied, output prompt information, the prompt information includes: increase the number of samples of this type in the basic parameters, adjust the training parameters or increase the sampling weight of samples of this type. Coverage frequency E in the monitoring and verification step step The calculation formula is as follows:
[0095] E step =step total ×[1-(1-P class ) B ]
[0096] Among them, E step : The number of steps in which this type of sample is expected to appear during the training process;
[0097] step total : total number of training steps, step total =(N total / batch_size)×epoch;
[0098] B: The number of samples in each step, i.e. batch_size.
[0099] In the monitoring and verification steps, the monitoring data sources include: training logs, training platform event hooks, and middle-layer sampling records. The following measures are recommended:
[0100] Dynamically adjust sampling weight or frequency; push alerts to the task management platform; automatically trigger sample replenishment and rescheduling.
[0101] This solution is highly versatile and independent of specific model structures or task types. It is based solely on training iteration logic (batch sampling and epochs) and can be adapted to any deep learning platform. Its specific features are as follows:
[0102] 1. Task-independent: adaptable to any task type such as classification, generation, recommendation, and dialogue;
[0103] 2. Model structure-independent: Applicable to various architectures such as Transformer, CNN, RNN, and recommendation models;
[0104] 3. Data modality independent: supports multiple data types such as text, images, audio, tables, etc.
[0105] 4. Flexible deployment: can be deployed independently or combined as pre-training preprocessing module, training monitoring component, post-training evaluation module.
[0106] 5. This method only relies on batch sampling and epoch mechanism during training, without modifying the model structure or task label system, with good platform portability and system engineering landing.
[0107] The implementation of the present scheme will also bring the following significant technical effects:
[0108] 1. Accurate and controllable training set construction: avoid "brainstorming" data preparation, ensure that each class of samples meets the training target through quantitative derivation; ensure that long-tail, edge or key class samples also have sufficient learning opportunities in the training process;
[0109] 2. Improve the learning ability of the model for small class samples: prevent long-tail or key class samples from being ineffective due to insufficient training frequency;
[0110] 3. Adapt to multi-task collaborative training scenarios: each class of samples can be individually configured T class value, supporting fine-grained training strategies;
[0111] 4. Real-time feedback and optimization during training: the system can dynamically detect sample step coverage, and automatically supplement or reschedule when insufficient;
[0112] 5. Enhance model robustness and generalization ability: by ensuring the linkage of sample distribution and training frequency, more comprehensive and stable model performance improvement is achieved.
[0113] A model training data construction system based on minimum training step coverage frequency, which uses the method as described above, as shown in Figure 2 , comprising a server, the server comprising the following modules:
[0114] Training configuration input module: for receiving basic parameters of training task; basic parameters include total training sample number N class , sample number in each training step batch_size, training round epoch, and target frequency T class of each class of samples expected in training; the acquisition of the basic parameters includes: obtaining from the front-end visual form interface; automatically reading from the training platform configuration file; calling through REST API interface. Realize small class sample guarantee mechanism: through the minimum step coverage frequency T class , ensure that long-tail, edge or key class samples have sufficient learning opportunities in the training process.
[0115] Sample structure and function classification module: used for dividing the original data set into structured sample categories; including but not limited to: main task sample: core task target related sample; expression variant sample: synonymous expression, style difference, word order change and other samples; edge / long tail sample: low frequency, complex structure task sample; adversarial sample: abnormal input, noise sample, misleading input; prompt style sample: instruction template, expression format diversified sample.
[0116] Sample classification methods include: based on preset label field; combined with metadata; internal semantic classification model automatic determination; optional: rule engine sets sample identification rules;
[0117] After classification, each type of sample is bound to the corresponding Tclass value, and participates in subsequent sample construction calculation.
[0118] Sample quantity reverse calculation module: used for calling preset formula to reversely calculate the minimum sample quantity N class and minimum sample proportion P class of each type of sample according to the input basic parameters; the preset formula includes sample quantity reverse calculation formula and sample proportion reverse calculation formula; the sample quantity reverse calculation formula is as follows:
[0119]
[0120] N class : the number of samples of a certain type that should be prepared at least in the training set;
[0121] T class : the minimum number or target frequency of the sample of the type in the training;
[0122] batch_size: the number of samples contained in each training step;
[0123] epoch: the number of rounds of the data set in the training.
[0124] The sample proportion reverse calculation formula is as follows:
[0125]
[0126] P class : the minimum proportion of the sample of the type in the training set;
[0127] N total : the total number of samples in the training set.
[0128] Sample configuration output module: used for writing the minimum sample quantity N class and minimum sample proportion P class into the sample construction configuration file, guiding the selection and generation of subsequent training sample data;
[0129] Sample screening and supplement module: used to ensure that the actual number of each type of sample meets the N required by backstepping; specific process as follows: class Request; the specific process is as follows:
[0130] Connect the data warehouse and screen according to the sample categories output by the classification module;
[0131] Analyze whether the number of a certain type of sample is less than N class , if less than, trigger the sample supplement strategy, which includes:
[0132] Text samples: call pre-trained language models for semantic equivalent transformation and style diversification generation;
[0133] Image samples: use image enhancement algorithms and GAN models to generate samples;
[0134] Multimodal samples: supplement samples through intra-modal data enhancement or cross-modal generation;
[0135] All generated samples record the following metadata:
[0136] Original sample ID; generation method; generation model version; similarity score or data quality score.
[0137] Monitoring and verification module: used to monitor the actual step coverage frequency E step of each type of sample in the simulated training process after sample preparation is completed; then compare E step with the set target frequency T class , judge whether the type of sample meets the training target frequency T class ; if not, output prompt information, which includes: increase the number of this type of sample in the basic parameters, adjust the training parameters or increase the sampling weight of this type of sample. The calculation formula of the coverage frequency E step in the monitoring and verification step is as follows:
[0138] E step = step total ×[1-(1-P class ) B ]
[0139] Wherein, E step : the number of steps that the type of sample is expected to appear in the training process;
[0140] step total : total training steps, step total =(N total / batch_size)×epoch;
[0141] B: the number of samples in each step, i.e. batch_size.
[0142] In the monitoring and verification module, the monitoring data sources include training logs, training platform event hooks, and intermediate layer sampling records, and the following measures are recommended:
[0143] Adjusting the sampling weight or frequency dynamically, pushing early warning to the task management platform, and automatically triggering sample supplement and rescheduling.
[0144] The present scheme is suitable for automatic construction of training data sets in deep learning training scenarios, especially for pre-training or fine-tuning processes of multiple tasks and multiple sample types. The system can be deployed as an independent module of the training platform or embedded in the data preprocessing pipeline before training to ensure that different types of samples obtain sufficient step coverage frequency in the training process, thereby improving the training effect and avoiding learning failure of key sample types.
[0145] The above is only an embodiment of the present application, and well-known specific structures and characteristics in the scheme are not described in detail. The person skilled in the art knows all the ordinary technical knowledge in the field of the present application before the filing date or the priority date, can know all the prior art in the field, and has the ability to apply conventional experimental means before that date. The person skilled in the art can improve and implement the present scheme based on their own ability under the guidance of the present application. Some typical known structures or known methods should not be an obstacle for the person skilled in the art to implement the present application. It should be noted that for those skilled in the art, without departing from the structure of the present application, a number of modifications and improvements can be made, which should also be considered within the scope of protection of the present application. The effectiveness and practicality of the present application will not be affected. The scope of protection claimed in the present application should be subject to the content of its claims, and the specific implementation mode and the like in the specification can be used to explain the content of the claims.
Claims
1. A method for constructing model training data based on minimum training step coverage frequency, characterized by , including the following steps: Training configuration input step: receive the basic parameters of the training task; the basic parameters include the total number of training samples, the number of samples in each training step, the number of training rounds, and the target frequency T of each type of sample in the training. class ; Sample quantity reverse calculation step: Based on the input basic parameters, call the preset formula to reversely calculate the minimum sample quantity N that each type of sample should have class and the minimum sample proportion P class ; The preset formula includes the sample quantity inverse formula and the sample proportion inverse formula; Sample configuration output step: the minimum sample size N class and the minimum sample proportion P class Write the sample construction configuration file to guide the selection and generation of subsequent training sample data; Monitoring and verification steps: After the sample preparation is completed, the simulation training process is carried out to monitor the actual step coverage frequency E of various samples in real time. step ; Compare with E step and the target frequency T class , determine whether the sample meets the training target frequency T class ; If not, a prompt message is output, which includes: increasing the number of samples of this type in the basic parameters, adjusting the training parameters, or increasing the sampling weight of samples of this type.
2. The method for constructing model training data based on minimum training step coverage frequency according to claim 1, characterized in that: The sample size inverse formula is as follows: N class : The minimum number of samples of a certain type that should be prepared in the training set; T class : The minimum number of times or target frequency of samples of this type expected to appear in training; Batch_size: the number of samples included in each training step; epoch: The number of rounds of the dataset in training.
3. The method for constructing model training data based on minimum training step coverage frequency according to claim 2, characterized in that: The reverse formula for the sample proportion is as follows: P class : This type of samples should account for the minimum proportion of the training set; N total : The total number of samples in the training set.
4. The method for constructing model training data based on minimum training step coverage frequency according to claim 3, characterized in that: The monitoring and verification steps cover the frequency E step The calculation formula is as follows: E step =step total ×[1-(1 - P class ) B ] Among them, E step : The number of steps in which this type of sample is expected to appear during the training process; step total : total number of training steps, step total =(N total / batch_size)×epoch; B: The number of samples in each step, i.e. batch_size.
5. The method and system for constructing model training data based on minimum training step coverage frequency according to claim 4, characterized in that: The acquisition of the basic parameters includes: obtaining from the front-end visual form interface; automatically reading from the training platform configuration file; and calling through the REST API interface.
6. The method and system for constructing model training data based on minimum training step coverage frequency according to claim 1, characterized in that: The following steps are also included: Sample structure and function classification steps: Divide the original dataset into structured sample categories; including but not limited to: main task samples: samples related to the core task objectives; expression variant samples: samples with synonymous expressions, style differences, word order changes, etc.; edge / long-tail samples: task samples with low frequency and complex structure; adversarial samples: abnormal inputs, noise samples, misleading inputs; prompt style samples: instruction templates, samples with diverse expression formats. Sample classification methods include: based on preset tag fields; combined with metadata; automatic determination by the internal semantic classification model; optional: setting sample identification rules by the rule engine; After classification, each class of samples is bound to the corresponding T class Value, participate in subsequent sample construction calculation.
7. The method and system for constructing model training data based on minimum training step coverage frequency according to claim 6, characterized in that: The following steps are also included: Sample screening and supplementation steps: Ensure that the actual number of samples in each category meets the N obtained by reverse deduction class Requirements; the specific process is as follows: Connect to the data warehouse and filter the sample categories according to the output of the sample structure and function classification steps; Analyze whether the number of samples of a certain type is less than N class If it is less than , the resampling strategy is triggered. The resampling strategy includes: Text samples: Use pre-trained language models to perform semantic equivalence transformation and style diversity generation; Image samples: Generate samples using image enhancement algorithms and GAN models; Multimodal samples: Generate supplementary samples through intra-modal data augmentation or cross-modal data augmentation; All generated samples record the following metadata: Original sample ID; generation method; generation model version; similarity score or data quality score.
8. The method and system for constructing model training data based on minimum training step coverage frequency according to claim 7, characterized in that: In the monitoring and verification steps, the monitoring data sources include: training logs, training platform event hooks, and middle-layer sampling records. The following measures are recommended: Dynamically adjust sampling weight or frequency; push alerts to the task management platform; automatically trigger sample replenishment and rescheduling.
9. A model training data construction system based on minimum training step coverage frequency, characterized by: The method according to any one of claims 1 to 8 above is used, comprising the following modules: Training configuration input module: used to receive the basic parameters of the training task; the basic parameters include the total number of training samples, the number of samples in each training step, the number of training rounds, and the target frequency T of each type of sample in the training. class ; Sample quantity reverse calculation module: used to call the preset formula to reversely calculate the minimum number of samples N that each type of sample should have based on the input basic parameters class and the minimum sample proportion P class ; The preset formula includes the sample quantity inverse formula and the sample proportion inverse formula; Sample configuration output module: used to convert the minimum sample number N class and the minimum sample proportion P class Write the sample construction configuration file to guide the selection and generation of subsequent training sample data; Monitoring and verification module: used to monitor the actual step coverage frequency E of various samples in real time during the simulation training process after sample preparation is completed step ; Compare with E step and the target frequency T class , determine whether the sample meets the training target frequency T class ; If not, a prompt message is output, which includes: increasing the number of samples of this type in the basic parameters, adjusting the training parameters, or increasing the sampling weight of samples of this type.
10. The model training data construction system based on minimum training step coverage frequency according to claim 9, characterized in that: Also includes the following modules: Sample structure and function classification module: used to divide the original dataset into structured sample categories; including but not limited to: main task samples: samples related to core task objectives; expression variant samples: samples such as synonymous expressions, style differences, word order changes, etc.; edge / long-tail samples: task samples with low frequency and complex structure; adversarial samples: abnormal inputs, noise samples, misleading inputs; prompt style samples: instruction templates, samples with diverse expression formats. Sample classification methods include: based on preset tag fields; combined with metadata; automatic determination by the internal semantic classification model; optional: setting sample identification rules by the rule engine; After classification, each type of sample is bound to the corresponding Tclass value and participates in subsequent sample construction calculations.
11. The model training data construction system based on minimum training step coverage frequency according to claim 10, characterized in that: Also includes the following modules: Sample screening and supplementation module: used to ensure that the actual number of samples of each type meets the N obtained by reverse deduction class Requirements; the specific process is as follows: Connect to the data warehouse and filter the sample categories according to the sample structure and function classification module output; Analyze whether the number of samples of a certain type is less than N class If it is less than , the resampling strategy is triggered. The resampling strategy includes: Text samples: Use pre-trained language models to perform semantic equivalence transformation and style diversity generation; Image samples: Generate samples using image enhancement algorithms and GAN models; Multimodal samples: Generate supplementary samples through intra-modal data augmentation or cross-modal data augmentation; All generated samples record the following metadata: Original sample ID; generation method; generation model version; similarity score or data quality score.