Data generation method based on size model cooperation for classification task

Abstract

This invention belongs to the field of data generation technology and discloses a data generation method based on large and small model collaboration for classification tasks. The method includes: Step 1, collecting the original dataset; Step 2, determining the scale of the generated data; Step 3, generating new generated data using a large language model; Step 4, verifying the logical consistency of the generated data; Step 5, scoring the quality of the new generated data to form a comprehensive score specific to the new generated data; and Step 6, training a small model based on the original training dataset and the new generated data. This application achieves precise and adaptive adjustment of data generation, preventing low-quality data from flowing into the training stage from the source, realizing accurate quantitative scoring of the generated data, and achieving precise application of the synthesized data through stage-adaptive rule adjustments.

Description

Technical Field

[0001] This invention belongs to the field of data generation technology, specifically relating to a data generation method based on size model collaboration for classification tasks. Background Technology

[0002] In the widespread application of artificial intelligence, data quality and quantity directly affect model performance. Data augmentation, as a key technology for improving model performance, is widely used in various task scenarios. In recent years, with the development of Large Language Models (LLM) and generative AI technologies, the application of large models in the field of data augmentation has gradually increased. Currently, the AI ​​field is actively exploring new paradigms for collaboration between large and small models. By leveraging the powerful general generative capabilities of large models, richer and higher-quality synthetic sample sets can be generated; small models, on the other hand, focus on leveraging their specialized business capabilities, being trained based on synthetic samples generated by large models and original data; using knowledge distillation techniques, large models transfer knowledge to small models, thereby enhancing the capabilities of small models and achieving complementary advantages.

[0003] However, the collaboration between large and small models is still in its early stages, mainly manifested in the following ways: 1) Large and small models are usually trained independently, and the collaboration method is mainly one-way invocation, lacking exploration of more complex collaboration mechanisms. The generation process and the training process of the small model are isolated from each other, failing to form a dynamic feedback loop, making it difficult to adjust and optimize the generation strategy of the large model according to the real-time performance of the small model. 2) The difference in model size leads to differences in the knowledge level between general-purpose and specialized models, making it difficult to achieve cross-model knowledge fusion. The general knowledge of the large model cannot be accurately transformed into effective data adapted to the specific tasks of the small model, resulting in a low degree of matching between the generated data and the task requirements of the small model. 3) Existing technologies mostly use fixed static indicators to evaluate the generated data, without considering the differentiated data quality requirements of the small model at different training stages, failing to fully leverage the role of data augmentation in improving model performance.

[0004] In summary, while current data augmentation techniques that combine large and small models show some potential, they suffer from limitations due to core issues such as a single collaborative mechanism, insufficient knowledge integration, and static evaluation and selection. These limitations make it difficult to dynamically adapt the generated data to the training needs of the small model, thus restricting the effectiveness of data augmentation and failing to fully unleash the advantages of combining large and small models. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a data generation method based on large and small model collaboration for classification tasks. This application systematically standardizes the entire data augmentation process, effectively improving the data processing efficiency, training convergence speed, and prediction accuracy and stability of artificial intelligence models in practical application scenarios.

[0006] To achieve the above objectives, this application employs the following technical solution:

[0007] This application discloses a data generation method based on size-model collaboration for classification tasks, characterized in that the data generation method specifically includes the following steps:

[0008] Step 1, Data Preparation: Collect the original dataset and preprocess it, including data cleaning and feature extraction, to ensure the basic quality and usability of the data. Classify and label the preprocessed data to clarify the category to which each data belongs and divide it into the original training dataset and the original test dataset. The original training dataset is used for training small models and generating examples for large models, while the original test dataset is used to evaluate the performance of small models.

[0009] Step 2: Determine the number of newly generated data sets: Combine the sample imbalance evaluation results of the original training dataset with the performance differences of small models to determine the number of newly generated data sets for each category;

[0010] Step 3, Data Generation: Based on the new data scale for different categories determined in Step 2, the large language model is used to generate a corresponding number of new data for different categories. This generation process is iterative, generating a batch of new data each time.

[0011] Step 4: Data consistency verification: Based on the reverse generation and forward derivation verification method, verify the logical consistency of the newly generated data in Step 3 to ensure the reliability of the new data finally used for model training.

[0012] Step 5: Based on the performance of the small model during the training process of the newly generated data, a quality score is given to the newly generated data to form a comprehensive score that is only for the newly generated data;

[0013] Step 6: Train a small model based on the original training dataset and the newly generated data.

[0014] A further improvement of this application is that step 2 specifically includes the following steps:

[0015] Step 2.1: Count the number of data points in each category in the original training dataset;

[0016] Step 2.2: Use the original category ratio and Gini coefficient The multi-dimensional indicators comprehensively quantify the quantity of data in each category, forming an assessment result of the imbalance in the overall distribution of the original training dataset.

[0017] Step 2.3: Set an imbalance threshold. Based on the imbalance assessment results of the original training dataset obtained in Step 2.2, determine the data generation quantity allocation strategy and generate basic generation weights. When the degree of imbalance is in the low range, i.e., the Gini coefficient When the imbalance level is less than or equal to the Gini coefficient threshold, an average distribution strategy is adopted. When the imbalance level is in the high range, i.e., the Gini coefficient is lower, the average distribution strategy is adopted. When the imbalance level exceeds the threshold, a proportional allocation strategy is adopted, that is, the number of data generated in each category is inversely proportional to the category ratio.

[0018] Step 2.4: Based on the assessment results of the degree of uneven distribution, preliminarily determine the amount of basic generated data for each category;

[0019] Step 2.5: Introduce a small model performance feedback dimension, combined with basic generation weights. Based on the performance metrics of the small model, a weighted model for the generated data samples is constructed to achieve a secondary adjustment of the amount of basic generated data, specifically as follows:

[0020]

[0021] in, For category The basic generation weights, For the adjusted category The generation weights, The total number of categories in the original training dataset. As a quantity balance factor, Performance metrics of the small model on the original test dataset The performance adjustment factor is based on the generated weights after secondary adjustment. Calculate the amount of generated data for each category. :

[0022]

[0023] in, To generate the total amount of data;

[0024] Step 2.6: Establish real-time monitoring and dynamic optimization of the generation process: During the overall data generation process, continuously track the changes in the distribution imbalance of the generated data and the performance fluctuations of the small model on the new data. Using a preset imbalance improvement evaluation function, evaluate the effectiveness of the current generation strategy in real time. The evaluation result is calculated using the following formula:

[0025]

[0026] in, The Gini coefficient for the currently generated data. This represents the improvement in F1-score for the smaller model on the new data. and These are the weighting parameters for quantity balancing and performance improvement, respectively. If the evaluation results... If the value falls below the validity assessment threshold, the parameter generation iteration mechanism is automatically triggered.

[0027] A further improvement of this application is that step 2.2 specifically includes the following steps:

[0028] Step 2.2.1: For a certain category in the original training dataset The proportion of its original categories :

[0029]

[0030] in, It is a category The amount of data, It represents the total number of all data in the original training dataset. By calculating the proportion of all data categories, we can intuitively understand the percentage of each category's sample size in the total dataset.

[0031] Step 2.2.2, Gini coefficient ,in, Both are categories Category ratio, It is the total number of categories in the original training dataset, and the Gini coefficient. The value of Gini coefficient ranges from 0 to 1. The larger the Gini coefficient, the more severe the imbalance in the data.

[0032] A further improvement in this application lies in the data generation quantity allocation strategy in step 2.3:

[0033] The average distribution strategy is as follows: if the number of data generated for each category is the same, then the basic generation weight for each category is... : ;

[0034] The proportional allocation strategy specifically involves: adjusting the original category proportions... Weights are generated based on mapping :

[0035]

[0036] in, As a quantity balance factor, Quantity balance factor It controls the degree of nonlinearity of the mapping. The larger the value, the faster the weight of the subcategories increases.

[0037] A further improvement in this application is that step 4 specifically comprises:

[0038] Step 4.1, Basic Quality Check: Check the format, length, sensitive information, and similarity of the data in the original training dataset, and remove invalid data;

[0039] Step 4.2: Remove invalid data and verify logical consistency: Input the feature part of each generated data into the large language model to generate a predicted label for that data. Compare the original label and the predicted label of the generated data: If the original label and the predicted label are completely consistent, the data is determined to be logically consistent and is retained in the dataset that has passed the verification. If the original label and the predicted label are inconsistent, it indicates that there is a logical problem of semantic contradiction or misassociation of labels in the data. The data sample is marked as low-quality data and removed.

[0040] A further improvement of this application is that, in step 5, a comprehensive score is formed only for the newly generated data, specifically including the following steps:

[0041] Step 5.1: Set up a specific training cycle separately. The small model is trained using only newly generated data, and the prediction results and confidence scores of the small model for each newly generated data are recorded in real time during the training process.

[0042] Step 5.2: Determine if the number of training sessions has reached the training cycle. If the training cycle is not reached Continue iterative training until the training cycle is reached. Complete the targeted training of the small model;

[0043] Step 5.3: After the training period ends, calculate the average confidence score of the small model on the generated data. Variability Correctness Forgetting rate Threshold proximity :

[0044]

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] in, This represents the mathematical expectation operation. This represents the total number of training cycles. It generates a sequence of confidence scores for all data throughout the entire training period. To generate data Confidence level for each period, This represents the standard deviation calculation. Indicates the indicator function, if the first The value is 1 if the predicted label for a period matches the original label, and 0 otherwise. Indicates the small model in the first... The periodicity of the data's predictive label; The original labels representing the generated data. Indicates the first The cycle prediction was correct. Indicates the first The cycle prediction was incorrect; This represents the logical AND operation. Indicates the logical NOT operation. This is the decision threshold;

[0051] Step 5.4: Use the average confidence level calculated in Step 5.3. Variability Correctness Forgetting rate Threshold proximity The overall score for each piece of composite data is calculated using a weighted summation formula:

[0052] .

[0053] A further improvement of this application is that step 6 specifically includes the following steps:

[0054] Step 6.1, Data Initialization: Merge the newly generated data with the original data to form the initial training dataset. ,in For the newly generated data, This is the original data;

[0055] Step 6.2, Model Training and Monitoring: Using the initial training dataset Iteratively train a small model while monitoring its performance metrics, such as accuracy, recall, and loss function value, denoted as... , For the number of indicators;

[0056] Step 6.3, Judging the Number of Training Iterations: Determine whether the preset number of training iterations has been reached. The count for each iteration is ,when If the number of training iterations has been reached, training ends; otherwise, iteration continues.

[0057] Step 6.4, Assessment Cycle Determination: Determine whether the assessment cycle has been reached. ,Every Each iteration performs a training state evaluation; when the condition is met... (When the period is measured by the number of iterations), if the evaluation period has been reached, the model state evaluation process is triggered; if "No", return to step 6.2 to continue iterating; if "Yes", enter the model training state evaluation, and divide the model training state into the cold start phase, optimization phase, and convergence phase. For the modulo operation;

[0058] Step 6.5: Filter data based on sample scores for the cold start, optimization, and convergence phases: Set differentiated screening thresholds for the cold start, optimization, and convergence phases, based on the comprehensive scores. Synthetic data below the differential screening threshold are removed to form the total sample sequence;

[0059] Step 6.6: Retrieve the number of data generated for each category. Clearly define each category The required number of targets to be screened is determined by dividing the total sample sequence generated in step 6.5 into segments based on sample labels. Each category's scoring pool , ,…, each Includes categories All samples, and according to Sort the score pools in descending order for each category. Select samples sequentially, starting with the highest-rated samples, until all samples are selected. 1 sample, if The total number of samples is insufficient If this happens, new data generation is triggered, and steps 3-5 are repeated. The newly generated data is scored based on the weight values ​​of each indicator in the current training phase, and samples are selected sequentially from the highest-scoring samples according to category until all samples are selected. Sample;

[0060] Step 6.7: Mix the selected high-quality generated data with the original training data to form the final training set and train the small model.

[0061] A further improvement in this application is that, in step 6.4, the training state of the small model is divided into a cold start phase, an optimization phase, and a convergence phase, with different phases determined based on the changing trends of the small model's performance indicators. Specifically:

[0062] Cold Start Phase: During this phase, the small model learns rapidly from its initial state, and its performance metrics change quickly. If, in several consecutive iterations, the key performance metrics of the small model, such as the rate of decrease in the loss function value or the rate of improvement in accuracy, exceed a set threshold... ,Right now If so, it is determined to be the cold start phase.

[0063] Optimization phase: When the rate of change of the performance index of the small model tends to stabilize, that is, when the rate of change is less than the threshold of the cold start phase. ,Right now Furthermore, there is still room for optimization, and it is determined to be in the optimization stage.

[0064] Convergence Phase: The rate of change of the performance index of the small model is extremely small, less than the set convergence phase threshold. ( ),Right now This indicates that the process is in the convergence phase.

[0065] Set the weight values ​​for the synthetic data scoring metrics during the startup, optimization, and convergence phases:

[0066] 1) Startup Phase: During this phase, the model parameters are undergoing rapid adjustment, creating an urgent need for data diversity. Therefore, variability is assigned the highest weight of 30% to increase its inclusiveness towards data with different features, allowing the model to access more diverse data. The correctness weight and threshold proximity weight are set to 20% to ensure the quality of basic data while avoiding over-selection; the confidence weight and forgetting rate are set to only 15%, weakening the requirement for predictive certainty and allowing the small model to access more samples with exploratory value.

[0067]

[0068] 2) Optimization Phase: Once the small model enters a stable optimization period, this phase should focus on selecting data with distinct features that can be stably learned by the model, further optimizing the performance of the small model. Therefore, this application increases the correctness weight to 35%, strengthening the screening requirements for data accuracy; the confidence weight is simultaneously increased to 25%, driving the model to focus on samples with stable predictive capabilities; the variability weight is appropriately reduced to 20%, maintaining necessary data diversity; the forgetting rate weight is 15%, and the threshold proximity weight is 5%. This phase focuses on stable learning, emphasizing the elimination of unstable samples that easily cause fluctuations in model memory, forming a screening guideline that emphasizes both accuracy and stability.

[0069]

[0070] 3) Convergence Phase: Given the tendency of small model parameters to stabilize, this phase requires rigorous screening of high-confidence, low-uncertainty data to ensure the small model focuses on high-quality, stable data at the end of training, avoiding performance fluctuations caused by introducing low-quality data. This application adopts a "confidence-driven - correctness-calibrated" weighting strategy: the confidence weight is increased to 40%, forcibly screening high-deterministic samples to solidify the model's decision boundaries; the correctness weight is maintained at 30% to ensure a baseline of data quality; the threshold proximity weight is increased to 15%, strictly filtering samples with ambiguous model classification; the variability weight is compressed to 10%, retaining only key samples that significantly improve generalization ability; and the forgetting factor is 5%, completing model memory consolidation. Simultaneously, a dynamic weight correction factor is introduced. When the validation set experiences three consecutive performance fluctuations, a weight fine-tuning procedure is automatically triggered, adaptively adjusting the weights of fluctuation-related indicators by ±5%.

[0071]

[0072] The beneficial effects of this application are:

[0073] The dual-dimensional adaptive imbalance-aware data generation method proposed in this application breaks through the limitations of traditional single-dimensional data generation. Starting from two key dimensions, namely the number of samples and model performance, it accurately determines the direction and scale of data generation, avoiding data redundancy or missing key category data caused by blind generation. This ensures that data augmentation closely matches the model training requirements, reduces interference from invalid training data, effectively improves model training efficiency, and shortens the training cycle.

[0074] This application employs a logical consistency verification method combining reverse generation and forward derivation, constructing a rigorous data quality screening checkpoint. Through bidirectional verification, it prevents low-quality data, such as those with semantic contradictions and label noise, from entering the training phase, providing logically consistent, accurate, and reliable data samples for model training. This reduces the interference of data noise on model learning, enhances the model's stability during training, and lowers the probability of incorrect judgments and predictions.

[0075] This application presents a data quality assessment method based on small model performance. It introduces multi-dimensional performance indicators to comprehensively score the generated data, overcoming the limitations of traditional single-fixed indicator evaluation. This method enables a comprehensive and detailed assessment of data quality, accurately identifying high-quality data. Combined with a training process-driven dynamic indicator selection mechanism, the selection strategy is dynamically adjusted according to different stages of model training, achieving refined management and efficient utilization of data resources. This avoids the problem of poor training results caused by low-quality data occupying training resources. Attached Figure Description

[0076] Figure 1 This is a flowchart of the data generation method for this application.

[0077] Figure 2 This is a flowchart showing the results of the imbalance assessment and the determination of the generated data scale based on the performance differences of the small models in this application.

[0078] Figure 3 This is a flowchart illustrating the data generation and verification process for this application.

[0079] Figure 4 This is a flowchart illustrating the process of verifying the logical consistency of the generated data in this application.

[0080] Figure 5 This is a flowchart illustrating step 5 of this application, which generates a comprehensive score based solely on new data.

[0081] Figure 6 This is a flowchart illustrating the process of training a small model based on dynamic sample selection using features from the training phase in step 6 of this application. Detailed Implementation

[0082] The embodiments of this application will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details should not be used to limit this application. That is, in some embodiments of this application, these practical details are not necessary. In addition, for the sake of simplicity, some conventional structures and components will be shown in the drawings in a simple schematic manner.

[0083] This application takes the intelligent work order processing application scenario in network operation and maintenance as an implementation case. For the text classification task of network operation and maintenance work orders, the input data is work order data that combines structured and unstructured data. The large model uses Llama3-8B, and the small model uses the lightweight text classification model TextCNN.

[0084] like Figure 1 As shown, this application presents a data generation method based on size-model collaboration for classification tasks, specifically including the following steps:

[0085] Step 1: Data Preparation: Collect the raw dataset. This includes original network maintenance work order texts (UTF-8 encoded), containing user repair requests such as "Broadband outage, error code 678," totaling 5000 entries. See the example below:

[0086] [{"ticket_id": "TK20240501001", "content": "My home broadband suddenly went down, restarting the router didn't help, it shows dial-up failed", "label": ""},{"ticket_id": "TK20240501002", "content": "I've exceeded my mobile data limit, but my data plan shows 5GB remaining. Please verify", "label": ""}]

[0087] The work order classification and labeling system is shown in the following example:

[0088] "0": "Category#Personal Services# Subcategory#Internet Services","1": "Category#Personal Services# Subcategory#Service-related SMS Reminders","2": "Category#Personal Services# Subcategory#Personal Landline","3": "Category#Personal Services# Subcategory#Suspension and Reactivation","4": "Category#Personal Services# Subcategory#Secondary Number Services","5": "Category#Personal Services# Subcategory#Number Recycling","6": "Category#Personal Services# Subcategory#Basic Planning","7": "Category#Personal Services# Subcategory#Basic Products","8": "Category#Personal Services# Subcategory#Value-Added Planning","9": "Category#Personal Services# Subcategory#Customer Data Management","10": "Category#Personal Services# Subcategory#Account Opening","11": "Category#Personal Services# Subcategory#Number Portability","12": "Category#Personal Services# Subcategory#Intra-Provincial Number Portability","13": "Category#Personal Services# Subcategory#Business Inquiry","14": "Category#Personal Services# Subcategory#SIM Card Replacement","15" Category #Personal Business# Subcategory #Transfer of Ownership

[0089] Preprocessing the raw dataset includes data cleaning, feature extraction, label annotation, and dataset partitioning to ensure basic data quality and usability.

[0090] 1) Data Cleaning: This includes removing sensitive information such as phone numbers, IP addresses, and special symbols like "@#￥", standardizing text formatting (converting to lowercase), and removing redundant spaces. For example: Original text: 138xxxx5678 Fault report: Broadband outage!! Restarted the router, still no luck. After cleaning: Fault report: Broadband outage! Restarted the router, still no luck.

[0091] 2) Feature extraction: This includes key field extraction, text feature extraction, and length feature statistics. For example, key field extraction refers to extracting the content and label fields from the original work order text; text feature extraction converts the text into a feature vector with a dimension of 500, reflecting the weight of keywords, such as the weight of core words like "internet outage," "traffic," and "dialing"; length feature extraction calculates the number of characters in the text, reflecting the level of detail in the work order description, and is used to assist in screening effective samples.

[0092] 3) Labeling: Manual labeling or automatic labeling combined with manual verification can be used. Based on the labeling system, the cleaned work order texts are manually categorized and labeled. For example, the text "Fault Report: Broadband outage! I restarted the router, but it still doesn't work" is labeled as "Category #Network Fault #Subcategory #Broadband Outage".

[0093] 4) Dataset partitioning: For example, randomly split the labeled samples in an 8:2 ratio, resulting in 4000 original training samples and 1000 original test samples. The original training dataset is used for training the small model and generating examples for the large model, while the original test dataset is used to evaluate the performance of the small model. [

[0095] {"text": "I can't cancel the data sharing feature for phone number 13645658905. I don't know which phone number it was shared with in September 2023. This caused my number to be suspended, saying the number was already in use. I'm requesting to cancel. I tried subscribing to a data package but still can't get a refund. Please help me resolve this, thank you.", "label": "Category #Personal Services #Subcategory #Internet Services"},

[0096] {"text": "A user called to say they do not accept the mobile channel. The front-end does not have permission to handle this. Please assist with the back-end. Thank you.", "label": "Category #Personal Services# Subcategory #Internet Services"},

[0097] {"text": "User 13846955237 is a distributor. The system shows multiple refund orders. When processing the refund in the past, it prompted "The customer has outstanding shared data for the current month; please expedite the process." The multiple refunds cannot be processed because they were handled by an agent. The user subscribed to a 20GB shared data service, but refunds are still not possible.", "label": "Category #Personal Services# Subcategory #Internet Services"} ]

[0099] Step 2: Determine the scale of the generated data: Based on the evaluation results of the sample imbalance of the original training dataset and the performance differences of small models, determine the scale of the newly generated data for different categories, such as... Figure 2 As shown, the specific steps include:

[0100] Step 2.1: Count the number of data points in each category in the original training dataset;

[0101] Step 2.2: Use the original category ratio and Gini coefficient The multi-dimensional indicators comprehensively quantify the quantity of data in each category, forming an assessment result of the imbalance in the overall distribution of the original training dataset.

[0102] Step 2.2.1: For a certain category in the original training dataset The proportion of its original categories :

[0103]

[0104] in, It is a category The amount of data, It represents the total number of all data in the original training dataset. By calculating the proportion of all data categories, we can intuitively understand the percentage of each category's sample size in the total dataset.

[0105] Step 2.2.2, Gini coefficient ,in, Also a category Category ratio, It is the total number of categories in the original training dataset, and the Gini coefficient. The value of Gini coefficient ranges from 0 to 1. The larger the Gini coefficient, the more severe the imbalance in the data.

[0106] Step 2.3: Set an imbalance threshold. Based on the imbalance assessment results of the original training dataset obtained in Step 2.2, decide on a data generation allocation strategy. When the imbalance is in the low range, i.e., the Gini coefficient... When the imbalance level is less than or equal to the Gini coefficient threshold, an average distribution strategy is adopted. When the imbalance level is in the high range, i.e., the Gini coefficient is lower, the average distribution strategy is adopted. When the imbalance level exceeds the threshold, a proportional allocation strategy is adopted, meaning that the number of data generated for each category is inversely proportional to the category ratio.

[0107] In step 2.3, the data generation quantity allocation strategy is as follows:

[0108] The average distribution strategy is as follows: if the number of data generated for each category is the same, then the basic generation weight for each category is... : ;

[0109] The proportional allocation strategy specifically involves: adjusting the original category proportions... Weights are generated based on mapping :

[0110]

[0111] in, As a quantity balance factor, Quantity balance factor It controls the degree of nonlinearity in the mapping. The larger the value, the faster the weight of the subcategories increases.

[0112] Step 2.4: Based on the assessment results of the degree of distribution imbalance, preliminarily determine the amount of basic generated data for each category to ensure that the generated data initially alleviates the category imbalance problem in terms of quantity.

[0113] Step 2.5: Introduce a small model performance feedback dimension. Combining the basic generation weights and small model performance metrics, construct a generated data sample weight model to achieve a secondary adjustment of the amount of basic generated data. Specifically:

[0114]

[0115] in, Generate weights based on the base. The total number of categories in the original training dataset. As a quantitative balancing factor, this invention preferentially uses the F1-score index to measure the model's performance across categories, denoted as . In addition, this application may also use metrics such as Recall and AUC. This is a performance adjustment factor used to control the adjustment intensity. The larger the value, the greater the adjustment for categories with poor performance. The normalized adjusted weights are used to calculate the number of generated data points for each category. :

[0116]

[0117] in, To generate the total amount of data;

[0118] Step 2.6: Establish real-time monitoring and dynamic optimization of the generation process: During the overall data generation process, continuously track the degree of imbalance in the distribution of generated data and the performance fluctuations of small models on new data. Using a preset imbalance improvement evaluation function, evaluate the effectiveness of the current generation strategy in real time. The evaluation result is calculated using the following formula:

[0119]

[0120] in, The Gini coefficient for the currently generated data. This represents the improvement in F1-score for the smaller model on the new data. and These are the weighting parameters for quantity balancing and performance improvement, respectively. If the evaluation results... If the value falls below the evaluation validity threshold, an automatic parameter generation iteration mechanism is triggered. The calculation weights and performance adjustment coefficients of the base generation quantity are dynamically adjusted. This is a quantity balancing parameter used to control... The proportion of the assessment Performance is a performance-enhancing weight parameter used to control... The evaluation proportion is determined by achieving a synergistic optimization of sample quantity balance and model performance balance, ensuring that the generated data accurately matches the dynamic needs of model training. If the improvement in the generated data quantity balance index is insufficient, the quantity balance factor in the calculation of the basic generation amount is increased. This increases the number of generated subcategories; however, for cases where performance improvement is not significant for small models, such as... <3%, increase performance adjustment factor This strengthens the positive adjustment of the number of classes generated with poor performance, supplementing with high-quality samples. Assume the F1-score of the small model (TextCNN) on the test set is:

[0121]

[0122] According to the formula for calculating the weights of the generated samples: In this embodiment, the performance balancing factor The adjusted generation weights are:

[0123]

[0124] Finally, the number of generated records is allocated based on the generation weight. Assuming a total generation quantity M = 1000 records, then: "Dial-up failure: 510 records, Package balance error: 290 records, Traffic metering anomaly: 110 records, Broadband disconnection: 90 records."

[0125] Step 3: Based on the calculated distribution of generated data, use a large model such as Llama3-8B to generate new data for each category. The prompt word template is shown below:

[0126] Task: Generate network maintenance work order text (Internet access service type).

[0127] need:

[0128] 1. Tag constraints: The text must belong to the category "#Personal Services# Subcategory#Internet Services", focusing on the Internet problems encountered by users, such as abnormal data sharing, speed limit, signal failure, etc.

[0129] 2. Content Requirements:

[0130] Include the specific mobile phone number (which can be virtual, such as 138xxxx5678) and the time / scene of the problem (such as "at home in September 2024").

[0131] Describe user actions such as "restarting the router" or "trying to subscribe to a data package" and unresolved issues (such as "data cannot be refunded" or "speed limit cannot be lifted").

[0132] The ending should clearly state the request, such as "Please help me resolve this, thank you."

[0133] 3. Formatting requirements:

[0134] Text length: 50-150 words (concise and clear, in line with the style of a real work order).

[0135] Disable sensitive information: Hides real name and ID number, but retains the first 3 digits and last 4 digits of mobile phone number (e.g., 1385678).

[0136] Example (can be used as a reference but requires innovation):

[0137] User 1385678 reported: Unable to unsubscribe from data sharing. In September 2024, it was unclear which secondary SIM card the data was shared with. The primary SIM card is being speed-limited due to the sharing issue. Attempts to subscribe to data packages have also failed. Requesting assistance. Output: Generate one compliant work order text (strictly adhering to tags and format).

[0138] Step 4: Data Consistency Verification: Based on the reverse generation and forward derivation verification methods, verify the logical consistency of the new generated data from Step 3 to ensure the reliability of the new data ultimately used for model training. For example... Figure 4 As shown, basic quality checks are performed on new data, including format specification checks, data length limit checks, sensitive information security checks, similarity checks, and other basic verification methods, to quickly remove obviously invalid data.

[0139] Each generated data text after filtering is input into Llama3-8B for predicted labels. If the predicted label matches the original label, it is retained; otherwise, it is discarded. The predicted label generation prompts are shown below:

[0140] Task: Generate corresponding tags for network maintenance work order text.

[0141] Background: There exists a predefined work order classification tag system, formatted as "Category#Main Category Name#Subcategory Name", for example:

[0142] #Network Failure# Broadband Outage

[0143] #Network Failure# Dial-up failed

[0144] #Traffic Issues# Traffic metering anomalies

[0145] #Data usage issue# Incorrect remaining data allowance in data plan

[0146] Input: Network maintenance work order text content

[0147] {text content}

[0148] Require:

[0149] Analyze the core issues of the text description and select the most matching tags from the above tag system.

[0150] If the text content matches multiple tags, select the one that best matches.

[0151] Output only the predicted labels, strictly following the format "class#major category name#minor category name", without adding any additional explanations.

[0152] Output example:

[0153] #Network Failure# Dial-up failed

[0154] Step 5: Based on the performance of the small model during the training process of the newly generated data, score the quality of the newly generated data to form a comprehensive score specific to the newly generated data. For example... Figure 5 As shown, the directional training period is set to 5 epochs. A TextCNN mini-model is trained on the newly generated data after verification in step 7. The prediction results and multi-dimensional performance indicators of the mini-model for each synthetic data point are tracked and recorded within this directional training period. After the evaluation period, the average confidence, variability, accuracy, forgetting rate, and threshold proximity are calculated.

[0155] Taking a newly generated sample as an example: "Dial-up failed, error code 678, network cable connection is normal, optical signal is stable but cannot access the internet," the original label is "Category #Network Failure# Subcategory #Dial-up Failure". Assume that the confidence level of the small model's prediction results for this sample over 5 training epochs is as shown in the table below, and the prediction results for all 5 epochs are consistent with the original label.

[0156] Average confidence level:

[0157]

[0158] Variability:

[0159]

[0160] Correctness:

[0161]

[0162] Forgetting rate:

[0163]

[0164] Calculate the average confidence level of the small model for the generated data. Variability Correctness Forgetting rate Threshold proximity :

[0165] Threshold proximity: Assuming a decision threshold It is 0.5.

[0166]

[0167] in, This represents the mathematical expectation operation. This represents the total number of training cycles. It generates a sequence of confidence scores for all data throughout the entire training period. To generate data Confidence level for each period, This represents the standard deviation calculation. Indicates the indicator function, if the first The value is 1 if the predicted label for a period matches the original label, and 0 otherwise. Indicates the small model in the first... The periodicity of the data's predictive label; The original labels representing the generated data. Indicates the first The cycle prediction was correct. Indicates the first The cycle prediction was incorrect; This represents the logical AND operation. Indicates the logical NOT operation. This is the decision threshold;

[0168] Using the calculated average confidence level Variability Correctness Forgetting rate Threshold proximity The overall score for each piece of composite data is calculated using a weighted summation formula:

[0169] .

[0170] Step 6: Dynamic sample selection based on training phase features, training a small model based on the original training dataset and newly generated data. This includes the following steps:

[0171] Step 6.1, Data Initialization: Merge the newly generated data with the original data to form the initial training dataset. ,in For the newly generated data, This is the original data;

[0172] Step 6.2, Model Training and Monitoring: Using the initial training dataset Iteratively train a small model while monitoring its performance metrics, such as accuracy, recall, and loss function value, denoted as... , For the number of indicators;

[0173] Step 6.3, Judging the Number of Training Iterations: Determine whether the preset number of training iterations has been reached. The count for each iteration is ,when If the number of training iterations has been reached, training ends; otherwise, iteration continues.

[0174] Step 6.4, Assessment Cycle Determination: Determine whether the assessment cycle has been reached. ,Every Each iteration performs a training state evaluation; when the condition is met... (When the period is measured by the number of iterations), it is determined that the evaluation period has been reached, and the model state evaluation process is triggered; if "no", return to step 6.2 to continue iterating; if "yes", enter the model training state evaluation, and divide the model training state into the cold start stage, the optimization stage, and the convergence stage.

[0175] Step 6.5: Filter data based on sample scores for the cold start, optimization, and convergence phases: Set differentiated screening thresholds for the cold start, optimization, and convergence phases, based on the comprehensive scores. Synthetic data below the differential screening threshold are removed to form the total sample sequence;

[0176] Step 6.6: Retrieve the number of data generated for each category. Clearly define each category The required number of targets to be screened is determined by dividing the total sample sequence generated in step 6.5 into segments based on sample labels. Each category's scoring pool , ,…, Each Includes categories All samples, and according to Sort the score pools in descending order for each category. Select samples sequentially, starting with the highest-rated samples, until all samples are selected. 1 sample, if The total number of samples is insufficient If this happens, new data generation is triggered, and steps 3-5 are repeated. The newly generated data is scored based on the weight values ​​of each indicator in the current training phase, and samples are selected sequentially from the highest-scoring samples according to category until all samples are selected. Sample;

[0177] Step 6.7: Mix the selected high-quality generated data with the original training data to form the final training set and train the small model.

[0178] This application dynamically adjusts the weights of each indicator based on the different data requirements at different stages of model training and the differences in classification tasks. In this embodiment, the indicator weights for each stage are shown in the table below:

[0179]

[0180] 1) Initial Phase: Emphasis is placed on data diversity, relaxing the requirements for accuracy and reducing its weight, while increasing the weight of variability and threshold proximity. Therefore, the weights of each indicator are:

[0181] , , , ,

[0182]

[0183] 2) Optimization Phase: Emphasizing high reliability, significantly improving the weighting of accuracy and average confidence, requiring high levels of both label matching and prediction confidence. Therefore, the weights of each indicator are:

[0184] , , , ,

[0185]

[0186] 3) Convergence Phase: Emphasizing stability, strengthening the weighting of variability and forgetting rate, and reducing dependence on threshold proximity. Therefore, the weights of each indicator are:

[0187] , , , ,

[0188]

[0189] The average score, KL divergence, and defect rate of the sample set generated in this iteration are calculated and fed back to the large model, thereby helping the large model optimize its generation strategy and further improve the quality of the synthesized samples. The specific implementation is as follows:

[0190] Average score: Based on the quality score of each generated sample, the average score of all generated data in this round is calculated.

[0191] KL divergence: This calculates the deviation of the feature distributions between generated synthetic samples of the same type and the original real samples. A smaller KL divergence value indicates that the distributions of the two samples are closer. Implementation details:

[0192] Core features of similar samples, such as keywords, sentence length, and part-of-speech ratio, were selected, and the feature frequency distributions of the original real samples and synthetic samples were statistically analyzed. For example, keywords for a certain category of data include: network failure, latency, server, connection failure, insufficient bandwidth, etc. The probability of the same keywords appearing in the original dataset and the synthetic dataset was statistically analyzed, as shown in the table below:

[0193]

[0194] in, As a key feature, The feature probability distribution of the original real sample. This represents the characteristic probability distribution of the synthesized sample.

[0195] Calculate the KL divergence components for each feature and sum them to obtain the total KL divergence. The calculation formula is as follows:

[0196]

[0197] In this example, the KL divergence is approximately 0.02 (≤0.1), indicating that the keyword distribution of the synthetic sample is close to that of the real sample.

[0198] Defect rate calculation: The defect rate for this iteration is calculated based on the number of generated samples that were removed and the total number of generated samples.

[0199]

[0200] When the quality indicators of a certain type of sample meet the following conditions during the targeted training and evaluation period:

[0201] Average score ≥ target value, such as 0.9;

[0202] The KL divergence between the feature distribution of similar samples and the real data is ≤0.1, which is close to the real distribution;

[0203] Defect rate ≤ 5%;

[0204] The current generation strategy, such as prompt word templates and parameter configurations, will be solidified as the baseline strategy for this category, and adjustments will only be triggered when business requirements change. Otherwise, the large model can adjust the generation strategy in stages based on the specific defect type of the feedback signal.

[0205] Different training stages are used to select synthetic data. First, differentiated selection thresholds are set for each training stage, and then the samples from each stage are scored accordingly. Synthetic data below the current training stage screening threshold will be removed, and suitable synthetic data will be selected for each training stage until the specified number of data to be generated for each category is reached; otherwise, a new round of data generation will be triggered.

[0206] The selected high-quality generated data is mixed with the original training data to form the final training set, and the small model is then fully trained.

[0207] During training, key metrics such as loss function value, accuracy improvement rate, and parameter update magnitude are collected in real time to construct stage identification feature vectors. Based on preset stage determination rules, training stages are automatically divided. Training sets selected for each stage are used in different training stages.

[0208] This application employs a dynamic balancing generation strategy based on a dual-dimensional sample-performance approach, integrating sample imbalance and small model performance differences to achieve precise and adaptive data generation. It utilizes logical consistency verification through reverse generation and forward derivation to prevent low-quality data from entering the training phase. A synthetic data quality scoring method based on small model performance feedback enables precise quantitative scoring of the generated data. Furthermore, a dynamic sample selection mechanism based on training phase features allows for precise application of the synthetic data through phase-adaptive rule adjustments.

[0209] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A data generation method based on size-model collaboration for classification tasks, characterized in that: The data generation method specifically includes the following steps: Step 1, Data Preparation: Collect the original dataset, preprocess the original dataset, classify and label the preprocessed data, clarify the category to which each data belongs, and divide it into the original training dataset and the original test dataset. Step 2: Determine the number of newly generated data sets: Combine the sample imbalance evaluation results of the original training dataset with the performance differences of small models to determine the number of newly generated data sets for each category; Step 3, Data Generation: Based on the scale of new data generated for different categories determined in Step 2, the large language model is used to generate a corresponding number of new data for different categories. This generation process is iterative, generating a batch of new data each time. Step 4: Generate Data Consistency Verification: Based on the reverse generation and forward derivation verification method, verify the logical consistency of the newly generated data generated in Step 3; Step 5: Based on the performance of the small model during the training process of the newly generated data, a quality score is given to the newly generated data to form a comprehensive score that is only for the newly generated data; Step 6: Train a small model based on the original training dataset and the newly generated data, wherein: Step 2 specifically includes the following steps: Step 2.1: Count the number of data points in each category in the original training dataset; Step 2.2: Use the original category ratio and Gini coefficient The multi-dimensional indicators comprehensively quantify the quantity of data in each category, forming an assessment result of the imbalance in the distribution of the original training dataset. Step 2.3: Set an imbalance threshold. Based on the imbalance assessment results of the original training dataset obtained in Step 2.2, determine the data generation quantity allocation strategy and generate basic generation weights. When the degree of imbalance is in the low range, i.e., the Gini coefficient When the imbalance level is less than or equal to the Gini coefficient threshold, an average distribution strategy is adopted. When the imbalance level is in the high range, i.e., the Gini coefficient is lower, the average distribution strategy is adopted. When the imbalance level exceeds the threshold, a proportional allocation strategy is adopted, that is, the number of data generated in each category is inversely proportional to the category ratio. Step 2.4: Based on the assessment results of the degree of uneven distribution, preliminarily determine the amount of basic generated data for each category; Step 2.5: Introduce a small model performance feedback dimension, combined with basic generation weights. Based on the performance metrics of the small model, a weighted model for the generated data samples is constructed to achieve a secondary adjustment of the amount of basic generated data, specifically as follows: ， in, For category The basic generation weights, For the adjusted category The generation weights, The total number of categories in the original training dataset. As a quantity balance factor, Performance metrics of the small model on the original test dataset The performance adjustment factor is based on the generated weights after secondary adjustment. Calculate the amount of generated data for each category. : ， in, To generate the total amount of data; Step 2.6: Establish real-time monitoring and dynamic optimization of the generation process: Continuously track the degree of imbalance in the distribution of generated data and the performance fluctuations of small models on new data. Using a preset imbalance improvement evaluation function, evaluate the effectiveness of the current generation strategy in real time. The evaluation result is calculated using the following formula: ， in, The Gini coefficient for the currently generated data. For small models on new data Increase value, and These are the weighting parameters for quantity balancing and performance improvement, respectively. If the evaluation results... If the value falls below the validity assessment threshold, the parameter generation iteration mechanism is automatically triggered.

2. The data generation method based on size-model collaboration for classification tasks according to claim 1, characterized in that: Step 2.2 specifically includes the following steps: Step 2.2.1: For a certain category in the original training dataset The proportion of its original categories : ， in, It is a category The amount of data, It is the total number of all data in the original training dataset; Step 2.2.2, Gini coefficient ,in, The Gini coefficient represents the total number of classes in the original training dataset. The higher the Gini coefficient, the more severe the data imbalance.

3. The data generation method based on size-model collaboration for classification tasks according to claim 1, characterized in that: In step 2.3, the data generation quantity allocation strategy is as follows: The average distribution strategy is as follows: if the number of data generated for each category is the same, then the basic generation weight for each category is... : ; The proportional allocation strategy specifically involves: adjusting the original category proportions... Weights are generated based on mapping : ， in, As a quantity balance factor, .

4. The data generation method based on size-model collaboration for classification tasks according to claim 1, characterized in that: Step 4 specifically involves: inputting the feature portion of each generated data point into the large language model to generate a predicted label for that data point; comparing the original label and the predicted label of the generated data point: if the original label and the predicted label are completely consistent, the data is determined to be logically consistent and is retained in the dataset that has passed the verification; if the original label and the predicted label are inconsistent, it indicates that there is a logical problem of semantic contradiction or misassociation of labels in the data, and the data sample is marked as low-quality data and removed.

5. The data generation method based on size-model collaboration for classification tasks according to claim 1, characterized in that: In step 5, a comprehensive score is generated specifically for the newly generated data, which includes the following steps: Step 5.1: Set a specific training period T and use only the newly generated data to train the small model in a targeted manner. During the training process, record the prediction results and confidence levels of the small model for each newly generated data in real time. Step 5.2: Determine if the number of training sessions has reached the training cycle. If the training cycle is not reached Continue iterative training until the training cycle is reached. Complete the targeted training of the small model; Step 5.3: After the training period ends, calculate the average confidence score of the small model on the generated data. Variability Correctness Forgetting rate Threshold proximity : ， ， ， ， ， ， in, This represents the mathematical expectation operation. This represents the total number of training cycles. It generates a sequence of confidence scores for all data throughout the entire training period. To generate data Confidence level for each period, This represents the standard deviation calculation. Indicates the indicator function, if the first The value is 1 if the predicted label for a period matches the original label, and 0 otherwise. Indicates the small model in the first... The periodicity of the data's predictive label; The original labels representing the generated data. Indicates the first The cycle prediction was correct. Indicates the first The cycle prediction was incorrect; This represents the logical AND operation. Indicates the logical NOT operation. This is the decision threshold; Step 5.4: Use the average confidence level calculated in Step 5.

3. Variability Correctness Forgetting rate Threshold proximity The overall score for each piece of composite data is calculated using a weighted summation formula: 。 6. The data generation method based on size-model collaboration for classification tasks according to claim 5, characterized in that: Step 6 specifically includes the following steps: Step 6.1, Data Initialization: Merge the newly generated data with the original data to form the initial training dataset; Step 6.2, Model Training and Monitoring: Iteratively train the small model using the initial training dataset, while monitoring the performance metrics of the small model, denoted as... , For the number of indicators; Step 6.3, Judging the Number of Training Iterations: Determine whether the preset number of training iterations has been reached. The count for each iteration is ,when If the number of training iterations has been reached, training ends; otherwise, iteration continues. Step 6.4, Assessment Cycle Determination: Determine whether the assessment cycle has been reached. ,Every Each iteration performs a training state evaluation; when the condition is met... If the evaluation cycle has been reached, the model state evaluation process is triggered; if the evaluation result is negative, return to step 6.2 to continue iterating. If the judgment result is yes, proceed to the model training state evaluation, and divide the model training state into the cold start stage, the optimization stage, and the convergence stage. Step 6.5: Filter data for the cold start, optimization, and convergence phases: Set differentiated filtering thresholds for the cold start, optimization, and convergence phases, based on the comprehensive score. Data generated below the differential screening threshold is removed to form the total sample sequence; Step 6.6: Retrieve the number of data generated for each category. Clearly define each category The required number of targets to be screened is determined by dividing the total sample sequence generated in step 6.5 into segments based on sample labels. Each category's scoring pool , ,…, Each Includes categories All samples, and according to Sort the score pools in descending order for each category. Select samples sequentially, starting with the highest-rated samples, until all samples are selected. 1 sample, if The total number of samples is insufficient If this happens, new data generation is triggered, and steps 3-5 are repeated. The newly generated data is scored based on the weight values ​​of each indicator in the current training phase, and samples are selected sequentially from the highest-scoring samples according to category until all samples are selected. Sample; Step 6.7: Mix the selected high-quality generated data with the original training data to form the final training set and train the small model.

Images