A dataset construction method for post-training of a vertical large model
By identifying and processing outliers in the training dataset of large vertical models, performing data augmentation and discarding, the problem of dataset imbalance was solved, the model's coverage and generalization ability in key edge scenarios were improved, and a high-precision large vertical model in the field of hydropower construction and energy storage was trained.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- POWERCHINA BEIJING ENG CORP
- Filing Date
- 2026-03-31
- Publication Date
- 2026-06-09
AI Technical Summary
The training dataset for large-scale vertical models in the fields of hydropower construction and energy storage suffers from uneven data distribution and a small number of long-tail data samples, resulting in insufficient coverage of key edge scenarios and limited generalization ability of the models.
By analyzing the frequency of each sample category in the statistical dataset, outliers are identified and differentiated. Data augmentation is performed on valuable long-tail categories, while noisy or invalid data categories are discarded. New samples are generated using a large-scale language model guided by experts to ensure the balance and effectiveness of the dataset.
It achieves a balanced distribution of the dataset, improves the coverage of long-tail categories, avoids noise data pollution, trains a more accurate vertical category model, and solves the problems of insufficient coverage and limited generalization ability of the model in key edge scenarios.
Smart Images
Figure CN122173931A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of vertical large model technology, specifically involving a method for constructing a dataset for post-training of a vertical large model. Background Technology
[0002] In the training of large-scale vertical models in the field of hydropower construction and energy storage, the current training dataset suffers from severe imbalance in data distribution, exhibiting a clear long-tail data distribution characteristic: a large number of training samples are concentrated in common business scenarios in this field, such as lithium battery energy storage system operation and maintenance, photovoltaic power plant design specifications, and standard substation inspection reports; while training data related to rare faults, new technology applications, and niche equipment in this field are extremely scarce. Rare faults include data related to large-scale energy storage fires and thermal runaway chain reactions, new technology applications include data related to hydrogen energy storage, flow batteries, and grid-type inverters, and niche equipment includes data related to domestic flywheels and off-grid microgrid controllers.
[0003] The long-tail distribution of the aforementioned datasets results in insufficient coverage of key edge scenarios when using such datasets to train large-scale vertical models in the fields of hydropower construction and energy storage. This severely limits the model's generalization ability and fails to meet the actual training and application needs of large-scale vertical models in this field.
[0004] In view of this, the present invention is hereby proposed. Summary of the Invention
[0005] To address the aforementioned technical problems in existing technologies, this invention provides a method for constructing a dataset for post-training of a large vertical model, which solves the problems of a small number of long-tail samples, noisy data, and unbalanced data distribution in the large vertical model dataset.
[0006] To achieve the above objectives, the technical solution of the present invention is as follows:
[0007] A method for constructing a dataset for post-training of a large vertical model includes: S1. Calculate the mean of the dataset based on the frequency of each sample category in the data set. and standard deviation ; S2, based on the mean and standard deviation Identify the outlier categories in the dataset; S3. Perform attribute determination on the outlier category to identify it as a valuable long-tail category or a noise / invalid data category. S4. Perform differential processing on the judgment results, perform data augmentation on valuable long-tail categories, and discard samples of noisy / invalid data categories; S5. After completing the differentiation process, a balanced large-scale vertical model is obtained, followed by the training dataset.
[0008] Furthermore, in step S1, the frequency of each sample category in the statistical dataset is... ,in Indicate category The number of samples.
[0009] Furthermore, the mean The calculation formula is:
[0010] The standard deviation The calculation formula is:
[0011] in, For the amount of data, For the number of data points.
[0012] Furthermore, the specific criteria for outlier identification are as follows: For each sample category Calculate the deviation between the frequency and the mean, if it satisfies Then determine the category. This is an outlier.
[0013] Furthermore, the determination of outlier attributes specifically involves: Determine the category of outliers It can be a valuable long-tail category or a noise / invalid data category; wherein, the noise / invalid data category includes the sample category corresponding to sensor fault logs.
[0014] Furthermore, the valuable long-tail categories are sample categories related to rare faults, new technology applications, or niche equipment in the field of hydropower construction and energy storage business; The rare faults include large-scale energy storage fires and thermal runaway chain reactions; the new technology applications include hydrogen energy storage, flow batteries, and grid-connected inverters; and the niche devices include domestically produced flywheels and off-grid microgrid controllers.
[0015] Furthermore, outliers identified as noise / invalid data are discarded, specifically: Simply remove all samples in the noisy / invalid data category to obtain a new dataset after discarding that category. ,and ,in, For the original dataset, The number of samples for the noise / invalid data category.
[0016] Furthermore, data augmentation is performed on outliers identified as valuable long-tail categories, specifically including: New samples are generated using a large, expert-guided language model; the number of new samples needed to be generated is then calculated. and the generated A new sample is added to this category, causing the frequency of samples in this category to fall within the range. Inside.
[0017] Furthermore, the formula for calculating the number of new samples k to be generated is:
[0018] After adding new samples, the frequency of the categories was enhanced. ,and .
[0019] Furthermore, after data augmentation, the valuable long-tail categories satisfy... ; in, This represents the mean of the augmented dataset for this category. The standard deviation of the dataset after augmentation for this category of data.
[0020] The beneficial effects of this invention are as follows: pass The algorithm modulates the data distribution of the dataset trained on the vertical large-scale model, effectively solving problems such as the small number of long-tail samples, noisy data, and unbalanced data distribution in the dataset, making the data distribution of the dataset more reasonable. At the same time, by augmenting the data of valuable long-tail categories and discarding noisy and invalid data categories, it achieves the quantification of outliers, which not only improves the coverage of long-tail categories, but also avoids the pollution of the dataset by noisy data. Finally, by training the vertical large-scale model using the optimized balanced dataset, it can solve the problems of insufficient coverage of key edge scenarios and limited generalization ability of the original model, and train a vertical large-scale model with higher accuracy in the field of hydropower construction and energy storage. Attached Figure Description
[0021] Figure 1 This is a flowchart illustrating the method for constructing a dataset for post-training of a large vertical model, as provided in an embodiment of the present invention. Detailed Implementation
[0022] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0023] It should be noted that, unless otherwise specifically stated, the relative arrangement and numerical expressions of the components and steps described in these embodiments should not be construed as limiting the scope of the invention.
[0024] The following description of exemplary embodiments is merely illustrative and is not intended to limit the invention or its application or use in any way. Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail herein, but where applicable, such techniques, methods, and apparatus should be considered part of this specification.
[0025] Example See Figure 1 , Figure 1 This is a flowchart illustrating a method for constructing a dataset for post-training of a large vertical model, as proposed in this invention. The algorithm quantifies and controls the frequency distribution of samples in the training dataset D of the large vertical model. By completing the steps of statistical calculation, outlier identification, outlier separation, and differential data processing, it achieves balanced construction of the dataset and solves the problems of long-tail distribution, sample imbalance, and noise pollution in the dataset of this field.
[0026] In this embodiment, the normal distribution characteristic is relied upon to achieve... In practical applications of the algorithm, the probability density function of the normal distribution is:
[0027] In the above formula, The probability density value, The mean of the sample frequencies in the dataset. The standard deviation of the sample frequencies in the dataset. It is a natural constant; according to the inherent properties of the normal distribution, approximately 99.7% of the data will fall within the interval [0, 1]. Within this implementation, samples exceeding this range are identified as outliers, and differential processing is performed on these outliers. Specific steps may include: S1. Calculate the mean of the dataset based on the frequency of each sample category in the data set. and standard deviation Specifically, this includes: First, statistical analysis of large models in vertical categories is performed before training the dataset. The frequency of each sample category is used to obtain the sample frequency set. ,in Represents the dataset Medium category The number of samples.
[0028] Based on the above set of sample frequencies, the mean of the dataset was calculated. and standard deviation Among them, the mean The calculation formula is:
[0029] The formula for calculating the standard deviation σ is:
[0030] in, For the amount of data, For the amount of data, Assign an individual number, For a single sample value, For category The frequency of.
[0031] S2, based on the mean and standard deviation Identify the outlier categories in the dataset; specifically including: The mean calculated based on step S1 and standard deviation For the dataset Each sample category in Calculate the deviation between the frequency and the mean for each occurrence, and determine whether the deviation satisfies the formula. If the formula holds true, then the sample category is determined. For dataset Outliers are those sample classes that are outside the 99.7% confidence interval of the normal distribution, and are the main contributing factors to the imbalance in the dataset distribution.
[0032] S3. Perform attribute determination on the outlier categories to identify them as either valuable long-tail categories or noise / invalid data categories; specifically including: The outlier categories identified in step S2 Perform attribute determination to separate outliers. The specific determination criteria are: if the outlier category... Valuable long-tail categories belonging to the hydropower construction and energy storage business sector are identified as suitable for data augmentation; outlier categories... If the data belongs to the category of noise or invalid data, then the data in that category must be discarded.
[0033] The valuable long-tail categories specifically include sample categories related to rare faults, novel technology applications, and niche devices within the field. Rare faults include categories related to large-scale energy storage fires and thermal runaway chain reactions; novel technology applications include categories related to hydrogen energy storage, flow batteries, and grid-type inverters; and niche devices include categories related to domestic flywheels and off-grid microgrid controllers. Samples in this category are effective samples required for model training, with the only problem being the scarcity of samples. The noisy or invalid data categories specifically include sample categories such as sensor fault logs that have no practical training value. Samples in this category cannot provide effective information for model training and will also cause noise pollution to the dataset.
[0034] Different processing methods are determined for outlier categories with different attributes: For valuable long-tail categories, new samples with semantic fidelity and technical compliance are generated using expert-guided large language models (LLM) to achieve data augmentation for this category; for noisy or invalid data categories, no sample generation is performed, and data is directly discarded.
[0035] Valuable long-tail data types are single-digit negative samples in business scenarios; noise data types are dirty data that appears in the environment or during data collection.
[0036] S4. Perform differential processing on the judgment results, perform data augmentation on valuable long-tail categories, and discard samples from noisy / invalid data categories; specifically including: For the valuable long-tail categories determined in step S3 The core of data augmentation is to generate a specified number of new samples so that the frequency of samples in that category falls within a certain range after augmentation. Within this range, it meets the 99.7% confidence interval requirement for a normal distribution.
[0037] First, calculate the number of new samples k to be generated, using the following formula:
[0038] in, For floor operations, Three times the standard deviation, The frequency of category j (the original sample frequency of this valuable long-tail category); will generate After adding a new sample to the category, the frequency of the enhanced samples in that category is obtained. , .
[0039] After completing the supplementation of new samples, it is necessary to ensure that the frequency of the enhanced samples meets the following requirements:
[0040] in, For the enhanced category Sample frequency; For the enhanced category The mean; This represents the enhanced standard deviation.
[0041] S5. After completing the differentiation process, a balanced large-scale training dataset for each vertical category is obtained; specifically including: For the noise or invalid data category determined in step S3 Perform data discarding and directly remove the dataset. Medium category All samples were used to obtain a new dataset after discarding that category. The calculation formula is as follows:
[0042] in, This is the training dataset for the large vertical model after removing noisy or invalid data categories. For the original dataset, This is a set of samples for the noise category.
[0043] In summary, the specific implementation of this invention is based on the 99.7% confidence interval of the normal distribution. It uses the 3σ algorithm to quantitatively identify and process outliers in the dataset, achieves accurate data augmentation for valuable long-tail categories, and thoroughly removes noisy or invalid data categories. Ultimately, it achieves a balanced distribution of the post-training dataset for large-scale vertical models in the hydropower construction and energy storage business field. This effectively improves the sample coverage of long-tail categories in this field and avoids the pollution of the dataset by noisy data from the source. The balanced dataset constructed by this implementation can be directly used for post-training of large-scale vertical models in the hydropower construction and energy storage business field, solving the problems of insufficient coverage and limited generalization ability of the model in key edge scenarios.
[0044] The above specific embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for constructing a dataset for post-training of a large vertical model, characterized in that, include: S1. Calculate the mean of the dataset based on the frequency of each sample category in the data set. and standard deviation ; S2, based on the mean and standard deviation Identify the outlier categories in the dataset; S3. Perform attribute determination on the outlier category to identify it as a valuable long-tail category or a noise / invalid data category. S4. Perform differential processing on the judgment results, perform data augmentation on valuable long-tail categories, and discard samples of noisy / invalid data categories; S5. After completing the differentiation process, a balanced large-scale vertical model is obtained, followed by the training dataset.
2. The method for constructing a dataset for post-training of a large vertical model according to claim 1, characterized in that, In step S1, the frequency of each sample category in the dataset is statistically analyzed. ,in Indicates category The number of samples.
3. The method for constructing a dataset for post-training of a large vertical model according to claim 2, characterized in that, The mean The calculation formula is: The standard deviation The calculation formula is: in, For the amount of data, For the number of data points.
4. The method for constructing a dataset for post-training of a large vertical model according to claim 2, characterized in that, The specific criteria for outlier identification are as follows: For each sample category Calculate the deviation between the frequency and the mean, if it satisfies Then determine the category. This is an outlier.
5. The method for constructing a dataset for post-training of a large vertical model according to claim 1, characterized in that, The outlier attribute determination is specifically as follows: Determine the category of outliers It can be a valuable long-tail category or a noise / invalid data category; wherein, the noise / invalid data category includes the sample category corresponding to sensor fault logs.
6. The method for constructing a dataset for post-training of a large vertical model according to claim 5, characterized in that, The valuable long-tail categories are sample categories related to rare faults, new technology applications, or niche equipment in the field of hydropower construction and energy storage business. The rare faults include large-scale energy storage fires and thermal runaway chain reactions; the new technology applications include hydrogen energy storage, flow batteries, and grid-connected inverters; and the niche devices include domestically produced flywheels and off-grid microgrid controllers.
7. The method for constructing a dataset for post-training of a large vertical model according to claim 5, characterized in that, Outliers identified as noise / invalid data will be discarded, specifically: Simply remove all samples in the noisy / invalid data category to obtain a new dataset after discarding that category. ,and ,in, For the original dataset, The number of samples for the noise / invalid data category.
8. The method for constructing a dataset for post-training of a large vertical model according to claim 5, characterized in that, Data augmentation is performed on outliers identified as valuable long-tail categories, specifically including: New samples are generated using a large, expert-guided language model; the number of new samples needed to be generated is calculated. and the generated A new sample is added to this category, causing the frequency of samples in this category to fall within the range. Inside.
9. The method for constructing a dataset for post-training of a large vertical model according to claim 8, characterized in that, The formula for calculating the number of new samples k to be generated is: After adding new samples, the frequency of the categories was enhanced. ,and .
10. The method for constructing a dataset for post-training of a large vertical model according to claim 9, characterized in that, After data augmentation, the valuable long-tail categories meet the following requirements. ; in, This represents the mean of the augmented dataset for this category. The standard deviation of the dataset after augmentation for this category of data.