A training data acquisition method and device

Abstract

The application provides a training data acquisition method and device. The method comprises: acquiring an original data set, wherein the original data set comprises a plurality of training sample data; filtering the original data set based on a small model to obtain a filtered data set, wherein the small model is trained based on an aligned training data set, the aligned training data set comprises a plurality of training sample data with labels, the label is a quality score label, and the training sample data in the aligned training data set is sampled from the original data set based on sample uncertainty; and obtaining a target training data set based on the filtered data set. The application samples relatively key and small amount of training sample data from the original data set based on sample uncertainty, significantly reduces data labeling cost, quickly trains a small model with data discrimination ability by using the labeled sampling data set, filters the original data set by using the small model, and significantly reduces data filtering cost and improves data filtering efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for acquiring training data. Background Technology

[0002] With the development and application of large model technology, it has been found that the quality of the training data for large models is crucial to their performance. For example, Microsoft's Phi series of studies experimentally demonstrated that incremental pre-training of large language models (LLMs) using high-quality and educational data can significantly improve their higher-order capabilities (such as reasoning, mathematics, planning, and coding), allowing LLMs with fewer parameters to approach the performance of LLMs with larger parameter scales. Increasing research also demonstrates that constructing high-quality training datasets is key to developing the higher-order intelligent capabilities of large models.

[0003] Therefore, how to extract and screen high-quality pre-training corpora from massive amounts of internet data at low cost and high efficiency has become the core technology for training large models and is an essential path to improve the high-order specialized capabilities of large models. Summary of the Invention

[0004] The embodiments of this application provide a training data acquisition method and apparatus, which can realize the low-cost and efficient mining of high-quality training data required from massive amounts of raw data.

[0005] In a first aspect, this application provides a method for acquiring training data, including acquiring an original dataset, which includes multiple training sample data; filtering the original dataset based on a small model to obtain a filtered dataset, wherein the small model is trained based on an aligned training dataset, which includes multiple labeled training sample data, the labels being quality score labels, and the training sample data in the aligned training dataset being sampled from the original dataset based on sample uncertainty; and obtaining a target training dataset based on the filtered dataset.

[0006] This application obtains a relatively small number of key training samples from the original dataset by sampling uncertainty, which significantly reduces the cost of data annotation (such as the cost of calling a large language model for annotation or the cost of manual annotation). The annotated sampled dataset is used to quickly train a small model with data discrimination ability. The small model is then used to filter the original dataset. Compared with the existing data filtering scheme that uses a large language model, this application significantly reduces the cost of data filtering and improves the efficiency of data filtering.

[0007] In one possible implementation, sample uncertainty is derived from both epistemic uncertainty and aleatoric uncertainty of the samples.

[0008] In another possible implementation, the training sample data in the alignment training dataset is obtained by sampling from the original dataset through the following steps: initializing the prediction model, target model, and selection model; using the training sample data x in the original dataset as input to the prediction model, target model, and selection model, respectively, to obtain the outputs of the prediction model, target model, and selection model, where the output of the selection model includes the random uncertainty of the samples; determining the cognitive uncertainty of the samples based on the difference between the outputs of the prediction model and the target model; determining the random uncertainty of the samples based on the output of the selection model; determining the sample uncertainty of the training sample data x based on the cognitive uncertainty and random uncertainty of the samples; determining whether to sample the training sample data x based on the sample uncertainty; adjusting the weight parameters of the prediction model based on the training sample data x and the output of the target model; and obtaining the training sample data in the alignment training dataset by sampling from the original dataset after multiple iterations.

[0009] This application designs a method for estimating cognitive uncertainty of samples using random network distillation. The core idea is to randomly initialize a target model and use another predictive model to learn the target model. If a sample x differs little from previous samples (i.e., low cognitive uncertainty), then the predictive model has already learned similar samples, and its output will be similar to the target model's output. Conversely, if sample x differs greatly from previously seen samples, the predictive model cannot accurately predict the target model's output, resulting in a large error. Therefore, this application uses the output error of these two models as an estimate of cognitive uncertainty. Then, by selecting models (e.g., logistic regression, ensemble models, and Bayesian neural network models that support outputting random uncertainty) to determine the random uncertainty of the samples, and combining cognitive uncertainty and random uncertainty, the sample uncertainty is determined. Finally, a scheme based on sample uncertainty is adopted to dynamically select data samples to be labeled, efficiently and quickly training a small model with data discrimination capabilities.

[0010] In another possible implementation, the small model is trained based on the following steps: sampling from the original dataset based on sample uncertainty to obtain a sampled training dataset; labeling each training sample data in the sampled training dataset based on the large language model to obtain an aligned training dataset; and training the small model based on the aligned training dataset to obtain the trained small model.

[0011] In order to train a small model with the least possible annotation cost, this application adopts a scheme based on sample uncertainty to dynamically select data samples to be labeled from the original dataset, thereby efficiently and quickly training a small model with data discrimination capabilities.

[0012] In another possible implementation, the quality score labels are obtained based on the annotations of the large language model. This aligns the training dataset with the annotations of the large language model, and aligns the data filtering capabilities of the small model and the large language model. In this way, the small model can be used instead of the large language model to filter the massive original dataset, which greatly reduces the cost of calling the large language model for data filtering.

[0013] Optionally, the quality score labels can also be manually labeled, aligning the data discriminative ability of the small model with that of the human, and using the data discriminative ability of the small model to filter out high-quality training sample data from the original dataset.

[0014] In another possible implementation, the number of parameters of the small model is less than a preset threshold, for example, the number of parameters of the small model is less than 1 million. The small model with fewer parameters is used to replace the large language model for data filtering, which reduces the cost of data filtering and greatly improves the efficiency of data filtering.

[0015] In one possible implementation, the original dataset is filtered based on a small model to obtain the filtered dataset. A specific implementation is as follows: multiple training sample data in the original dataset are used as input to the small model, and the first quality score of each training sample data in the original dataset is output. The first quality score indicates the level of gain obtained by the target business model when trained with the training sample data in the target capability item; the training sample data in the original dataset with the first quality score less than a first threshold are filtered to obtain the filtered dataset.

[0016] Using a small model with fewer parameters to score the quality of massive raw datasets greatly reduces the cost of data screening compared to using a large language model. At the same time, the reasoning speed of the small model with fewer parameters is also faster than that of the large language model, increasing screening efficiency.

[0017] In another possible implementation, a specific way to obtain the target training dataset based on the filtered dataset is as follows: the training sample data in the filtered dataset is used as the input of the quality assessment model, and the output is a second quality score for each training sample data in the filtered dataset. The prediction accuracy of the quality assessment model is higher than that of the small model. Based on the second quality score, the filtered dataset is filtered again to obtain the target training dataset.

[0018] This application first uses a small data filtering model to coarsely screen the massive raw dataset, effectively reducing the amount of data processing. Compared with using a large language model to screen the massive raw dataset, this greatly reduces the screening cost. Then, based on the coarsely screened dataset, a more refined screening is performed to improve the efficiency of mining the required training data from the raw dataset and the accuracy of data mining.

[0019] In another possible implementation, the target training dataset is obtained by further filtering the filtered dataset based on the second quality score (also known as the quality prediction value). Specifically, the training sample data in the filtered dataset is sorted in descending order based on the second quality score; the top K training sample data in the filtered dataset are used as the target training dataset, where K is a positive integer.

[0020] In this possible implementation, this application utilizes a more powerful fine-screening model (i.e., a quality assessment model) to score, rank, and select the topk of the recalled data (i.e., the data after coarse screening) to obtain the target training dataset. This allows for convenient control over the amount of data obtained from the screening order, making it easier for practical use.

[0021] In another possible implementation, based on the second quality score, a further specific implementation of filtering from the filtered dataset to obtain the target training dataset is as follows: Training samples in the filtered dataset with a second quality score greater than or equal to a second threshold are used as the target training dataset. By setting a threshold, the dataset is quickly filtered to the desired target training dataset, increasing the efficiency of data mining.

[0022] In another possible implementation, using the training samples from the filtered dataset as input to the quality assessment model and outputting a second quality score for each training sample in the filtered dataset, a specific implementation involves: extracting features from each training sample in the filtered dataset to obtain a feature vector for each training sample; using the feature vector of each training sample as input to the trained regression model, and outputting a second quality score for each training sample. Using a regression model to predict the quality of the training sample data is more accurate.

[0023] In another possible implementation, a specific approach to extracting features from each training sample in the filtered dataset to obtain a feature vector for each training sample is as follows: multiple feature extraction methods are used to extract features from each training sample to obtain multiple semantic features corresponding to each training sample; these multiple semantic features are then fused to obtain a feature vector for each training sample.

[0024] Compared to general single-model feature extraction methods, feature extraction methods that fuse multiple semantic features can effectively improve the accuracy of model predictions, thereby enhancing the data mining effect.

[0025] In another possible implementation, the regression model is trained on an aligned training dataset. As described above, the aligned training dataset obtained in the preceding steps already includes multiple training sample data with quality score labels. Therefore, the aligned training data can be used to supervise the training of the regression model to obtain the regression model without having to label the training sample data again, thus reducing the data labeling cost.

[0026] In another possible implementation, the labels for each training sample in the aligned training dataset are obtained based on a large language model and target prompts. The target prompts are obtained based on each training sample and a target prompt template; the target prompt templates are derived through inference evolution based on the large language model. This eliminates the need for human experts to meticulously design prompts and provides accuracy, comprehensiveness, and design efficiency compared to manually designed prompts.

[0027] In another possible implementation, the target prompt template is derived through the following steps: First, data mining requirements and an initial prompt template are obtained. The data mining requirements indicate the target capability of the target business model that can be improved by the training sample data mined from the original dataset. The initial prompt template includes rule items, each containing several first rules, indicating the features of the training sample data that meet the data mining requirements. Next, N training sample data points are extracted from the original dataset, where N is a positive integer. The data mining requirements and the N first prompts are used as input to a large language model, outputting a third quality score for the N training sample data points. The N first prompts are derived based on the initial prompt template and the N training sample data points. Then, rules are summarized from the M training sample data points using the large language model, resulting in several second rules. The M training sample data points are the M training sample data points with the highest third quality scores among the N training sample data points. Finally, the rule items of the initial prompt template are updated based on these second rules. The evolution direction of the initial prompt template is verified, and a verification result is obtained. Based on the verification result, it is determined whether to accept the update of the initial prompt template. After several iterations, the target prompt template is obtained.

[0028] This application designs a data-driven prompt self-evolution direction to solve the problem of difficulty in defining the scope / quality of specific data. It uses a validation dataset to solve the problem of evolutionary deviation from the correct direction that is prone to occur during the evolution process, and realizes automatic evolution to obtain a comprehensive, accurate and practical prompt.

[0029] In another possible implementation, verifying the evolution direction of the initial prompt template to obtain the verification result is specifically implemented as follows: A second prompt is generated for each training sample in the verification dataset based on the updated initial prompt template. The verification dataset includes multiple training sample data with quality score labels. The second prompt is used as input to a large language model, outputting a fourth quality score for each training sample in the verification dataset. The verification result is determined based on a first ranking result and a second ranking result, where the first ranking result is obtained by ranking the multiple training sample data in the verification dataset based on the fourth quality score, and the second ranking result is obtained by ranking the multiple training sample data in the verification dataset based on the quality score labels.

[0030] In another possible implementation, a specific way to determine the verification result based on the first sorting result and the second sorting result is to: calculate the degree of consistency between the first sorting result and the second sorting result; and determine the verification result based on the degree of consistency.

[0031] Optionally, there are various methods to calculate the consistency between the first and second ranking results, including but not limited to rank correlation, edit distance, and cosine similarity. This application does not limit the specific method for calculating the consistency between the first and second ranking results, and appropriate consistency calculation methods can be selected as needed.

[0032] In another possible implementation, the target training dataset is used as a training dataset for the pre-training phase of the large language model; or, the target training dataset is used as a training dataset for the post-training / annealing phase of the large language model; or, the target training dataset is used as a training dataset for the fine-tuning phase of the large language model; or, the target training dataset is used as a training dataset for the alignment phase of the large language model; or, the target training dataset is used as a training dataset for the language model.

[0033] Secondly, this application also provides a training data acquisition device, which includes an acquisition module, a filtering module, and a determination module. The acquisition module is used to acquire an original dataset, which includes multiple training sample data. The filtering module is used to filter the original dataset based on a small model to obtain a filtered dataset. The small model is trained based on an aligned training dataset, which includes multiple labeled training sample data, with the labels being quality score labels. The training sample data in the aligned training dataset is sampled from the original dataset based on sample uncertainty. The determination module is used to obtain a target training dataset based on the filtered dataset.

[0034] In one possible implementation, sample uncertainty is derived from the cognitive uncertainty of the sample and the accidental uncertainty of the sample.

[0035] In another possible implementation, the training sample data in the alignment training dataset is obtained by sampling from the original dataset through the following steps: initializing the prediction model, target model, and selection model; using the training sample data x in the original dataset as input to the prediction model, target model, and selection model, respectively, to obtain the outputs of the prediction model, target model, and selection model, where the output of the selection model includes the random uncertainty of the samples; determining the cognitive uncertainty of the samples based on the difference between the outputs of the prediction model and the target model; determining the random uncertainty of the samples based on the output of the selection model; determining the sample uncertainty of the training sample data x based on the cognitive uncertainty and random uncertainty of the samples; determining whether to sample the training sample data x based on the sample uncertainty; adjusting the weight parameters of the prediction model based on the training sample data x and the output of the target model; and obtaining the training sample data in the alignment training dataset by sampling from the original dataset after multiple iterations.

[0036] In another possible implementation, the small model is trained based on the following steps: sampling from the original dataset based on sample uncertainty to obtain a sampled training dataset; labeling each training sample data in the sampled training dataset based on the large language model to obtain an aligned training dataset; and training the small model based on the aligned training dataset to obtain the trained small model.

[0037] In another possible implementation, the quality score labels are obtained based on annotations from a large language model or manually.

[0038] In another possible implementation, the number of parameters in the small model is less than a preset threshold.

[0039] In one possible implementation, the determining module is specifically used to: take multiple training sample data from the original dataset as input to a small model, output a first quality score for each training sample data in the original dataset, the first quality score indicating the level of gain obtained by the target business model when trained using the training sample data; filter the training sample data in the original dataset whose first quality score is less than a first threshold to obtain the filtered dataset.

[0040] In another possible implementation, a specific way to obtain the target training dataset based on the filtered dataset is as follows: the training sample data in the filtered dataset is used as the input of the quality assessment model, and the output is a second quality score for each training sample data in the filtered dataset. The prediction accuracy of the quality assessment model is higher than that of the small model. Based on the second quality score, the filtered dataset is filtered again to obtain the target training dataset.

[0041] In another possible implementation, a specific implementation of further filtering from the filtered dataset based on the second quality score to obtain the target training dataset is as follows: the training sample data in the filtered dataset are sorted in descending order based on the second quality score; the top K training sample data in the filtered dataset are used as the target training dataset, where K is a positive integer.

[0042] In another possible implementation, based on the second quality score, a further specific implementation of filtering from the filtered dataset to obtain the target training dataset is as follows: Training samples in the filtered dataset with a second quality score greater than or equal to a second threshold are used as the target training dataset. By setting a threshold, the dataset is quickly filtered to the desired target training dataset, increasing the efficiency of data mining.

[0043] In another possible implementation, the training sample data in the filtered dataset is used as input to the quality assessment model, and a specific implementation of outputting the second quality score of each training sample data in the filtered dataset is as follows: feature extraction is performed on each training sample data in the filtered dataset to obtain the feature vector of each training sample; the feature vector of each training sample is used as input to the trained regression model, and the second quality score of each training sample data is output.

[0044] In another possible implementation, a specific approach to extracting features from each training sample in the filtered dataset to obtain a feature vector for each training sample is as follows: multiple feature extraction methods are used to extract features from each training sample to obtain multiple semantic features corresponding to each training sample; these multiple semantic features are then fused to obtain a feature vector for each training sample.

[0045] In another possible implementation, the regression model is trained based on an aligned training dataset.

[0046] In another possible implementation, the labels of each training sample in the alignment training dataset are obtained based on a large language model and target prompts, and the target prompts are obtained based on each training sample and a target prompt template; the training data acquisition device provided in this application also includes a prompt self-evolution module, which is used to perform reasoning evolution based on the large language model to obtain the target prompt template.

[0047] In another possible implementation, the prompt self-evolution module is specifically used for: obtaining data mining requirements and an initial prompt template. The data mining requirements indicate the target capability of the target business model that can be improved by the training sample data mined from the original dataset. The initial prompt template includes rule items, each rule item including several first rules, which indicate the features of the training sample data that meet the data mining requirements; extracting N training sample data from the original dataset, where N is a positive integer; using the data mining requirements and the N first prompts as input to a large language model, outputting a second quality score for the N training sample data, where the N first prompts are obtained based on the initial prompt template and the N training sample data; calling the large language model to summarize the rules of M training sample data, obtaining several second rules, where the M training sample data are the M training sample data with the highest second quality scores among the N training sample data; updating the rule items of the initial prompt template based on the several second rules; verifying the evolution direction of the initial prompt template and obtaining the verification result; determining whether to accept the update of the initial prompt template based on the verification result; and obtaining the target prompt template after several iterations.

[0048] In another possible implementation, verifying the evolution direction of the initial prompt template to obtain the verification result is specifically implemented as follows: A second prompt is generated for each training sample in the verification dataset based on the updated initial prompt template. The verification dataset includes multiple training sample data with quality score labels. The second prompt is used as input to a large language model, which outputs a third quality score for each training sample in the verification dataset. The verification result is determined based on a first ranking result and a second ranking result, where the first ranking result is obtained by ranking the multiple training sample data in the verification dataset based on the third quality score, and the second ranking result is obtained by ranking the multiple training sample data in the verification dataset based on the quality score labels.

[0049] In another possible implementation, a specific way to determine the verification result based on the first sorting result and the second sorting result is to: calculate the degree of consistency between the first sorting result and the second sorting result; and determine the verification result based on the degree of consistency.

[0050] Optionally, there are multiple methods to calculate the consistency between the first and second sorting results, including but not limited to sorting relevance, editing examples, and cosine similarity. This application does not limit the specific method for calculating the consistency between the first and second sorting results, and appropriate consistency calculation methods can be selected as needed.

[0051] In another possible implementation, the target training dataset is used as a training dataset for the pre-training phase of the large language model; or, the target training dataset is used as a training dataset for the post-training / annealing phase of the large language model; or, the target training dataset is used as a training dataset for the fine-tuning phase of the large language model; or, the target training dataset is used as a training dataset for the alignment phase of the large language model; or, the target training dataset is used as a training dataset for the language model.

[0052] Thirdly, embodiments of this application provide a computing device, including a memory and a processor, wherein the memory stores instructions that, when executed by the processor, cause the method described in the first aspect or any possible implementation of the first aspect to be implemented.

[0053] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the method described in the first aspect or any possible implementation thereof to be implemented.

[0054] Fifthly, embodiments of this application also provide a computer program or computer program product, which includes instructions that, when executed, cause a computer to perform the method described in the first aspect or any possible implementation thereof.

[0055] In a sixth aspect, embodiments of this application also provide a chip including at least one processor and a communication interface, the processor being configured to perform the method described in the first aspect or any possible implementation thereof.

[0056] It is understandable that the beneficial effects of aspects two through six above can be found in the relevant descriptions in aspect one above, and for the sake of brevity, they will not be repeated here. Attached Figure Description

[0057] Figure 1 A schematic diagram of a system architecture is shown;

[0058] Figure 2 This diagram illustrates the data and training pipeline for a large language model.

[0059] Figure 3 A schematic diagram of the implementation architecture of the training data acquisition method provided in an embodiment of this application is shown;

[0060] Figure 4 This illustration shows a schematic diagram of the specific implementation process of the training data acquisition method provided in an embodiment of this application;

[0061] Figure 5This illustration shows a schematic diagram of the implementation process of the prompt self-evolution provided in an embodiment of this application;

[0062] Figure 6 A schematic diagram of the decision boundary of the small model is shown;

[0063] Figure 7 This illustration shows a schematic diagram of the method principle for estimating the cognitive uncertainty of samples using the random network distillation method proposed in an embodiment of this application;

[0064] Figure 8 A schematic diagram of a sampling process based on sample uncertainty is shown;

[0065] Figure 9 A schematic diagram illustrating the process of training a quality assessment model and performing data mining is shown.

[0066] Figure 10 A flowchart illustrating a training data acquisition method provided in an embodiment of this application;

[0067] Figure 11 A comparative diagram of F1, precision, and recall metrics using the alignment scheme, random sampling scheme, and Epsilon-greedy baseline scheme provided in the embodiments of this application is shown.

[0068] Figure 12 and Figure 13 Schematic diagrams of generalized inference-en regression results and generalized inference-zh regression results are shown respectively;

[0069] Figure 14 This is a schematic diagram of the structure of a training data acquisition device provided in an embodiment of this application;

[0070] Figure 15 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0071] The term "and / or" used in this article describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0072] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same properties in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such processes, methods, systems, products, or apparatus.

[0073] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0074] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0075] Related technologies have explored many approaches to data mining. For example, Microsoft's Phi series of works points to a new direction in the evolution of LLM: improving the quality and educational value of pre-training data to enhance the model's coding / mathematical capabilities. Phi-1 primarily targets the coding capabilities of large models, constructing 6B of "textbook-style" high-quality pre-training data + 1B of synthetic data using data labeled with commercial large models (OpenAI GPT-3.5). Textbook-style high-quality data refers to data that is clear, self-contained, with explicit instructions, and balanced diversity. Specifically, Phi-1 proposes: 1) crawling data from programming communities on the internet (such as Stack Overflow) and using the GPT3.5 model to score the quality of this data. A random forest-based classification model is trained based on the scored data to filter the crawled data, obtaining high-quality code data. 2) A large amount of user requirements and code implementation data is also synthesized using GPT3.5. These data together constitute the high-quality pre-training corpus of Phi-1. 3) Simultaneously, Phi also extracts comments from the code to simulate user requirements and forms Question-Answer data pairs with the corresponding code to construct SFT data for the fine-tuning stage of the large model. Phi-1.5 and Phi-2 further apply this approach to providing the model with language understanding and complex reasoning capabilities.

[0076] However, this approach constructs its dataset by scoring the quality of each data point using LLM (Limited Learning Model), and then filtering based on the scores. A similar approach is applied to constructing instruction compliance datasets. These methods require extensive LLM-based inference processes or the use of LLM APIs, making annotation costly. Furthermore, data quality heavily depends on the design of the scoring prompts, necessitating repeated manual optimization and adjustments.

[0077] The second related technical solution is Deepseek Math. Deepseek Math-7B, a mathematics-specific model launched by DeepSeek in 2024, boasts mathematical performance comparable to OpenAI's GPT4 model. The technical documentation for Deepseek Math discloses its data construction process as follows: Its core lies in leveraging the fact that mathematical data often clusters on specific websites (e.g., Stack Overflow's Math section). It proposes using web pages as the basic unit, employing a trained classification model to analyze the web page text and determine if it contains a large amount of mathematically relevant knowledge. If so, it is labeled as mathematically relevant. Then, through clustering / manual analysis, the proportion of mathematical content under each domain is calculated, and all web pages under domains with a high proportion of mathematically relevant content are fully retrieved, cleaned, and used as pre-training data for the Deepseek Math model.

[0078] This approach is only suitable for certain specific types of specialized data, requiring that the relevant content be distributed relatively centrally on the internet. However, for certain specialized data, such as reasoning and planning, the distribution on the internet is not centralized, making it difficult to apply this method.

[0079] The third related technical solution is a filtering scheme based on heuristic programming. Its core idea is to improve data quality by designing multiple heuristic programming-based filters to filter the source data layer by layer. For example, FineWeb has designed multiple planning filters, such as deleting text if the proportion of lines ending with punctuation marks is less than 0.12; deleting text if the character repetition rate is greater than 0.1; and deleting invalid content such as "lorem ipsum" in the current file / paragraph. Similarly, the C4 dataset construction process also employs a large number of heuristic filters.

[0080] The problem with this approach is that it employs numerous heuristic rule filters during dataset construction. The design and summarization of these rules require significant manual intervention and have limited generalization capabilities, making it difficult to guarantee broad applicability to all data.

[0081] As can be seen from the above description, the key problem in constructing high-quality pre-trained datasets is: how to construct high-quality pre-trained corpora for any specific domain category (such as logical reasoning, task planning, etc.)?

[0082] Specifically, the following problems need to be solved:

[0083] Operating efficiency: The amount of data on the Internet is large (the original Common Crawl is about 400TB / dump), and the computing power cannot support complex filtering models.

[0084] Construction Costs: Defining and scoring domain-specific data (e.g., reasoning / humor / emotion) requires high-order cognitive abilities, but manual / LLM annotation is expensive. For example, using GPT-3.5 annotation costs $0.5 / 1M Token, requiring $2,000,000 to annotate 4TTokens of data; using DeepSeek annotation costs $0.14 / 1M Token, requiring $560,000 to annotate 4TTokens of data.

[0085] Generalization: Existing domain data is often very limited and cannot cover the diverse range of data on the Internet.

[0086] To address these issues, embodiments of this application provide a training data acquisition method and apparatus. This method is applicable to any specific field and can efficiently and cost-effectively mine the required high-quality pre-training data from massive datasets.

[0087] It should be noted that the training data acquisition method provided in this application is not limited to pre-training data mining tasks for large language model pre-training. It can be applied to any other data construction task requiring targeted mining, including but not limited to training datasets used in the post-training / annealing stage of large language models, supervised fine-tuning (SFT) datasets for large language models, alignment data for large language models, or other targeted data mining tasks used for traditional language model training. Accordingly, the purpose of the target dataset obtained by the training data acquisition method provided in this application varies depending on the data mining task applied. For example, when the training data acquisition method provided in this application embodiment is applied to an arbitrary-direction data mining task in the pre-training of a large model, the obtained target training dataset is used as the training dataset in the pre-training stage of the large language model; when the training data acquisition method provided in this application embodiment is applied to an arbitrary-direction data mining task in the training dataset used in the post-training / annealing stage of the large language model, the obtained target training dataset is used as the training dataset in the post-training / annealing stage of the large language model; when the training data acquisition method provided in this application embodiment is applied to an arbitrary-direction data mining task in the training dataset of the fine-tuning stage of the large language model, the obtained target training dataset is used as the training dataset in the fine-tuning stage of the large language model; when the training data acquisition method provided in this application embodiment is applied to an arbitrary-direction data mining task in the alignment stage of the large language model, the obtained target training dataset is used as the training dataset in the alignment stage of the large language model; when the training data acquisition method provided in this application embodiment is applied to an arbitrary-direction data mining task in the training of a traditional language model, the obtained target training dataset is used as the training dataset in the training of the traditional language model.

[0088] The following example uses the targeted mining task of pre-training data for large language models to illustrate the detailed implementation of the training data acquisition method provided in this application. The implementation of other targeted mining task scenarios is similar and can be referred to for implementation. For the sake of brevity, it will not be described in detail.

[0089] To facilitate understanding of the training data acquisition method and apparatus provided in the embodiments of this application, some technical terms involved in the embodiments of this application will be briefly explained below.

[0090] Prompt: Also known as a prompt word, it is used to guide a large model to generate a specific response or perform a specific task. It is usually a clear text, such as "Please write a five-character quatrain with the theme of spring, which must rhyme and have five characters per line."

[0091] GPT stands for Generative Pre-trained Transformer, a pre-trained language model based on the Transformer architecture developed by OpenAI. Through pre-training on large amounts of text data, GPT models learn to generate coherent and logical text, and can be used for various natural language processing tasks, such as text generation, machine translation, text summarization, and question answering systems. Common GPT models include GPT-3.5 and GPT-4, representing the most advanced models currently available.

[0092] Token: In the training of large-scale language models, a token is the smallest unit of text processing. It can be a word, a character, or a subword. When processing natural language, the language model segments the input text into a series of tokens. By processing and learning these tokens, the model can understand what we say and provide corresponding responses.

[0093] Cognitive uncertainty stems from a lack of knowledge, specifically an insufficient understanding of the model's understanding of certain parameters or the system. It can be reduced by acquiring more information or data. For example, in machine learning, cognitive uncertainty arises if we lack sufficient understanding of the model's parameters or have insufficient data. Increasing the amount of data or improving the model structure can typically reduce this type of uncertainty.

[0094] Random uncertainty: This type of uncertainty is inherent and related to the randomness of the system. Even with a perfect model and infinite data, random uncertainty cannot be eliminated because it is caused by the random nature of the system itself. For example, the outcome of a coin toss has random uncertainty; even if we fully understand the properties of the coin, we cannot predict the result of a single toss.

[0095] The specific implementation of the training data acquisition method provided in this application embodiment is described in detail below with reference to the accompanying drawings and embodiments.

[0096] Figure 1 A schematic diagram of a system architecture is shown. The system includes a terminal 110, a server 130, and a network 120 that communicatively connects the terminal 110 and the server 130. The server 130 may include one or more servers (…). Figure 1 (Using only one server as an example for illustration), server 130 can provide one or more terminals 110 with arbitrary directional data mining tasks on the large language model pre-training set, thereby improving the quality of the large language model pre-training dataset.

[0097] In some embodiments, server 130 may also provide other services or software applications, including both non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to end users under a Software as a Service (SaaS) model.

[0098] exist Figure 1 In the system shown, server 130 may include one or more components that implement the functions performed by server 130. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. A user of operating terminal 110 may sequentially interact with server 130 and utilize the services provided by these components using one or more client applications. It should be understood that... Figure 1 This is merely one example of a system for implementing the various methods described in the embodiments of this application; other different configurations are possible, and the embodiments of this application do not limit this.

[0099] The terminal 110 may have data mining-related applications (such as a large language model training assistant) installed, or a data mining-related webpage opened. The aforementioned applications and webpages can provide an interactive interface. Optionally, the interactive interface includes a text box. The terminal 110 can receive data mining requirements entered by the user in the text box and send the data mining requirements to the server 130. The server 130 can perform data mining tasks based on the received data mining requirements using the training data acquisition method provided in this application embodiment to obtain a high-quality training dataset and improve a certain capability of the large language model.

[0100] Terminal 110 may include various types of computer devices, such as portable handheld devices (e.g., smartphones), general-purpose computers (e.g., personal computers or laptops), workstation computers, wearable devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (e.g., Google Chrome OS), Harmony OS; or various mobile operating systems, such as Microsoft Windows Mobile OS, Windowsphone, Android, iPhone OS, HarmonyOS NEXT, etc. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays and smartwatches, etc.

[0101] Network 120 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, 3G, 4G, 5G, etc., to support data communication. For example, one or more networks 120 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, the public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, Wi-Fi), and / or any combination of these and / or other networks.

[0102] Server 130 may include one or more general-purpose computers, special-purpose server computers (e.g., personal computer servers, UNIX servers, terminal servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement or combination. Server 330 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for the server. In various embodiments, server 130 may run one or more services or software applications that provide the functionality described below.

[0103] In some implementations, server 130 can be a server for a distributed system or a server integrated with blockchain. Server 130 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and virtual private servers (VPS) services, such as high management difficulty and weak business scalability.

[0104] It should be noted that in some other examples, the terminal 110 can also execute the training data acquisition method provided in the embodiments of this application on its own, and mine a high-quality pre-training dataset from a massive raw dataset, without the cooperation of the server 130. Alternatively, the server 130 can also execute the training data acquisition method provided in the embodiments of this application on its own, and mine a high-quality pre-training dataset from a massive raw dataset, without the cooperation of the terminal 110. This application does not make any specific limitations on this.

[0105] Figure 2 A schematic diagram of the data and training pipeline for a large language model is shown. Figure 2As shown, the data pipeline for a large language model includes data cleaning, data deduplication, and dataset construction. Dataset construction yields a pre-training dataset and an SFT-aligned dataset. The training pipeline includes pre-training, post-training, supervised fine-tuning, and alignment, ultimately resulting in a trained large language model. The training data acquisition method and apparatus provided in this application are applied to dataset construction within the data pipeline. By mining massive amounts of data, high-quality specialized datasets are constructed, improving the specialized capabilities of the large language model (such as reasoning, planning, mathematical, and code generation capabilities). In other words, the training data acquisition method and apparatus provided in this application are tools for constructing and producing specialized datasets for large language models, and can exist as software in the production cycle of the large language model.

[0106] Figure 3 A schematic diagram illustrating the implementation architecture of the training data acquisition method provided in an embodiment of this application is shown. Figure 3 As shown, the implementation architecture of the training data acquisition method provided in this application mainly includes four parts: self-evolving prompt, alignment, coarse filtering, ranking, and fine-grained filtering. In the self-evolving prompt part, this application addresses the difficulty of defining the boundaries and quality of specialized data by designing a data-driven self-evolving prompt technique. It automatically summarizes and generalizes patterns in the data using a large language model, automatically reflecting on and optimizing prompts with high quality scores. Through multiple iterations, the scoring prompts are gradually updated (it can be understood that the prompts here refer to prompt templates, which are combined with specific data to generate specific scoring prompts). In the alignment part, to minimize reliance on the large language model during data construction and reduce annotation costs, we use a smaller model with fewer parameters to replace the large language model for data annotation (i.e., quality score annotation of the data). To train the smaller model with minimal annotation costs, this application adopts a scheme based on sample uncertainty to dynamically select the data samples to be annotated, efficiently and quickly aligning the data filtering capabilities of the smaller model with those of the large language model. In the coarse screening section, considering the massive amount of data to be screened (the commonly used internet dataset Common Crawl exceeds 100T tokens), this application adopts a combination of coarse and fine screening: first, a coarse screening model (i.e., a small model) filters the massive amount of data, greatly reducing the amount of data to be processed. In the ranking and fine screening section, a more powerful fine screening model (i.e., a quality scoring model) is used to score, rank, and select the Top K from the recalled data, ultimately obtaining a high-quality specialized training dataset.

[0107] Figure 4 A schematic diagram illustrating the specific implementation flow of the training data acquisition method provided in an embodiment of this application is shown. Figure 4 As shown in the embodiments of this application, the specific implementation of the training data acquisition method can be divided into three main steps: prompting self-evolution, model alignment screening, and specialized data mining. The specific implementation of these three main steps is described in detail below.

[0108] In the prompt self-evolution step, this application implements data-driven prompt self-evolution to automatically describe the characteristics of the target data, which facilitates subsequent guidance of the large language model to accurately label the data for quality scoring.

[0109] When constructing high-level specialized datasets for large language models, it is necessary to use the large language model to score the quality of the data. A primary challenge is the difficulty in accurately describing data quality. Taking reasoning data as an example, the following difficulties arise when defining quality scoring cues: 1) How to define the boundaries of reasoning data? That is, what constitutes reasoning, and what does not? 2) From which dimensions should the quality of reasoning be measured? What are the characteristics of good reasoning data? 3) How to score the reasoning quality of the data? How to accurately define the distinguishing criteria for each score range? Similarly, specialized datasets for other high-level capabilities of large language models, such as task planning, face the same problems. These difficulties make defining descriptive cues for data quality scores challenging, requiring repeated iterations by human experts. The entire process involves significant human intervention, and quality cannot be guaranteed.

[0110] To address this issue and enable automatic optimization of prompts based on the features of the corpus to be mined, this application proposes a data-driven prompt self-evolution strategy to automatically describe the features of the target data.

[0111] The self-evolving prompts ultimately yield target prompts used to perform targeted quality scoring on the original data through a large language model. For example, if our current task is to improve the "reasoning" ability of the large language model, we need to mine a large amount of data related to reasoning or data that can improve reasoning ability. In this case, the "prompts" are used to score whether the original data can improve the reasoning ability of the large language model. For instance, we define the scoring range as 1-10 points, with 8 points and above indicating a significant improvement in the reasoning ability of the large language model. Therefore, how these scoring prompts are written greatly affects the quality of data mining. Currently, the industry generally relies on experts to manually write these prompts, but because the boundaries and quality of targeted data are difficult to define, manually writing prompts is challenging and accuracy cannot be guaranteed. Therefore, this application proposes a data-driven self-evolving prompt method. By allowing the large language model to summarize the patterns in the original data and self-evolve and optimize the prompts, it achieves a high degree of automation, and the evolved prompts have advantages such as comprehensiveness, accuracy, and strong practicality.

[0112] Figure 5 A schematic diagram illustrating the implementation process of the prompt self-evolution provided in an embodiment of this application is shown.

[0113] First, obtain the input data, which includes the data mining requirements, initial hints (also known as initial hint templates), and the original dataset D. raw and validation dataset D validation The data mining requirement specifies how the mined data can improve the specific capabilities of the large language model. For example, it may require that the mined data can be used to improve the reasoning ability of the large language model. The data mining requirement can be user input. For instance, a user can open a data mining client on a terminal device and input the data mining requirement on the client's interactive interface. For example, the interactive interface may provide an input text box where the user can input the data mining requirement, such as "the mined data can be used to improve the reasoning ability of the large language model." Alternatively, the interactive interface may provide multiple common options and a custom option. Common options may include reasoning, planning, mathematics, code generation, etc. The user can select an option that matches the mining requirement, increasing the user-friendliness. If no option matches the user's mining requirement, the user can click "custom" and input the mining requirement from the pop-up input box. The terminal device sends the user-inputted mining requirement to a server deployed with the training data acquisition device provided in this embodiment. The server receives the data sent by the terminal device and obtains the data mining requirement.

[0114] The initial prompt (pinitial) can optionally be a manually written prompt, such as a simple description by a person of what kind of data can improve the reasoning ability of a large language model. Optionally, in some other examples, the initial prompt (p) can be... initial It also includes several examples, such as a feasible initial prompt p initial The specific rules to be mined are: 1. If the text contains the "because, therefore" field, then the text has high data quality and strong inference data; 2. If the text contains a clear conclusion at the end, and the conclusion is obvious, then the quality is high; Here are a few examples for reference: Example 1, a passage from a mystery novel, which has strong inference, has a high quality score, for example, 10 points; Example 2, a movie review (the review mainly expresses emotions, such as liking or disliking the movie, and does not contain inference), then the quality score is very low, for example, 1 point; Finally, now you are given a new text: ***, please output the quality score of this text.

[0115] Optional, initial prompt p initial It can also be generated initially rather than manually written, such as the initial prompt p. initialThe specific rules section is empty; the specific rules summarized by the large language model will be filled in during the subsequent reasoning and evolution process of the large language model.

[0116] In one example, the initial prompt p initial It also includes a mining requirement field, which is generated based on the mining requirements input by the user. For example, it can be "requirements," which is used to guide the large language model to mine and infer relevant data.

[0117] Original dataset D raw To crawl massive amounts of text data from the internet, such as books, academic papers, forum posts, movie reviews, and any other text data from the internet. The specific original dataset D... raw This can be a common crawl dataset, approximately 400TB / dump. Original dataset D raw Typically, the datasets are extremely large, ranging from tens to hundreds of TB or even more. Using a large language model to filter high-quality data that meets the requirements would be extremely costly and computationally expensive, almost unbearable. Therefore, this application proposes using a smaller model with fewer parameters on the original dataset D. raw The screening process greatly reduces screening costs and improves screening efficiency. The small model will be introduced in more detail below.

[0118] Validate dataset D validation Used to verify whether the direction of the cue evolution has gone astray. Validation dataset D validation These can be high-quality datasets, either manually selected or open-source, and are generally small in size. For example, the validation dataset D. validation It includes 1000 text data points, each with a quality score, and the data is sorted in descending order of quality score. Optional, a validation dataset D is also included. validation The quality scores of the data are either manually labeled or labeled by models with high inference accuracy.

[0119] In another example, the input data also includes a prompt format that indicates the format of the final evolved target prompt.

[0120] After obtaining the input data, randomly select from the original dataset D raw Extract N data points, and then based on the data mining requirements and initial hints p... initial Based on a large language model, a quality score is assigned to each data point in N data sets to obtain the quality score for each day's data. For example, based on the initial prompt p... initialGenerate N prompts for N data points. Take the data mining requirements and the N prompts as input to the large language model, and output the quality scores corresponding to the N prompts, that is, the quality scores of each data point in the N data points.

[0121] Then, the large language model rule extraction operation is performed, which involves analyzing the data with high quality scores, summarizing rules, and then using the summarized rules to update the initial prompt p. initial .

[0122] For example, N data points are sorted in descending order based on quality scores, and the top M data points are selected. Then, a large language model is used to summarize the patterns in these M data points to obtain their characteristics. Finally, the summarized patterns are used to apply to the initial prompt p. initial The rules are updated accordingly. For example, a prompt is generated for M data entries using a preset prompt template, such as "The current prompt is "The current initial prompt p". initial According to the current initial prompt p initial We score N data points to obtain N data point quality scores. The M data points with the highest quality scores are listed below: "****". Do you think these M data points share any common characteristics or rules? Please summarize them. Based on this preset prompt template, we generate prompts for these M data points. These prompts are then input into a large language model to guide the model in summarizing patterns from these M data points and using the summarized rules to update the initial prompt p. initial .

[0123] Initial prompt p initial The update methods include, but are not limited to, the following: adding one or more new prompt rules; deleting one or more existing prompt rules; modifying one or more existing prompt rules; and summarizing multiple existing prompt rules into one or more.

[0124] See also Figure 5 In large language model rule extraction, update initial hint p initial Then, the initial prompt p is validated using the validation dataset. initial The evolution direction is verified. If the verification fails, the evolution result of this round is discarded, that is, the initial prompt p is not used. initial Perform an update; if the verification passes, accept the result of this round of evolution, i.e., the initial prompt p. initial Make modifications and updates to avoid indicating that the evolution direction has deviated from the correct direction.

[0125] For example, the updated prompt p′ is used to score the quality of each data point in the validation dataset based on a large language model. For instance, a prompt is generated for each data point in the validation dataset based on the updated prompt p′, and this prompt is input into the large language model to guide it in scoring the quality of each data point in the validation dataset, resulting in a quality score for each data point. Then, the data in the validation dataset is sorted in descending order based on this quality score. The consistency between the original sorting and the sorting after scoring based on prompt p′ is calculated. If the consistency between the two sortings is completely identical, the direction of prompt evolution is considered to be without deviation; otherwise, it is considered to be deviated. Alternatively, if the consistency between the two sortings exceeds a threshold θ, the direction of prompt evolution is considered to be without deviation; otherwise, it is considered to be deviated.

[0126] Several methods can be used to calculate the similarity between two ranking results, including but not limited to ranking relevance, edit examples, and cosine similarity. The formula for ranking relevance is shown below:

[0127]

[0128] In the formula, r represents the relevance, with a value range of 0-1, and R(X) and R(Y) are the quality scores of the same data in the validation dataset based on the prompt p′ and the annotations in the validation dataset, respectively.

[0129] Then, based on whether there is a deviation, determine the verification result, or in other words, whether to accept the evolution result (i.e., whether to accept the update of the initial prompt). Optionally, the acceptance method includes, but is not limited to, the greedy acceptance method and the ∈-greedy acceptance method. The greedy acceptance method specifically states: as long as the direction of the prompt evolution is determined not to have deviated, the initial prompt p is accepted. initial The update; the ∈-greedy acceptance method is as follows: if the direction of the prompt evolution has not deviated, it is accepted; at the same time, it also accepts the evolution result when the direction of the prompt evolution deviates with a certain probability.

[0130] After multiple iterations, the final target prompt is obtained. For example, it checks whether the stopping condition has been reached; if so, it outputs the current prompt p, which is the final prompt p obtained through evolution. * If the target is not reached, the process returns to continuous iterative evolution until a stopping condition is met. Optional stopping conditions include, but are not limited to: reaching a preset number of iterations, reaching a preset iteration time, and no change in Prompt after multiple rounds of evolution.

[0131] Back Figure 4 Following the self-evolution step, a model alignment step is executed, implementing a sampling method based on multiple uncertainties, which enables the alignment of large language models with only a small amount of data.

[0132] The trade-off between generalization and operational efficiency: To effectively filter complex and diverse internet data, it is often necessary to score the quality of each data point. However, the amount of internet data is enormous (for example, the Common Crawl dataset contains over 430TB of data). If every data point is labeled using a large language model, it is not only costly but also requires a significant amount of inference time. The solution proposed in this application is to label some data using a large language model and then use this data to train an efficient small model (such as traditional machine learning models like RandomForest / XGBoost, with fewer parameters, for example, less than 1 million parameters, hereinafter referred to as a small model). Finally, this small model is used to filter the massive amount of data. The key technical challenge is: how to train an efficient small model with minimal cost of labeling with a large language model, so that its data discrimination ability can rival the filtering ability of a large language model (since filtering ability depends on the ability to score the quality of data, alignment can also be understood as aligning the data quality scoring abilities of the small model and the large language model).

[0133] To achieve this goal, the key lies in deciding which data needs to be labeled using a large language model. This application proposes that the following two types of data should be labeled using a large language model: samples whose predictions by the current small model are uncertain, and data that differs significantly from samples seen by the current small model.

[0134] Figure 6 A schematic diagram of the decision boundary of a small model is shown. For example... Figure 6 The pentagrams representing aleatoric samples and the triangles representing epistemic samples are the samples that require annotation by a large language model. After annotation, these samples constitute the aligned training dataset D. a Using the aligned training dataset D a Supervised training of small models is used to align the data filtering capabilities of small models with those of large language models.

[0135] It should be noted that the large language model used in the alignment step and the large language model used in the prompting evolution step can be the same large language model or different large language models, for example... Figure 3 As shown, the large language model used in the prompt evolution step is GPT-4, and the large language model used in the alignment step is GPT-3.5.

[0136] To determine the cognitive uncertainty of a sample, this application proposes a method using random network distillation to estimate it. The core idea is to randomly initialize a target model φ(x) and use another predictive model ψ(x) to learn the target model. If a sample x differs little from previous samples (i.e., has low cognitive uncertainty), then the predictive model has already learned similar samples, and its output will be similar to the target model's output. Conversely, if sample x differs greatly from previously seen samples, the predictive model cannot accurately predict the target model's output, resulting in a large error. Therefore, the output error of these two models can be used as an estimate of cognitive uncertainty.

[0137] Figure 7 This illustration shows a schematic diagram of the method principle for estimating the cognitive uncertainty of samples using a random network distillation method proposed in an embodiment of this application. Figure 7 If, for the same input x, the outputs of the target model and the prediction model differ significantly, then the cognitive uncertainty of that sample is relatively high.

[0138] This application also designs a screening model, also known as a filter model, to predict the quality score label of sample x, while outputting the uncertainty of the predicted value and using it as an estimate of random uncertainty.

[0139] This application's embodiments can dynamically select samples to be labeled based on a large language model based on two types of uncertainty: cognitive uncertainty and accidental uncertainty. This allows for the training of an efficient small model with minimal sample labeling, enabling its data filtering performance to approach that of a large language model.

[0140] The meaning of "the performance of the small model in data filtering is close to that of the large language model" can be understood as follows: the data filtered from the original dataset by the small model is not much different from the data filtered from the original dataset by the large language model.

[0141] Figure 8 A schematic diagram of a sampling process based on sample uncertainty is shown. Figure 8 As shown, three models are first randomly initialized: a prediction model φ(x), a target model ψ(x), and a selection model f(x). Then, any data sample x is randomly drawn from the massive data of the original dataset, and the difference between the outputs of the prediction model and the target model is calculated. (where l is the scaling factor), and use it as a measure of sample cognitive uncertainty; predict the label of sample x by screening the model, and use the uncertainty of the prediction as a measure of random uncertainty; where the uncertainty of the prediction may be implemented by various models, including but not limited to: logistic regression, ensemble models and Bayesian neural networks (BNN).

[0142] Then, using a sampler and combining multiple sample uncertainty measures, it is determined whether to label sample x. Possible methods include: an upper confidence bound algorithm, α(x) = f(x) + βu(x), which calculates the uncertainty of sample x based on its cognitive uncertainty and accidental uncertainty. If the uncertainty of sample x exceeds a certain threshold, then it is determined whether to label sample x based on the large language model. For example, a target cue p obtained through a cue self-evolutionary step could be used. * A hint is generated for sample x, and the hint is input into the large language model to guide the large language model to generate a quality score s for sample x, thus obtaining sample x and quality score label s.

[0143] For example, the decision to label sample x can also be made by calculating entropy based on random uncertainty.

[0144] Add the labeled samples (x, s) to the labeling database and update the weight parameters of the selection model using the labeling database; update the prediction model using samples: x, ψ(x); loop until convergence or the preset labeling budget is exhausted (i.e., the cost budget spent on labeling based on the large language model); at the end of the loop, output the selection model as a small model and the labeling database as the alignment training dataset D. a .

[0145] See also Figure 4 After performing the model alignment screening step, a specialized data mining step is performed to build a high-quality dataset for any domain.

[0146] Considering the massive amount of data to be filtered (the commonly used internet dataset Common Crawl exceeds 100TB of tokens), this application adopts a combination of coarse and fine screening: First, a coarse screening model (i.e., a small model) filters the massive amount of data, greatly reducing the amount of data to be processed. Then, a more powerful fine screening model (i.e., a quality scoring model) is used to score, rank, and select the Top K from the recalled data, ultimately mining the required high-quality specialized dataset (i.e., the target training dataset). This high-quality specialized dataset is then used as the pre-training dataset for the large language model, effectively improving the model's specific capabilities, such as reasoning ability, mathematical ability, and coding ability.

[0147] In other words, the training sample data in the target training dataset mined in this application embodiment has a relatively high quality score. The higher the quality score, the higher the gain of the target business model's target-specific capabilities after training with that training sample data. For example, if the target business model can be a code generation model (a large language model with code generation capabilities), and the target capability is code generation capability, then training the code generation model with training sample data with a higher quality score will improve the code generation capability of the code generation model, meaning the code generated by the code generation model will be more accurate and have fewer bugs. Conversely, training the code generation model with training data with a low quality score will result in a lower quality score. The code generation capability gain is low or non-existent, meaning there is no improvement in the accuracy of the code generated by the code generation model. For example, if the target business model is a route planning model and the target capability is route planning, then training the route planning model with training sample data that has a higher quality score will improve the route planning capability of the model, meaning the quality of the route planning results output by the model will be higher (e.g., the planned route is more reasonable, with less congestion). Conversely, training the route planning model with training data that has a low quality score will result in low or no gain in the route planning capability, meaning there is no improvement in the quality of the route planning results output by the model.

[0148] First, a small model obtained from the model alignment step is used to perform a coarse screening of the massive original dataset, recalling data to obtain a coarsely screened dataset. For example, each data point in the original dataset is used as input to the small model, which outputs a quality score for each data point in the original dataset. Then, data with quality scores below a preset threshold are filtered out, and the remaining data are recalled to obtain the coarsely screened dataset.

[0149] Then, the quality scoring model is used to predict the quality score of the coarsely screened dataset, and the quality prediction value of each data in the coarsely screened dataset is obtained. Based on the quality prediction value, the coarsely screened dataset is screened again to obtain the target training dataset.

[0150] Optionally, the quality scoring model can be a regression model, utilizing an aligned training dataset D. a The regression model is trained under supervision to obtain a trained regression model. Then, this trained regression model is used to predict the quality of each data point in the coarsely filtered dataset, resulting in predicted quality values ​​for each data point in the coarsely filtered dataset. Finally, the data in the coarsely filtered dataset is sorted based on the quality prediction results, and a specified number of data points are filtered as needed. For example, the top k data points with the highest predicted quality values ​​are selected as the target training dataset.

[0151] Figure 9 A schematic diagram illustrating the process of training a quality assessment model and performing data mining is shown. Figure 9 As shown, the process includes two parts: training and mining. The training and mining processes are described in detail below.

[0152] I. Training Process

[0153] First, obtain the alignment training dataset D. a As can be seen from the above, the training dataset D is aligned with the set. a This includes multiple datasets labeled with quality scores for the large language model. In other words, it aligns the training dataset D. a This includes multiple text data entries with quality score labels, thus saving on additional annotation costs for the dataset.

[0154] Then, align the training dataset D. a Feature extraction is performed on each text data in the dataset to obtain the aligned training dataset D. a The feature vector of each data point is used to facilitate the training of the subsequent quality scoring model.

[0155] Optionally, embodiments of this application target the aligned training dataset D. a The methods for feature extraction from any text data x include, but are not limited to:

[0156] 1) Use a single pre-trained language model to extract text features from text data x, such as the fasttext model.

[0157] 2) Use manually written rules to extract text features from text data x.

[0158] 3) Use multiple pre-trained language models to extract text features from text data x. For example, use multiple models such as fasttext, all-MiniLM-L6-v2, all-roberta-large-v1, and multi.wiki.bpe to extract features from text data x to obtain multiple semantic features of text data x. Then, fuse these multiple semantic features to obtain the text features of text data x. These text features are the feature vector of text data x.

[0159] 4) Use multiple pre-trained language models and manually written rules to extract text features from text data x. For example, use multiple models such as fasttext model, all-MiniLM-L6-v2 model, all-roberta-large-v1 model, multi.wiki.bpe model and manually written rules to extract features from text data x to obtain multiple semantic features of text data x. Then, fuse these multiple semantic features to obtain the text features of text data x, which is the feature vector of text data x.

[0160] Finally, based on the features and corresponding annotations of each extracted text data, a regression model is trained using a supervised training method. The regression model includes, but is not limited to, linear models (e.g., linear regression models), tree models (e.g., random forests, XGBoost, etc.), and multilayer perceptron models.

[0161] II. Excavation Process

[0162] First, obtain the dataset after coarse screening. Then, perform feature extraction on each text data in the coarse screening dataset (using the same feature extraction method as the training process) to obtain the feature vector of each text data. Then, use the feature vector of each text data as the input of the trained regression model and output the quality prediction value of each text data in the coarse screening dataset.

[0163] Then, based on the quality prediction values, the data in the coarse dataset are sorted in descending order, and a specified number of data are filtered as mining results. The filtering methods include, but are not limited to, taking the top k% of the sorted data as mining results and taking data with quality prediction values ​​greater than or equal to the threshold λ as mining results, to obtain high-quality specialized data. The high-quality specialized data obtained from mining is used to train the large language model, which effectively improves the specialized capabilities of the large language model, such as reasoning ability, planning ability, and code generation ability.

[0164] Figure 10This is a flowchart illustrating a training data acquisition method provided in an embodiment of this application. This method can be executed by any device, equipment, platform, or cluster of devices with computing capabilities. This application does not specifically limit the specific computing device executing the method; a suitable computing device can be selected as needed. For example, it can be executed by… Figure 1 The implementation is executed on server 130, providing users with arbitrary-targeted data mining services. Alternatively, the implementation can be executed collaboratively by terminal 11 and server 130. The following description uses a collaborative implementation by a terminal device and server as an example to illustrate the training data acquisition method provided in this application embodiment. Figure 10 As shown, the data acquisition method provided in this application embodiment includes at least steps S1001 to S1003.

[0165] In step S1001, the original dataset is obtained.

[0166] In this embodiment of the application, the original dataset D raw It can crawl massive amounts of text data from the internet, such as books, academic papers, forum posts, movie reviews, and any other text data from the internet. The specific original dataset D... raw It can be the common crawl dataset.

[0167] For example, when a user needs to perform data mining, they can open a data mining client on a terminal device (e.g., a mobile phone) and input data mining requirements on the client's interactive interface. For instance, the interface provides an input text box where the user can input their data mining requirements, such as the mined data being used to improve the reasoning ability of a large language model. Alternatively, the interface can provide multiple common options and a custom option. Common options include reasoning, planning, mathematics, code generation, etc. The user can select an option that matches their mining requirements, increasing user-friendliness. If no option matches the user's mining requirements, the user can click "custom" and input their mining requirements from the pop-up input box. After inputting the information, the user clicks "OK" or the "Start Mining" button. The terminal device then sends a mining request to a server equipped with the training data acquisition device provided in this embodiment. This mining request carries the user-inputted data mining requirements. The server receives the mining request from the terminal device, parses it, obtains the data mining requirements, responds to the request, reads the common crawl dataset, and obtains the original dataset.

[0168] In step S1002, the original dataset is filtered based on the small model to obtain the filtered dataset.

[0169] As described above, the original dataset contains a massive amount of text data. If it is filtered based on a large language model, the cost would be enormous and unbearable. Therefore, in order to reduce the filtering cost, this embodiment of the application uses a small model that has been trained and has a similar filtering effect to the large language model to perform a rough filtering of the original dataset, which effectively reduces the amount of subsequent data processing and increases the efficiency of data mining.

[0170] For example, each text data point in the original dataset is used as input to a small model, which outputs a quality score for each text data point in the original dataset. Then, those with quality scores below a preset threshold are filtered out, resulting in a coarsely filtered dataset. By using a small model with fewer parameters (e.g., less than 1 million parameters) to filter the original dataset, the data filtering cost is effectively reduced and the filtering efficiency is improved.

[0171] To further improve efficiency and achieve the ability to quickly align small and large language models, this application proposes a sampling method based on sample uncertainty, which achieves the ability to align large language models using only a small amount of data. The sampled data is then labeled with quality scores. Optionally, the labeling method used for large language models can be used to label the sampled data with quality scores, or manual labeling can be used.

[0172] It is understandable that if the sampled data is labeled using a large language model, the self-evolutionary steps mentioned above need to be executed. However, if the sampled data is labeled using a manual labeling method, the self-evolutionary steps mentioned above do not need to be executed.

[0173] For specific sampling methods based on sample uncertainty, please refer to the relevant descriptions above. For the sake of brevity, they will not be repeated here.

[0174] In step S1003, the target training dataset is obtained based on the filtered dataset.

[0175] In one example, the filtered dataset can be directly used as the target training dataset, and then the target business model can be trained using the target training dataset to improve the specific capabilities of the target business model.

[0176] In another example, after obtaining the dataset after filtering the small models, a more detailed secondary filtering is performed on the filtered dataset to further improve the quality of the training data obtained.

[0177] For example, the training sample data in the filtered dataset is used as the input to the quality assessment model, and the output is the quality prediction value of each training sample data in the filtered dataset. Then, based on the quality prediction value, the filtered dataset is filtered again to obtain the target training dataset.

[0178] Optionally, the quality assessment model can be a trained regression model, which takes each text data in the filtered dataset as input and outputs the predicted quality value of each text data in the filtered dataset.

[0179] For details on the specific structure and training process of the regression model, please refer to the above text. Figure 9 For the sake of brevity, the description will not be repeated here.

[0180] A specific implementation of further filtering the filtered dataset based on the quality prediction value to obtain the target training dataset can be: directly taking text data with a quality prediction value greater than a preset threshold λ as the target training dataset, and quickly mining the target training dataset.

[0181] In another example, the data in the coarsened dataset can be sorted in descending order based on the quality prediction value, and then the top K text data can be taken as the target dataset to achieve on-demand filtering of a specified number of data as the mining results.

[0182] Using the training data acquisition method provided in this application, 15B token-based reasoning-specific data was constructed, and a 13B large language model was trained using this data. Table 1 shows a comparison between the large language model trained using the reasoning-specific data mined by the training data acquisition method provided in this application and the large language model trained using randomly sampled data. The comparison reveals that the reasoning-related ability of the large language model trained using the reasoning-specific data mined by the training data acquisition method provided in this application is significantly improved.

[0183]

[0184] This application embodiment validates the effectiveness of self-evolved target prompts, expert-written prompts, and random sampling through data analysis. The effectiveness scores are shown in Table 2.

[0185]

[0186] Table 2

[0187] As can be seen from Table 2, the target prompts obtained through self-evolution provided in the embodiments of this application are significantly better than those obtained through random sampling and expert prompts.

[0188] Compared to baseline schemes such as random sampling / Epsilon-greedy, the alignment scheme in this application embodiment can efficiently and quickly align the data filtering capabilities of small models with those of large models.

[0189] Figure 11 A schematic diagram comparing the alignment scheme, random sampling scheme, and Epsilon-greedy baseline scheme provided in the embodiments of this application in terms of F1, precision, and recall is shown.

[0190] like Figure 11 As shown, the alignment scheme in this application embodiment, compared with baseline schemes such as random sampling / Epsilon-greedy, can efficiently and quickly align the data filtering capabilities of small models with those of large models.

[0191] Figure 12 and Figure 13 Schematic diagrams of the generalized inference-en regression results and the generalized inference-zh regression results are shown respectively. (From...) Figure 13 As can be seen, compared with general classification methods, the regression-based ranking method provided in this application provides higher quality data and allows for convenient control of the amount of data obtained, making it convenient for practical use.

[0192] Based on the same concept as the aforementioned embodiment of a training data acquisition method, this application also provides a training data acquisition device 1400. This device 1400 can be deployed on a server or terminal device to provide efficient and low-cost data mining services using high-quality data. The training data acquisition device 1400 includes components for implementing... Figure 3-13 The units or modules of each step in the training data acquisition method shown.

[0193] Figure 14 This is a schematic diagram of a training data acquisition device provided in an embodiment of this application. Figure 14 As shown, the training data acquisition device 1400 includes an acquisition module 1401, a filtering module 1402, and a determination module 1403. The acquisition module 1401 is used to acquire the original dataset, which includes multiple training sample data. The filtering module 1402 is used to filter the original dataset based on a small model to obtain a filtered dataset. The small model is trained based on an aligned training dataset, which includes multiple labeled training sample data. The labels are quality score labels. The training sample data in the aligned training dataset is sampled from the original dataset based on sample uncertainty. The determination module 1403 is used to obtain the target training dataset based on the filtered dataset.

[0194] In one possible implementation, sample uncertainty is derived from the cognitive uncertainty of the sample and the accidental uncertainty of the sample.

[0195] In another possible implementation, the training sample data in the alignment training dataset is obtained by sampling from the original dataset through the following steps: initializing the prediction model, target model, and selection model; using the training sample data x in the original dataset as input to the prediction model, target model, and selection model, respectively, to obtain the outputs of the prediction model, target model, and selection model, where the output of the selection model includes the random uncertainty of the samples; determining the cognitive uncertainty of the samples based on the difference between the outputs of the prediction model and the target model; determining the random uncertainty of the samples based on the output of the selection model; determining the sample uncertainty of the training sample data x based on the cognitive uncertainty and random uncertainty of the samples; determining whether to sample the training sample data x based on the sample uncertainty; adjusting the weight parameters of the prediction model based on the training sample data x and the output of the target model; and obtaining the training sample data in the alignment training dataset by sampling from the original dataset after multiple iterations.

[0196] In another possible implementation, the small model is trained based on the following steps: sampling from the original dataset based on sample uncertainty to obtain a sampled training dataset; labeling each training sample data in the sampled training dataset based on the large language model to obtain an aligned training dataset; and training the small model based on the aligned training dataset to obtain the trained small model.

[0197] In another possible implementation, the quality score labels are obtained based on annotations from a large language model or manually.

[0198] In another possible implementation, the number of parameters in the small model is less than a preset threshold.

[0199] In one possible implementation, the determining module 1403 is specifically used to: take multiple training sample data in the original dataset as input to a small model, output a first quality score for each training sample data in the original dataset, the first quality score indicating the level of gain obtained by the target business model when trained using the training sample data; filter the training sample data in the original dataset whose first quality score is less than a first threshold to obtain the filtered dataset.

[0200] In another possible implementation, a specific way to obtain the target training dataset based on the filtered dataset is as follows: the training sample data in the filtered dataset is used as the input of the quality assessment model, and the output is a second quality score for each training sample data in the filtered dataset. The prediction accuracy of the quality assessment model is higher than that of the small model. Based on the second quality score, the filtered dataset is filtered again to obtain the target training dataset.

[0201] In another possible implementation, a specific implementation of further filtering from the filtered dataset based on the second quality score to obtain the target training dataset is as follows: the training sample data in the filtered dataset are sorted in descending order based on the second quality score; the top K training sample data in the filtered dataset are used as the target training dataset, where K is a positive integer.

[0202] In another possible implementation, based on the second quality score, a further specific implementation of filtering from the filtered dataset to obtain the target training dataset is as follows: Training samples in the filtered dataset with a second quality score greater than or equal to a second threshold are used as the target training dataset. By setting a threshold, the dataset is quickly filtered to the desired target training dataset, increasing the efficiency of data mining.

[0203] In another possible implementation, the training sample data in the filtered dataset is used as input to the quality assessment model, and a specific implementation of outputting the second quality score of each training sample data in the filtered dataset is as follows: feature extraction is performed on each training sample data in the filtered dataset to obtain the feature vector of each training sample; the feature vector of each training sample is used as input to the trained regression model, and the second quality score of each training sample data is output.

[0204] In another possible implementation, a specific approach to extracting features from each training sample in the filtered dataset to obtain a feature vector for each training sample is as follows: multiple feature extraction methods are used to extract features from each training sample to obtain multiple semantic features corresponding to each training sample; these multiple semantic features are then fused to obtain a feature vector for each training sample.

[0205] In another possible implementation, the regression model is trained based on an aligned training dataset.

[0206] In another possible implementation, the labels of each training sample in the alignment training dataset are obtained based on a large language model and target prompts, and the target prompts are obtained based on each training sample and a target prompt template; the training data acquisition device 1400 provided in this application also includes a prompt self-evolution module 1404, which is used to obtain the target prompt template by reasoning evolution based on the large language model.

[0207] In another possible implementation, the prompt self-evolution module 1404 is specifically used for: obtaining data mining requirements and an initial prompt template. The data mining requirements indicate the target capability of the target business model that can be improved by the training sample data mined from the original dataset. The initial prompt template includes rule items, each rule item including several first rules, which indicate the features of the training sample data that meet the data mining requirements; extracting N training sample data from the original dataset, where N is a positive integer; using the data mining requirements and the N first prompts as input to a large language model, outputting a second quality score for the N training sample data, where the N first prompts are obtained based on the initial prompt template and the N training sample data; calling the large language model to summarize the rules of M training sample data, obtaining several second rules, where the M training sample data are the M training sample data with the highest second quality score among the N training sample data; updating the rule items of the initial prompt template based on the several second rules; verifying the evolution direction of the initial prompt template and obtaining the verification result; determining whether to accept the update of the initial prompt template based on the verification result; and obtaining the target prompt template after several iterations.

[0208] In another possible implementation, verifying the evolution direction of the initial prompt template to obtain the verification result is specifically implemented as follows: A second prompt is generated for each training sample in the verification dataset based on the updated initial prompt template. The verification dataset includes multiple training sample data with quality score labels. The second prompt is used as input to a large language model, which outputs a third quality score for each training sample in the verification dataset. The verification result is determined based on a first ranking result and a second ranking result, where the first ranking result is obtained by ranking the multiple training sample data in the verification dataset based on the third quality score, and the second ranking result is obtained by ranking the multiple training sample data in the verification dataset based on the quality score labels.

[0209] In another possible implementation, a specific way to determine the verification result based on the first sorting result and the second sorting result is to: calculate the degree of consistency between the first sorting result and the second sorting result; and determine the verification result based on the degree of consistency.

[0210] Optionally, there are multiple methods to calculate the consistency between the first and second sorting results, including but not limited to sorting relevance, editing examples, and cosine similarity. This application does not limit the specific method for calculating the consistency between the first and second sorting results, and appropriate consistency calculation methods can be selected as needed.

[0211] In another possible implementation, the target training dataset is used as a training dataset for the pre-training phase of the large language model; or, the target training dataset is used as a training dataset for the post-training / annealing phase of the large language model; or, the target training dataset is used as a training dataset for the fine-tuning phase of the large language model; or, the target training dataset is used as a training dataset for the alignment phase of the large language model; or, the target training dataset is used as a training dataset for the language model.

[0212] The training data acquisition device 1400 according to the embodiments of this application can correspond to the execution of the method described in the embodiments of this application, and the above and other operations and / or functions of each module in the training data acquisition device 1400 are respectively for implementing Figure 3-13 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0213] This application embodiment also provides a computing device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute... Figure 3-13 The method described.

[0214] Figure 15 A schematic diagram of the structure of a computing device provided in an embodiment of this application.

[0215] like Figure 15 As shown, the computing device 1500 includes at least one processor 1501, a memory 1502, and a communication interface 1503. The processor 1501, memory 1502, and communication interface 1503 are communicatively connected, which can be achieved via a wired (e.g., bus) or wireless connection. The communication interface 1503 is used to send and / or receive data from other devices. The memory 1502 stores computer instructions, which the processor 1501 executes to perform the methods described in the foregoing method embodiments, thereby achieving efficient and low-cost execution of data mining tasks that yield high-quality data.

[0216] It should be understood that, in the embodiments of this application, the processor 1501 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0217] The memory 1502 may include read-only memory and random access memory, and provides instructions and data to the processor 1501. The memory 1502 may also include non-volatile random access memory. Optionally, the random access memory may be, for example, high bandwidth memory (HBM).

[0218] The memory 1502 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0219] It should be understood that the computing device 1500 according to the embodiments of this application can perform the implementation of the embodiments of this application. Figure 3-13 The method shown is described in detail above, and will not be repeated here for the sake of brevity.

[0220] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer instructions are executed by a processor, the aforementioned method is implemented.

[0221] An embodiment of this application provides a chip including at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0222] Embodiments of this application provide a computer program or computer program product that includes instructions that, when executed, cause a computer to perform the methods mentioned above.

[0223] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0224] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented using hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0225] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for acquiring training data, characterized in that, include: Obtain the original dataset, which includes multiple training sample data; The original dataset is filtered based on the small model to obtain the filtered dataset small model. The small model is trained on the aligned training dataset, which includes multiple labeled training sample data. The labels are quality score labels. The training sample data in the aligned training dataset is sampled from the original dataset based on sample uncertainty. Based on the filtered dataset, the target training dataset is obtained.

2. The method according to claim 1, characterized in that, The sample uncertainty is derived from the cognitive uncertainty and the accidental uncertainty of the sample.

3. The method according to claim 2, characterized in that, The training sample data in the aligned training dataset is obtained by sampling from the original dataset through the following steps: Initialization yields the prediction model, target model, and selection model; The training sample data x in the original dataset is used as the input to the prediction model, the target model, and the screening model, respectively, to obtain the output of the prediction model, the output of the target model, and the output of the screening model. The output of the screening model includes the random uncertainty of the sample. The cognitive uncertainty of the sample is determined based on the difference between the output of the prediction model and the output of the target model. Based on the cognitive uncertainty and the accidental uncertainty of the sample, the sample uncertainty of the training sample data x is determined; Based on the aforementioned sample uncertainty, determine whether to sample the training sample data x. Based on the training sample data x and the output of the target model, the weight parameters of the prediction model are adjusted; After multiple iterations, training sample data in the aligned training dataset is obtained by sampling from the original dataset.

4. The method according to any one of claims 1-3, characterized in that, The small model is trained based on the following steps: Based on the aforementioned sample uncertainty, a sampled training dataset is obtained by sampling from the original dataset. The alignment training dataset is obtained by labeling each training sample data in the sampling training dataset based on the large language model. The small model is trained based on the alignment training dataset to obtain the trained small model.

5. The method according to any one of claims 1-4, characterized in that, The quality scoring labels are obtained based on large language model annotations or manual annotations.

6. The method according to any one of claims 1-5, characterized in that, The number of parameters in the small model is less than a preset threshold.

7. The method according to any one of claims 1-6, characterized in that, The process of filtering the original dataset based on a small model to obtain a filtered dataset includes: The original dataset contains multiple training sample data as input to the small model, and the first quality score of each training sample data in the original dataset is output. The first quality score indicates the level of gain obtained by the target business model in the target capability item after training with the training sample data. The training sample data in the original dataset with the first quality score less than the first threshold are filtered to obtain the filtered dataset.

8. The method according to any one of claims 1-7, characterized in that, The target training dataset obtained based on the filtered dataset includes: The training sample data in the filtered dataset is used as the input to the quality assessment model, and the second quality score of each training sample data in the filtered dataset is output. The prediction accuracy of the quality assessment model is higher than that of the small model. Based on the second quality score, the filtered dataset is further filtered to obtain the target training dataset.

9. The method according to any one of claims 1-8, characterized in that, The step of further filtering from the filtered dataset based on the second quality score to obtain the target training dataset includes: The training sample data in the filtered dataset are sorted in descending order based on the predicted quality values. The top K training samples from the filtered dataset are used as the target training dataset, where K is a positive integer.

10. The method according to any one of claims 1-9, characterized in that, The step of filtering from the filtered dataset based on the second quality score to obtain the target training dataset includes: The training sample data in the filtered dataset whose second quality score is greater than or equal to the second threshold are used as the target training dataset.

11. The method according to any one of claims 8-10, characterized in that, The step of using the training sample data in the filtered dataset as input to the quality assessment model and outputting a second quality score for each training sample data in the filtered dataset includes: Feature extraction is performed on each training sample data in the filtered dataset to obtain the feature vector of each training sample. The feature vector of each training sample is used as the input to the trained regression model, and a second quality score for each training sample data is output.

12. The method according to claim 11, characterized in that, The step of extracting features from each training sample in the filtered dataset to obtain a feature vector for each training sample includes: Multiple feature extraction methods are used to extract features from each training sample data to obtain multiple semantic features corresponding to each training sample data. The multiple semantic features are fused to obtain the feature vector of each training sample data.

13. The method according to claim 11 or 12, wherein the regression model is trained based on the aligned training dataset.

14. The method according to any one of claims 1-13, characterized in that, The label for each training sample in the alignment training dataset is obtained based on the large language model and the target prompt, and the target prompt is obtained based on each training sample and the target prompt template. The target prompt template is obtained through reasoning and evolution based on the large language model.

15. The method according to claim 14, characterized in that, The target prompt template is derived through reasoning and evolution based on the following steps: Obtain data mining requirements and an initial prompt template. The data mining requirements indicate the target capabilities that the training sample data mined from the original dataset can improve the target business model. The initial prompt template includes rule items, each rule item including several first rules. The rule items indicate the features of the training sample data that meet the data mining requirements. N training sample data are extracted from the original dataset, where N is a positive integer; The data mining requirements and the N first prompts are used as input to the large language model, and the third quality score of the N training sample data is output. The N first prompts are obtained based on the initial prompt template and the N training sample data. Based on the large language model, rules are summarized from the M training sample data to obtain several second rules. The M training sample data are the M training sample data with the highest third quality score among the N training sample data. The rule items of the initial prompt template are updated based on the aforementioned second rules; The evolution direction of the initial prompt template is verified, and the verification result is obtained; Based on the verification results, determine whether to accept the update of the initial prompt template; After several rounds of iteration, the target prompt template is obtained.

16. The method according to claim 15, characterized in that, The verification of the evolution direction of the initial prompt template, and the resulting verification results, include: A second prompt is generated for each training sample in the validation dataset based on the updated initial prompt template, the validation dataset including multiple training sample data with quality score labels; The second prompt is used as input to the large language model, and the output is a fourth quality score for each training sample in the validation dataset. The verification result is determined based on the first and second sorting results. The first sorting result is obtained by sorting multiple training sample data in the verification dataset based on the fourth quality score, and the second sorting result is obtained by sorting multiple training sample data in the verification dataset based on the quality score label.

17. The method according to claim 16, characterized in that, Determining the verification result based on the first and second sorting results includes: Calculate the degree of consistency between the first sorting result and the second sorting result; The verification result is determined based on the degree of consistency.

18. The method according to any one of claims 1-17, characterized in that, The target training dataset is used as the training dataset for the pre-training stage of the large language model; Alternatively, the target training dataset can be used as the training dataset for the post-training / annealing stage of a large language model. Alternatively, the target training dataset can be used as a training dataset for the fine-tuning phase of a large language model. Alternatively, the target training dataset can be used as the training dataset for the alignment phase of a large language model. Alternatively, the target training dataset can be used as the training dataset for the language model.

19. A training data acquisition device, characterized in that, include: The acquisition module is used to acquire the original dataset, which includes multiple training sample data. The filtering module is used to filter the original dataset based on the small model to obtain the filtered dataset. The small model is trained on the alignment training dataset, which includes multiple labeled training sample data. The labels are quality score labels. The training sample data in the alignment training dataset is sampled from the original dataset based on sample uncertainty. The determination module is used to obtain the target training dataset based on the filtered dataset.

20. A computing device, comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-18 to be implemented.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the method as described in any one of claims 1-18 to be implemented.

Images