Large model pre-training data screening method and system based on conformal inference, terminal equipment and medium

Abstract

The invention discloses a large model pre-training data screening method and system based on conformal inference, terminal equipment and a medium, and relates to the technical field of model training data screening. The method comprises the following steps: collecting original unlabeled data to construct a target data set; selecting a part of samples from the target data set, and constructing a calibration set containing real quality labels with high-quality or low-quality labeled samples; calculating uncertainty scores of low-quality samples of the calibration set and to-be-screened samples of the target data set by using a pre-training quality classifier; calculating a conformal test value of each to-be-screened sample based on the two types of scores; and determining a test value condition through the dynamic threshold, and screening the to-be-screened sample meeting the condition as a high-quality sample. Statistical theory support is provided based on conformal inference, the proportion of low-quality samples after training data screening is controllable, the quality and the screening amount of the training data can be balanced, the obtaining cost of high-quality training data is reduced, and the large model pre-training requirement is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model training data screening, in particular to a large model pre-training data screening method and system based on conformal inference, a terminal device and a medium. BACKGROUND

[0002] In the pre-training process of large language models, multi-modal models and other large models, high-quality training data is a key factor in determining model performance. However, the quality of large-scale raw data from the Internet or synthesis is uneven, and needs to be screened before being used for model training.

[0003] In the prior art, there are limitations in the screening method of training data. First, the training data relies on manual screening, which is costly and has poor scalability. Second, although the automatic screening method based on the model can reduce the cost, it is easy to introduce low-quality samples to pollute the data. Although there is a selective screening method in the prior art that retains high-confidence samples through a classifier, it lacks statistical theory support and cannot theoretically guarantee the quality of the screened data. Moreover, the bias of the model easily leads to the mixing of low-quality data in high-confidence samples, resulting in poor quality of training data. Furthermore, the prior art cannot maximize the screening amount while controlling the proportion of low-quality samples, making it difficult to meet the high standard requirements of data quality for industrial-level large model pre-training. Conformal inference is a statistical inference framework that does not require assumptions about the specific distribution of data or models. By using calibration data with real labels to quantify prediction uncertainty, it can provide reliable statistical guarantees such as controlling error rates and determining confidence ranges for machine learning, data screening and other tasks, and adapt to various scenarios such as black-box models. However, the prior art does not use conformal inference technology for training data screening of models.

[0004] Therefore, there is an urgent need for a large model pre-training data screening method to fill the gap in the prior art. SUMMARY

[0005] The technical problem to be solved by the present application is in the field of model training data screening. The existing methods lack statistical theory support, resulting in no guarantee of the quality of the screened data, the model bias easily mixes low-quality data, and the proportion of low-quality samples cannot be dynamically controlled to balance data quality and screening amount, making it difficult to adapt to industrial-level requirements. Therefore, there is an urgent need for an effective solution to solve the above technical problems.

[0006] To solve the above technical problems, the technical solution adopted by the present application is as follows: In a first aspect, the present application provides a large model pre-training data screening method based on conformal inference, comprising: Collecting raw unlabeled data to construct a target data set; selecting part of samples from the target dataset to construct a calibration set containing real quality labels; wherein the real quality labels mark the samples as high quality or low quality; using the pre-trained quality classifier, calculating the uncertainty scores of the low-quality samples in the calibration set, and calculating the uncertainty scores of the to-be-screened samples in the target dataset; For each to-be-screened sample, based on the uncertainty scores of the low-quality samples in the calibration set and the uncertainty score of the to-be-screened sample, calculating the conformal inspection value of the to-be-screened sample; determining the conformal inspection value condition through a dynamic threshold, and screening the to-be-screened samples in the target dataset that satisfy the conformal inspection value condition as high-quality samples.

[0007] In an implementation manner, the selecting part of samples from the target dataset to construct a calibration set containing real quality labels comprises: randomly selecting a preset proportion of data samples based on the target dataset; wherein the preset proportion is 5% to 10% of the total sample quantity of the target dataset; manually or domain expertly annotating the selected data samples to determine whether the data samples are high-quality or low-quality, and obtaining the calibration set containing real quality labels.

[0008] In an implementation manner, the step of obtaining the pre-trained quality classifier comprises: selecting part of unlabeled data samples from the target dataset; manually annotating the selected unlabeled data samples to determine whether the samples are high-quality or low-quality, and constructing a quality classifier training set; based on the quality classifier training set, fine-tuning a selected basic pre-trained model, or using a large language model to perform few-shot learning or zero-shot learning on the quality classifier training set, to obtain the pre-trained quality classifier; wherein the selected basic pre-trained model comprises a BERT model and a RoBERTa model, and the pre-trained quality classifier is used to calculate the confidence of a sample being high-quality.

[0009] In an implementation manner, the using the pre-trained quality classifier, calculating the uncertainty scores of the low-quality samples in the calibration set, and calculating the uncertainty scores of the to-be-screened samples in the target dataset comprises: using the pre-trained quality classifier, respectively performing quality prediction on the low-quality samples in the calibration set and the to-be-screened samples in the target dataset to obtain the confidence of each sample being high-quality; Based on the confidence of each sample being high quality, the uncertainty scores of the low-quality samples in the calibration set and the uncertainty scores of the samples to be screened in the target data set are calculated respectively by a complementary confidence algorithm.

[0010] In an implementation manner, the conformal inspection value of each sample to be screened is calculated based on the uncertainty scores of the low-quality samples in the calibration set and the uncertainty score of the sample to be screened, comprising: For any sample to be screened, the uncertainty score of the sample to be screened is extracted, defined as a target score; The number of samples in the calibration set whose uncertainty scores are greater than or equal to the target score is counted, defined as a matching number; The total number of low-quality samples in the calibration set is obtained, defined as a total number of low-quality samples; The conformal inspection value of the sample to be screened is calculated by statistical comparison based on the matching number and the total number of low-quality samples; wherein the conformal inspection value is negatively correlated with the confidence of the sample to be screened being a high-quality sample, and the conformal inspection value is used to reflect the similarity between the sample to be screened and the low-quality samples in the calibration set.

[0011] In an implementation manner, the condition of determining the conformal inspection value by a dynamic threshold value comprises: A low-quality sample proportion threshold value is preset; wherein the low-quality sample proportion threshold value is used to limit the maximum allowed proportion of low-quality samples in the high-quality sample set after screening; The conformal inspection values of all samples to be screened in the target data set are obtained, and the conformal inspection values are arranged in ascending order to obtain an ordered conformal inspection value sequence; The ordered conformal inspection value sequence is traversed, and for the conformal inspection value corresponding to each index in the sequence, whether the conformal inspection value corresponding to the index meets the screening requirement is judged based on the low-quality sample proportion threshold value and the total number of low-quality samples by a statistical constraint condition, and the maximum index meeting the statistical constraint condition is determined; The conformal inspection value corresponding to the maximum index in the ordered conformal inspection value sequence is defined as a dynamic threshold value, and a conformal inspection value condition is obtained; wherein the conformal inspection value condition is that the conformal inspection value of the sample to be screened is less than or equal to the dynamic threshold value.

[0012] In an implementation manner, the samples to be screened in the target data set whose conformal inspection values meet the conformal inspection value condition are screened as high-quality samples, comprising: For each sample to be screened in the target data set, the sample to be screened is conditionally judged by using the conformal inspection value condition. If the conformal test value of the sample to be screened is less than or equal to the dynamic threshold value, the sample to be screened is marked as a high-quality sample.

[0013] In a second aspect, the embodiments of the present application also provide a large model pre-training data screening system based on conformal inference, which comprises: A target data set acquisition module is configured to collect original unlabeled data and construct a target data set. A calibration set acquisition module is configured to select part of samples from the target data set and construct a calibration set containing real quality labels; wherein the real quality labels mark the samples as high-quality or low-quality. An uncertainty score calculation module is configured to calculate the uncertainty scores of low-quality samples in the calibration set using a pre-trained quality classifier, and calculate the uncertainty scores of samples to be screened in the target data set. A conformal test value calculation module is configured to calculate, for each sample to be screened, a conformal test value of the sample to be screened based on the uncertainty scores of low-quality samples in the calibration set and the uncertainty score of the sample to be screened. A screening module is configured to determine a conformal test value condition through a dynamic threshold value, and screen samples to be screened in the target data set that satisfy the conformal test value condition as high-quality samples.

[0014] In a third aspect, the embodiments of the present application also provide a terminal device, which comprises a memory, a processor, and a large model pre-training data screening program based on conformal inference stored in the memory and executable on the processor. When the processor executes the large model pre-training data screening program based on conformal inference, the steps of the large model pre-training data screening method based on conformal inference in any of the above solutions are implemented.

[0015] In a fourth aspect, the embodiments of the present application also provide a computer readable storage medium, which stores a large model pre-training data screening program based on conformal inference. When the processor executes the large model pre-training data screening program based on conformal inference, the steps of the large model pre-training data screening method based on conformal inference in any of the above solutions are implemented.

[0016] Beneficial effects: The application discloses a large model pre-training data screening method and system based on conformal inference, a terminal device and a medium, and relates to the technical field of model training data screening. The method first collects original unlabeled data to construct a target data set. Then, part of the samples in the target data set are selected to construct a calibration set containing real quality labels; wherein the real quality labels mark the samples as high quality or low quality. Then, a pre-trained quality classifier is used to calculate the uncertainty scores of the low-quality samples in the calibration set and the uncertainty scores of the samples to be screened in the target data set. Next, for each sample to be screened, based on the uncertainty scores of the low-quality samples in the calibration set and the uncertainty scores of the sample to be screened, the conformal test value of the sample to be screened is calculated. Finally, the conformal test value condition is determined by a dynamic threshold, and the samples to be screened in the target data set that meet the conformal test value condition are screened as high-quality samples. The application provides statistical theory support based on conformal inference, theoretically guarantees that the proportion of low-quality samples in the high-quality sample set after screening does not exceed the preset threshold, solves the problem of lack of quality guarantee in existing methods, and balances the quality and screening amount through a dynamic threshold to maximize the amount of screened data under the premise of quality control. In addition, the method is suitable for various large models such as large language models and multi-modal models, has strong expansibility, does not require a large amount of manual screening, significantly reduces the cost of obtaining high-quality data, and meets the high-standard demand of industrial-level large model pre-training for data quality. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The flowchart of the specific implementation mode of the large model pre-training data screening method based on conformal inference provided by the embodiments of the application is shown.

[0018] Figure 2 The algorithm flowchart of the large model pre-training data screening method based on conformal inference provided by the embodiments of the application is shown.

[0019] Figure 3 The principle block diagram of the large model pre-training data screening device based on conformal inference provided by the embodiments of the application is shown.

[0020] Figure 4 The internal structure principle block diagram of the terminal device provided by the embodiments of the application is shown. DETAILED DESCRIPTION

[0021] To make the purpose, technical scheme and effect of the application clearer and more explicit, the application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0022] The flowcharts shown in the drawings are merely illustrative and do not necessarily include all contents and operations or steps, nor are they necessarily executed in the order described. For example, some operations or steps can be further decomposed, combined or partially merged, so the actual execution order can be changed according to actual conditions.

[0023] It should be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second", etc. are used to distinguish the same or similar items with basically the same function and role. For example, the first control information and the second control information are only used to distinguish different control information, and do not limit the order.

[0025] Those skilled in the art can understand that the terms "first", "second", etc. do not limit the quantity and execution order, and the terms "first", "second", etc. do not necessarily mean different.

[0026] It should also be understood that the term "and / or" used in the specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0027] In the field of artificial intelligence, the performance improvement of large models (including but not limited to large language models, visual models and multi-modal models) highly depends on high-quality data in the pre-training stage. High-quality data can effectively optimize the model parameter update efficiency and reduce model overfitting or bias. However, various low-quality data such as samples containing factual errors, semantic redundancy and logical confusion can directly pollute the training process, leading to decreased reasoning accuracy and weakened generalization ability of the model, and even producing unexpected output results.

[0028] Currently, the original data for pre-training of large models mainly comes from Internet public resources or model synthesis generation. Such data often has the characteristics of coexistence of massiveness and quality difference. On the one hand, the data size can reach tens of billions or even hundreds of billions, providing a basis for large models to learn complex features; on the other hand, the data quality is uneven, containing both high-quality samples that can significantly improve the performance of the model and a large number of low-quality samples that are worthless or harmful, so the original data must be screened and processed before pre-training.

[0029] Existing data screening techniques mainly fall into three categories, but they all have limitations that are difficult to overcome.

[0030] The first type is artificial screening technology, which specifically labels the quality level of each sample by artificial or field experts and screens. Although it can ensure screening accuracy, it has problems of high cost and poor scalability. In the face of billions of large model pre-training data, artificial screening requires a lot of manpower and time, and it is difficult to complete batch processing in a short time, and it is completely unable to adapt to the rapid iteration needs of industrial large models.

[0031] The second type is traditional model automatic screening technology, which specifically uses the sample quality confidence output by simple classification models such as logistic regression and shallow neural networks to set a fixed threshold to screen high-confidence samples. Although this technology reduces labor costs, it is easy to introduce low-quality samples. Because the model may make systematic mistakes for specific types of samples such as rare field text and special format data, the confidence distribution will shift, resulting in a large number of low-quality data mixed in high-confidence samples. For example, the model may judge a text with smooth semantics but factual errors as a high-confidence high-quality sample, ultimately harming the subsequent pre-training effect.

[0032] The third type is selective screening technology, which is specifically an improved solution to traditional model screening technology. The core idea is to only keep high-confidence samples determined by the classifier, and low-confidence samples are subject to secondary human review. However, this technology still has defects, including lack of theoretical quality guarantee, prominent model bias problem, and inability to balance quality and quantity. Specifically, because the screening logic is mostly based on heuristic strategies without strict statistical theory support, it cannot guarantee from a mathematical perspective that the proportion of low-quality samples in the screened sample set is within a controllable range, that is, it cannot quantify the quality risk of screened data; when the classifier has cognitive bias for certain samples, the confidence output cannot truly reflect the sample quality, resulting in the high-confidence sample pool being contaminated by low-quality samples; the threshold of traditional selective screening technology is fixed, and if the threshold is increased to ensure quality, a large number of high-quality samples will be mistakenly excluded, reducing data utilization; if the threshold is lowered to retain more samples, more low-quality samples will be introduced, and it is always impossible to balance the proportion of low-quality samples and maximize the amount of screened data.

[0033] Therefore, existing large model pre-training data screening technologies cannot simultaneously meet the needs of guaranteeing the quality of screened data in a statistical sense, adapting to batch processing of massive data, and balancing quality and quantity, and cannot meet the high standard requirements of industrial large model pre-training for data quality.

[0034] To address this, a method for screening high-quality pre-trained data for large models using conformal inference is proposed. This method is based on a conformal inference and error detection rate control framework. Its core idea is based on a classifier that can discriminate data quality. First, the confidence distribution of low-quality samples in a calibration set containing real labels is statistically analyzed to construct a statistical test metric. Then, this test metric is applied to unscreened samples to achieve an objective judgment of the reliability of their prediction results.

[0035] This method can automatically select samples that are statistically proven to have high confidence and low error rate as high-quality training data, thereby maximizing the amount of automatically screenable data while ensuring data quality. This method is applicable to training data screening scenarios for large language models, visual models, and multimodal models, and can significantly reduce the cost of acquiring high-quality data while providing quantifiable assurance of the quality of the screened data.

[0036] This embodiment provides a method for filtering pre-training data for large models based on conformal inference, such as... Figure 1 As shown, the specific steps include the following: Step S100: Collect raw, unlabeled data and construct the target dataset.

[0037] In this embodiment, raw unlabeled data refers to raw data that has not yet been labeled with quality attributes by manual or automated tools. Specifically, it can be various types of data such as text, images, and audio obtained from various channels such as publicly available internet platforms, professional databases, and large-scale model synthesis. The target dataset is a massive collection of data formed after collecting, organizing, deduplicating, and other preprocessing of this raw unlabeled data, and it is a potential source of pre-training data for large models.

[0038] By collecting data from multiple channels, a sufficient data foundation can be provided for subsequent screening, ensuring that there is a enough amount of high-quality data to support the pre-training of large models after data screening, and avoiding performance limitations caused by insufficient original data. At the same time, the deduplication operation in the preprocessing process can reduce the impact of redundant data on screening efficiency and improve the execution efficiency of subsequent steps.

[0039] Step S200: Select a portion of samples from the target dataset to construct a calibration set containing real quality labels; wherein the samples labeled with real quality labels are either high quality or low quality.

[0040] In this embodiment, the true quality label is an identifier used to clearly define the quality attributes of a sample, and can be determined according to a preset quality standard. In this embodiment, the true quality label includes high-quality labels and low-quality labels, used to indicate whether the sample data is of high or low quality. High-quality samples typically refer to samples without factual errors, semantically coherent, with valid information, and meeting pre-training requirements, while low-quality samples include samples with factual bias, semantic redundancy, logical inconsistencies, or no practical value. The calibration set is a dataset formed by selecting a portion of samples from the target dataset and labeling them with true quality labels. Its purpose is to provide a reliable reference distribution for subsequent conformal inference and to evaluate the prediction uncertainty of the quality classifier.

[0041] By introducing a calibration set with real labels, the limitations of existing technologies that rely on heuristic strategies are overcome, providing a foundation for the construction of subsequent statistical test measures. This allows the screening results to be supported by statistical theory and solves the problem that existing technologies cannot guarantee the quality of screening data.

[0042] In one implementation, the step of selecting a subset of samples from the target dataset to construct a calibration set containing true quality labels specifically includes the following steps: Step S210: Based on the target dataset, randomly select a preset proportion of data samples; wherein the preset proportion is 5% to 10% of the total sample size of the target dataset; Step S220: Manually label or label the selected data samples with domain experts to determine whether the data samples are of high quality or low quality, and obtain a calibration set containing real quality labels.

[0043] In this embodiment, a predetermined proportion of samples is first randomly selected from the target dataset. This predetermined proportion is set to 5% to 10% of the total sample size of the target dataset. This proportion ensures the representativeness of the calibration set while fully reflecting the quality distribution characteristics of the target dataset, and controls the annotation cost, avoiding excessive workload and inefficiency caused by selecting too many samples for manual or expert annotation. The random selection method avoids sample selection bias, ensuring that the samples in the calibration set can objectively reflect the overall quality of the target dataset, providing an unbiased reference distribution for subsequent statistical inference.

[0044] The selected samples are then labeled manually or by domain experts. Manual labeling is suitable for scenarios with general quality standards, while domain expert labeling is suitable for professional fields with high requirements for quality judgment. The two labeling methods are adapted to different scenario needs, ensuring the accuracy of the labeling results.

[0045] Specifically, such as Figure 2 As shown in the input steps, the complete calibration set is defined as follows:

[0046] in, To calibrate the data samples in the set, , For data sample indexing, To calibrate the number of data samples in the data set.

[0047] The unlabeled samples to be filtered in the target dataset are defined as follows:

[0048] in, For data sample indexing, This represents the number of unlabeled samples to be filtered in the target dataset.

[0049] The constructed calibration set is characterized by strong representativeness and accurate labeling, which can provide reliable basic data for conformal inference, further strengthening the statistical guarantee of the screening results, making the subsequent uncertainty calculation and conformal test value construction based on the calibration set more accurate, and ultimately improving the quality and reliability of the screening data.

[0050] Step S300: Using a pre-trained quality classifier, calculate the uncertainty score of low-quality samples in the calibration set, and calculate the uncertainty score of samples to be screened in the target dataset.

[0051] In this embodiment, the pre-trained quality classifier is a model capable of judging sample quality. Through a specific training process, it acquires the ability to judge the high-quality attributes of samples and can output a confidence level that a sample is of high quality. The uncertainty score is an indicator that quantifies the uncertainty of the model's judgment on sample quality. It is specifically derived based on the high-quality confidence level using a specific algorithm and is used to reflect the reliability of the model's judgment on sample quality.

[0052] By quantifying uncertainty, unreliable regions in model judgments can be effectively identified, the impact of model bias can be reduced, and low-quality samples can be avoided from being mixed into high-confidence sets due to systematic misjudgments by the model. This provides accurate input data for the subsequent calculation of conformal test values ​​and ensures the accuracy of statistical inference.

[0053] In one implementation, the step of obtaining the pre-trained quality classifier specifically includes: Step S310: Select a portion of unlabeled data samples from the target dataset; Step S320: Manually label the selected unlabeled data samples to determine whether the samples are of high quality or low quality, and construct a training set for the quality classifier. Step S330: Based on the quality classifier training set, fine-tune the selected basic pre-trained model, or use a large language model to perform few-shot or zero-shot learning on the quality classifier training set to obtain the pre-trained quality classifier; wherein, the selected basic pre-trained model includes BERT model and RoBERTa model, and the pre-trained quality classifier is used to calculate the confidence that the sample is of high quality.

[0054] In this embodiment, a portion of unlabeled samples are first selected from the target dataset and manually labeled to construct a training set for the quality classifier. The training set is constructed based on the logic that the quality of labeled samples determines the classifier's performance. Specifically, manual labeling clarifies the high-quality and low-quality attributes of samples, providing supervisory signals for classifier training. Subsequently, based on this training set, the classifier can be trained in two ways: one is to fine-tune a selected base pre-trained model, and the other is to use a large language model for few-shot or zero-shot learning. The selected base pre-trained models include BERT and RoBERTa models. These models possess powerful semantic understanding capabilities and, after fine-tuning, can quickly adapt to specific tasks related to sample quality assessment, making them suitable for scenarios with a certain amount of labeled samples and sufficient computational resources. Few-shot learning using a large language model is suitable for situations with few labeled samples, while zero-shot learning requires no additional labeled training samples; the model can understand the quality assessment task simply through natural language instructions, making it suitable for scenarios with extremely high labeling costs or scarce data.

[0055] Specifically, such as Figure 2 As shown in the input steps, the pre-trained quality classifier is defined as... Its output This represents the probability that the sample is of high quality, i.e., the confidence level that the sample is of high quality.

[0056] By selecting appropriate training methods based on available annotation and computational resources, a classifier can be used to calculate high-quality confidence scores for samples, providing data input for subsequent uncertainty score calculations. The technical advantage lies in ensuring that the quality classifier possesses reliable quality discrimination capabilities across various scenarios through diverse training methods, providing high-quality confidence outputs for the entire screening process and reducing screening errors caused by insufficient classifier performance.

[0057] Furthermore, the classifier can be replaced with lighter and faster small models such as recurrent neural networks (RNN) and long short-term memory networks (LSTM), further expanding the applicability of the solution and adapting it to scenarios with different computing resource constraints.

[0058] In one implementation, the step of using a pre-trained quality classifier to calculate the uncertainty score of low-quality samples in the calibration set and to calculate the uncertainty score of samples to be screened in the target dataset specifically includes the following steps: Step S340: Using the pre-trained quality classifier, perform quality prediction on the low-quality samples in the calibration set and the samples to be screened in the target dataset, respectively, to obtain the confidence score of each sample as high quality; Step S350: Based on the confidence that each sample is of high quality, the uncertainty scores of the low-quality samples in the calibration set and the uncertainty scores of the samples to be screened in the target dataset are calculated using the complementary confidence algorithm.

[0059] In this embodiment, a pre-trained quality classifier is first used to predict the quality of low-quality samples in the calibration set and the samples to be screened in the target dataset, outputting a confidence score for each sample to be of high quality. This confidence score is a quantification of the classifier's confidence in the sample belonging to the high-quality category based on its own model parameters and training experience, ranging from 0 to 1. A higher confidence score indicates that the classifier believes the sample is more likely to be of high quality. Subsequently, based on this confidence score, an uncertainty score is calculated using a complementary confidence algorithm. This algorithm subtracts the confidence score of the sample being of high quality from 1, utilizing the complementary relationship between confidence and uncertainty to transform the classifier's confidence in the high-quality attribute into uncertainty about the judgment result. For example, if the confidence score of a sample for high quality is 0.8, its uncertainty score is 0.2, indicating that the classifier's judgment of the sample's high quality has a 20% uncertainty; if the confidence score is 0.3, the uncertainty score is 0.7, indicating that the classifier's judgment of the sample's quality is extremely unreliable.

[0060] Specifically, such as Figure 2 As shown in step 1, low-quality samples in the calibration set are identified. Based on the true labels, a subset of low-quality samples is extracted, represented as:

[0061] Among them, let This indicates the number of low-quality samples.

[0062] Subsequently, as Figure 2 As shown in step 2, calculate the uncertainty score.

[0063] For all low-quality calibration samples, arrive The uncertainty score is expressed as:

[0064] For all samples to be screened, for arrive The uncertainty score is expressed as:

[0065] By employing a complementary confidence algorithm, the abstract confidence level is transformed into an intuitive uncertainty score. This score directly reflects the reliability of the classifier's judgment, providing a unified and accurate quantitative indicator for the subsequent calculation of conformal test values. Furthermore, this calculation method is simple and efficient, supports batch data processing, and is suitable for industrial-grade massive data screening scenarios, ensuring the computational efficiency of the entire screening process.

[0066] Step S400: For each sample to be screened, calculate the conformal test value of the sample to be screened based on the uncertainty score of the low-quality samples in the calibration set and the uncertainty score of the sample to be screened.

[0067] In this embodiment, the conformal test value is a statistical test metric constructed based on the uncertainty scores of low-quality samples in the calibration set and the sample to be screened. It is used to objectively reflect the similarity between the sample to be screened and the low-quality samples. Its calculation logic is to determine the relative position of the sample to be screened in the distribution of low-quality samples by comparing the uncertainty score of the sample to be screened with the uncertainty score distribution of low-quality samples in the calibration set, thereby quantifying the risk that it is a low-quality sample.

[0068] The construction of conformal test values ​​does not depend on specific distribution assumptions of the data or model, possessing distribution-independent characteristics and seamlessly adapting to black-box models. This provides rigorous statistical theoretical support for the screening process, overcoming the lack of quality assurance in existing technologies. By quantifying the similarity between the samples to be screened and low-quality samples, it provides interpretable and verifiable criteria for subsequent screening.

[0069] In one implementation, the step of calculating the conformal test value of each sample to be screened, based on the uncertainty score of the low-quality samples in the calibration set and the uncertainty score of the sample to be screened, specifically includes the following steps: Step S410: For any sample to be screened, extract the uncertainty score of the sample to be screened, and define it as the target score; Step S420: Count the number of samples in the calibration set whose uncertainty score is greater than or equal to the target score, and define it as the number of matches; Step S430: Obtain the total number of low-quality samples in the calibration set, which is defined as the total number of low-quality samples; Step S440: Combining the number of matches with the total number of low-quality samples, the conformal test value of the sample to be screened is calculated through statistical comparison; wherein, the magnitude of the conformal test value is negatively correlated with the confidence that the sample to be screened is a high-quality sample, and the conformal test value is used to reflect the similarity between the sample to be screened and the low-quality samples in the calibration set.

[0070] In this embodiment, firstly, for any sample to be screened, its uncertainty score is extracted and defined as the target score. This target score is the core indicator for measuring the uncertainty of the sample's quality. Then, the number of samples with uncertainty scores greater than or equal to the target score among all low-quality samples in the calibration set is counted; this is the matching number. This number reflects the position of the sample's uncertainty score within the uncertainty distribution of low-quality samples. A higher matching number indicates that the uncertainty of the sample to be screened is similar to that of more low-quality samples, and the higher the risk of it being a low-quality sample. Next, the total number of low-quality samples in the calibration set is obtained; this value is the basis for calculating the statistical proportion. Finally, combining the matching number and the total number of low-quality samples, a conformal test value is calculated through statistical comparison. Specifically, it is calculated by adding 1 to the sum of the matching numbers and dividing by the sum of the total number of low-quality samples plus 1. This test value is negatively correlated with the confidence that the sample to be screened is of high quality. The smaller the test value, the greater the difference between the sample to be screened and the low-quality samples, and the higher the confidence; the larger the test value, the higher the similarity between the sample and the low-quality samples, and the higher the risk.

[0071] Specifically, such as Figure 2 As shown in step 3, construct the conformal test value. For each sample to be screened arrive Calculate the conformal test value of this sample relative to the distribution of low-quality samples. , is represented as:

[0072] By using clear steps and quantitative calculations, the process of calculating conformal test values ​​is standardized and reproducible, ensuring that the test value of each sample to be screened can objectively reflect its correlation with low-quality samples.

[0073] In addition, new conformal test values ​​can be constructed using high-quality data from the calibration set, or by simultaneously predicting conformal test values ​​using multiple models, thereby enhancing the robustness of the screening process and further expanding the flexibility of the scheme.

[0074] Step S500: Determine the conformal test value condition through a dynamic threshold, and select the samples in the target dataset whose conformal test values ​​meet the conformal test value condition as high-quality samples.

[0075] In this embodiment, the dynamic threshold is a screening critical value adaptively determined by a specific algorithm based on a preset low-quality sample proportion threshold. Unlike traditional fixed thresholds, it can be dynamically adjusted according to the conformal test value distribution of the samples to be screened. The conformal test value condition is a screening rule set based on the dynamic threshold, used to determine whether the samples to be screened meet the high-quality standard.

[0076] The advantage of using dynamic thresholds to determine screening criteria lies in achieving a balance between quality control and data volume. By setting a threshold for the proportion of low-quality samples, the maximum allowable proportion of low-quality samples is limited. This allows the dynamic threshold algorithm to maximize the number of high-quality samples selected while meeting the quality constraints, solving the problems of quality control failure or low data utilization in traditional fixed threshold methods. It is also suitable for the needs of industrial-grade large-scale model pre-training for massive amounts of high-quality data.

[0077] In one implementation, determining the conformal test value condition through a dynamic threshold specifically includes the following steps: Step S510: Preset a low-quality sample ratio threshold; wherein, the low-quality sample ratio threshold is used to limit the maximum allowable ratio of low-quality samples in the high-quality sample set after screening; Step S520: Obtain the conformal test values ​​of all samples to be screened in the target dataset, and arrange the conformal test values ​​in ascending order to obtain an ordered conformal test value sequence; Step S530: Traverse the ordered conformal test value sequence. For each index in the sequence, based on the low-quality sample proportion threshold and the total number of low-quality samples, determine whether the conformal test value corresponding to the index meets the screening requirements through statistical constraints, and determine the largest index that satisfies the statistical constraints. Step S540: Define the conformal test value corresponding to the maximum index in the ordered conformal test value sequence as a dynamic threshold to obtain the conformal test value condition; wherein, the conformal test value condition is that the conformal test value of the sample to be screened is less than or equal to the dynamic threshold.

[0078] In this embodiment, a low-quality sample proportion threshold is first preset, with a value ranging from 0 to 1. Its function is to limit the maximum allowable proportion of low-quality samples in the high-quality sample set after screening. For example, a low-quality sample proportion threshold of 0.05 means that the proportion of low-quality samples in the screened samples must not exceed 5%. Users can set this threshold according to their actual quality requirements. Then, the conformal test values ​​of all samples to be screened are obtained and arranged in ascending order to form an ordered conformal test value sequence. The purpose of this sorting is to screen samples from high to low confidence, ensuring that samples with the lowest risk are retained first. Next, the sequence is traversed. For each index, the test value is used to determine whether the screening requirements are met based on the low-quality sample proportion threshold and the total number of low-quality samples, using statistical constraints. During the traversal, the maximum index that satisfies this condition needs to be determined. The test value corresponding to this index is the dynamic threshold. The selection of the maximum index ensures that as many samples as possible are screened while satisfying the low-quality sample proportion constraint.

[0079] Specifically, such as Figure 2 The input steps are shown in step 4. First, define the proportion of target low-quality data, expressed as:

[0080] Subsequently, dynamic threshold filtering based on FDR (False Discovery Rate) control was completed, targeting... arrive All Sort in ascending order to get:

[0081] Subsequently, the largest index that satisfies the low-quality data ratio control was found. :

[0082] If the set is empty, then .

[0083] The dynamic threshold algorithm enables adaptive adjustment of the threshold. Compared with traditional fixed threshold and multiple hypothesis testing algorithms, it can maximize the amount of data screened while strictly controlling the proportion of low-quality samples, thus solving the problem that existing technologies cannot balance quality and quantity.

[0084] In addition, variations of multiple hypothesis testing can be used, such as the BH procedure, Storey-BH procedure, and other multiple hypothesis testing methods, to further improve the adaptability of the scheme. Among them, the BH procedure is a commonly used multiple hypothesis testing correction method in statistics, which is used to control the false discovery rate.

[0085] In one implementation, the step of selecting samples from the target dataset whose conformity test values ​​satisfy the conformity test value conditions as high-quality samples specifically includes the following steps: Step S550: For each sample to be screened in the target dataset, use the conformal test value condition to make a conditional judgment on the sample to be screened; Step S560: If the conformal test value of the sample to be screened is less than or equal to the dynamic threshold, then the sample to be screened is marked as a high-quality sample.

[0086] In this embodiment, for each sample to be screened in the target dataset, the conformal test value condition determined earlier is used for judgment. This condition is whether the conformal test value of the sample is less than or equal to a dynamic threshold. The judgment process adopts a sample-by-sample verification method to ensure that each sample can obtain an accurate quality judgment. It supports parallel computing, can adapt to batch processing scenarios of massive data, and improves screening efficiency. If the conformal test value of the sample to be screened meets the condition, it means that the sample has a low similarity to low-quality samples, and the confidence of it being a high-quality sample is high, so it is marked as a high-quality sample; if the condition is not met, it means that the risk of the sample being a low-quality sample exceeds the allowable range, and it is not included in the high-quality sample set.

[0087] Specifically, such as Figure 2 As shown in the output steps, a high-quality sample selection set is output. :

[0088] Through explicit conditional judgments and labeling logic, high-quality samples are accurately screened. The screening results strictly adhere to the pre-defined constraint on the proportion of low-quality samples, providing controllable data for the pre-training of large models. Furthermore, the selected high-quality samples can directly enter the pre-training stage without additional manual secondary screening, reducing the cost of acquiring high-quality data and meeting the needs of rapid iteration for industrial-grade large models.

[0089] In addition, this screening process can be combined with a manual review system to form a complete workflow that combines AI pre-screening with expert review, which can further improve data quality while minimizing manpower costs.

[0090] In summary, this embodiment proposes a method for filtering pre-trained data of large models based on conformal inference. This method achieves controllable quality, model independence, strong scalability, and high interpretability in filtering pre-trained data of large models. Specifically, this method ensures, statistically speaking, that the expected proportion of low-quality data in the filtering results does not exceed a preset threshold. Furthermore, this method relies solely on the model prediction confidence level and can be adapted to any type of large model. It also supports parallel computation and batch data processing, making it suitable for industrial-grade data filtering platforms. Moreover, based on the conformal inference framework, this method possesses strict statistical guarantees and interpretability.

[0091] Existing methods for large model data screening and automatic labeling primarily rely on traditional heuristic confidence-based screening strategies, such as setting fixed confidence thresholds or directly screening samples based on model output probabilities. These methods cannot guarantee the quality of the screened data. This embodiment introduces a conformal inference framework into the field of large model data screening for the first time. By constructing calibrable statistical test values, it theoretically ensures that the expected proportion of low-quality data in the screening results is below a preset threshold, achieving controllable data screening. The core of this theory lies in its distribution-independent property, meaning that the validity of its conclusions does not depend on specific distribution assumptions about the model's internal parameters or the data generation process. This characteristic allows it to be seamlessly applied to black-box models with unknown or uninterpretable structures. Furthermore, this embodiment proposes a dynamic screening algorithm that can automatically determine the optimal confidence threshold while controlling the proportion of low-quality data, achieving the goal of maximizing the amount of screenable data under the constraint of low-quality data proportion. Compared to traditional multiple hypothesis testing screening algorithms, this algorithm can select more data.

[0092] like Figure 3 As shown in the figure, this embodiment of the invention provides a large model pre-training data screening system based on conformal inference. The system includes: a target dataset acquisition module 10, a calibration set acquisition module 20, an uncertainty score calculation module 30, a conformal test value calculation module 40, and a screening module 50.

[0093] Specifically, the target dataset acquisition module 10 is used to collect raw unlabeled data and construct a target dataset; the calibration set acquisition module 20 is used to select a portion of samples from the target dataset and construct a calibration set containing real quality labels; wherein, the samples labeled with real quality labels are high-quality or low-quality; the uncertainty score calculation module 30 is used to calculate the uncertainty score of low-quality samples in the calibration set using a pre-trained quality classifier, and to calculate the uncertainty score of the samples to be screened in the target dataset; the conformal test value calculation module 40 is used to calculate the conformal test value of each sample to be screened based on the uncertainty score of low-quality samples in the calibration set and the uncertainty score of the sample to be screened; the screening module 50 is used to determine the conformal test value conditions through a dynamic threshold, and to screen the samples in the target dataset whose conformal test values ​​meet the conformal test value conditions as high-quality samples.

[0094] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 4 As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a large model pre-training data filtering method based on conformal inference. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0095] Those skilled in the art will understand that Figure 4 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0096] In one embodiment, a terminal device is provided, including a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs including instructions for performing operations as described in the embodiments of the methods above.

[0097] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0098] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for filtering pre-training data for large models based on conformal inference, characterized in that, The method includes: Collect raw, unlabeled data and construct the target dataset; A subset of samples is selected from the target dataset to construct a calibration set containing true quality labels; wherein the true quality labels indicate whether the samples are of high or low quality. Using a pre-trained quality classifier, the uncertainty score of low-quality samples in the calibration set is calculated, and the uncertainty score of the samples to be screened in the target dataset is calculated. For each sample to be screened, the conformal test value of the sample to be screened is calculated based on the uncertainty score of the low-quality samples in the calibration set and the uncertainty score of the sample to be screened. The conformal test value condition is determined by a dynamic threshold, and the samples in the target dataset whose conformal test values ​​meet the conformal test value condition are selected as high-quality samples.

2. The method for filtering pre-training data of large models based on conformal inference according to claim 1, characterized in that, The step of selecting a subset of samples from the target dataset to construct a calibration set containing true quality labels includes: Based on the target dataset, a preset proportion of data samples are randomly selected; wherein, the preset proportion is 5% to 10% of the total sample size of the target dataset. The selected data samples are manually labeled or labeled by domain experts to determine whether the data samples are of high or low quality, thus obtaining a calibration set containing true quality labels.

3. The method for filtering pre-training data of large models based on conformal inference according to claim 1, characterized in that, The steps for obtaining the pre-trained quality classifier include: Select a portion of unlabeled data samples from the target dataset; Manually label the selected unlabeled data samples to determine whether the samples are of high or low quality, and construct a training set for the quality classifier. Based on the training set of the quality classifier, the selected basic pre-trained model is fine-tuned, or a large language model is used to perform few-shot or zero-shot learning on the training set of the quality classifier to obtain the pre-trained quality classifier; wherein, the selected basic pre-trained model includes the BERT model and the RoBERTa model, and the pre-trained quality classifier is used to calculate the confidence that the sample is of high quality.

4. The method for filtering pre-training data of large models based on conformal inference according to claim 1, characterized in that, The process of using a pre-trained quality classifier to calculate the uncertainty score of low-quality samples in the calibration set and to calculate the uncertainty score of samples to be screened in the target dataset includes: Using the pre-trained quality classifier, quality predictions are performed on low-quality samples in the calibration set and samples to be screened in the target dataset, respectively, to obtain the confidence score of each sample as high quality. Based on the confidence level that each sample is of high quality, the uncertainty scores of low-quality samples in the calibration set and the uncertainty scores of samples to be screened in the target dataset are calculated using a complementary confidence algorithm.

5. The method for filtering pre-training data of large models based on conformal inference according to claim 1, characterized in that, For each sample to be screened, based on the uncertainty scores of the low-quality samples in the calibration set and the uncertainty scores of the sample to be screened, the conformal test value of the sample to be screened is calculated, including: For any sample to be screened, the uncertainty score of the sample to be screened is extracted and defined as the target score; The number of samples in the calibration set whose uncertainty score is greater than or equal to the target score is defined as the number of matches. The total number of low-quality samples in the calibration set is obtained and defined as the total number of low-quality samples. By combining the number of matches with the total number of low-quality samples, the conformal test value of the sample to be screened is calculated through statistical comparison; wherein, the magnitude of the conformal test value is negatively correlated with the confidence that the sample to be screened is a high-quality sample, and the conformal test value is used to reflect the similarity between the sample to be screened and the low-quality samples in the calibration set.

6. The method for filtering pre-training data of large models based on conformal inference according to claim 5, characterized in that, The condition for determining the conformal test value through dynamic thresholding includes: A preset low-quality sample ratio threshold is set; wherein, the low-quality sample ratio threshold is used to limit the maximum allowable ratio of low-quality samples in the high-quality sample set after screening. Obtain the conformal test values ​​of all samples to be screened in the target dataset, and arrange the conformal test values ​​in ascending order to obtain an ordered conformal test value sequence; Traverse the ordered conformal test value sequence, and for each index in the sequence, determine whether the conformal test value corresponding to the index meets the screening requirements based on the low-quality sample proportion threshold and the total number of low-quality samples, and determine the largest index that satisfies the statistical constraints. The conformal test value corresponding to the largest index in the ordered conformal test value sequence is defined as the dynamic threshold to obtain the conformal test value condition; wherein, the conformal test value condition is that the conformal test value of the sample to be screened is less than or equal to the dynamic threshold.

7. The method for filtering pre-training data of large models based on conformal inference according to claim 6, characterized in that, The step of selecting high-quality samples from the target dataset whose conformity test values ​​meet the conformity test value conditions includes: For each sample to be screened in the target dataset, the conformal test value condition is used to make a conditional judgment on the sample to be screened. If the conformal test value of the sample to be screened is less than or equal to the dynamic threshold, then the sample to be screened is marked as a high-quality sample.

8. A large model pre-training data filtering system based on conformal inference, characterized in that, The system includes: The target dataset acquisition module is used to collect raw, unlabeled data and construct the target dataset. The calibration set acquisition module is used to select a portion of samples from the target dataset to construct a calibration set containing real quality labels; wherein the real quality labels indicate whether the samples are of high or low quality. The uncertainty score calculation module is used to calculate the uncertainty score of low-quality samples in the calibration set using a pre-trained quality classifier, and to calculate the uncertainty score of samples to be screened in the target dataset. The conformal test value calculation module is used to calculate the conformal test value of each sample to be screened based on the uncertainty score of the low-quality samples in the calibration set and the uncertainty score of the sample to be screened. The filtering module is used to determine the conformal test value conditions through a dynamic threshold, and to filter the samples in the target dataset whose conformal test values ​​meet the conformal test value conditions as high-quality samples.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a large model pre-training data filtering program based on conformal inference stored in the memory and executable on the processor. When the processor executes the large model pre-training data filtering program based on conformal inference, it implements the steps of the large model pre-training data filtering method based on conformal inference as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a large model pre-training data filtering program based on conformal inference. When the large model pre-training data filtering program based on conformal inference is executed by a processor, it implements the steps of the large model pre-training data filtering method based on conformal inference as described in any one of claims 1-7.

Images