Decoupling screening method, device, medium and product for large model pre-training corpus

Abstract

The application discloses a decoupling screening method and device for large model pre-training corpus, a medium and a product, and belongs to the technical field of data processing. The application effectively avoids the problem of repeated selection of information homogeneous samples caused by high correlation of original score indexes by decoupling multi-dimensional quality scores into main factors with low correlation with each other, and improves the effective information density of training data. By dynamically allocating data budget to each main factor and combining coverage gain and redundancy penalty to calculate comprehensive utility value, samples are screened under total data budget and coverage constraint, realizing joint optimization of data quality, coverage diversity and low redundancy within fixed training budget. Through iterative optimization of data budget allocation based on training feedback of candidate training data set, a closed-loop mechanism is formed, so that the screening strategy can adaptively approach the direction of maximizing the real training benefit, and the manual adjustment and trial-and-error cost is significantly reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to decoupling screening methods, devices, media and products for large model pre-training corpora. Background Technology

[0002] In the pre-training process of large language models, the quality, coverage, and redundancy of the training corpus jointly determine the model's knowledge acquisition efficiency and generalization ability. Existing pre-training corpus construction schemes have the following shortcomings: In terms of quality evaluation, existing techniques usually use multi-dimensional scoring followed by weighted fusion or fixed threshold screening, but they do not consider the high correlation between indicators. Simple fusion can easily lead to repeated selection of texts with homogeneous information, reducing the effective information density. In terms of coverage balance, existing techniques mostly adopt a two-stage process of first quality filtering and then domain screening, or use domain classification and cluster analysis to guide sampling, but they usually rely on manual experience to set the sampling ratio, lacking unified quantification and dynamic modeling of sample redundancy, domain scarcity, and generalization benefits. In terms of redundancy control, existing techniques mainly use hard deduplication or sampling based on embedding density. Hard deduplication will discard some samples with local difference value, while density sampling cannot handle the cross-dimensional duplication problem where the same sample is repeatedly selected because it scores high on multiple evaluation dimensions. In summary, existing technologies lack a unified model for multidimensional score correlation, sample redundancy, domain scarcity, and generalization gains. There is an urgent need for a data optimization method that can jointly achieve multidimensional score decoupling, coverage constraints, redundancy suppression, and training feedback closed-loop optimization. Summary of the Invention

[0003] To address the problems in existing pre-training corpus construction methods, such as information homogeneity caused by the correlation of multidimensional scoring indicators, lack of dynamic modeling due to reliance on human experience for coverage balance, and information loss and difficulty in suppressing cross-dimensional repetition caused by hard deduplication or density sampling, this paper presents a decoupled screening method, device, medium, and product for large-scale model pre-training corpora. This method aims to achieve high-quality training data screening through multidimensional scoring decoupling, dynamic budget allocation under coverage constraints, soft suppression of redundancy, and closed-loop optimization of training feedback.

[0004] This application provides a decoupled filtering method for pre-training corpora of large models, including:

[0005] The multidimensional quality scores of the reference corpus are decoupled to obtain several decoupled principal factors and their prediction models.

[0006] Based on the prediction model, the predicted principal factor scores of each sample in the target corpus are obtained;

[0007] Allocate data budgets to each principal factor and calculate the overall utility value based on the predicted principal factor scores, coverage gain, and redundancy penalty for each sample;

[0008] Under the total data budget and coverage constraints, samples are selected based on the comprehensive utility value to form a candidate training dataset;

[0009] Based on the training feedback from the candidate training dataset, the allocation of the data budget is iteratively optimized until the stopping condition is met.

[0010] Optionally, the decoupling process for the multidimensional quality score of the reference corpus includes:

[0011] Based on the multidimensional ratings of each sample in the reference corpus, the credibility weight of each rating dimension is calculated.

[0012] Construct a weighted covariance matrix based on the aforementioned credibility weights;

[0013] The weighted covariance matrix is ​​subjected to eigenvalue decomposition, and the top K eigenvectors are selected as the decoupling principal factors based on a preset cumulative explained variance threshold.

[0014] Optionally, allocating data budgets to each principal factor includes:

[0015] Based on the performance indicators and cumulative return indicators of each principal factor, the budget allocation weight of each principal factor is dynamically calculated.

[0016] Based on the budget allocation weights and the total data budget, the sub-data budgets allocated to each principal factor are determined.

[0017] Optionally, the coverage gain is:

[0018] ,

[0019] in, Indicates candidate samples Compared to the currently selected sample set The marginal coverage gain is used to measure the degree of improvement in the overall coverage distribution after including the sample; For a predefined set of domains, topics, or categories, As one of the specific categories; For category The priority weight reflects the relative importance of the category in the coverage target; For category The target coverage percentage to be achieved in the final training dataset; For the currently selected sample set Medium category The actual coverage ratio; As an indicator function, when the sample Category The time value is Otherwise ; Ensure only when category A positive gain is generated only when the current coverage ratio has not yet reached the target, and the magnitude of the gain is proportional to the coverage gap.

[0020] Optionally, the redundancy penalty is calculated based on the sample commonness penalty value and the cross-overlap penalty value;

[0021] The sample commonness penalty value is determined based on one or more of the sample's n-gram statistical probability, semantic embedding space neighborhood density, or MinHash approximate repetition.

[0022] The cross-over penalty value is used to suppress the repeated selection of the same sample due to its high scores on multiple main factors.

[0023] Optionally, the step of iteratively optimizing the allocation of the data budget based on the training feedback of the candidate training dataset includes:

[0024] The agent model is trained using the candidate training dataset, and the actual training benefits are obtained.

[0025] Based on the actual training gains, update the cumulative gains metrics for each main factor.

[0026] In the next iteration, the budget allocation weights will be recalculated based on the updated cumulative revenue metrics.

[0027] Optionally, updating the cumulative return metrics of each main factor based on the actual training returns includes:

[0028] Evaluate the reduction in validation loss or the improvement in performance metrics for specific downstream tasks of the surrogate model after training.

[0029] The reduction in verification loss or the improvement in performance indicators are attributed to different principal factors, and the contribution value of the current round of returns corresponding to each principal factor is calculated.

[0030] Based on the contribution value of this round of revenue, the corresponding cumulative revenue indicator is updated using either exponential smoothing or direct assignment.

[0031] This application provides an electronic device, the electronic device comprising:

[0032] One or more processors; and

[0033] A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the decoupling screening method for large model pre-training corpora described above.

[0034] This application provides a computer-readable medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-described decoupling screening method for large model pre-training corpora.

[0035] This application provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the above-described decoupling screening method for large model pre-training corpora.

[0036] The beneficial effects of the above technical solution are as follows:

[0037] This application's decoupled screening method for large model pre-training corpora effectively avoids the problem of repeated selection of homogeneous samples due to the high correlation of the original scoring indicators by decoupling multidimensional quality scores into mutually low-correlation principal factors, thereby improving the effective information density of the training data. By dynamically allocating data budgets to each principal factor and calculating the comprehensive utility value by combining coverage gain and redundancy penalty, samples are screened under the total data budget and coverage constraints. This achieves joint optimization of data quality, coverage diversity, and low redundancy within a fixed training budget, overcoming the shortcomings of existing technologies that rely on manual experience to set sampling ratios and are difficult to balance the coverage needs of high-frequency and low-frequency domains. By iteratively optimizing the data budget allocation based on training feedback from candidate training datasets, a closed-loop mechanism of "screening-training-feedback-rescreening" is formed, enabling the screening strategy to adaptively approach the direction of maximizing real training benefits, significantly reducing the cost of manual intervention and trial and error. Attached Figure Description

[0038] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0039] Figure 1 This is a flowchart of one embodiment of the decoupling and filtering method for large model pre-training corpora described in this application;

[0040] Figure 2 This is a flowchart of a method for decoupling the multidimensional quality score of a reference corpus as described in this application.

[0041] Figure 3 This is a flowchart illustrating an embodiment of the method for allocating data budgets to each principal factor in this application;

[0042] Figure 4 A flowchart illustrating an embodiment of a method for iteratively optimizing a data budget based on training feedback from candidate training datasets;

[0043] Figure 5 A flowchart illustrating an embodiment of updating the cumulative revenue metric based on actual training gains;

[0044] Figure 6 This is an exemplary structural diagram of the electronic device of this application. Detailed Implementation

[0045] The advantages of this application are further illustrated below with reference to the accompanying drawings and specific embodiments.

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0047] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0048] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0049] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0050] This application can be applied to the automated selection of pre-training corpora for large-scale language models, such as constructing high-quality pre-training datasets for large language models with hundreds of billions or trillion parameters from massive amounts of heterogeneous data from multiple sources, including web page texts, e-books, academic papers, code repositories, and question-and-answer communities. A specific embodiment is described below.

[0051] The goal is to construct a pre-training dataset for a general-purpose Chinese model containing approximately 100 bytes of tokens. The original corpus contains approximately 500 bytes of Chinese text from various domains, including web pages, books, encyclopedia entries, and code. First, approximately 500,000 texts are stratified by domain from the original corpus to construct a reference corpus. Each reference text is then evaluated for quality across dimensions such as language coherence, conciseness, spelling accuracy, knowledge depth, knowledge richness, reasoning ability, educational value, practicality, comprehension difficulty, factual accuracy, and information completeness. For each dimension, the mean score, score fluctuation, and sample-dimensional reliability (r) are obtained through repeated evaluation or cross-evaluation by multiple reviewers. ij Subsequently, robust standardization was performed on the mean scores, based on r. ij Sample-level credibility weight w i Or dimension-level credibility weight v j A weighted covariance matrix is ​​constructed and eigenvalue decomposition is performed. The top K least correlated or approximately orthogonal principal factors are retained based on the cumulative explained variance threshold. Each principal factor is a statistical direction formed by combining multiple original scoring dimensions. Its correlation with these dimensions can be used for auxiliary interpretation; for example, principal factors highly correlated with knowledge depth, reasoning, and comprehension difficulty can be interpreted as directions with higher cognitive load, but this interpretation does not constitute a limitation on the meaning of the principal factors. After directional calibration for each principal factor, a principal factor predictor is trained using reference corpus text as input and the corresponding principal factor projection score as the supervision label. Reasoning is then performed on all samples in the target corpus to obtain the predicted score for each sample on each principal factor. During the selection phase, sub-data budgets are dynamically allocated based on the explanatory power, domain scarcity, coverage gain, and surrogate training benefit of each principal factor, forming candidate sample pools under each principal factor. The comprehensive utility value is used for ranking within the candidate pool and handling conflict merging, rather than replacing the selection of sub-principal factors. Subsequently, the candidate pools are merged under the combined effects of the total data budget, the budgets of each principal factor, coverage constraints, commonality penalties, cross-over penalties, and fact risk penalties to form a candidate training dataset. A small-scale surrogate model is then trained using this candidate training dataset, attributing the reduction in validation loss or the improvement in downstream benchmark tasks to each principal factor and updating the cumulative return metric. After several iterations, the final training dataset is output when the stopping condition is met.

[0052] like Figure 1 As shown, this embodiment provides a decoupling screening method for large model pre-training corpora, including the following steps:

[0053] S1. Decouple the multidimensional quality scores of the reference corpus to obtain several decoupled principal factors and their prediction models;

[0054] S2. Based on the prediction model, obtain the predicted principal factor scores for each sample in the target corpus;

[0055] S3. Allocate data budgets to each principal factor and calculate the overall utility value based on the predicted principal factor score, coverage gain, and redundancy penalty of each sample. The overall utility value is used to quantify the overall screening priority of the samples.

[0056] S4. Under the total data budget and coverage constraints, samples are selected based on the comprehensive utility value to form a candidate training dataset;

[0057] S5. Based on the training feedback of the candidate training dataset, iteratively optimize the allocation of the data budget until the stopping condition is met.

[0058] In this embodiment, the decoupled screening method for large model pre-training corpora effectively avoids the problem of repeated selection of homogeneous samples due to the high correlation of the original scoring indicators by decoupling multidimensional quality scores into mutually low-correlation principal factors, thereby improving the effective information density of the training data. By dynamically allocating data budgets to each principal factor and calculating the comprehensive utility value by combining coverage gain and redundancy penalty, samples are screened under the total data budget and coverage constraints. This achieves joint optimization of data quality, coverage diversity, and low redundancy within a fixed training budget, overcoming the shortcomings of existing technologies that rely on manual experience to set sampling ratios and are difficult to balance the coverage needs of high-frequency and low-frequency domains. By introducing factual risk penalty in the calculation of comprehensive utility value, it can effectively suppress the erroneous retention of high-risk samples such as factual errors and security risks, further improving the quality and reliability of the training dataset. By iteratively optimizing the data budget allocation based on training feedback of candidate training datasets, a closed-loop mechanism of "screening-training-feedback-rescreening" is formed, enabling the screening strategy to adaptively approach the direction of maximizing real training benefits, significantly reducing the cost of manual intervention and trial and error.

[0059] In an alternative embodiment, such as Figure 2 As shown, step S1 decouples the multidimensional quality scores of the reference corpus, which may include the following steps:

[0060] S11. Based on the multidimensional ratings of each sample in the reference corpus, calculate the credibility weight of each rating dimension;

[0061] S12. Construct a weighted covariance matrix based on the aforementioned credibility weights;

[0062] S13. Perform eigenvalue decomposition on the weighted covariance matrix, and select the top K eigenvectors as the decoupling principal factors based on a preset cumulative explained variance threshold.

[0063] In this embodiment, by introducing a credibility weight, the influence of dimensions with large score fluctuations and poor stability is automatically weakened when constructing the covariance matrix. As a result, the extracted principal factors can better reflect the true direction of data value and avoid the deviation of principal factor direction caused by large score noise in some evaluation dimensions. At the same time, by controlling the number of principal factors through the cumulative explained variance threshold, the complexity of subsequent calculations and predictions is reduced while retaining core value information, thereby improving the processing efficiency of large-scale target corpora.

[0064] In an alternative embodiment, such as Figure 3 As shown, step S3 allocates data budgets for each principal factor, including the following steps:

[0065] S31. Calculate the budget allocation weight of each principal factor dynamically based on its performance indicators and cumulative return indicators.

[0066] S32. Based on the budget allocation weights and the total data budget, determine the sub-data budgets allocated to each principal factor.

[0067] Furthermore, the performance indicators are determined based on one or more of the following: the explained variance of the corresponding principal factor, the domain scarcity of the associated samples, and the expected gain on the overall data coverage target.

[0068] In this embodiment, by introducing explained variance, domain scarcity, and expected gain as components of the performance indicators, the budget allocation can comprehensively measure the value of each principal factor in multiple dimensions such as statistical importance, domain balance, and coverage contribution. At the same time, the performance indicators and cumulative benefit indicators jointly drive the dynamic calculation of budget allocation weights, so that the screening strategy can be reasonably allocated based on prior value indicators in the first round, and can be continuously corrected in subsequent iterations based on the actual benefit feedback of agent training, gradually converging to the budget configuration that maximizes training benefits. This effectively overcomes the shortcomings of static allocation in the prior art, which relies on human experience and cannot be adaptively adjusted.

[0069] In an optional embodiment, the decoupling process of the multidimensional quality scores of the reference corpus further includes a preprocessing step of the original scores.

[0070] The preprocessing steps include robust normalization and credibility calculation.

[0071] The robust normalization is calculated using formula (1):

[0072] (1),

[0073] in, Indicates the first The sample at the th Original scores across each evaluation dimension; Indicates the first The median of all sample scores across each evaluation dimension; Indicates the first Interquartile range of all sample scores across each evaluation dimension; This is a preset positive small number used to prevent division by zero errors.

[0074] This robust standardization removes bias and unifies the scale of the original scores by using the median and interquartile range, eliminating the influence of different evaluation dimensions and outliers, and making the data from different scoring dimensions comparable in subsequent processing.

[0075] The credibility calculation method is formula (2):

[0076] (2),

[0077] in, Indicates the first The sample at the th The credibility of the ratings across all evaluation dimensions; Indicates the first The sample at the th The variance of the scores on each evaluation dimension is obtained by performing multiple independent scores on the same sample on the same dimension or by cross-evaluation by multiple reviewers. These are the preset adjustment parameters. It is a preset positive small quantity.

[0078] This credibility calculation converts rating fluctuations into credibility weights. The greater the rating fluctuation, the lower the credibility, thus automatically weakening the influence of unstable ratings in subsequent decoupling processes.

[0079] The weighted covariance matrix is ​​constructed as shown in formulas (3) and (4):

[0080] (3),

[0081] (4),

[0082] in, This represents the confidence-weighted covariance matrix; Indicates the first The score vector of each sample after robust standardization; This represents the mean vector of all samples; Indicates the first The overall weight of a sample is obtained by calculating the average credibility of that sample across all evaluation dimensions; This represents the total number of evaluation dimensions.

[0083] When estimating the correlation between indicators, this weighted covariance matrix assigns greater weight to samples with more stable scores and less weight to samples with more score noise.

[0084] The eigenvalue decomposition of the weighted covariance matrix is ​​shown in formulas (5) and (6):

[0085] (5),

[0086] (6),

[0087] in, This represents the eigenvector matrix, where the column vectors represent the directions of the principal factors. Represents an eigenvalue diagonal matrix. Indicates the first The explained variance of each principal factor is used to measure the amount of information carried by that principal factor.

[0088] The projection of a sample into the principal factor space is calculated as shown in formulas (7) and (8):

[0089] (7),

[0090] (8),

[0091] in, Indicates the first The projected score vector of each sample in the decoupled principal factor space; Indicates from the previous The projection matrix composed of eigenvectors; Indicates the first The i-th feature vector, corresponding to the i-th feature vector The direction of each principal factor; The number of principal factors selected based on a preset cumulative explained variance threshold.

[0092] The above projection maps the samples from the original high-dimensional evaluation space to at least a few core value directions, which facilitates subsequent budget allocation and sample selection.

[0093] In an optional embodiment, when allocating data budgets for each principal factor in step S3, the budget allocation is calculated as follows:

[0094] (9),

[0095] in: Indicates the first Sub-data budgeting for each principal factor allocation; Indicates the total data budget; Indicates the first The variance explained by each principal factor; Indicates the first The domain scarcity of each principal factor is used to measure the extent to which the principal factor complements the scarce domain. Indicates the first The coverage gain of each principal factor is used to measure the extent to which the principal factor fills gaps in the data distribution. Indicates the first The proxy training benefit of each principal factor; , , , These are the adjustable weight coefficients for each item; This represents the total number of the decoupled principal factors.

[0096] The budget allocation function incorporates explained variance, domain scarcity, coverage gain, and surrogate training gains into the calculation, so that more of the budget flows to the main factors that can represent the overall corpus variation, supplement scarce domains, and have been verified to have benefits through surrogate training.

[0097] In an optional embodiment, the coverage gain is as shown in formula (10):

[0098] (10)

[0099] in, Indicates candidate samples Compared to the currently selected sample set The marginal coverage gain is used to measure the degree of improvement in the overall coverage distribution after including the sample; For a predefined set of domains, topics, or categories, As one of the specific categories; For category The priority weight reflects the relative importance of the category in the coverage target; For category The target coverage percentage to be achieved in the final training dataset; For the currently selected sample set Medium category The actual coverage ratio; As an indicator function, when the sample Category The time value is Otherwise ; Ensure only when category A positive gain is generated only when the current coverage ratio has not yet reached the target, and the magnitude of the gain is proportional to the coverage gap.

[0100] In this embodiment, by quantifying the coverage gain into a weighted sum of the coverage gap of each category and the priority weight, it is possible to accurately identify and prioritize samples belonging to the currently insufficiently covered categories during sample selection. This effectively improves the distribution balance of training data in the domain, topic, or category dimensions under the constraint of total data budget, avoids excessive crowding of low-frequency professional sample selection opportunities by high-frequency general samples, and ensures that the final training dataset has more comprehensive knowledge coverage and stronger generalization potential.

[0101] In an optional embodiment, the redundancy penalty is calculated based on a sample commonness penalty value and a cross-overlap penalty value;

[0102] The sample commonness penalty value is determined based on one or more of the sample's n-gram statistical probability, semantic embedding space neighborhood density, or MinHash approximate repetition.

[0103] The cross-over penalty value is used to suppress the repeated selection of the same sample due to its high scores on multiple main factors.

[0104] In this embodiment, the commonness penalty value quantifies the repetition and density of samples in n-gram patterns, embedding space distributions, or hash signatures. It deducts the utility value of templated and formulaic low-information incremental samples, achieving soft suppression rather than direct deletion. This reduces redundancy while retaining similar texts with local differences in value. The cross-overlap penalty value deducts the utility value by detecting the repeated occurrence of the same sample in multiple main factor candidate pools, effectively avoiding the problem of repeated selection due to samples scoring high in different value directions. The combined effect of these two penalty mechanisms ensures that the final training dataset, under a fixed budget constraint, possesses both low redundancy and high information density. This overcomes the shortcomings of existing hard deduplication methods, such as information loss, and the inability of single-density sampling to handle cross-dimensional repeated selections.

[0105] In an optional embodiment, the sample commonness is calculated as shown in formula (11):

[0106] (11),

[0107] in, Indicates sample The frequency penalty value; Represents the high-frequency template probability based on n-gram statistics; This represents the neighborhood density of a sample in the semantic embedding space; Indicates the degree of near-duplication based on MinHash; , , These are the adjustable weight coefficients for each item.

[0108] This commonness calculation quantifies the degree of templated or repetitive nature of a sample by combining the repetition of three dimensions: n-gram pattern, embedding distribution, and hash signature. The larger the value, the more "common" the sample is.

[0109] In an optional embodiment, the comprehensive utility value is calculated as shown in formula (12):

[0110] (12)

[0111] in, Indicates candidate samples The overall utility value; Indicates sample In the Scores on each principal factor; This is a pre-defined fraction mapping function; For the first The utility weight coefficients of each principal factor; The coverage gain; This is the sample commonness penalty value; This is the penalty value for cross-overlap. The fact risk penalty value is estimated by an external fact checker, rule checker, or anomaly detection model. , , , These are the adjustable weight coefficients for each item.

[0112] This comprehensive utility function uses the main factor score and coverage gain as positive contributors, and commonality penalty, cross-over penalty, and factual risk penalty as negative deductions, thus achieving a comprehensive quantitative evaluation of sample value. By adjusting the weight coefficients, the selection preferences can be flexibly adjusted according to the actual corpus quality distribution and training objectives: when the corpus has high factual noise, the risk penalty coefficient can be increased. When the problem of data duplication is severe, the redundancy penalty coefficient can be increased. and .

[0113] In an optional embodiment, the cross-over penalty is calculated as shown in formula (13):

[0114] (13)

[0115] in, Indicates sample The cross-overlap penalty value; and They represent the first The and the first The candidate sample set corresponding to each principal factor; As an indicator function, when the sample When it appears in the candidate pools of two different main factors, its value is... Otherwise .

[0116] This cross-overlap penalty reduces the utility value of samples that are repeatedly selected across multiple main factor candidate pools by counting the number of times the same sample appears, achieving soft suppression rather than hard deletion. When it is necessary to retain some high-value samples across factors, the corresponding penalty coefficient can be adjusted. Set to a non-zero finite value to balance redundancy suppression and value preservation.

[0117] In an alternative embodiment, such as Figure 4 As shown, step S5 iteratively optimizes the allocation of the data budget based on the training feedback of the candidate training dataset, and may include the following steps:

[0118] S51. Train the proxy model using the candidate training dataset and obtain the actual training benefits;

[0119] S52. Based on the actual training gains, update the cumulative gains metrics for each main factor;

[0120] S53. In the next iteration, the budget allocation weights are recalculated based on the updated cumulative revenue metrics.

[0121] In this embodiment, a surrogate model is used to quickly verify the actual training effect of the current round of screening results. The actual training gains, such as the reduction in verification loss or the improvement in downstream task performance, are fed back to the cumulative benefit indicators corresponding to each principal factor, thereby correcting the calculation basis for the budget allocation weights in the next round. Under this closed-loop mechanism, principal factors that can continuously generate positive training gains will receive more data budgets, while principal factors with low input-output ratios will be automatically suppressed. This allows the overall budget allocation to gradually approach the direction of maximizing training gains after multiple iterations, significantly reducing the cost of manual adjustment and empirical trial and error, and solving the defect of existing static screening methods that cannot adaptively adjust according to actual training effects.

[0122] Furthermore, such as Figure 5 As shown, in step S52, updating the cumulative return metrics of each main factor based on the actual training returns may include the following steps:

[0123] S521. Evaluate the reduction in validation loss or the improvement in performance metrics for specific downstream tasks of the surrogate model after training;

[0124] S522. Attribute the reduction in verification loss or the improvement in performance indicators to different principal factors, and calculate the current round return contribution value corresponding to each principal factor;

[0125] S523. Based on the current round of revenue contribution value, update the corresponding cumulative revenue indicator using either exponential smoothing or direct assignment.

[0126] In this embodiment, by attributing the overall training gains of the surrogate model to different principal factors, the actual marginal contribution of each principal factor's corresponding data direction to the model's performance improvement can be accurately quantified. This provides a basis for subsequent budget adjustments and avoids budget mismatch caused by averaging data from different value directions. Furthermore, updating the cumulative gains metric using exponential smoothing effectively integrates historical accumulated gains with new gains from the current round, reducing metric oscillations caused by random fluctuations in single-round training. This makes the cumulative gains metric more stable and reliable, ensuring that the budget allocation weights converge steadily towards maximizing training gains across multiple iterations, thus improving the robustness and efficiency of the entire closed-loop optimization process.

[0127] In an optional embodiment, step S4, under the total data budget and coverage constraints, filters samples based on comprehensive utility values, including:

[0128] Under the constraints of the total data budget and the data budget of each principal factor, samples are selected from high to low based on the comprehensive utility value until the preset coverage constraint is met.

[0129] In this embodiment, the dual constraints of the total data budget and the budgets of each principal factor sub-data ensure a balanced allocation of global resources and value orientations in sample selection, preventing high-frequency general samples from excessively crowding out the budget and resulting in insufficient selection of samples from scarce domains. Simultaneously, a comprehensive utility value is used as the unified basis for sample ranking and selection. This utility value integrates multi-dimensional information such as principal factor scores, coverage gain, and redundancy penalties, enabling the selection process to prioritize samples that contribute most to data quality and coverage balance. Guided by the coverage constraint, the final candidate training dataset achieves a preset balance across multiple dimensions, including domain distribution, topic coverage, and difficulty level, effectively overcoming the shortcomings of existing technologies where Top-k selection favors dominant style samples, leading to insufficient knowledge coverage.

[0130] In an optional embodiment, the optimization objective for selecting samples based on comprehensive utility value under the total data budget and coverage constraints is as shown in formula (14):

[0131] (14)

[0132] The constraints are satisfied as shown in formula (15):

[0133] (15)

[0134] in: This is the final selected training dataset; For the selected dataset Total number of tokens; The total data budget; For any two distinct principal factors, the corresponding subsets and The Jaccard similarity between them is constrained to not exceed the overlap threshold. ; The overall coverage of the selected dataset must meet a preset coverage constraint threshold. .

[0135] Under the triple constraints of total data budget, controlled cross-overlap, and overall coverage achievement, the optimization objective selects the sample set with the largest total utility as the final training dataset, unifying multidimensional scoring decoupling, dynamic budget allocation, coverage gain calculation, redundancy and risk suppression into a solvable constraint optimization framework.

[0136] In an optional embodiment, the method further includes the following abnormal scenario handling strategies:

[0137] When the target corpus has extreme class imbalance, increase the weight of the coverage item. Weighting the scarcity factor enhances the filtering of low-frequency categories; when the corpus has severe duplication issues, the redundancy penalty coefficient is increased. and overlap penalty coefficient When the corpus has high factual noise, increase the risk penalty coefficient. .

[0138] When the proxy experiment budget is limited, feedback training is not performed in every round, but rather triggered at fixed rounds, such as after each round. After each screening, a proxy feedback is performed; or only local feedback updates are performed on the few principal factors with the largest budget changes to reduce computational costs.

[0139] For open web page corpora without explicit domain labels, pseudo-domain labels are first obtained through clustering, topic modeling, embedding bucketing, or metadata rules. Then, the coverage ratio and scarcity are estimated using the pseudo-domain labels, so that coverage constraints can still be implemented without manual fine annotation.

[0140] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0141] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices.

[0142] The electronic device includes: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the steps of the methods provided in any one or more of the above embodiments. Figure 6 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0143] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103 and output device 1104 may be connected by a bus or other means, as shown in the figure, which is connected by a bus.

[0144] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0145] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0146] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0147] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0148] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0149] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0150] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0151] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0153] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0154] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0155] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0156] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A decoupled screening method for pre-training corpora of large models, characterized in that, include: The multidimensional quality scores of the reference corpus are decoupled to obtain several decoupled principal factors and their prediction models. Based on the prediction model, the predicted principal factor scores of each sample in the target corpus are obtained; Allocate data budgets to each principal factor and calculate the overall utility value based on the predicted principal factor scores, coverage gain, and redundancy penalty for each sample; Under the total data budget and coverage constraints, samples are selected based on the comprehensive utility value to form a candidate training dataset; Based on the training feedback from the candidate training dataset, the allocation of the data budget is iteratively optimized until the stopping condition is met.

2. The decoupling screening method for large model pre-training corpora according to claim 1, characterized in that, The decoupling process for the multidimensional quality scoring of the reference corpus includes: Based on the multidimensional ratings of each sample in the reference corpus, the credibility weight of each rating dimension is calculated. Construct a weighted covariance matrix based on the aforementioned credibility weights; The weighted covariance matrix is ​​subjected to eigenvalue decomposition, and the top K eigenvectors are selected as the decoupling principal factors based on a preset cumulative explained variance threshold.

3. The decoupling screening method for large model pre-training corpora according to claim 1, characterized in that, The allocation of data budgets for each principal factor includes: Based on the performance indicators and cumulative return indicators of each principal factor, the budget allocation weight of each principal factor is dynamically calculated. Based on the budget allocation weights and the total data budget, the sub-data budgets allocated to each principal factor are determined.

4. The decoupling screening method for large model pre-training corpora according to claim 1, characterized in that, The coverage gain is: ， in, Indicates candidate samples Compared to the currently selected sample set The marginal coverage gain is used to measure the degree of improvement in the overall coverage distribution after including the sample; For a predefined set of domains, topics, or categories, As one of the specific categories; For category The priority weight reflects the relative importance of the category in the coverage target; For category The target coverage percentage to be achieved in the final training dataset; For the currently selected sample set Medium category The actual coverage ratio; As an indicator function, when the sample Category The time value is Otherwise ; Ensure only when category A positive gain is generated only when the current coverage ratio has not yet reached the target, and the magnitude of the gain is proportional to the coverage gap.

5. The decoupling screening method for large model pre-training corpora according to claim 1, characterized in that, The redundancy penalty is calculated based on the sample commonness penalty value and the cross-over penalty value; The sample commonness penalty value is determined based on one or more of the sample's n-gram statistical probability, semantic embedding space neighborhood density, or MinHash approximate repetition. The cross-over penalty value is used to suppress the repeated selection of the same sample due to its high scores on multiple main factors.

6. The decoupling screening method for large model pre-training corpora according to claim 3, characterized in that, The step of iteratively optimizing the allocation of the data budget based on the training feedback of the candidate training dataset includes: The agent model is trained using the candidate training dataset, and the actual training benefits are obtained. Based on the actual training gains, update the cumulative gains metrics for each main factor. In the next iteration, the budget allocation weights will be recalculated based on the updated cumulative revenue metrics.

7. The decoupling screening method for large model pre-training corpora according to claim 6, characterized in that, The step of updating the cumulative return metrics for each main factor based on the actual training returns includes: Evaluate the reduction in validation loss or the improvement in performance metrics for specific downstream tasks of the surrogate model after training. The reduction in verification loss or the improvement in performance indicators are attributed to different principal factors, and the contribution value of the current round of returns corresponding to each principal factor is calculated. Based on the contribution value of this round of revenue, the corresponding cumulative revenue indicator is updated using either exponential smoothing or direct assignment.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the decoupling screening method for large model pre-training corpora as described in any one of claims 1 to 7.

9. A computer-readable medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the decoupling screening method for large model pre-training corpora as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the decoupling screening method for large model pre-training corpora as described in any one of claims 1 to 7.

Images