A general-purpose and security-performance-considered large model training data metric and screening method

Abstract

The application discloses a kind of general-purpose and safety performance big model training data metric and screening method. By constructing a machine learning model, a multidimensional feature vector is input, and the real comprehensive quality score is labeled for training. The machine learning model acts as a low-cost, high-efficiency proxy model, which only needs to perform feature extraction on the new data set to predict its quality score, thereby successfully replacing and bypassing the traditional process of supervised fine-tuning, benchmark evaluation and score calculation, which has high computational cost and high time consumption, achieving millisecond-level data set screening. This method can overcome the defects of weak correlation between existing data set indicators and model final performance, ignoring the trade-off between generality and security, and subjective index weight allocation, achieving objective and comprehensive evaluation of data set quality.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and in particular to a method for measuring and filtering large model training data that balances versatility and security performance. Background Technology

[0002] In recent years, generative artificial intelligence technologies, represented by LLM (Limited Learning Model), have achieved rapid development and have been widely applied in various scenarios such as search engines, intelligent customer service, and office assistants. However, as the capabilities of these models have increased, their safety and alignment issues have become increasingly prominent. Therefore, "safety alignment" has become an indispensable and crucial step in model development, and in this process, the quality of the safety dataset used for fine-tuning (SFT) or reinforcement learning (RLHF) is widely recognized as a core factor determining the final safety performance of the model.

[0003] Currently, the industry primarily relies on manual review, heuristic rules based on keywords or sources, or isolated, low-order computational metrics (such as text fluency metrics, data diversity scores, or specific statistical features) for evaluating and screening secure datasets. However, these existing techniques have serious shortcomings. On the one hand, the correlation between these "low-order metrics" and the "high-order performance" (i.e., the model's actual security capabilities) ultimately exhibited after model fine-tuning is weak and lacks verification. On the other hand, existing methods ignore the crucial trade-off between security and general performance; that is, while improving model security, it may severely impair its general capabilities in mathematics, reasoning, coding, etc., leading to a "catastrophic forgetting of capabilities," and existing screening methods lack a comprehensive consideration of this "side effect." Furthermore, when using multiple metrics to evaluate datasets, there is a lack of objective weighting mechanisms, relying mainly on subjective settings. Therefore, there is an urgent need in this field for a general method that can quantitatively, comprehensively (balancing security and generality), and objectively evaluate the quality of secure datasets for large language models.

[0004] The existing technology has the following technical problems:

[0005] 1) The representation metrics of the dataset are inconsistent with the final performance of the model: The representation metrics that existing methods rely on, which are directly computed on the dataset (such as PPL, diversity or textual statistical features), have a weak correlation with the actual performance (i.e., safety and generality in downstream tasks) shown by the model after fine-tuning with the data. There is a lack of a verified and reliable mapping relationship.

[0006] 2) Ignoring the performance trade-off between security and generality: Existing methods, when evaluating secure datasets, often prioritize security metrics while neglecting the potential damage these datasets can inflict on the model's general capabilities in mathematics, reasoning, and coding—a problem known as "catastrophic forgetting of capabilities." This results in a lack of a comprehensive mechanism that can balance security and generality.

[0007] 3) Lack of objective automated weighting mechanism: When using multiple indicators for comprehensive evaluation, how to assign weights to these indicators to combine them into a final quality score currently relies heavily on human experience or subjectively set heuristic rules, lacking an objective, data-driven automated weighting learning mechanism.

[0008] 4) High evaluation costs and limited scalability: Existing high-quality evaluation methods (such as manual review) are costly, inefficient, and highly subjective, making them difficult to scale; while automated methods (such as keyword filtering) are too crude. There is an urgent lack in this field of methods for rapid, low-cost, and objectively quantifiable evaluation and screening of new datasets.

[0009] 5) Limited to single-dimensional metrics and superficial statistical features: Existing metrics are mostly superficial statistical features (such as text length, keyword frequency, etc.), which cannot deeply characterize the key features of the dataset at the semantic and distribution levels. For example, existing methods lack effective measures to measure the difference between the dataset's answers and the model's original probability distribution, and also cannot assess the distributional balance of the dataset in terms of sample difficulty. Summary of the Invention

[0010] To address the aforementioned problems, this invention provides a method for measuring and selecting large-scale model training data that balances versatility and security. This method overcomes the shortcomings of existing technologies, such as weak correlation between dataset metrics and final model performance, neglect of the trade-off between versatility and security, and subjective allocation of metric weights, thus achieving an objective and comprehensive evaluation of dataset quality. This invention is achieved through the following technical solution:

[0011] This invention discloses a method for measuring and filtering large model training data that balances versatility and security, including:

[0012] S1: Obtain multiple security datasets to be evaluated, and at least one benchmark large language model;

[0013] S2: Supervised fine-tuning of the benchmark large language model was performed using the security dataset to be evaluated to obtain the fine-tuned model; the uniform configuration of the fine-tuning was maintained to ensure comparability;

[0014] S3: On the basis of the general performance evaluation benchmark and the security performance evaluation benchmark, evaluate the model that has been fine-tuned in S2, and calculate the general performance score and security performance score of each security dataset to be evaluated;

[0015] S4: Combine the scores from S3 to calculate the overall quality score for each security dataset to be evaluated;

[0016] S5: Extract multi-dimensional feature index vectors for each security dataset to be evaluated;

[0017] S6: Use the multi-dimensional feature index vector of S5 as input and the comprehensive quality score of S4 as label to train the machine learning model;

[0018] S7: Obtain a new dataset to be evaluated, and perform step S5 to extract its multi-dimensional feature index vector; input the multi-dimensional feature index vector into the S6 machine learning model to obtain the predicted comprehensive quality score.

[0019] The steps for training a machine learning model in S6 include: pairing the multi-dimensional feature index vectors of each security dataset to be evaluated in S5 with their corresponding S4 comprehensive quality scores to construct a training set; and using the training set to fit the machine learning model.

[0020] As a further improvement, in S3 of this invention, the general performance evaluation benchmarks are MMLU, HumanEval, and MATH-500 benchmarks; the safety performance evaluation benchmarks are AdvBench and StrongReject benchmarks.

[0021] As a further improvement, in S5 of this invention, the multi-dimensional feature index vector is composed of the following indicators:

[0022] Predictability metrics for language models;

[0023] Safety response distribution variability index;

[0024] Toxicity style index based on external classifier;

[0025] Diversity indicators;

[0026] Effective information density index.

[0027] As a further improvement, in S3 of the present invention, the general performance score is obtained by calculating the average of the general performance scores obtained by one or more fine-tuned models fine-tuned from the same dataset to be evaluated; the security performance score is obtained by calculating the average of the corresponding security performance scores.

[0028] As a further improvement, in S4 of the present invention, the overall quality score is the weighted sum of the general performance score and the safety performance score of S3.

[0029] As a further improvement, the predictability index of the language model described in this invention is the arithmetic mean of the conditional perplexity of the benchmark large language model for all samples in a given dataset to be evaluated; the safety response distribution dissimilarity index is the decaying Kullback-Leibler divergence.

[0030] As a further improvement, the diversity metrics described in this invention include Self-BLEU-4 and embedding dispersion.

[0031] As a further improvement, in S6 of the present invention, the machine learning model is an L1 regularized linear regression model or an L2 regularized linear regression model.

[0032] As a further improvement, in S2 of the present invention, in order to ensure the fairness of the evaluation, all fine-tuning processes use the same hyperparameter settings, that is, maintain a unified configuration for fine-tuning; the unified configuration includes learning rate, learning rate scheduler, number of training rounds, batch size, optimizer, weight decay, training accuracy, maximum sequence length, and global random seed.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] 1. Low-cost alternative to expensive fine-tuning evaluation process based on a "surrogate model": This invention constructs a machine learning model through S1 to S6. This model is trained using the multi-dimensional feature vector described in S5 as input and the true comprehensive quality score described in S4 as the label. In the application stage of S7, this machine learning model acts as a low-cost, high-efficiency surrogate model. It only needs to perform feature extraction in S5 on a new dataset to predict its quality score, thus successfully replacing and bypassing the entire high-computing-cost and time-consuming traditional process of supervised fine-tuning in S2, benchmark evaluation in S3, and score calculation in S4, achieving millisecond-level dataset selection.

[0035] 2. Objective Quantification of Feature Weights Using L1 Regularized Linear Regression: In S6, this invention employs an L1 regularized linear regression model (or an L2 regularized linear regression model) to fit the mapping relationship between dataset features and overall quality scores. By minimizing the sum of the mean squared error and the L1 penalty term (or L2 penalty term), this method can automatically select the key features that have the most significant impact on the final performance of the model and assign them objective weight values. This avoids the subjectivity of relying on human experience or heuristic rules to set indicator weights in existing technologies, and realizes a completely data-driven evaluation system.

[0036] 3. Constructing a dual-objective comprehensive utility function that balances generality and security: In S3 and S4, this invention constructs a comprehensive utility function that includes a "general performance score" (composed of the mean values ​​of MMLU, HumanEval, and MATH-500) and a "security performance score" (composed of the mean values ​​of AdvBench and StrongReject). By calculating the weighted sum of these two scores in S4 as the final label, this invention ensures that the selected dataset not only possesses high security but also retains the mathematical, code, and logical reasoning capabilities of the model to the greatest extent possible, effectively solving the common problem of "catastrophic forgetting of capabilities" in secure alignment.

[0037] 4. Introduction of a novel "Safety Response Distribution Divergence Index (DKL)" based on decaying KL divergence: In S5, this invention proposes a novel index, DKL. This index quantifies the "degree of distortion" of the model's original beliefs by calculating the decaying KL divergence between the probability distribution of the baseline model in a freely generated state and its probability distribution under the constraints of a safe dataset. Compared to traditional indices, DKL can more sensitively identify low-quality, forcibly aligned data that severely conflict with the model's pre-training knowledge and may lead to model collapse.

[0038] 5. Introducing toxicity style indicators and diversity indicators: On the one hand, the 95th percentile of toxicity score is used in S5 ( Instead of using the average value, a toxic style metric (TSS) is calculated to effectively capture and penalize a very small number of "long-tail" highly toxic samples in the dataset. On the other hand, diversity is measured by combining Self-BLEU-4 (N-gram overlap) and embedding diffusion (semantic distance) as dual dimensions. This combined metric significantly improves the robustness of the evaluation results to extreme samples and its discriminative power for content richness. Attached Figure Description

[0039] Figure 1 This embodiment provides an algorithm flowchart for a method of filtering high-quality datasets for secure alignment of large models. Detailed Implementation

[0040] This invention discloses a method for measuring and filtering large model training data that balances versatility and security, including:

[0041] S1: Obtain multiple security datasets to be evaluated, and at least one benchmark large language model;

[0042] S2: Supervised fine-tuning of the benchmark large language model is performed using the safety dataset to be evaluated to obtain the fine-tuned model; the uniform configuration of fine-tuning is maintained to ensure comparability; to ensure the fairness of the evaluation, all fine-tuning processes use the same hyperparameter settings, that is, the uniform configuration of fine-tuning is maintained; the uniform configuration includes learning rate, learning rate scheduler, number of training rounds, batch size, optimizer, weight decay, training accuracy, maximum sequence length and global random seed.

[0043] S3: Evaluate the fine-tuned models from S2 on both the general performance benchmark and the security performance benchmark, calculating the general performance score and security performance score for each security dataset to be evaluated. The general performance benchmarks are MMLU, HumanEval, and MATH-500; the security performance benchmarks are AdvBench and StrongReject. The general performance score is obtained by averaging the general performance scores of one or more fine-tuned models fine-tuned from the same dataset to be evaluated; the security performance score is obtained by averaging the corresponding security performance scores.

[0044] S4: Combine the S3 scores to calculate the overall quality score for each security dataset to be evaluated; the overall quality score is the weighted sum of the S3 general performance score and the security performance score.

[0045] S5: For each security dataset to be evaluated, extract its multi-dimensional feature index vector; the multi-dimensional feature index vector consists of the following indicators:

[0046] Predictability metrics for language models;

[0047] Safety response distribution variability index;

[0048] Toxicity style index based on external classifier;

[0049] Diversity indicators;

[0050] Effective information density index.

[0051] The predictability metric for a language model is the arithmetic mean of the conditional perplexity of the benchmark large language model across all samples in a given dataset to be evaluated; the dissimilarity metric for the distribution of secure responses is the decaying Kullback-Leibler divergence. Diversity metrics include Self-BLEU-4 and embedding diffusion.

[0052] S6: Use the multi-dimensional feature index vector of S5 as input and the comprehensive quality score of S4 as label to train the machine learning model; the machine learning model is an L1 regularized linear regression model or an L2 regularized linear regression model.

[0053] S7: Obtain a new dataset to be evaluated, and perform step S5 to extract its multi-dimensional feature index vector; input the multi-dimensional feature index vector into the machine learning model S6 to obtain the predicted comprehensive quality score.

[0054] The steps for training a machine learning model in S6 include: pairing the multi-dimensional feature index vectors of each security dataset to be evaluated in S5 with their corresponding S4 comprehensive quality scores to construct a training set; and using the training set to fit the machine learning model.

[0055] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are merely intended to assist those skilled in the art in understanding the principles and knowledge of the present invention, and are not intended to limit the scope of the invention, nor should they be considered as limiting the application scenarios of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various modifications or alterations to the invention, but any variations, changes, and transformations made to the embodiments based on the principles and spirit of the invention also fall within the scope defined by the appended claims. And it is obvious that this specification only uses preferred embodiments as examples, and does not need to exhaustively describe all embodiments.

[0056] Figure 1 This is a flowchart illustrating a method for selecting high-quality, secure datasets for large language models, as provided in this embodiment.

[0057] This invention discloses a method for selecting high-quality, secure datasets for large language models, comprising:

[0058] S1: Obtain multiple security datasets to be evaluated (e.g., dataset A, dataset B, and dataset C), and at least one benchmark large language model; the benchmark large language model can be an open-source model that is the state-of-the-art (SOTA) in the field, such as Llama-3.1-8B, Qwen3-8B, or GPT-OSS. For example, Llama-3.1-8B can be selected as the benchmark large language model.

[0059] S2: Supervised fine-tuning of the benchmark large language model is performed on the security dataset to be evaluated to obtain the fine-tuned model (e.g., model A, model B, and model C) corresponding to each security dataset to be evaluated. To ensure the fairness of the evaluation, all fine-tuning processes can use the same hyperparameter settings, including the same learning rate, learning rate scheduler, number of training rounds, batch size, optimizer, weight decay, training accuracy, maximum sequence length, and global random seed.

[0060] S3: Evaluate the fine-tuned model obtained in S2 on the general performance evaluation benchmark and the safety performance evaluation benchmark to obtain the general performance score and the safety performance score respectively; the general performance evaluation benchmark includes MMLU, HumanEval and MATH-500 benchmark; the safety performance evaluation benchmark includes AdvBench and StrongReject benchmark.

[0061] Specifically:

[0062] S3-1. Input the fine-tuned models (Model A, Model B, and Model C) obtained in S2 into the MMLU, HumanEval, and MATH-500 general performance benchmarks respectively, and obtain the scores of each fine-tuned model on each general performance benchmark. , and The general performance score is obtained by calculating the average of the scores of each benchmark. ,Right now ;

[0063] S3-2. Input the fine-tuned models (Model A, Model B, and Model C) obtained in S2 into the AdvBench and StrongReject security performance evaluation benchmarks to obtain the scores of each fine-tuned model on each security performance evaluation benchmark. and The safety performance score is obtained by calculating the average of the scores from each benchmark. ,Right now .

[0064] S4: Combining the aforementioned general performance score and safety performance score Using a pre-defined utility function, the overall quality score corresponding to each security dataset to be evaluated (Model A, Model B, and Model C) is calculated. The utility function is preferably a weighted sum of general performance score and security performance score, for example... ,in and Preset weights.

[0065] S5: For each security dataset to be evaluated in S1 (dataset A, dataset B, and dataset C), extract its multi-dimensional feature indicator vector based on the pre-defined indicator system. The multi-dimensional feature index vector includes:

[0066] 1) Predictability metrics for language models;

[0067] 2) Safety response distribution difference index;

[0068] 3) Toxicity style index based on external classifier;

[0069] 4) Diversity indicators;

[0070] 5) Effective information density index.

[0071] Specifically:

[0072] S5-1, Predictability Indicator of Computational Language Model: Given each (or several randomly sampled) sample (“question-answer pair”) from the security dataset to be evaluated, input into the baseline large language model Llama-3.1-8B of S5, the computational model should perform a given problem... Below, generate the answer. The degree of confusion due to conditions.

[0073] .

[0074] in, This refers to the model having problems. and the answer before After considering the context of the word, predict the first word. Word The probability of.

[0075] S5-2. Calculate the safety response distribution difference index:

[0076] 1) For each (or a randomly sampled number of) "question-answer pairs" in the security dataset to be evaluated:

[0077] make As a baseline large language model (Llama-3.1-8B), given a question (i.e., a prompt), it is input... Perform a forward propagation to obtain the complete vocabulary probability distribution of the model for the next token with only the question context (i.e., the output of the Softmax layer); this distribution is denoted as... ,Right now ; This represents the model's "original" or "first reaction" distribution to the prompt. In subsequent calculations, The current sample serves as a fixed reference benchmark.

[0078] Obtain the standard "safe response" corresponding to this Prompt. ,in The length of the token in the answer. For each time step. (from arrive ): Construct the context of the current time step .(when When the context is Prompt, the context is the same as the context. Input benchmark model To obtain the model's understanding of the first element in this context. The probability distribution of the complete vocabulary of each token; this distribution is denoted as ,Right now .

[0079] 2) Let γ ∈ (0,1) be the time discount factor, and calculate each time step according to the following formula. Calculate and sum the decaying KL divergence:

[0080] .

[0081] 3) Take the average of the DKL values ​​for all "question-answer pairs" as the deviation index for the dataset.

[0082] S5-3. Calculate toxicity style metrics: Use a pre-trained toxicity classifier, such as Detoxify, to score the answers in the dataset:

[0083] 1) Input the answers from the "question-answer pairs" into the toxicity classifier (Detoxify) in sequence to obtain the toxicity score. ;

[0084] 2) Calculate the 95th percentile of the toxicity score. ;

[0085] 3) Define toxicity style score:

[0086] .

[0087] A higher value indicates that the dataset is more "safe" overall (high score = low toxicity).

[0088] S5-4. Calculate the diversity index:

[0089] Diversity among responses is measured from two perspectives: n-gram overlap and semantic space dispersion.

[0090] 1) Self-BLEU-4

[0091] Each answer in the dataset is treated as a candidate sentence, and all other answers in the dataset are treated as reference sentences.

[0092] The sentence score is calculated using the sentence_bleu method from the NLTK library (nltk.translate.bleu_score) in Python, with its standard Cumulative BLEU-4 setting (i.e., the geometric mean of 1-gram to 4-gram weights);

[0093] The arithmetic mean of all samples is taken to obtain Self-BLEU-4.

[0094] A higher Self-BLEU-4 indicates more similar response styles and lower diversity. (1 - Self-BLEU-4) will be used later as a feature where "higher diversity is better".

[0095] 2) Embedding Dispersion

[0096] Using the SentenceTransformer library in Python and loading a specific pre-trained model (e.g., all-MiniLM-L6-v2), all answers in the dataset are encoded to obtain sentence vectors (Embeddings).

[0097] Calculate the cosine similarity between each pair of all samples (question-answer pairs). and take As semantic distance;

[0098] The embedding diffusion score is obtained by averaging the semantic distances of all samples. The higher the score, the greater the semantic differences between the answers, i.e., the better the semantic diversity.

[0099] S5-5, Calculate the effective information density index:

[0100] 1) Use the same tokenizer to tokenize the answer;

[0101] 2) Calculate the information density for each answer:

[0102] ;

[0103] 3) For all samples Take the average to obtain the average information density value;

[0104] S6, Training Machine Learning Models

[0105] Using the comprehensive quality score described in S4 And the multi-dimensional feature index vector described in S5 Train an interpretable machine learning model to learn the weight of each metric in the overall quality assessment.

[0106] 1) Constructing the training dataset

[0107] Based on the outputs of S4 and S5, construct a training set for the regression task:

[0108] Input features For each security dataset to be evaluated Multi-dimensional feature index vectors extracted using S5 .

[0109] Target Label Using the values ​​calculated in S4, and Corresponding overall quality score The final training set consists of... Sample composition.

[0110] 2) Training an L1 regularized linear regression model

[0111] Using the training dataset constructed in step 1, train an L1 regularized linear regression model. What is learned is from the eigenvectors To the overall quality score Linear mapping:

[0112] ;

[0113] Its objective function is to minimize the sum of the mean squared error (MSE) and the L1 penalty term (the L1 norm of the weights):

[0114] ;

[0115] in: It represents the total number of data samples.

[0116] It is the feature weight vector of the model. It is a bias term.

[0117] It is a hyperparameter that controls the strength of regularization.

[0118] It is the L1 norm of the weight vector.

[0119] S7: Obtain a new secure dataset to be filtered (e.g., dataset...) Extract its feature index vector according to the steps described in S5. The dataset is then input into the machine learning model obtained in S6. The dataset quality evaluator outputs a predicted quality score. Based on the predicted quality score, determine whether the new dataset to be screened (dataset D) is a high-quality dataset. For example, a quality threshold can be set. ,when Greater than this threshold When the dataset is deemed high-quality, it is considered a high-quality dataset.

[0120] The above description is merely a preferred embodiment of the present invention. Although the preferred embodiments have been disclosed above, they are not intended to limit the present invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention's technical solutions still fall within the protection scope of the present invention.

Claims

1. A general-purpose and security performance balanced large model training data metric and screening method, characterized in that, The method comprises the following steps: S1: obtaining a plurality of to-be-evaluated safety data sets and at least one benchmark large language model; The samples in the to-be-evaluated safety data set are "question-answer pairs"; S2: using the to-be-evaluated safety data set to supervise the fine-tuning of the benchmark large language model respectively to obtain a fine-tuned model; and keeping the uniform configuration of fine-tuning to ensure comparability; S3: evaluating the fine-tuned model in S2 on a general performance evaluation benchmark and a safety performance evaluation benchmark to calculate the general performance score and the safety performance score of each to-be-evaluated safety data set; the general performance score is obtained by calculating the average value of the general performance scores obtained by one or more fine-tuned models fine-tuned by the same to-be-evaluated data set; The safety performance score is obtained by calculating the average value of the corresponding safety performance scores; S4: combining the scores in S3 to calculate the comprehensive quality score corresponding to each to-be-evaluated safety data set; The comprehensive quality score is the weighted sum of the general performance score and the safety performance score in S3; S5: extracting a multi-dimensional feature index vector for each to-be-evaluated safety data set; The multi-dimensional feature index vector is composed of the following indexes: an anticipability index of the language model; a safety response distribution difference index; a toxicity style index based on an external classifier; a diversity index; an effective information density index; S6: training a machine learning model by taking the multi-dimensional feature index vector in S5 as input and the comprehensive quality score in S4 as label; S7: obtaining a new to-be-evaluated data set, executing the step of extracting the multi-dimensional feature index vector in S5, and inputting the multi-dimensional feature index vector into the machine learning model in S6 to obtain a predicted comprehensive quality score.

2. The general-purpose and security performance balanced large model training data measurement and screening method according to claim 1, characterized in that, The step of training the machine learning model in S6 comprises: pairing the multi-dimensional feature index vector of each to-be-evaluated safety data set in S5 with the corresponding comprehensive quality score in S4 to construct a training set; and fitting the machine learning model using the training set.

3. The general-purpose and security performance balanced large model training data measurement and screening method according to claim 1, characterized in that, In S3, the general performance evaluation benchmark is MMLU, HumanEval and MATH-500 benchmark; and the safety performance evaluation benchmark is AdvBench and StrongReject benchmark.

4. The general-purpose and security performance balanced large model training data measurement and screening method according to claim 1, characterized in that, The anticipability index of the language model is the arithmetic average of the conditional perplexity of the benchmark large language model on all samples in the to-be-evaluated data set; and the safety response distribution difference index is the decay Kullback-Leibler divergence.

5. The general-purpose and security performance balanced large model training data measurement and screening method according to claim 1, characterized in that, The diversity index includes Self-BLEU-4 and embedding dispersion.

6. The general-purpose and security performance balanced large model training data measurement and screening method according to claims 1-5, characterized in that, In S6, the machine learning model is an L1 regularized linear regression model or an L2 regularized linear regression model.

7. The general-purpose and security performance balanced large model training data measurement and screening method according to claim 6, characterized in that, In S2, to ensure evaluation fairness, the same hyperparameter setting is used in all fine-tuning processes, i.e., the uniform configuration of fine-tuning is kept; and the uniform configuration includes learning rate, learning rate scheduler, training round number, batch size, optimizer, weight decay, training precision, maximum sequence length and global random seed.

Images