Fine-grained data selection method and apparatus

Abstract

This invention provides a fine-grained data selection method and apparatus, relating to the field of data processing technology, comprising: acquiring a dataset to be screened; an instruction sequence and an answer sequence; for the instruction-answer data pair, obtaining the loss change and entropy change of each word in the answer sequence based on a pre-trained large language model; obtaining a composite perceptual index of each word based on the loss change and entropy change of each word; obtaining a comprehensive quality score of the instruction-answer data pair based on the composite perceptual index of each word; wherein the comprehensive quality score is determined based on the sequence-level statistical characteristics of the composite perceptual index and the fluctuation characteristics of the composite perceptual index along the answer sequence; and screening the dataset to be screened based on the comprehensive quality score to obtain a high-quality data subset.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a fine-grained data selection method and apparatus. Background Technology

[0002] After large-scale pre-training, large language models typically require supervised fine-tuning to align with human command intent, enabling them to accurately understand and execute diverse user commands. During supervised fine-tuning, the quality of the training data directly determines the upper limit of the model's performance. High-quality fine-tuning data effectively guides the model to establish correct command response patterns, while low-quality or redundant data not only wastes computational resources but may also introduce noise interference, leading to a decline in model performance.

[0003] With the continuous growth of the parameter scale of large language models and the continuous expansion of the candidate fine-tuning data scale, how to efficiently and accurately select high-quality samples with real deep training value from massive candidate data has become a key technical challenge to improve the efficiency of supervised fine-tuning and the final alignment performance of large language models.

[0004] Existing data screening methods are insufficient in terms of evaluation dimensions and assessment granularity when measuring sample quality. They are unable to fully and meticulously characterize the actual contribution value of data to model fine-tuning. As a result, the selected dataset is prone to contain samples with low logical value, which in turn restricts the performance of the fine-tuned model in instruction following and logical reasoning. Summary of the Invention

[0005] This invention provides a fine-grained data selection method and apparatus to address the problem that existing data screening methods in the prior art have insufficient evaluation dimensions and assessment granularity when measuring sample quality.

[0006] This invention provides a fine-grained data selection method, comprising: Obtain the dataset to be filtered; wherein, the dataset to be filtered includes: multiple instruction-response data pairs, and the instruction-response data pairs include: instruction sequences and response sequences; For the instruction-response data pair, the loss change and entropy change of each word in the response sequence are obtained based on a pre-trained large language model; wherein, the loss change and the entropy change are determined based on a comparison condition of whether the instruction sequence exists or not. Based on the loss change and entropy change of each word, a composite perception index for each word is obtained. Based on the composite perception index of each word, a comprehensive quality score is obtained for the instruction-response data pair; wherein, the comprehensive quality score is determined based on the sequence-level statistical characteristics of the composite perception index and the fluctuation characteristics of the composite perception index along the response sequence. Based on the comprehensive quality score, the dataset to be screened is filtered to obtain a high-quality data subset.

[0007] According to a fine-grained data selection method provided by the present invention, obtaining the loss change and entropy change of each word in the response sequence includes: Obtain the first negative log-likelihood under the condition of including the instruction sequence, and the second negative log-likelihood under the condition of missing the instruction sequence; The change in loss is determined based on the difference between the first negative log-likelihood and the second negative log-likelihood; wherein the change in loss is used to characterize the probability difference of the predicted target word. Obtain the first distribution information entropy under the condition that the instruction sequence is included, and the second distribution information entropy under the condition that the instruction sequence is missing; The entropy change is determined based on the difference between the first distribution information entropy and the second distribution information entropy; wherein the entropy change is used to characterize the uncertainty difference in the probability distribution of the whole word list.

[0008] According to a fine-grained data selection method provided by the present invention, obtaining the composite perceptual index of each word element includes: The loss change and entropy change of each word are weighted and fused to obtain the composite perception index of each word.

[0009] According to a fine-grained data selection method provided by the present invention, the sequence-level statistical characteristics of the composite perception index include: the average composite perception index of the response sequence; The fluctuation characteristics of the composite perception index along the response sequence include: the rate of change of the sign of the composite perception index; The symbol change rate is determined based on the composite perception index, specifically the statistical frequency of positive-to-negative symbol flips between adjacent lexical units in the response sequence.

[0010] According to a fine-grained data selection method provided by the present invention, obtaining a comprehensive quality score for the instruction-response data pair includes: The average composite sensing index is subjected to exponential mapping to obtain the gain intensity value; The overall quality score is determined by multiplying the gain intensity value with the sign change rate.

[0011] According to a fine-grained data selection method provided by the present invention, the dataset to be screened is filtered based on the comprehensive quality score to obtain a high-quality data subset, including: Based on the loss change and entropy change of each word, the average loss change and average entropy change of the response sequence are obtained respectively. If the instruction-response data pair satisfies a preset dual threshold condition, the instruction-response data pair is added to the candidate set; wherein, the preset dual threshold condition includes that the average loss change is less than zero and the average entropy change is less than zero. Based on the comprehensive quality score, the data pairs in the candidate set are sorted and truncated, and the data pairs at the top of the list with a predetermined proportion are extracted as the high-quality data subset.

[0012] According to a fine-grained data selection method provided by the present invention, after obtaining the high-quality data subset, the method further includes: The high-quality data subset is used as fine-tuning samples and input into the large language model to be trained. Supervised fine-tuning is performed on the large language model to be trained based on the high-quality data subset.

[0013] The present invention also provides a fine-grained data selection device, comprising the following modules: The first acquisition module is used to acquire a dataset to be filtered; wherein, the dataset to be filtered includes: multiple instruction-response data pairs, and the instruction-response data pairs include: instruction sequences and response sequences; The second acquisition module is used to acquire the loss change and entropy change of each word in the response sequence based on a pre-trained large language model for the instruction-response data pair; wherein the loss change and the entropy change are determined based on a comparison condition of whether the instruction sequence exists or not. The calculation module is used to obtain the composite perception index of each word based on the loss change and the entropy change of each word. The determination module is used to obtain a comprehensive quality score of the instruction-response data pair based on the composite perception index of each word; wherein the comprehensive quality score is determined based on the sequence-level statistical characteristics of the composite perception index and the fluctuation characteristics of the composite perception index along the response sequence. The filtering module is used to filter the dataset to be filtered based on the comprehensive quality score to obtain a high-quality data subset.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the fine-grained data selection method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the fine-grained data selection method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the fine-grained data selection method as described above.

[0017] This invention provides a fine-grained data selection method and apparatus. By comparing the presence or absence of a command sequence, the loss and entropy changes of each word in the response sequence are obtained and fused into a composite perceptual index. This overcomes the limitation of relying solely on a single probability dimension to evaluate sample value. It simultaneously captures target probability guidance information and the evolutionary information of uncertainty in the full vocabulary distribution at the word level, effectively identifying and excluding simple samples lacking logical challenge. Furthermore, by combining the sequence-level statistical characteristics of the composite perceptual index with its fluctuation characteristics along the response sequence to determine the comprehensive quality score, the smoothing masking of deep cognitive reshaping signals within the sequence by the mean evaluation mechanism is avoided. This allows the comprehensive quality score to simultaneously reflect the information gain strength and the frequency of directional adjustments in the model's predicted state, thereby accurately identifying high-quality samples with both strong correction potential and deep logical challenge, significantly improving the data selection accuracy and model alignment performance of supervised fine-tuning. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the fine-grained data selection method provided by the present invention; Figure 2 This is a schematic diagram of the data filtering process provided by the present invention; Figure 3 A flowchart illustrating the calculation instructions and response data for the overall quality score provided by this invention; Figure 4 This is a schematic diagram of the fine-grained data selection device provided by the present invention; Figure 5This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] The fine-grained data selection method provided in this invention can be applied to electronic devices with data processing capabilities. These devices can be servers, computers, workstations, or other computing devices, or distributed computing systems composed of multiple computing nodes. The electronic device is equipped with a processor, which can be a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor (TPU), or a combination thereof, to execute the various computational and data processing operations involved in this invention. In the following embodiments, the processor is used as the execution subject of the method of this invention.

[0022] Figure 1 This is a flowchart illustrating the fine-grained data selection method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 110, obtain the dataset to be filtered; wherein, the dataset to be filtered includes: multiple instruction-response data pairs, and the instruction-response data pair includes: instruction sequence and response sequence; In this step, the processor acquires a dataset to be screened. This dataset is a candidate data set for supervised fine-tuning of a large language model, containing multiple instruction-response data pairs.

[0023] Each instruction-response data pair consists of two parts: an instruction sequence and a response sequence. The instruction sequence refers to the natural language instruction text proposed by the user or task designer, used to describe the task that the model is expected to perform or the question to answer, such as: Please summarize the main points of the following article, or translate the following English text into Chinese.

[0024] The response sequence is the target response text corresponding to the instruction sequence. It consists of one or more tokens arranged in sequence to represent the expected response to the instruction.

[0025] A lexical is the smallest unit of text that a language model processes. It can be a complete word, a word fragment, or a character. The specific way it is segmented depends on the word segmenter used by the language model.

[0026] The dataset to be selected can be obtained in ways including but not limited to: downloading from publicly available instruction fine-tuning datasets, collecting from data annotation platforms, automatically generating from large language models, or aggregating from multiple heterogeneous data sources after preprocessing. The number of instruction-response pairs in the dataset to be selected can be determined according to the actual application scenario, for example, it can contain tens of thousands to millions of instruction-response pairs.

[0027] In a specific implementation scenario, the dataset to be screened can be denoted as: :,in N The total number of response pairs in the dataset to be filtered. For the first i A sequence of instructions in an instruction-response data pair. For the first i The sequence of responses in an instruction-response data pair.

[0028] Step 120: For the instruction-response data pair, obtain the loss change and entropy change of each word in the response sequence based on the pre-trained large language model; wherein, the loss change and the entropy change are determined based on the comparison condition of whether the instruction sequence exists or not. In this step, for each instruction-response data pair in the dataset to be filtered, the processor uses a pre-trained large language model to obtain the loss change and entropy change of each word in the response sequence of the data pair.

[0029] A pre-trained large language model refers to a language model that has been pre-trained on a large-scale corpus and possesses the ability to probabilistically model natural language text. This pre-trained large language model can be any available pre-trained language model, such as the LLaMA series models. It should be noted that this pre-trained large language model is used to evaluate data quality; it itself does not require additional training or fine-tuning, but only utilizes its forward inference capabilities to obtain the necessary probabilistic information.

[0030] The loss change is a scalar value calculated for each word position in the response sequence. It is used to measure the change in the prediction performance of the pre-trained large language model at the corresponding word positions under two different input conditions: the presence or absence of the instruction sequence.

[0031] Specifically, the change in loss reflects the degree to which the difficulty of predicting the target word at a given position changes when the instruction sequence is introduced, compared to the situation without instructions. A negative change in loss indicates that the introduction of the instruction sequence reduces the loss for the model to generate the correct target word at that position, meaning that the instruction has a positive guiding effect on the model's prediction.

[0032] The entropy change is also a scalar value calculated for each word position in the response sequence. It is used to measure the change in the uncertainty of the probability distribution at the corresponding word position of the pre-trained large language model under two different input conditions: the presence or absence of the instruction sequence.

[0033] The entropy change characterizes the impact of the instruction sequence on the model's predicted state from the perspective of the overall probability distribution. When the entropy change is negative, it indicates that the introduction of the instruction sequence reduces the model's prediction uncertainty at that position, meaning that the instruction makes the model's probability distribution more concentrated.

[0034] The comparison condition determination based on the existence or non-existence of the instruction sequence refers to the processor performing forward inference of the pre-trained large language model under two conditions for the same word position in the response sequence: one is to use both the instruction sequence and the response sequence as input (i.e., the condition that the instruction sequence exists), and the other is to use only the response sequence as input (i.e., the condition that the instruction sequence is missing).

[0035] By comparing the differences in the model's predictions at the same lexical position under these two conditions, the changes in loss and entropy for that lexical unit are determined. This comparison mechanism can separate the influence of the instruction sequence on the model's predictive behavior from the language pattern of the response sequence itself, thereby accurately evaluating the guiding effect of the instructions on each position in the response generation process.

[0036] When calculating the change in loss, it can be determined based on the target word prediction probabilities output by the pre-trained large language model under the two conditions mentioned above.

[0037] For example, it can be obtained by calculating the difference between the negative log-likelihood values ​​corresponding to the target lexical under two conditions.

[0038] When calculating the change in entropy, it can be determined based on the information entropy of the vocabulary probability distribution output by the pre-trained large language model under the two conditions mentioned above, for example, by calculating the difference between the information entropy of the probability distribution under the two conditions.

[0039] In this application, the specific calculation methods for the change in loss and the change in entropy are not limited to the examples above. Any calculation method that can measure the impact of the instruction sequence on the model's performance in predicting target terms and the uncertainty of distribution is applicable.

[0040] In a specific embodiment, for a given pre-trained large language model A candidate instruction-response data pair ,in This is the instruction text. For the corresponding answer text sequence, for the answer sequence Each lexical position in ( ), calculate in the given instruction Under the given conditions, the change in the negative log-likelihood (loss) of the model generating the correct word relative to the loss under the no-instruction condition. and entropy change .

[0041] In this application, the change in loss and the change in entropy characterize the impact of the instruction sequence on the model's generated response sequence from different dimensions. The change in loss measures the guiding effect of the instruction from the perspective of the prediction probability of the target lexical unit, while the change in entropy measures the impact of the instruction on the predictive certainty of the model from the perspective of the uncertainty of the probability distribution of the entire vocabulary. The combination of the two can provide a more comprehensive evaluation of the training value of the data.

[0042] Step 130: Based on the loss change and entropy change of each word, obtain the composite perception index of each word. In this application, the changes in loss and entropy of each word are combined to obtain a composite perceptual index for each word. This composite perceptual index is a fusion result of the changes in loss and entropy, used to comprehensively reflect the multidimensional impact of the instruction sequence on the model's predictive behavior at the word level.

[0043] Specific fusion methods can include weighted summation, product operation, nonlinear combination, or other mathematical operations that can integrate two variables into a single indicator.

[0044] By integrating the change in loss and the change in entropy into a composite perceptual index, we can simultaneously consider the target word probability guidance information and the uncertainty evolution information of the whole word list distribution in a unified index system, thereby more accurately measuring the training value of each word position for model fine-tuning.

[0045] In a specific implementation example, for the first... t Each word position can be used to calculate the change in loss at that position. and entropy change The composite sensing index at this position is calculated using a pre-defined fusion rule. Thus, the composite sensing indices for all word positions in the answer sequence constitute an index sequence. This indicator sequence characterizes the instruction sequence on a word-by-word basis. The combined impact on each position in the process of the model generating the entire response sequence.

[0046] Step 140: Based on the composite perception index of each word, obtain the comprehensive quality score of the instruction-response data pair; wherein, the comprehensive quality score is determined based on the sequence-level statistical characteristics of the composite perception index and the fluctuation characteristics of the composite perception index along the response sequence. In this application, the overall quality score is determined based on the sequence-level statistical characteristics of the composite perception index and the fluctuation characteristics of the composite perception index along the response sequence.

[0047] In this application, the comprehensive quality score of the instruction-response data pair is calculated based on the composite perception index of each word obtained in step 130.

[0048] The overall quality score is a quantitative evaluation of the training value of the entire instruction-response data. Its determination process takes into account both the sequence-level statistical characteristics of the composite perceptual index and the fluctuation characteristics of the composite perceptual index along the response sequence.

[0049] The sequence-level statistical features of the composite perception index refer to the feature quantities obtained by performing sequence-level statistical calculations on the composite perception index of all words in the response sequence. These feature quantities characterize the overall influence of the instruction sequence on the generation of the entire response sequence. Sequence-level statistical features may include, but are not limited to, statistics such as mean, median, and weighted mean.

[0050] The fluctuation characteristics of the composite perception index along the response sequence refer to the change pattern and fluctuation degree of the composite perception index during the word-by-word meta-progression of the response sequence.

[0051] This fluctuation feature reflects whether the model's predicted state has undergone frequent directional adjustments during the word-by-word meta-generation of the answer.

[0052] In the supervised fine-tuning scenario of large language models, if the composite perception index of the model fluctuates frequently along the response sequence when processing a certain instruction-response data pair, it indicates that the data pair can trigger repeated adjustments to the model's predicted state, which means that the data contains high cognitive reshaping value and has strong training significance for correcting the model's inherent biases.

[0053] By combining sequence-level statistical features and fluctuation features to determine the overall quality score, we can avoid the smoothing and masking of deep signals within the sequence when relying solely on statistical measures such as the mean. This allows the overall quality score to simultaneously reflect the training value of the data at both the information gain strength and the frequency of directional adjustments in the model's predicted state.

[0054] In a specific implementation example, we can first base it on a composite sensing index sequence. Calculate its sequence-level statistical characteristics (e.g., average value). S Simultaneously, the fluctuation characteristics of the indicator sequence along the response sequence are calculated, such as the statistical measure of the change in indicator values ​​between adjacent words. Then, the two are coupled through a pre-defined operation to determine the final comprehensive quality score. .

[0055] Step 150: Based on the comprehensive quality score, the dataset to be screened is filtered to obtain a high-quality data subset.

[0056] In this application, based on the comprehensive quality score, all instruction-response data pairs in the dataset to be screened are screened, and data pairs with high training value are extracted to form a high-quality data subset.

[0057] The filtering methods may include, but are not limited to: sorting based on comprehensive quality scores and extracting data pairs with higher scores, filtering based on preset thresholds, and performing multi-level filtering in combination with other auxiliary conditions.

[0058] Through the filtering process, the data samples with the highest training value for supervised fine-tuning of the model can be accurately identified from the massive candidate data. The high-quality data subset obtained can be directly used for subsequent supervised fine-tuning training of the model.

[0059] In a specific implementation example, the server can sort all instruction-response pairs in the dataset to be filtered according to their overall quality score from high to low, and extract a preset proportion of the top-ranked pairs (e.g., the top...). K The data pairs representing % are considered as a high-quality subset.

[0060] In addition, before sorting and filtering, the data can be initially filtered based on preset conditions to remove data pairs that obviously do not meet the requirements, and then sorted and truncated in the filtered candidate set.

[0061] The fine-grained data selection method provided in this invention obtains the loss change and entropy change of each word in the response sequence based on the comparison condition of the presence or absence of the instruction sequence, and merges them into a composite perceptual index. This overcomes the limitation of relying solely on a single probability dimension to evaluate sample value. It simultaneously captures target probability guidance information and evolutionary information of uncertainty in the full vocabulary distribution at the word level, effectively identifying and excluding simple samples lacking logical challenge. Furthermore, by combining the sequence-level statistical characteristics of the composite perceptual index with its fluctuation characteristics along the response sequence to determine the comprehensive quality score, the smoothing masking of deep cognitive reshaping signals within the sequence by the mean evaluation mechanism is avoided. This allows the comprehensive quality score to simultaneously reflect the information gain strength and the frequency of directional adjustment of the model's predicted state, thereby accurately identifying high-quality samples with both strong correction potential and deep logical challenge, significantly improving the data selection accuracy and model alignment performance of supervised fine-tuning.

[0062] Optionally, obtaining the loss change and entropy change of each word in the response sequence includes: Obtain the first negative log-likelihood under the condition of including the instruction sequence, and the second negative log-likelihood under the condition of missing the instruction sequence; The change in loss is determined based on the difference between the first negative log-likelihood and the second negative log-likelihood; wherein the change in loss is used to characterize the probability difference of the predicted target word. Obtain the first distribution information entropy under the condition that the instruction sequence is included, and the second distribution information entropy under the condition that the instruction sequence is missing; The entropy change is determined based on the difference between the first distribution information entropy and the second distribution information entropy; wherein the entropy change is used to characterize the uncertainty difference in the probability distribution of the whole word list.

[0063] In this application, for each lexical position in the response sequence t The server performs forward inference on the pre-trained large language model under two different conditions.

[0064] Given a sequence of instructions, the instruction sequence... x and position in the answer sequence t All previous morphemes As input, obtain the model at location t Predict target word The negative log-likelihood of is denoted as the first negative log-likelihood, i.e. .

[0065] In the case of missing instruction sequences, only the positions in the response sequence will be considered. t All previous morphemes As input, obtain the model at location t Predict target word The negative log-likelihood of is denoted as the second negative log-likelihood, i.e. .

[0066] In this application, the negative log-likelihood is the negative log-value of the probability that the language model predicts the target word under given context. The larger the value, the more uncertain the model's prediction of the target word, i.e., the lower the prediction probability; the smaller the value, the more certain the model's prediction of the target word, i.e., the higher the prediction probability.

[0067] The change in loss is determined based on the difference between the first negative log-likelihood and the second negative log-likelihood. Specifically, the change in loss can be determined by calculating the difference between the first negative log-likelihood and the second negative log-likelihood.

[0068] In a specific implementation example, the change in loss at position t The calculation method is as follows: ; in, Indicates position All previous morphemes, Representation Model The predicted probability. This value Indication of instructions Reduced the model's position Generate correct lexical units The loss. That is, the correct prediction of the target word at that position by the instruction sequence has a positive guiding effect. When the instruction sequence is added, it indicates that the addition of the instruction sequence actually increases the prediction loss at that position, suggesting a conflict between the instruction information and the model's prediction for that position.

[0069] In this application, for each lexical position in the response sequence t The server obtains the information entropy of the model's predicted probability distribution at a given location under two conditions. Under the condition of including the instruction sequence, the server obtains the model's information entropy at location... t The information entropy of the predicted probability distribution is denoted as the first distribution information entropy, i.e. .

[0070] Under the condition of missing instruction sequence, obtain the model at position. t The information entropy of the predicted probability distribution is denoted as the second distribution information entropy, i.e. .

[0071] Distributional information entropy is a measure of the uncertainty of a probability distribution, and its calculation is based on the probability distribution of the model at that lexical position over the entire vocabulary.

[0072] Specifically, the first distribution information entropy is calculated as follows: The second distribution information entropy is calculated as follows: in, The vocabulary set for the model. This is any word element in the vocabulary. Indication of instructions Reduced the model's position Predict the entropy of the lexicon.

[0073] The larger the information entropy value, the more dispersed the probability distribution and the less uncertain the model's prediction; the smaller the information entropy value, the more concentrated the probability distribution and the more certain the model's prediction.

[0074] The entropy change is determined based on the difference between the first distribution information entropy and the second distribution information entropy. In a specific implementation example, the location... t entropy change The calculation method is as follows: ; This change in entropy characterizes how the uncertainty of the full vocabulary probability distribution at that position changes after the instruction sequence is added.

[0075] when This indicates that the addition of the instruction sequence reduces the entropy of the full vocabulary distribution of the predicted words at the position, meaning that the instruction sequence makes the model's prediction distribution more concentrated and reduces uncertainty.

[0076] More specifically, the changes in loss and entropy reflect the impact of instructions on model predictions at different granular levels.

[0077] The change in loss focuses on the change in the predicted probability of a single target word, while the change in entropy focuses on the structural change in the probability distribution across the entire vocabulary.

[0078] For example, at certain lexical positions, there might be a small decrease in loss, meaning the probability of the target lexical term doesn't change much; however, the entropy decrease might be large, indicating a significant reduction in the uncertainty of the entire vocabulary distribution. This suggests that the instruction easily eliminated the model's prediction ambiguity, and the actual logical value of the data at that position is low. By simultaneously obtaining the changes in loss and entropy, the training value of each lexical position can be evaluated more comprehensively.

[0079] Optionally, obtaining the composite perception index of each word element includes: The loss change and entropy change of each word are weighted and fused to obtain the composite perception index of each word.

[0080] In this application, the loss change and entropy change at each word position in the response sequence are weighted and fused, and the two are combined according to a preset weight ratio to obtain the composite perception index of that word position.

[0081] The weighted fusion process refers to multiplying the change in loss and the change in entropy by their respective weight coefficients and then summing them. By setting different weight coefficients, the relative contributions of the change in loss and the change in entropy in the composite sensing index can be adjusted to adapt to different data filtering needs and application scenarios.

[0082] In a specific implementation example, for the first... t Each word position, its composite perception index The calculation method is as follows: ; in, Preset weighting coefficients ( ), used to adjust the relative importance of the change in loss and the change in entropy.

[0083] In this application, through weighted fusion processing, the composite perception index simultaneously encodes the information changes of the target word probability guidance dimension and the uncertainty dimension of the full word list distribution at the word level, so that the subsequent quality score can be calculated based on a more comprehensive word-level evaluation result, thereby improving the accuracy of data screening.

[0084] Optionally, the sequence-level statistical features of the composite perception index include: the average composite perception index of the response sequence; The fluctuation characteristics of the composite perception index along the response sequence include: the rate of change of the sign of the composite perception index; The symbol change rate is determined based on the composite perception index, specifically the statistical frequency of positive-to-negative symbol flips between adjacent lexical units in the response sequence.

[0085] In this application, the average composite sensing index is the result obtained by taking the arithmetic mean of the composite sensing indices at all word positions in the response sequence. It is used to reflect the overall information gain strength of the instruction sequence in generating the entire response sequence. In a specific implementation example, the average composite sensing index of the response sequence... S The calculation method is as follows: in, T To answer the question of the length of the sequence, For the first t A composite perceptual index based on the position of each word element. When When the instruction sequence brings comprehensive information gain to the generation of the entire response sequence, it means that the instruction sequence reduces the prediction loss and prediction distribution uncertainty of the model as a whole.

[0086] The sign change rate is a statistical measure of the reversal of the positive and negative signs of numerical symbols between adjacent words in a compound sensing index sequence. Specifically, for a compound sensing index sequence... For each pair of adjacent word positions (t, t+1), determine... and Whether the numerical sign has undergone a sign flip (i.e., one is positive or zero and the other is negative, or vice versa); count the number of times the sign flip occurs, and divide that number by the total number of adjacent word pairs to obtain the sign change rate R. in, For indicator functions, the value is 1 when the condition inside the parentheses is true, and 0 otherwise; for symbolic functions... Defined as: rate of change of sign The larger the value, the more frequently the composite perception index alternates between positive and negative in the response sequence, reflecting that the model's predicted state undergoes more directional adjustments during the word-by-word generation of the response. This frequent directional adjustment means that the model faces strong cognitive conflict when processing this instruction-response data pair, requiring repeated adjustments to its prediction strategy. Therefore, this data pair has high cognitive reshaping value and correction potential.

[0087] on the contrary, R The smaller the value, the more stable the model's predicted state is throughout the entire response sequence, and the more consistent the direction of change. This data has a weaker effect on intervening in the model's inherent biases.

[0088] By introducing the rate of change of sign as a fluctuation feature, this scheme can capture the deep training signals inside the sequence that are masked by the traditional mean evaluation system, thereby more accurately identifying high-value samples that can effectively intervene in and correct the inherent bias of the model.

[0089] Optionally, a comprehensive quality score is obtained for the instruction-response data pairs, including: The average composite sensing index is subjected to exponential mapping to obtain the gain intensity value; The overall quality score is determined by multiplying the gain intensity value with the sign change rate.

[0090] In this application, the average composite sensing index is first subjected to exponential mapping processing. This exponential mapping processing refers to using the average composite sensing index as the independent variable of an exponential function to calculate the corresponding exponential value, i.e., the gain intensity value.

[0091] The role of exponential mapping is to transform the average composite sensing index from linear space to exponential space, so that the differences in the average composite sensing index are nonlinearly amplified in exponential space, thereby better distinguishing data samples with different gain intensity levels.

[0092] In a specific implementation example, the gain intensity value is calculated as follows: ,in This represents the average composite perception index. When... At that time, the gain intensity value The value range of is (0,1), and The larger the absolute value of (i.e., the stronger the overall information gain), the closer the gain strength value is to 0; when When the gain strength value is 1, it means that the instruction sequence does not bring any overall information gain to the generation of the response sequence.

[0093] The server then multiplies the gain strength value by the sign rate of change to determine the overall quality score. In a specific implementation example, the overall quality score is... The calculation method is as follows: This comprehensive quality score integrates the gain strength value and the sign change rate into a scalar value through multiplicative coupling, thereby achieving a comprehensive quantification of the training value of the data samples.

[0094] Specifically, the overall quality score considers information from two levels: gain strength value and other factors. This reflects the overall information gain strength of the instruction sequence on the response sequence generation process, and the rate of change of sign. This reflects the frequency with which the predicted state is adjusted during the inference process.

[0095] A high overall quality score is achieved only when a command-response data pair possesses both high information gain and high frequency of prediction state adjustments. This ensures that the final selected data samples are both logically challenging and can effectively intervene in the inherent biases of the model.

[0096] By employing an exponential mapping and multiplicative coupling calculation method, the comprehensive quality score can effectively avoid the dimensional confusion problem that may be caused by simple linear superposition, so that the evaluation information of two different levels can be reasonably integrated in the quality score, thereby improving the discrimination and accuracy of data screening.

[0097] Optionally, the dataset to be screened is filtered based on the comprehensive quality score to obtain a high-quality subset of data, including: Based on the loss change and entropy change of each word, the average loss change and average entropy change of the response sequence are obtained respectively. If the instruction-response data pair satisfies a preset dual threshold condition, the instruction-response data pair is added to the candidate set; wherein, the preset dual threshold condition includes that the average loss change is less than zero and the average entropy change is less than zero. Based on the comprehensive quality score, the data pairs in the candidate set are sorted and truncated, and the data pairs at the top of the list with a predetermined proportion are extracted as the high-quality data subset.

[0098] In this application, the arithmetic mean of the loss changes at all word positions in the response sequence is taken to obtain the average loss change of the instruction-response data pair; at the same time, the arithmetic mean of the entropy changes at all word positions in the response sequence is taken to obtain the average entropy change of the instruction-response data pair.

[0099] In a specific implementation example, the average loss change and the change in average entropy The calculation methods are as follows: in, To answer the sequence The length of the value. Indication of instructions For generating the entire answer sequence This results in a comprehensive information gain (manifested as a reduction in loss and a reduction in entropy).

[0100] More specifically, a dual threshold judgment is applied to the average change in loss and the average change in entropy for each instruction-response data pair. Only when an instruction-response data pair simultaneously satisfies the average change in loss... And the change in average entropy Only when these two conditions are met will the data pair be added to the candidate set.

[0101] In this application, the design principle of the dual threshold condition is that the change in average loss is less than zero, which ensures that the instruction sequence reduces the loss of the model's predicted target words as a whole, that is, the instruction sequence has a positive guiding effect on the model to generate correct answers; the change in average entropy is less than zero, which ensures that the instruction sequence reduces the uncertainty of the model's prediction distribution as a whole, that is, the instruction sequence makes the model's predictions more focused and certain.

[0102] Only data pairs that simultaneously meet both conditions are considered valid samples where the instruction sequence has played a positive role in both probabilistic guidance and distribution determinism, and are suitable as candidate data for supervised fine-tuning. Data pairs that do not meet either condition will be filtered out to avoid negatively impacting subsequent fine-tuning training.

[0103] In this application, all instruction-response data pairs that pass the dual threshold conditions in the candidate set are sorted in descending order according to their comprehensive quality scores. Subsequently, a predetermined proportion of data pairs ranking at the top are extracted from the sorted candidate set to form the final high-quality data subset.

[0104] Preset ratio The settings can be adjusted based on the actual computing resource budget and fine-tuning data requirements, such as setting it to the top 10%, top 20%, or top 30%.

[0105] In a specific implementation example Figure 2 This is a schematic diagram of the data filtering process provided by the present invention, such as... Figure 2 As shown, the server first loads the pre-trained large language model. And set weighting coefficients Then, the dataset to be filtered Each instruction-response data pair in the process is used to calculate the average change in loss. Average entropy change and overall quality score Next, a dual threshold test is performed on each data pair, retaining... and The instruction-response pair is added to the candidate set, and data pairs that do not meet the conditions are discarded.

[0106] Finally, the data pairs in the candidate set are sorted from high to low according to their overall quality scores, and the top ones are retained. The data pairs are output as a high-quality subset of data.

[0107] By employing a two-stage screening mechanism that combines initial screening with a comprehensive quality score ranking truncation, this scheme can first exclude data that is unqualified in terms of probability guidance or distribution determinism, and then accurately select the samples with the highest training value from the qualified candidates, thus establishing a systematic high-quality data identification process and effectively improving the reliability and accuracy of the screening results.

[0108] Optionally, after obtaining the high-quality data subset, the method further includes: The high-quality data subset is used as fine-tuning samples and input into the large language model to be trained. Supervised fine-tuning is performed on the large language model to be trained based on the high-quality data subset.

[0109] In this embodiment, after the server completes data filtering and obtains a high-quality data subset, it uses this high-quality data subset as training data for supervised fine-tuning and performs supervised fine-tuning operations on the large language model to be trained.

[0110] The large language model to be trained refers to the target model that needs to be fine-tuned under supervised supervision to align with human command intentions. This large language model to be trained can be the same as the pre-trained large language model used for data screening and evaluation, or it can be a different model.

[0111] In some implementation scenarios, a smaller pre-trained large language model can be used as the evaluation model for data screening, while a larger pre-trained large language model can be used as the large language model to be trained for supervised fine-tuning.

[0112] Supervised fine-tuning refers to the process of updating and optimizing the parameters of a large language model to be trained using supervised learning methods, with instruction-response data pairs from a high-quality subset of data as training samples.

[0113] During supervised fine-tuning, the instruction sequence is used as input and the response sequence is used as the desired output target. By minimizing the loss function (e.g., cross-entropy loss) between the model-generated response and the target response, the model parameters are iteratively updated, enabling the model to learn to generate high-quality responses that meet expectations based on the instructions.

[0114] Since the data in the high-quality data subset are samples with high training value selected after fine-grained screening, compared with fine-tuning using all unscreened data, supervised fine-tuning using the high-quality data subset can effectively improve the model's performance in instruction following and logical reasoning while reducing computational costs, thus achieving more efficient model alignment.

[0115] Figure 3 The flowchart illustrating the calculation instructions and response data for the overall quality score provided by this invention is as follows: Figure 3 As shown, First, input the instruction to be processed – a question-answer pair (x, y); for this question-answer pair, calculate the change in loss for each word in the response sequence. With entropy change Both of these changes were derived by comparing model predictions under conditions including and lacking instruction sequences.

[0116] The loss change and entropy change are weighted and fused to calculate the composite perceptual index of each word in the response sequence. ,in These are preset weighting coefficients.

[0117] Based on the calculation results of each word, the average change in loss of the response sequence is calculated. Average entropy change and average composite perception index On the other hand, the frequency of positive and negative flips of the compound perception index between adjacent word units is statistically analyzed, and the sign change rate of the compound perception index is calculated. .

[0118] The gain strength of the sequence, i.e., the exponential mapping of the average composite sensing index, is used. The degree of tortuosity of the predicted state (rate of change of sign) Perform multiplication to calculate the final overall quality score. The process ends. The higher the score, the greater the value of the sample for fine-tuning large language models.

[0119] The fine-grained data selection device provided by the present invention will be described below. The fine-grained data selection device described below and the fine-grained data selection method described above can be referred to in correspondence.

[0120] Figure 4This is a schematic diagram of the fine-grained data selection device provided by the present invention, as shown below. Figure 4 As shown, it includes: The first acquisition module 410 is used to acquire a dataset to be filtered; wherein, the dataset to be filtered includes: multiple instruction-response data pairs, and the instruction-response data pairs include: instruction sequences and response sequences; The second acquisition module 420 is used to acquire the loss change and entropy change of each word in the response sequence based on a pre-trained large language model for the instruction-response data pair; wherein the loss change and the entropy change are determined based on a comparison condition of whether the instruction sequence exists or not. The calculation module 430 is used to obtain the composite perception index of each word based on the loss change and the entropy change of each word. The determining module 440 is used to obtain a comprehensive quality score of the instruction-response data pair based on the composite perception index of each word; wherein, the comprehensive quality score is determined based on the sequence-level statistical features of the composite perception index and the fluctuation features of the composite perception index along the response sequence. The filtering module 450 is used to filter the dataset to be filtered based on the comprehensive quality score to obtain a high-quality data subset.

[0121] In this application, the loss and entropy changes of each word in the response sequence are obtained based on the comparison condition of the presence or absence of the instruction sequence, and then fused into a composite perceptual index. This overcomes the limitation of relying solely on a single probability dimension to evaluate sample value. It simultaneously captures target probability guidance information and evolutionary information of uncertainty in the full vocabulary distribution at the word level, effectively identifying and excluding simple samples lacking logical challenge. Furthermore, by combining the sequence-level statistical features of the composite perceptual index with its fluctuation features along the response sequence to determine the comprehensive quality score, the smoothing masking of deep cognitive reshaping signals within the sequence by the mean evaluation mechanism is avoided. This allows the comprehensive quality score to simultaneously reflect the information gain strength and the frequency of directional adjustment of the model's predicted state, thereby accurately identifying high-quality samples with both strong correction potential and deep logical challenge, significantly improving the data selection accuracy and model alignment performance of supervised fine-tuning.

[0122] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can invoke logical instructions in the memory 530 to execute a fine-grained data selection method, which includes: acquiring a dataset to be filtered; wherein the dataset to be filtered includes: multiple instruction-response data pairs, and the instruction-response data pairs include: an instruction sequence and a response sequence; For the instruction-response data pair, the loss change and entropy change of each word in the response sequence are obtained based on a pre-trained large language model; wherein, the loss change and the entropy change are determined based on a comparison condition of whether the instruction sequence exists or not. Based on the loss change and entropy change of each word, a composite perception index for each word is obtained. Based on the composite perception index of each word, a comprehensive quality score is obtained for the instruction-response data pair; wherein, the comprehensive quality score is determined based on the sequence-level statistical characteristics of the composite perception index and the fluctuation characteristics of the composite perception index along the response sequence. Based on the comprehensive quality score, the dataset to be screened is filtered to obtain a high-quality data subset.

[0123] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, the computer program being executed by a processor, the computer being able to execute the fine-grained data selection method provided by the above methods, the method including: acquiring a dataset to be filtered; wherein, the dataset to be filtered includes: multiple instruction-response data pairs, the instruction-response data pair including: an instruction sequence and a response sequence; For the instruction-response data pair, the loss change and entropy change of each word in the response sequence are obtained based on a pre-trained large language model; wherein, the loss change and the entropy change are determined based on a comparison condition of whether the instruction sequence exists or not. Based on the loss change and entropy change of each word, a composite perception index for each word is obtained. Based on the composite perception index of each word, a comprehensive quality score is obtained for the instruction-response data pair; wherein, the comprehensive quality score is determined based on the sequence-level statistical characteristics of the composite perception index and the fluctuation characteristics of the composite perception index along the response sequence. Based on the comprehensive quality score, the dataset to be screened is filtered to obtain a high-quality data subset.

[0125] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a fine-grained data selection method provided by the methods described above, the method comprising: acquiring a dataset to be filtered; wherein the dataset to be filtered comprises: a plurality of instruction-response data pairs, the instruction-response data pairs comprising: an instruction sequence and a response sequence; For the instruction-response data pair, the loss change and entropy change of each word in the response sequence are obtained based on a pre-trained large language model; wherein, the loss change and the entropy change are determined based on a comparison condition of whether the instruction sequence exists or not. Based on the loss change and entropy change of each word, a composite perception index for each word is obtained. Based on the composite perception index of each word, a comprehensive quality score is obtained for the instruction-response data pair; wherein, the comprehensive quality score is determined based on the sequence-level statistical characteristics of the composite perception index and the fluctuation characteristics of the composite perception index along the response sequence. Based on the comprehensive quality score, the dataset to be screened is filtered to obtain a high-quality data subset.

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fine-grained data selection method, characterized in that, include: Obtain the dataset to be filtered; wherein, the dataset to be filtered includes: multiple instruction-response data pairs, and the instruction-response data pairs include: instruction sequences and response sequences; For the instruction-response data pair, the loss change and entropy change of each word in the response sequence are obtained based on a pre-trained large language model; wherein, the loss change and the entropy change are determined based on a comparison condition of whether the instruction sequence exists or not. Based on the loss change and entropy change of each word, a composite perception index for each word is obtained. Based on the composite perception index of each word, a comprehensive quality score is obtained for the instruction-response data pair; wherein, the comprehensive quality score is determined based on the sequence-level statistical characteristics of the composite perception index and the fluctuation characteristics of the composite perception index along the response sequence. Based on the comprehensive quality score, the dataset to be screened is filtered to obtain a high-quality data subset.

2. The fine-grained data selection method according to claim 1, characterized in that, Obtaining the loss change and entropy change of each word in the response sequence includes: Obtain the first negative log-likelihood under the condition of including the instruction sequence, and the second negative log-likelihood under the condition of missing the instruction sequence; The change in loss is determined based on the difference between the first negative log-likelihood and the second negative log-likelihood; wherein the change in loss is used to characterize the probability difference of the predicted target word. Obtain the first distribution information entropy under the condition that the instruction sequence is included, and the second distribution information entropy under the condition that the instruction sequence is missing; The entropy change is determined based on the difference between the first distribution information entropy and the second distribution information entropy; wherein the entropy change is used to characterize the uncertainty difference in the probability distribution of the whole word list.

3. The fine-grained data selection method according to claim 1, characterized in that, The process of obtaining the composite perception index for each word element includes: The loss change and entropy change of each word are weighted and fused to obtain the composite perception index of each word.

4. The fine-grained data selection method according to claim 1, characterized in that, The sequence-level statistical features of the composite perception index include: the average composite perception index of the response sequence; The fluctuation characteristics of the composite perception index along the response sequence include: the rate of change of the sign of the composite perception index; The symbol change rate is determined based on the composite perception index, specifically the statistical frequency of positive-to-negative symbol flips between adjacent lexical units in the response sequence.

5. The fine-grained data selection method according to claim 4, characterized in that, Obtain the overall quality score of the instruction-response data pair, including: The average composite sensing index is subjected to exponential mapping to obtain the gain intensity value; The overall quality score is determined by multiplying the gain intensity value with the sign change rate.

6. The fine-grained data selection method according to claim 1, characterized in that, The dataset to be screened is filtered based on the comprehensive quality score to obtain a high-quality subset of data, including: Based on the loss change and entropy change of each word, the average loss change and average entropy change of the response sequence are obtained respectively. If the instruction-response data pair satisfies a preset dual threshold condition, the instruction-response data pair is added to the candidate set; wherein, the preset dual threshold condition includes that the average loss change is less than zero and the average entropy change is less than zero. Based on the comprehensive quality score, the data pairs in the candidate set are sorted and truncated, and the data pairs at the top of the list with a predetermined proportion are extracted as the high-quality data subset.

7. The fine-grained data selection method according to any one of claims 1 to 6, characterized in that, After obtaining the high-quality data subset, the method further includes: The high-quality data subset is used as fine-tuning samples and input into the large language model to be trained. Supervised fine-tuning is performed on the large language model to be trained based on the high-quality data subset.

8. A fine-grained data selection device, characterized in that, include: The first acquisition module is used to acquire a dataset to be filtered; wherein, the dataset to be filtered includes: multiple instruction-response data pairs, and the instruction-response data pairs include: instruction sequences and response sequences; The second acquisition module is used to acquire the loss change and entropy change of each word in the response sequence based on a pre-trained large language model for the instruction-response data pair; wherein the loss change and the entropy change are determined based on a comparison condition of whether the instruction sequence exists or not. The calculation module is used to obtain the composite perception index of each word based on the loss change and the entropy change of each word. The determination module is used to obtain a comprehensive quality score of the instruction-response data pair based on the composite perception index of each word; wherein the comprehensive quality score is determined based on the sequence-level statistical characteristics of the composite perception index and the fluctuation characteristics of the composite perception index along the response sequence. The filtering module is used to filter the dataset to be filtered based on the comprehensive quality score to obtain a high-quality data subset.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the fine-grained data selection method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the fine-grained data selection method as described in any one of claims 1 to 7.

Images