Instruction data screening method and device, equipment, storage medium and product

Abstract

The invention discloses an instruction data screening method and device, equipment, a storage medium and a product, and relates to the technical field of data screening, and the instruction data screening method comprises the following steps: obtaining instruction data to be screened, and carrying out multi-dimensional quality evaluation on the instruction data to obtain a quality evaluation result, the multi-dimensional quality evaluation comprises quality scoring, difficulty scoring and diversity evaluation; based on a preset sorting rule and the quality evaluation result, sorting the instruction data to obtain a sorted data set; and performing label driving sampling on the sorted data set to obtain the sampled instruction data. According to the method and the device, the data quality can be intuitively known, the sorted data set sorted according to the preset sorting rule and the quality evaluation result is subjected to label driving sampling, the sampled instruction data is obtained, and the data quality can be intuitively known through multi-dimensional quality evaluation.

Description

Technical Field

[0001] This application relates to the field of data filtering technology, and in particular to a method, apparatus, device, storage medium and product for filtering instruction data. Background Technology

[0002] With the development of Large Language Models (LLM), the number and richness of open-source instruction fine-tuning datasets available for large model fine-tuning are increasing. Currently, it is generally believed that in large model fine-tuning scenarios, data quality is far more important than data quantity. High-quality data can significantly improve model performance, reduce overfitting, and enhance generalization ability.

[0003] In related technologies, there is no unified standard and evaluation method for instruction data quality, which makes it impossible to intuitively understand the data quality when selecting and preparing data for training models. Summary of the Invention

[0004] The main purpose of this application is to provide a method, apparatus, device, storage medium and product for filtering instruction data, which aims to solve the technical problem that the lack of a unified standard and evaluation method for instruction data quality leads to an inability to intuitively understand data quality when selecting and preparing data for training models.

[0005] To achieve the above objectives, this application proposes an instruction data filtering method, which includes: Obtain instruction data to be filtered, and perform multi-dimensional quality assessment on the instruction data to obtain quality assessment results. The multi-dimensional quality assessment includes quality score, difficulty score and diversity evaluation. Based on the preset sorting rules and the quality assessment results, the instruction data is sorted to obtain a sorted dataset; Label-driven sampling is performed on the sorted dataset to obtain the sampled instruction data.

[0006] In one embodiment, the step of performing label-driven sampling on the sorted dataset to obtain the sampled instruction data includes: The intent labels in the samples of the sorted dataset are judged sequentially. If there is an intent label that does not appear in the filtering pool, the sample is stored in the filtering pool. When the number of samples in the filtering pool is greater than a preset number, the instruction data is constructed based on the samples in the filtering pool.

[0007] In one embodiment, the steps prior to obtaining the instruction data to be filtered and performing a multi-dimensional quality assessment on the instruction data to obtain the quality assessment result include: Obtain sample instruction data related to quality score and difficulty score, and the score results corresponding to the sample instruction data; The instruction data and the scoring results are respectively input into the initial quality scoring model and the initial difficulty scoring model for model training to obtain the target quality scoring model and the target difficulty scoring model that meet the accuracy requirements. The initial quality scoring model and the initial difficulty scoring model are based on the score prediction of the regression task head.

[0008] In one embodiment, the quality assessment result includes a quality score, a difficulty score, and a diversity evaluation. The step of performing a multi-dimensional quality assessment on the instruction data to obtain the quality assessment result includes: The instruction data is tagged using a preset tagging model to obtain the diversified evaluation corresponding to each instruction data, wherein the diversified evaluation includes a tag set; The complexity of the instruction data is quantified using a target difficulty scoring model to obtain the difficulty score; The quality score is obtained by quantifying the data quality of the instruction data using a target quality scoring model.

[0009] In one embodiment, the step of quantifying the complexity of the instruction data and obtaining the difficulty score using a target difficulty scoring model includes: The execution difficulty of the instruction data is quantified using a target difficulty scoring model to obtain a subjective execution difficulty score. The knowledge graph complexity score is obtained by calculating the number of entities involved in the instruction data and the weighted sum of the relationships between the entities. The minimum number of subtasks required to complete the instructions in the instruction data is quantified to obtain the task decomposition machine complexity score. The logical reasoning depth of the quality data is quantified to obtain a reasoning chain length score; The difficulty score is determined based on the subjective execution difficulty score, the knowledge graph complexity score, the task decomposition machine complexity score, and the above.

[0010] In one embodiment, the step of quantifying the data quality of the instruction data using a target quality scoring model to obtain a quality score includes: The accuracy of the instruction data is quantified using a target quality scoring model to obtain an accuracy score. The completeness score is obtained by quantifying the matching degree between the answers and questions in the instruction data. The educational value of the answers and questions is quantified to obtain an educational value score; The security of the answers and questions is quantified to obtain a security score; The quality score is determined based on the accuracy score, the completeness score, the educational value score, and the safety score.

[0011] Furthermore, to achieve the above objectives, this application also proposes an instruction data filtering device, which includes: An evaluation module is used to acquire instruction data to be screened and to perform multi-dimensional quality evaluation on the instruction data to obtain quality evaluation results. The multi-dimensional quality evaluation includes quality score, difficulty score and diversity evaluation. The sorting module is used to sort the instruction data based on preset sorting rules and the quality assessment results to obtain a sorted dataset. The sampling module is used to perform label-driven sampling on the sorted dataset to obtain the sampled instruction data.

[0012] In addition, to achieve the above objectives, this application also proposes an instruction data filtering device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the instruction data filtering method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the instruction data filtering method described above.

[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the instruction data filtering method described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: In contrast to related technologies, a unified standard and evaluation method for instruction data quality has not yet been established, making it difficult to intuitively understand data quality when selecting and preparing data for model training. This application acquires instruction data to be screened and performs a multi-dimensional quality assessment on the instruction data to obtain quality assessment results. The multi-dimensional quality assessment includes quality score, difficulty score, and diversity evaluation. Based on a preset sorting rule and the quality assessment results, the instruction data is sorted to obtain a sorted dataset. Label-driven sampling is then performed on the sorted dataset to obtain sampled instruction data. This application performs a multi-dimensional quality assessment on the instruction data, including quality score, difficulty score, and diversity evaluation, allowing for a direct understanding of data quality. Furthermore, label-driven sampling is performed on the sorted dataset according to the preset sorting rule and quality assessment results to obtain sampled instruction data. This multi-dimensional quality assessment provides a direct understanding of data quality, thereby enabling the screening of instruction data. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the data filtering method for the instructions in this application. Figure 2 This is a flowchart illustrating the data filtering method for the instructions in this application. Figure 3 This is a flowchart of the instruction data filtering method of this application; Figure 4 This is a flowchart illustrating Embodiment 2 of the data filtering method for the instructions in this application. Figure 5 This is a model design diagram of the data filtering method for the instructions in this application; Figure 6 This is a schematic diagram of the module structure of the instruction data filtering device according to an embodiment of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the instruction data filtering method in the embodiments of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of this application embodiment is: to obtain instruction data to be screened, and to perform multi-dimensional quality assessment on the instruction data to obtain quality assessment results, wherein the multi-dimensional quality assessment includes quality score, difficulty score and diversity evaluation; to sort the instruction data based on preset sorting rules and the quality assessment results to obtain sorted dataset; and to perform label-driven sampling on the sorted dataset to obtain sampled instruction data.

[0023] In related technologies, there is no unified standard and evaluation method for instruction data quality, which makes it impossible to intuitively understand the data quality when selecting and preparing data for training models.

[0024] This application performs a multi-dimensional quality assessment on the instruction data, including quality score, difficulty score, and diversity evaluation, which provides an intuitive understanding of the data quality. It also performs label-driven sampling on the sorted dataset according to preset sorting rules and quality assessment results to obtain the sampled instruction data. Through multi-dimensional quality assessment, the data quality can be intuitively understood, thereby completing the filtering of instruction data.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or instruction data filtering device capable of performing the above functions. The following description uses an instruction data filtering device as an example to illustrate this embodiment and the subsequent embodiments.

[0026] Based on this, embodiments of this application provide a method for filtering instruction data, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the instruction data filtering method of this application.

[0027] In this embodiment, the instruction data filtering method includes steps S10 to S30: Step S10: Obtain instruction data to be filtered, and perform multi-dimensional quality assessment on the instruction data to obtain quality assessment results. The multi-dimensional quality assessment includes quality score, difficulty score and diversity evaluation. It should be noted that the execution entity in this embodiment is the instruction data filtering device. Instruction data refers to the original instruction data that needs to be evaluated and filtered in the large language model fine-tuning scenario. The instruction data filtering device acquires the instruction data to be filtered and performs multi-dimensional quality evaluation, specifically including three core indicators: quality score, difficulty score, and diversity evaluation. The quantitative output obtained after the above multi-dimensional evaluation is the quality evaluation result, which realizes a comprehensive quantification of the quality, difficulty, and diversity of instruction data. This provides a standardized basis for the subsequent efficient and accurate screening of high-quality data, thereby improving the effect and efficiency of large model fine-tuning and reducing resource consumption.

[0028] Step S20: Based on the preset sorting rules and the quality assessment results, sort the instruction data to obtain a sorted dataset; Understandably, the preset sorting rules refer to the pre-defined criteria used to determine the order of instruction data. The instruction data filtering device sorts all instruction data according to these pre-defined criteria, resulting in a rearranged set of instruction data. By systematically sorting the quality assessment results based on the preset sorting rules, it achieves efficient organization and prioritization of large-scale instruction data. This allows high-quality, high-difficulty, or highly diverse data to be prioritized for filtering, providing an ordered data foundation for subsequent label-driven sampling and significantly improving the targeting and efficiency of data filtering.

[0029] For example, after obtaining the quality assessment results, the instruction data filtering device sorts all instruction data according to a preset "comprehensive score descending order" rule (i.e., quality score × difficulty score).

[0030] Specifically, the preset sorting rules can have multiple dimensions, and you can flexibly choose one or a combination of the above dimensions as the filtering criteria according to the needs of different training tasks: Complexity dimension (based on the number of InsTag tags); Difficulty dimension (based on difficulty score results); Quality dimensions (based on quality score results); Overall dimension (based on the product of quality score and difficulty score: quality score x difficulty score).

[0031] Step S30: Perform label-driven sampling on the sorted dataset to obtain the sampled instruction data.

[0032] It should be noted that the sampled data refers to the subset of instruction data that has been finally selected through a label-driven sampling process, meeting the preset requirements in quantity and possessing high diversity in topic coverage. This represents the final training dataset that achieves a balance between quality, difficulty, and diversity. The instruction data selection device, through label-driven sampling, prioritizes samples that enrich the overall label diversity based on the sorted data, ensuring the final selected instruction data is as follows: Figure 2 , Figure 2 A data filtering flowchart is provided.

[0033] In one feasible implementation, step S30 includes: The intent labels in the samples of the sorted dataset are judged sequentially. If there is an intent label that does not appear in the filtering pool, the sample is stored in the filtering pool. Understandably, intent tags refer to topic classification tags obtained after automatically labeling instruction data using the InsTag tagging model, used to represent the semantic content and task type of the data. The filtering pool is a temporarily constructed dataset dynamically built during the diversity filtering process, used to store currently selected instruction data samples that possess diversity and representativeness. The instruction data filtering device processes each data sample one by one according to a predetermined order in the sorted dataset (usually sorted in descending order of comprehensive score), sequentially determining whether the intent tags can increase the tag diversity of the filtering pool; that is, whether there are samples whose intent tags are not present in the filtering pool. If so, samples that can increase the tag diversity of the filtering pool are stored in the filtering pool.

[0034] When the number of samples in the filtering pool is greater than a preset number, the instruction data is constructed based on the samples in the filtering pool.

[0035] It should be noted that the preset number refers to a pre-set threshold parameter used to control the size of the final output dataset. Its specific value depends on the needs of the actual training task and the availability of data resources. When the number of instruction data in the filtering pool exceeds the preset number, the instruction data filtering device officially determines the samples accumulated in the filtering pool as the final output training dataset for fine-tuning the large model.

[0036] Specifically, refer to Figure 3 , Figure 3A flowchart for instruction data filtering is provided. The instruction data filtering device sequentially extracts samples from the sorted dataset arranged in descending order of comprehensive score. For each sample, the device reads its InsTag intent label (such as "medical consultation", "programming guidance", etc.) and checks whether the label already exists in the label set of the filtering pool. If the label is a new label (such as the current sample label is "legal consultation" and there is no such label in the filtering pool), the sample is stored in the filtering pool, and the label set of the filtering pool is updated. When the number of samples in the filtering pool is detected to have reached the threshold, the sampling process is immediately terminated, and all samples in the current filtering pool are formally constructed into the final output instruction dataset. This dataset ensures both data quality and difficulty, and ensures the diversity of topics through the label-driven mechanism, which can be directly used for subsequent large model instruction fine-tuning training.

[0037] In this implementation, using tag information generated by the open-source InsTag model, a multi-dimensional screening method is proposed, employing complexity, difficulty score, quality score, and comprehensive score (quality score * difficulty score) as screening indicators. This method involves "batch labeling—multi-dimensional sorting—tag-driven sampling" for fine-tuning data screening. This approach not only ensures the balance between quality and difficulty in the selected data subset but also improves sample coverage and diversity through a tag-driven approach, effectively avoiding the problems of data homogenization and bias in traditional screening methods.

[0038] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Before step S10, the instruction data filtering method further includes steps S01~S02: Step S01: Obtain sample instruction data related to quality score and difficulty score, and the score results corresponding to the sample instruction data; It is understood that the sample instruction data refers to training samples specifically used to train the quality scoring model and the difficulty scoring model. The scoring results are labeled data generated based on preset scoring criteria, containing specific scores for each instruction data point across various dimensions of quality scoring (accuracy, completeness, educational value, and security) and difficulty scoring (subjective execution difficulty, knowledge graph complexity, task decomposition complexity, and inference chain length), ranging from 0 to 5 points. The instruction data filtering device, by acquiring specially labeled training data and its scoring results, provides high-quality supervision signals for the subsequent training of lightweight quality scoring and difficulty scoring models. This enables the trained evaluation models to accurately understand and quantify the quality characteristics and difficulty levels of the instruction data, laying the foundation for rapid evaluation of large-scale datasets.

[0039] Step S02: Input the sample instruction data and the scoring results into the initial quality scoring model and the initial difficulty scoring model respectively for model training to obtain the target quality scoring model and the target difficulty scoring model that meet the accuracy requirements. The initial quality scoring model and the initial difficulty scoring model are based on the regression task head to predict the score.

[0040] It should be noted that the initial quality scoring model and the initial difficulty scoring model refer to the models to be trained, built upon pre-trained language models (Qwen2.5-1.5B-Base and Qwen2.5-0.5B-Base, respectively). These models add a regression task head to the original model structure. The regression task head is a linear layer added after the last hidden state layer of the pre-trained language model. It maps the high-dimensional representation of the model to continuous scoring values, thus constructing the scoring task as a regression prediction problem. The instruction data filtering device inputs sample instruction data and scoring results into a model architecture based on a regression task head for specialized training, resulting in a final model that has completed training and passed validation. This model can automatically output corresponding quality and difficulty scores for the input instruction data. The mean absolute error (MAE) between the predicted scores and the actual scores on the test set of the trained model reaches a preset threshold, such as MAE ≤ 0.80 for the quality scoring model and MAE ≤ 0.62 for the difficulty scoring model. By adopting a model architecture based on a regression task head for specialized training, the complex subjective scoring task is transformed into a stable numerical prediction problem. This not only significantly improves scoring efficiency but also ensures the accuracy and consistency of the scoring results, overcoming the instability and high computational cost problems that exist when directly using large language models for scoring.

[0041] Specifically, refer to Figure 5 , Figure 5 The invention provides a model design diagram. Considering the model's language understanding capabilities and inference efficiency, this invention selects the Qwen2.5 series base model for fine-tuning. Specifically, the quality scoring model uses Qwen2.5-1.5B-Base, and the difficulty scoring model uses Qwen2.5-0.5B-Base. Given the instability of the generated model's output and the correlation between inference time and output token length, while the scoring task only needs to output a fixed value, this application adds a linear layer as a task head after the last hidden state layer of the transformer for direct score prediction. This task head can be configured as either classification or regression. In practice, this invention tested transforming the scoring task into a multi-class or regression task, and the regression model demonstrated superior performance. In this way, the model can efficiently complete quality and difficulty scoring tasks without large-scale modifications to its internal structure, reducing token consumption while ensuring the stability and accuracy of the prediction results.

[0042] In one feasible implementation, the step of performing a multi-dimensional quality assessment on the instruction data to obtain the quality assessment result includes: The instruction data is tagged using a preset tagging model to obtain the diversified evaluation corresponding to each instruction data, wherein the diversified evaluation includes a tag set; Understandably, the preset tagging model refers to the pre-selected and loaded InsTag tagging model, which has 1.8B parameters and is specifically designed for multi-topic intent recognition and tag generation of text data. The instruction data filtering device uses the tagging model to automatically analyze the input instruction data, identify its semantic content, and output corresponding topic tags.

[0043] For example, the instruction data filtering device loads the open-source InsTag-1.8B tagging model, as described in Part 5 "Diversity Filtering" of the disclosure document, to automatically process each input instruction data. The model outputs a set of tags for each data sample, such as {"cooking skills", "ingredient processing"} or {"programming problem", "Python", "debugging"}, etc. These tag sets fully represent the diversity characteristics of the data and, together with the quality score and difficulty score, constitute a complete quality assessment result, providing data support for subsequent diversity filtering.

[0044] The complexity of the instruction data is quantified using a target difficulty scoring model to obtain the difficulty score; It should be noted that the target difficulty scoring model refers to a difficulty assessment model that has been trained and validated, based on the Qwen2.5-0.5B-Base architecture, and uses a regression task head for prediction. The instruction data filtering device uses numerical results in the range of 0-5 points output by the specially trained target difficulty scoring model to characterize the overall complexity of the instruction data.

[0045] The quality score is obtained by quantifying the data quality of the instruction data using a target quality scoring model.

[0046] It is understandable that the target quality scoring model refers to a quality assessment model that has been trained and validated, based on the Qwen2.5-1.5B-Base architecture, and uses a regression task head for prediction. The instruction data filtering device systematically quantifies the data quality of the instruction data through a specially trained target quality scoring model, and outputs a numerical result in the range of 0-5 points to characterize the overall quality level of the instruction data.

[0047] In one feasible implementation, the step of quantifying the complexity of the instruction data and obtaining the difficulty score using a target difficulty scoring model includes: The execution difficulty of the instruction data is quantified using a target difficulty scoring model to obtain a subjective execution difficulty score. It should be noted that the subjective execution difficulty score refers to the specific evaluation result of the target difficulty scoring model on the subjective execution difficulty dimension of the instruction data, and the score range is 0-5. The instruction data screening device obtains the subjective execution difficulty score by quantifying the complexity of understanding the instruction, the cognitive load required for execution, and the difficulty of handling potential ambiguities through the target difficulty scoring model.

[0048] The knowledge graph complexity score is obtained by calculating the number of entities involved in the instruction data and the weighted sum of the relationships between the entities. Understandably, the number of entities refers to the specific number of core concepts or objects with independent semantic meaning appearing in the instruction data text, such as names of people, places, technical terms, and events. The relationships between entities refer to the semantic associations or logical connections that exist between these entities in the instruction context, including but not limited to causal relationships, inclusion relationships, temporal relationships, and attribute relationships. The instruction data filtering device measures the knowledge density of the task by calculating the weighted sum of the number of entities involved in the instruction and their relationships, thereby obtaining a knowledge graph complexity score. This indicator not only considers the number of independent knowledge points but also emphasizes the relationships between entities, thus assessing the degree to which the instruction requires knowledge integration capabilities.

[0049] The minimum number of subtasks required to complete the instructions in the instruction data is quantified to obtain the task decomposition machine complexity score. It should be noted that the minimum number of subtasks refers to the number of the most basic, indivisible operational steps that must be executed to complete the original instruction. These subtasks are logically independent and arranged in a specific order. Since a higher task decomposition complexity indicates that the instruction requires the coordinated completion of multiple steps or multiple sub-objectives, the instruction data filtering device obtains a task decomposition machine complexity score by measuring the minimum number of subtasks required to complete the instruction.

[0050] The logical reasoning depth of the quality data is quantified to obtain a reasoning chain length score; Understandably, the instruction data filtering device evaluates the logical reasoning depth of the instruction by calculating the shortest number of reasoning steps required to reach the target conclusion from the known conditions, and obtains a reasoning chain length score.

[0051] The difficulty score is determined based on the subjective execution difficulty score, the knowledge graph complexity score, the task decomposition machine complexity score, and the above.

[0052] It should be noted that the difficulty score is used to measure the complexity of the instruction task, ensuring a balance in the hierarchy and challenge of the training data, thereby improving the model's inference and generalization abilities. The instruction data filtering device generates a final difficulty score within the range of 0-5 by comprehensively considering the scoring results of four key dimensions.

[0053] In one feasible implementation, the step of quantifying the data quality of the instruction data and obtaining a quality score through a target quality scoring model includes: The accuracy of the instruction data is quantified using a target quality scoring model to obtain an accuracy score. Understandably, the instruction data filtering device quantifies the clarity of questions and answers in the instruction data through a target quality scoring model to obtain an accuracy score, ensuring that the content does not contain factual errors, logical contradictions, ambiguities, or semantic vagueness, while the answers must accurately respond to the question instructions and content requirements.

[0054] The completeness score is obtained by quantifying the matching degree between the answers and questions in the instruction data. It should be noted that the instruction data filtering device focuses on examining whether the answer fully covers all the task points required in the question instruction through the target quality scoring model. For multi-step or complex tasks, all necessary operation results must be fully presented in order to obtain a completeness score.

[0055] The educational value of the answers and questions is quantified to obtain an educational value score; Understandably, the instruction data filtering device uses a target quality scoring model to evaluate whether the content of questions and answers is inspiring and educational, thereby obtaining an educational value score.

[0056] The security of the answers and questions is quantified to obtain a security score; It should be noted that the instruction data screening device uses a target quality scoring model to assess whether the content of the questions and answers is unsafe, harmful, or unethical, thereby obtaining a safety score.

[0057] The quality score is determined based on the accuracy score, the completeness score, the educational value score, and the safety score.

[0058] Understandably, the instruction data filtering device can ensure that the data entering the training set has high reliability and effectiveness through the comprehensive score of these four dimensions, i.e., the quality score.

[0059] This implementation proposes a method that fine-tunes a pre-trained language model by adding only a classification head, transforming the complex quality and difficulty assessment task into a classification or regression prediction problem. This method significantly reduces reliance on computational power while maintaining assessment accuracy, overcoming the high cost and low efficiency bottlenecks of existing methods that rely on direct scoring with large language models. It is suitable for rapid assessment and filtering of large datasets with millions of records.

[0060] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data filtering method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0061] This application also provides an instruction data filtering device, please refer to... Figure 6 The instruction data filtering device includes: Evaluation module 10 is used to acquire instruction data to be screened and to perform multi-dimensional quality evaluation on the instruction data to obtain quality evaluation results. The multi-dimensional quality evaluation includes quality score, difficulty score and diversity evaluation. The sorting module 20 is used to sort the instruction data based on preset sorting rules and the quality assessment results to obtain a sorted dataset; The sampling module 30 is used to perform label-driven sampling on the sorted dataset to obtain the sampled instruction data.

[0062] Optionally, the sampling module includes: A construction submodule is used to sequentially judge the intent labels in the samples of the sorted dataset. If there is an intent label that does not appear in the filtering pool, the sample is stored in the filtering pool. When the number of samples in the filtering pool is greater than a preset number, the instruction data is constructed based on the samples in the filtering pool.

[0063] Optionally, the evaluation module includes: The training submodule is used to acquire sample instruction data related to quality scores and difficulty scores, and the corresponding score results of the sample instruction data; the instruction data and the score results are respectively input into the initial quality score model and the initial difficulty score model for model training, to obtain the target quality score model and the target difficulty score model that meet the accuracy requirements, wherein the initial quality score model and the initial difficulty score model are based on the score prediction of the regression task head.

[0064] The quantization submodule is used to label the instruction data using a preset labeling model to obtain the diversified evaluation corresponding to each instruction data, wherein the diversified evaluation includes a set of labels; to quantify the complexity of the instruction data using a target difficulty scoring model to obtain the difficulty score; and to quantify the data quality of the instruction data using a target quality scoring model to obtain the quality score.

[0065] Optionally, the quantization submodule includes: The first quantification unit is used to quantify the execution difficulty of the instruction data using a target difficulty scoring model to obtain a subjective execution difficulty score; calculate the number of entities involved in the instruction data and the weighted sum of the relationships between the entities to obtain a knowledge graph complexity score; quantify the minimum number of subtasks required to complete the instructions in the instruction data to obtain a task decomposition machine complexity score; quantify the logical reasoning depth of the quality data to obtain a reasoning chain length score; and determine the difficulty score based on the subjective execution difficulty score, the knowledge graph complexity score, the task decomposition machine complexity score, and the above.

[0066] The second quantification unit is used to quantify the accuracy of the instruction data using a target quality scoring model to obtain an accuracy score; quantify the matching degree between the answers and questions in the instruction data to obtain a completeness score; quantify the educational value of the content of the answers and questions to obtain an educational value score; quantify the security of the content of the answers and questions to obtain a security score; and determine the quality score based on the accuracy score, the completeness score, the educational value score, and the security score.

[0067] The instruction data filtering device provided in this application, employing the instruction data filtering method in the above embodiments, can solve the technical problem of instruction data filtering. Compared with the prior art, the beneficial effects of the instruction data filtering device provided in this application are the same as those of the instruction data filtering method provided in the above embodiments, and other technical features in the instruction data filtering device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0068] This application provides an instruction data filtering device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the instruction data filtering method in the first embodiment described above.

[0069] The following is for reference. Figure 7The diagram illustrates a structural schematic of an instruction data filtering device suitable for implementing embodiments of this application. The instruction data filtering device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, tablets, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The instruction data filtering device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0070] like Figure 7 As shown, the instruction data filtering device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the instruction data filtering device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the instruction data filtering device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows instruction data filtering devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0071] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0072] The instruction data filtering device provided in this application, employing the instruction data filtering method in the above embodiments, can solve the technical problem of instruction data filtering. Compared with the prior art, the beneficial effects of the instruction data filtering device provided in this application are the same as those of the instruction data filtering method provided in the above embodiments, and other technical features in this instruction data filtering device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0073] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0074] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0075] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the instruction data filtering method in the above embodiments.

[0076] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0077] The aforementioned computer-readable storage medium may be included in the instruction data filtering device; or it may exist independently and not be assembled into the instruction data filtering device.

[0078] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by the instruction data filtering device, the instruction data filtering device: acquires instruction data to be filtered, performs a multi-dimensional quality assessment on the instruction data, and obtains a quality assessment result, wherein the multi-dimensional quality assessment includes a quality score, a difficulty score, and a diversity evaluation; sorts the instruction data based on a preset sorting rule and the quality assessment result, and obtains a sorted dataset; and performs label-driven sampling on the sorted dataset to obtain the sampled instruction data.

[0079] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0081] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0082] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described instruction data filtering method, thereby solving the technical problem of instruction data filtering. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the instruction data filtering method provided in the above embodiments, and will not be repeated here.

[0083] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the instruction data filtering method described above.

[0084] The computer program product provided in this application can solve the technical problem of instruction data filtering. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the instruction data filtering method provided in the above embodiments, and will not be repeated here.

[0085] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A method of filtering instruction data, the method comprising: The instruction data screening method comprises: acquiring instruction data to be screened, and performing multi-dimensional quality evaluation on the instruction data to obtain a quality evaluation result, wherein the multi-dimensional quality evaluation comprises quality score, difficulty score and diversity evaluation; based on a preset sorting rule and the quality evaluation result, sorting the instruction data to obtain a sorted data set; label-driven sampling is performed on the sorted data set to obtain sampled instruction data.

2. The method of claim 1, wherein, The step of performing label-driven sampling on the sorted data set to obtain sampled instruction data comprises: judging the intent label in the sample of the sorted data set in turn, and if the intent label does not appear in the screening pool, storing the sample in the screening pool; when the number of samples in the screening pool is greater than a preset number, constructing the instruction data based on the samples in the screening pool.

3. The method of claim 1, wherein the instruction data filtering method is characterized by, The step of acquiring instruction data to be screened and performing multi-dimensional quality evaluation on the instruction data to obtain a quality evaluation result comprises: acquiring sample instruction data related to quality score and difficulty score and score results corresponding to the sample instruction data; inputting the instruction data and the score results into an initial quality score model and an initial difficulty score model respectively for model training to obtain target quality score model and target difficulty score model meeting accuracy requirements, wherein the initial quality score model and the initial difficulty score model predict scores based on a regression task head.

4. The method of claim 1, wherein the instruction data filtering method is characterized by, The quality evaluation result comprises quality score, difficulty score and diversity evaluation, and the step of performing multi-dimensional quality evaluation on the instruction data to obtain a quality evaluation result comprises: labeling the instruction data by a preset labeling model to obtain a diversity evaluation corresponding to each instruction data, wherein the diversity evaluation comprises a label set; quantifying the complexity of the instruction data by a target difficulty score model to obtain the difficulty score; quantifying the data quality of the instruction data by a target quality score model to obtain the quality score.

5. The method of claim 4, wherein the instruction data screening method is characterized by, The step of quantifying the complexity of the instruction data by a target difficulty score model to obtain the difficulty score comprises: quantifying the execution difficulty of the instruction data by a target difficulty score model to obtain a subjective execution difficulty score; calculating the weighted sum of the number of entities involved in the instruction data and the relationship between the entities to obtain a knowledge graph complexity score; quantifying the minimum number of sub-tasks required to complete the instruction in the instruction data to obtain a task decomposition machine complexity score; quantifying the logical reasoning depth of the quality data to obtain an inference chain length score; determining the difficulty score based on the subjective execution difficulty score, the knowledge graph complexity score, the task decomposition machine complexity score and the inference chain length score.

6. The method of claim 4, wherein the instruction data filtering method is characterized by, The step of quantifying the data quality of the instruction data by a target quality score model to obtain the quality score comprises: quantifying the accuracy of the instruction data by a target quality score model to obtain an accuracy score; quantifying the matching degree of the answer and the question in the instruction data to obtain a completeness score; quantify an educational value of the content of the answer and the question to obtain an educational value score; quantify a security of the content of the answer and the question to obtain a security score; determine the quality score based on the accuracy score, the completeness score, the educational value score, and the security score.

7. An instruction data screening device, characterized by The apparatus comprises: an evaluation module configured to obtain instruction data to be screened and perform multi-dimensional quality evaluation on the instruction data to obtain a quality evaluation result, wherein the multi-dimensional quality evaluation comprises a quality score, a difficulty score, and a diversity evaluation; an ordering module configured to order the instruction data based on a preset ordering rule and the quality evaluation result to obtain an ordered data set; a sampling module configured to perform label-driven sampling on the ordered data set to obtain the sampled instruction data.

8. An instruction data screening device, characterized by The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the instruction data screening method according to any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the steps of the instruction data screening method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterised in that, The computer program product comprises a computer program. When the computer program is executed by a processor, the steps of the instruction data screening method according to any one of claims 1 to 6 are implemented.

Images