A method and apparatus for filtering data
By scoring and clustering the training data, high-quality training data is selected for instruction fine-tuning, which solves the problem of poor performance of large language models on specific tasks and achieves more efficient model capabilities and resource conservation.
Patent Information
- Application Number
- CN202510634101.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, when fine-tuning large language models, the lack of high-quality training data results in the model's performance being less accurate and effective in specific tasks or domains.
Multiple training data are scored using scoring parameters to select high-quality training data, including dimensions such as correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security. These data are then clustered and evolved to generate a small but diverse amount of training data for fine-tuning model instructions.
It improves the model's ability to work in different languages, reduces the difficulty of migrating to unfamiliar language platforms, saves computing resources, and improves the model's performance on specific tasks.
Smart Images

Figure CN123115784A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence, and more specifically, to a method and apparatus for filtering data. Background Technology
[0002] With the advancement of deep learning technology and the rapid development of language models, artificial intelligence has achieved revolutionary breakthroughs in fields such as natural language processing and reasoning tasks.
[0003] To improve the capabilities of large language models for specific tasks or domains, instruction fine-tuning can be performed on instruction pairs to make them more accurate and efficient in those tasks or domains. The quality of the instruction pairs has a significant impact on the effectiveness of instruction fine-tuning. Summary of the Invention
[0004] This application provides a method and apparatus for filtering data in order to obtain high-quality training data for fine-tuning instructions in a language model.
[0005] Firstly, a method for filtering data is provided, comprising: scoring each of a plurality of first training data according to scoring parameters to obtain a scoring result for each of the first training data, wherein the scoring parameters include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security; each of the first training data includes instruction pairs in different languages; determining n1 first training data from the plurality of first training data according to the scoring result, where n1 is a positive integer; clustering the plurality of first training data according to the number of the plurality of first training data to obtain k1 classes, each class including at least one of the first training data, where k1 is an integer greater than 1; determining n2 first training data from each class according to the scoring result, where n2 is a positive integer; and determining second training data based on the n1 first training data and k1 groups of the n2 first training data, wherein the second training data is used for instruction fine-tuning of the model.
[0006] In this application, the scoring parameters are the dimensions / angles / scoring criteria used to evaluate the first training data.
[0007] In the embodiments of this application, the scoring parameters for different first training data can be different.
[0008] For example, the scoring parameters used to evaluate the first training data #1 are consistency and satisfaction, while the scoring parameters used to evaluate the first training data #2 are information content or security.
[0009] In this application, the scoring results reflect the quality of the first training data.
[0010] For example, if the score of the first training data #1 is higher than the score of the first training data #2, then the quality of the first training data #1 is considered to be higher than that of the first training data #2.
[0011] In this application, the first training data includes instruction pairs in different languages, which enables the language model to learn the features of multiple languages during instruction fine-tuning. This helps the language model being fine-tuned to better understand instruction pairs, improves the model's capabilities in different languages, and reduces the difficulty of transferring the language model to a language platform it has not learned before.
[0012] Compared to directly discarding data with low scores in the overall dataset, selecting n² first training data from each class can improve the diversity of the selected data, or in other words, improve the quality of the selected data.
[0013] Compared to fine-tuning the language model using a large amount of raw data instructions, fine-tuning the language model using selected data instructions can save computational resources.
[0014] Based on the method provided in the embodiments of this application, by filtering multiple first training data to obtain n1+k1*n2 first training data, a small amount of high-quality, multilingual second training data for instruction fine-tuning of the language model can be obtained. The second training data helps to save computational resources for instruction fine-tuning of the language model, improve the model capability of the instruction-fine-tuned language model in different languages, and reduce the difficulty of migrating the instruction-fine-tuned language model to a language platform that has not been learned.
[0015] In conjunction with the first aspect, in some possible implementations of the first aspect, the method further includes: determining the scoring parameter based on the task to be processed after fine-tuning the model instructions.
[0016] Specifically, the scoring parameters can be determined based on the task that the fine-tuned model is used to process according to the instructions. Alternatively, the scoring parameters can be determined based on the characteristics of the task that the fine-tuned model is used to process according to the instructions.
[0017] For example, emotional question answering needs to pay more attention to the emotional tone of the answers, while legal consultation needs to pay more attention to the accuracy and reliability of the answers. Therefore, the model, fine-tuned according to instructions, can be used for emotional question answering to determine if the scoring parameters are human-centered; or, the model, fine-tuned according to instructions, can be used for legal consultation to determine if the scoring parameters are correct.
[0018] If the scoring parameters are the same or fixed regardless of what task the fine-tuned model is used to process, the scoring results of the first training data determined based on the scoring parameters may not be reasonable.
[0019] Based on the method provided in the embodiments of this application, by fine-tuning the task to be processed according to the model instructions and determining the scoring parameters, training data that is more suitable for the needs of different tasks can be obtained. Compared with fixed scoring parameters, the selected training data can be more reasonable.
[0020] In conjunction with the first aspect, in some possible implementations of the first aspect, determining the second training data based on the n1 first training data and k1 sets of the n2 first training data includes: performing evolutionary processing on the n1 first training data and k1 sets of the n2 first training data to obtain the second training data; the evolutionary processing includes at least one of the following: adding constraints; deepening understanding; concretizing; adding reasoning steps; complicating the input data; or generating new data based on the input data.
[0021] Based on the method provided in the embodiments of this application, by performing evolutionary processing on the n1 first training data and k1 sets of the n2 first training data to obtain the second training data, the complexity of the training data can be enhanced, resulting in more complex second training data and improving the quality of the filtered data.
[0022] In conjunction with the first aspect, in some possible implementations of the first aspect, determining the second training data based on the n1 first training data and k1 sets of the n2 first training data further includes: performing the evolutionary processing based on the cultural specificity of the language of the instruction pair.
[0023] For example, during evolutionary processing, it is necessary to take into account the cultural taboos or characteristics of the users of different instruction pairs. Evolutionary processing that does not consider the cultural taboos or characteristics of the language users may generate instruction pairs that violate cultural taboos or characteristics.
[0024] Based on the method provided in the embodiments of this application, by performing this evolutionary processing according to the cultural specificity of the language of the instruction pair, the training data obtained through evolution can be made more reasonable.
[0025] In conjunction with the first aspect, in some possible implementations of the first aspect, determining n1 first training data from the plurality of first training data based on the scoring result includes: determining the n1 first training data with the highest scoring result from the plurality of first training data based on the scoring result.
[0026] For example, determining the n1 first training data based on the scoring results can be done by arranging the first training data corresponding to the scoring results from high to low, and selecting the first n1 first training data.
[0027] Based on the method provided in the embodiments of this application, by determining the n1 first training data with the highest scores from the plurality of first training data according to the scoring results, the quality of the selected first training data can be guaranteed.
[0028] In conjunction with the first aspect, in some possible implementations of the first aspect, determining n2 first training data from each class based on the scoring result includes: determining the n2 first training data with the highest scoring result from each class based on the scoring result.
[0029] For example, determining the n2 first training data based on the scoring results can be done by arranging the first training data corresponding to the scoring results in each class from high to low, and selecting the first n2 first training data.
[0030] One of the n2 first training data points in a certain class may not belong to the first n1 data points in multiple first training data points.
[0031] Based on the method provided in the embodiments of this application, by determining the top n2 first training data from each class, it is possible to avoid determining n1 first training data based solely on the scoring results, which would result in the rejection of some first training data that are of good quality in the class but perform poorly in the dataset, thereby improving the diversity of the filtered data.
[0032] In conjunction with the first aspect, in some possible implementations of the first aspect, n1 > n2.
[0033] n1 > n2 helps ensure the quality of the first training data obtained through screening.
[0034] In conjunction with the first aspect, in some possible implementations of the first aspect, the method further includes: deleting duplicate data from the n1 first training data and the k1 sets of the n2 first training data.
[0035] For example, if there are n1+k1*n2 first training data, and the first training data #3 and the first training data #4 are completely identical, then the first training data #3 and the first training data #4 are considered to be duplicates, and either the first training data #3 or the first training data #4 can be deleted.
[0036] Based on the method provided in the embodiments of this application, redundant filtered data can be avoided by deleting duplicate data.
[0037] In conjunction with the first aspect, in some possible implementations of the first aspect, scoring each of the multiple first training data according to the scoring parameters to obtain a scoring result for each of the first training data includes: inputting the multiple first training data into a scoring model to obtain a scoring result for each of the first training data, wherein the number of model parameters of the scoring model is less than or equal to 1 billion.
[0038] Based on the method provided in the embodiments of this application, the scoring model used to score the first training data is a small model with fewer than or equal to 1 billion model parameters. This can help with the local deployment and industrial implementation of the data screening method, improve the stability of the data screening method application, and avoid the risks brought about by external application program interface (API) access.
[0039] Secondly, a method for fine-tuning instructions is provided. This method includes: fine-tuning a model based on second training data, which is determined based on n1 sets of first training data and k1 sets of n2 sets of first training data. The first training data includes instruction pairs in different languages, where n1 is a positive integer, n2 is a positive integer, and k1 is an integer greater than 1. The n1 sets of first training data are determined from multiple sets of first training data based on a scoring result. The k1 sets of n2 sets of first training data are determined from k1 classes based on the scoring result. The scoring result is obtained by scoring each set of first training data in the multiple sets of first training data according to scoring parameters, which include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security. The k1 classes are obtained by clustering the multiple sets of first training data based on their number, with each class including at least one set of first training data.
[0040] In conjunction with the second aspect, in some possible implementations of the second aspect, the scoring parameter is determined based on the task to be processed after fine-tuning the model instructions.
[0041] In conjunction with the second aspect, in some possible implementations of the second aspect, the second training data is obtained by performing an evolutionary process on the n1 sets of the first training data and the k1 sets of the n2 sets of the first training data; the evolutionary process includes at least one of the following: adding constraints; deepening understanding; concretizing; adding inference steps; complicating the input data; or generating new data based on the input data.
[0042] In conjunction with the second aspect, in some possible implementations of the second aspect, the second training data is obtained by performing this evolutionary processing based on the cultural specificity of the language of the instruction pair.
[0043] In conjunction with the second aspect, in some possible implementations of the second aspect, the n1 first training data are the n1 first training data with the highest score determined from the plurality of first training data based on the scoring result.
[0044] In conjunction with the second aspect, in some possible implementations of the second aspect, the n² first training data are the n² first training data with the highest scores determined from each class based on the scoring results.
[0045] In conjunction with the second aspect, in some possible implementations of the second aspect, the second data is obtained by removing duplicate data from the n1 first training data and k1 sets of the n2 first training data.
[0046] In conjunction with the second aspect, in some possible implementations of the second aspect, the scoring result of each of the first training data is obtained by inputting the multiple first training data into a scoring model, the number of model parameters of the scoring model being less than or equal to 1 billion.
[0047] The possible implementation methods and their technical effects of the second aspect above can be referred to the description of the first aspect and any of its implementation methods, and will not be repeated here.
[0048] Thirdly, an apparatus for filtering data is provided, the apparatus including a processing unit for performing the method in the first aspect or any possible implementation thereof.
[0049] The processing unit is configured to: score each of the multiple first training data according to scoring parameters to obtain a scoring result for each first training data, wherein the scoring parameters include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security; each first training data includes instruction pairs in different languages; the processing unit is further configured to: determine n1 first training data from the multiple first training data according to the scoring result, where n1 is a positive integer; the processing unit is further configured to: cluster the multiple first training data according to the number of data points to obtain k1 classes, each class including at least one first training data, where k1 is an integer greater than 1; the processing unit is further configured to: determine n2 first training data from each class according to the scoring result, where n2 is a positive integer; the processing unit is further configured to: determine second training data based on the n1 first training data and k1 sets of the n2 first training data, wherein the second training data is used for fine-tuning the model's instructions.
[0050] In conjunction with the third aspect, in some possible implementations of the third aspect, the processing unit is also used to: determine the scoring parameter based on the task to be processed after fine-tuning the model instructions.
[0051] In conjunction with the third aspect, in some possible implementations of the third aspect, the processing unit is specifically used to: perform evolutionary processing on the n1 first training data and k1 sets of the n2 first training data to obtain the second training data; the evolutionary processing includes at least one of the following: adding constraints; deepening understanding; concretizing; adding inference steps; complicating the input data; or generating new data based on the input data.
[0052] In conjunction with the third aspect, in some possible implementations of the third aspect, the processing unit is specifically used to: perform this evolutionary processing based on the cultural specificity of the language of the instruction pair.
[0053] In conjunction with the third aspect, in some possible implementations of the third aspect, the processing unit is specifically used to: determine the n1 first training data with the highest scores from the plurality of first training data based on the scoring results.
[0054] In conjunction with the third aspect, in some possible implementations of the third aspect, the processing unit is specifically used to: determine the n2 first training data with the highest scores from each class based on the scoring results.
[0055] In conjunction with the third aspect, in some possible implementations of the third aspect, the processing unit is also used to: delete duplicate data in the n1 first training data and k1 sets of the n2 first training data.
[0056] In conjunction with the third aspect, in some possible implementations of the third aspect, the processing unit is specifically used to: input the multiple first training data into the scoring model to obtain a scoring result for each of the first training data, wherein the number of model parameters of the scoring model is less than or equal to 1 billion.
[0057] In some possible implementations, the processing unit is a processing module or a processor.
[0058] In some possible implementations, the device also includes an output unit for outputting the second training data.
[0059] In some possible implementations, the output unit is an output module or a transceiver.
[0060] The possible implementation methods and technical effects of the third aspect above can be referred to the description of the first aspect and its possible implementation methods above, and will not be repeated here.
[0061] Fourthly, an instruction fine-tuning apparatus is provided, the apparatus including a processing unit for executing the method in the second aspect or any possible implementation thereof.
[0062] The processing unit is used to: fine-tune the model based on second training data, which is determined based on n1 first training data and k1 groups of n2 first training data. The first training data includes instruction pairs in different languages, where n1 is a positive integer, n2 is a positive integer, and k1 is an integer greater than 1. The n1 first training data are determined from multiple first training data based on scoring results, and the k1 groups of n2 first training data are determined from k1 classes based on the scoring results. The scoring results are obtained by scoring each of the multiple first training data according to scoring parameters, which include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security. The k1 classes are obtained by clustering the multiple first training data based on the number of data points, and each class includes at least one first training data point.
[0063] In some possible implementations, the processing unit is a processing module or a processor.
[0064] In conjunction with the fourth aspect, in some possible implementations of the fourth aspect, the scoring parameter is determined based on the task to be processed after fine-tuning the model instructions.
[0065] In conjunction with the fourth aspect, in some possible implementations of the fourth aspect, the second training data is obtained by performing an evolutionary process on the n1 sets of the first training data and the k1 sets of the n2 sets of the first training data; the evolutionary process includes at least one of the following: adding constraints; deepening understanding; concretizing; adding inference steps; complicating the input data; or generating new data based on the input data.
[0066] In conjunction with the fourth aspect, in some possible implementations of the fourth aspect, the second training data is obtained by performing this evolutionary processing based on the cultural specificity of the language of the instruction pair.
[0067] In conjunction with the fourth aspect, in some possible implementations of the fourth aspect, the n1 first training data are the n1 first training data with the highest score determined from the plurality of first training data based on the scoring result.
[0068] In conjunction with the fourth aspect, in some possible implementations of the fourth aspect, the n² first training data are the n² first training data with the highest scores determined from each class based on the scoring results.
[0069] In conjunction with the fourth aspect, in some possible implementations of the fourth aspect, the second data is obtained by removing duplicate data from the n1 first training data and k1 sets of the n2 first training data.
[0070] In conjunction with the fourth aspect, in some possible implementations of the fourth aspect, the scoring result of each of the first training data is obtained by inputting the multiple first training data into a scoring model, the number of model parameters of the scoring model being less than or equal to 1 billion.
[0071] The possible implementation methods and technical effects of the fourth aspect above can be referred to the description of the second aspect and its possible implementation methods above, and will not be repeated here.
[0072] Fifthly, a computing device is provided, including a processor and a memory, and in some possible implementations, an input / output interface. The processor controls the input / output interface to send and receive information, the memory stores a computer program, and the processor retrieves and runs the computer program from the memory, causing the computing device to perform the method described in the first aspect or any possible implementation thereof, or causing the computing device to perform the method described in the second aspect or any possible implementation thereof.
[0073] In some possible implementations, the processor can be a general-purpose processor, which can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc.; when implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.
[0074] A sixth aspect provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, causing the computing device cluster to perform a method as described in the first aspect or any possible implementation thereof, or to perform a method as described in the second aspect or any possible implementation thereof.
[0075] In a seventh aspect, a chip is provided that acquires and executes instructions to implement the method as described in the first aspect and any implementation thereof, or that acquires and executes instructions to implement the method as described in the second aspect and any implementation thereof.
[0076] In some possible implementations, the chip includes a processor and a data interface, through which the processor reads instructions stored in memory and executes the methods described in the first aspect and any implementation thereof, or executes the methods described in the second aspect and any implementation thereof.
[0077] In some possible implementations, the chip may further include a memory storing instructions, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform a method as described in the first aspect and any implementation thereof, or the processor is configured to perform a method as described in the second aspect and any implementation thereof.
[0078] Eighthly, a computer program product comprising instructions is provided, which, when executed by a computing device / processor, causes the computing device / processor to perform the method as described in the first aspect and any implementation thereof, or causes the computing device / processor to perform the method as described in the second aspect and any implementation thereof.
[0079] Ninthly, a computer program product containing instructions is provided, which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in the first aspect and any implementation thereof, or cause the cluster of computing devices to perform the method as described in the second aspect and any implementation thereof.
[0080] A tenth aspect provides a computer-readable storage medium including computer program instructions that, when executed by a computing device / processor, perform a method as described in the first aspect and any implementation thereof, or perform a method as described in the second aspect and any implementation thereof.
[0081] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.
[0082] In some possible implementations, the aforementioned storage medium may specifically be a non-volatile storage medium.
[0083] Eleventhly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a cluster of computing devices, perform a method as described in the first aspect and any implementation thereof, or perform a method as described in the second aspect and any implementation thereof.
[0084] As examples, these computer-readable storages include, but are not limited to, one or more of the following: ROM, PROM, EPROM, Flash memory, EEPROM, and hard disk drive.
[0085] In some possible implementations, the aforementioned storage medium may specifically be a non-volatile storage medium. Attached Figure Description
[0086] Figure 1 This is a schematic diagram of the architecture of a data processing system applicable to embodiments of this application.
[0087] Figure 2 This is a schematic block diagram of a cloud scenario applicable to embodiments of this application.
[0088] Figure 3 This is a schematic diagram illustrating the interaction between a tenant and an artificial intelligence infrastructure development platform applicable to embodiments of this application.
[0089] Figure 4 This is a schematic diagram of a data filtering method provided in an embodiment of this application.
[0090] Figure 5 This is a schematic diagram of a data filtering method provided in an embodiment of this application.
[0091] Figure 6 This is a schematic diagram of a data filtering method provided in an embodiment of this application.
[0092] Figure 7 This is a schematic diagram of an instruction quality scoring model provided in an embodiment of this application.
[0093] Figure 8 This is a schematic diagram of a clustering method provided in an embodiment of this application.
[0094] Figure 9 This is a schematic diagram illustrating an evolution provided in an embodiment of this application.
[0095] Figure 10 This is a schematic diagram illustrating the reliability of an instruction quality enhancement result provided in an embodiment of this application.
[0096] Figure 11This is a schematic diagram illustrating the performance of a trained model provided in an embodiment of this application.
[0097] Figure 12 This is a schematic diagram illustrating the reliability of an instruction quality enhancement result provided in an embodiment of this application.
[0098] Figure 13 This is a schematic diagram illustrating the reliability of a clustering and ranking method provided in an embodiment of this application.
[0099] Figure 14 This is a schematic block diagram of an apparatus provided in an embodiment of this application.
[0100] Figure 15 This is a schematic diagram of the architecture of a computing device provided in an embodiment of this application.
[0101] Figure 16 This is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application. Detailed Implementation
[0102] The technical solutions of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort should fall within the scope of protection of this application.
[0103] Before introducing the embodiments of this application, the following points should be made first.
[0104] In this application, terms such as “exemplary,” “for example,” and “taking…as an example” are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as an “example” in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the term “example” is intended to present concepts in a concrete manner. In embodiments of this application, “of,” “corresponding, relevant,” and “corresponding” may sometimes be used interchangeably, and it should be noted that their intended meanings are consistent unless their distinction is emphasized.
[0105] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of neural networks and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0106] References such as "in some possible implementations" as used in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in some possible implementations" appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized.
[0107] In this application, "at least one" or "at least one item" refers to one or more items, and "more than one" refers to two or more items. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0108] In this application, terms such as "first," "second," "#1," and "#2" are used merely for descriptive convenience to distinguish objects and are not intended to limit the scope of the embodiments of this application, nor to describe the order or sequence of features. It should be understood that such described objects can be interchanged where appropriate to describe solutions other than those in the embodiments of this application.
[0109] In this application, the terms "S410" are merely identifiers for the convenience of description and do not limit the order of execution steps.
[0110] To facilitate understanding of the embodiments of this application, the terms involved in this application will be briefly explained first.
[0111] It should be understood that the related conceptual explanations may be limited by the specific circumstances of the embodiments of this application, but it does not mean that this application can only be limited to the specific circumstances. The specific circumstances of different embodiments may also differ, which are not limited here.
[0112] 1. Artificial intelligence (AI)
[0113] AI (Artificial Intelligence) is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.
[0114] The basic principle of AI is to combine massive amounts of data with powerful computing capabilities and intelligent algorithms to build an AI model that solves specific problems. This AI model can automatically summarize and learn potential patterns or features from the data, thereby achieving a way of thinking that is close to that of humans.
[0115] AI models, also known as AI algorithms, are a collective term for mathematical algorithms built upon the principles of artificial intelligence. They form the foundation for using AI to solve specific problems. Depending on the specific methods and / or technologies used to implement artificial intelligence, AI models can also be called machine learning models or deep learning models.
[0116] 2. Language Model (LM)
[0117] Language models are a type of machine learning model that can be used to process and predict natural language data. Language models can include various types, such as large language models, compact language models, or small language models, for processing and predicting natural language data.
[0118] Large language models (LLMs) are neural network models trained on large corpora with a vast number of parameters. They can understand, generate, process, or analyze natural language text. Specifically, LLMs are typically based on neural network techniques, learning the syntax, semantics, and contextual information of a language through training on large amounts of text data. During training, the model continuously optimizes its parameters to improve its ability to understand, generate, process, or analyze text. Due to their powerful capabilities in understanding and processing natural language, LLMs have been widely applied in many fields to solve problems related to natural language understanding.
[0119] Small language models (SLMs) typically refer to models with a relatively small number of parameters. They are designed with efficiency and practicality in mind, are small in size, and easily adaptable to resource-constrained and computationally limited environments, such as mobile devices or embedded systems. Compared to LLMs, SLMs have a significantly reduced number of parameters, making them more economical in terms of storage and computing resource requirements. Due to the simplified parameters and model structure, SLMs often respond faster when processing requests. SLMs may be optimized for specific application scenarios or tasks to provide good performance with limited resources. Despite their smaller number of parameters, SLMs can still achieve generalization capabilities across a variety of tasks through careful design and training.
[0120] In the embodiments of this application, a large language model and a small language model can refer to two language models with a significantly different number of parameters. For example, a large language model is a model with tens of billions of parameters, while a small language model is a model with hundreds of millions of parameters; or, the parameter difference between a large language model and a small language model can be hundreds of times.
[0121] 3. Prompt
[0122] Prompt is an interaction method based on natural language processing. It enables communication between users and machines by parsing natural language. The main implementation of prompt is to convert natural language into machine-recognizable instructions by building a corresponding corpus and semantic parsing model. For example, prompt can generate more granular tasks by piecing together prompts from the original text based on context, thought processes, or cue words, guiding the larger model to complete the task better.
[0123] The design principles of prompts include: writing clear and specific instructions, and giving the model time to think.
[0124] The key design elements of a prompt include: instruction, context, constraint, example, and input. Instruction provides the task instructions and states the specific goals. Context allows the AI (or large model) to understand the task from the user's perspective and provides context around the task. Constraint specifies the steps to complete the task (e.g., a chain of thought, COT), clearly defines the language style, and specifies the output template. Example provides concrete examples.
[0125] It should be noted that the division of the design points of the prompt here is only an example, and other division methods can also be used for the prompt. This application does not limit this.
[0126] 4. Instruction tuning (IT)
[0127] General-purpose large language models (such as GPT-4 and Llama-3.1) are deep learning models pre-trained on large amounts of unlabeled text data. They possess general natural language processing capabilities and can handle various tasks, such as text generation, translation, question answering, and sentiment analysis. However, these large language models are generally not specifically trained for any particular task or domain during training. Each industry and different application scenarios have their own specific and personalized needs, and pre-trained large language models often struggle to comprehensively meet these requirements.
[0128] To improve the capabilities of large language models for specific tasks or domains, they can be fine-tuned to make them more accurate and effective in those tasks or domains.
[0129] Instruction fine-tuning is a fine-tuning technique used to adapt a pre-trained base model to downstream tasks. For example, instruction fine-tuning is a technique for fine-tuning a large model on a labeled dataset with instruction prompts and corresponding outputs. By providing a set of guidelines or instructions outlining the required actions, pre-trained models can be adapted to perform specific tasks, improving their performance on those tasks and enhancing their overall ability to follow instructions, thus facilitating the practical application of the pre-trained model. Large language models can be fine-tuned for a variety of purposes, from style customization to supplementing the core knowledge and vocabulary of the pre-trained model, to optimizing performance for specific use cases. Language models can gain the ability to follow instructions through instruction fine-tuning, aligning the language model with key human standards such as security, privacy, or legal compliance.
[0130] Instruction fine-tuning requires supervised learning on labeled (input, output) pairs. These labeled (input, output) pairs can also be referred to as instruction data or instruction pairs used for instruction fine-tuning.
[0131] The quality or quantity of instruction pairs has a significant impact on the effectiveness of instruction fine-tuning.
[0132] To obtain high-quality instruction pairs, experienced experts can label the (input and output) pairs to ensure their quality and diversity. Alternatively, a language model can be used to generate instruction pairs to build an IT dataset. Or, to obtain a large number of instruction pairs, a language model can be used to filter existing instruction pairs to build the IT dataset.
[0133] The following text combines Figure 1 Describe possible implementations for training or using the model.
[0134] Figure 1 This is a schematic diagram of the architecture 100 of a data processing system applicable to embodiments of this application.
[0135] like Figure 1 As shown, the data acquisition device 160 is used to collect training data and store the training data in the database 130. The training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.
[0136] It should be noted that the data acquisition device 160, the execution device 110, and the training device 120 may be the same or different devices. The training data may include instruction pairs.
[0137] Model training refers to the process of using a specified initial model to calculate training data, and then adjusting the parameters of the initial model based on the calculation results, so that the model gradually learns certain patterns and acquires specific functions. Once trained and possessing stable functionality, the model can be used to filter or generate data. Model data filtering involves using the trained model to score and filter input data to obtain filtered data; model data generation involves using the trained model and the input data to generate new data based on the input data, thus obtaining generated data.
[0138] The following describes in more detail how the training device 120 obtains the target model / rule 101 based on training data. This training data may be second training data obtained based on the data filtering method 400 provided in the embodiments of this application; or, the training data may be data used to train a scoring model for performing the data filtering method 400 provided in the embodiments of this application; or, the training data may be data used to train a model for performing the data filtering method 400 provided in the embodiments of this application.
[0139] It should be noted that in practical applications, the training data maintained in database 130 can come from data acquisition device 160 or other devices. Training device 120 does not necessarily train the target model / rule 101 entirely based on the training data maintained in database 130; it can also obtain training data from the cloud or other sources for model training / instruction fine-tuning. The above description should not be construed as limiting the embodiments of this application. The target model / rule 101 trained by training device 120 can be applied to different systems or devices, for example, applied to... Figure 1 The execution device 110 shown.
[0140] It should be understood that the execution device 110 may be a terminal. For example, the terminal may be a digital camera, video recording device, mobile phone, personal computer (PC), laptop computer, server, tablet computer, smart TV, in-vehicle terminal, mobile internet device (MID), wearable device, virtual reality (VR) device, augmented reality (AR) device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc., or it may be an edge device (e.g., a box carrying a chip with processing capabilities); the execution device 110 may also be a server or cloud, etc.
[0141] The execution device 110 is equipped with an I / O interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. Furthermore, this input data can be user-inputted data, data uploaded by the user through the device, or data from a database; this application does not impose any limitations on this.
[0142] As one possible embodiment, the execution device 110 and the training device 120 are different processors deployed on different physical devices (e.g., servers in a server or cluster). For example, the execution device 110 may be a graphics processing unit (GPU), a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The training device 120 may be a CPU, NPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits (the integrated circuit is used to perform model training / instruction fine-tuning based on the second training data obtained according to the data filtering method 400 provided in the embodiments of this application; or, the integrated circuit is used to train the model to obtain the scoring model in the data filtering method 400 provided in the embodiments of this application; or, the integrated circuit is used to train the model to obtain a model for executing the data filtering method 400 provided in the embodiments of this application). The training device 120 can configure the trained neural network to multiple execution devices 110. Each execution device 110 uses the trained neural network to perform functions such as data filtering, scoring, clustering, or evolution.
[0143] In another possible embodiment, the execution device 110 and the training device 120 are deployed on the same physical device, or the execution device 110 and the training device 120 are on the same physical device. The computing device can configure the trained neural network to itself and use the trained neural network to perform functions such as data filtering, scoring, clustering, or evolution.
[0144] Training device 120 is used to train a neural network using training data until the loss function in the neural network converges and its value is less than a specific threshold, at which point the neural network training is complete, thus achieving a certain level of accuracy. Alternatively, all training data in database 130 can be used for training, completing the neural network training and enabling it to perform functions such as data filtering, scoring, clustering, or evolution. Then, training device 120 configures the trained neural network to execution device 110. Execution device 110 is used to perform functions such as data filtering, scoring, clustering, or evolution on user input based on the trained neural network.
[0145] The preprocessing module 113 is used to preprocess the input data received by the I / O interface 112. In this embodiment, the preprocessing module 113 can be used to obtain user input (e.g., from conversations, text messages, memos, clipboard, etc.). During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the processed data, instructions, etc., into the data storage system 150. Finally, the I / O interface 112 returns the processing result (output data) to the client device 140, thereby providing it to the user.
[0146] It should be understood that Figure 1 This is merely a schematic diagram of a system architecture. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 1 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.
[0147] Before any AI model can be used to solve a specific technical problem, it needs to be trained. Training is mainly divided into supervised training and unsupervised training. Supervised training is also known as supervised learning, and unsupervised training is also known as unsupervised learning.
[0148] Supervised training is a method of training models using labeled data. Labeled data refers to data where each input sample has a corresponding target output (label). The goal of the model is to learn the mapping relationship between inputs and outputs, thereby making accurate predictions about new input data. Taking text as an example, in supervised training of an AI model, the training texts in the training text set are labeled. These training texts are sequentially input into the AI model, which learns the mapping relationship between inputs and labels until it can accurately judge or recognize new input data.
[0149] Unsupervised training is a method of training models using unlabeled data. The goal of the model is to discover hidden structures, patterns, or relationships within the data, rather than predicting labels. Taking text as an example, in unsupervised training of an AI model, the training texts in the training set are unlabeled. These training texts are sequentially input into the AI model, which gradually identifies the relationships and latent rules between them until it can determine or recognize the type or features of the input text.
[0150] Different AI models can be used for different task types. Some AI models can only be trained using supervised learning, some can only be trained using unsupervised learning, and some can be trained using both supervised and unsupervised learning.
[0151] As an example, in one possible implementation, the method provided in this application embodiment can be applied to a cloud service scenario, where the method is executed by a cloud management platform within the cloud service scenario. For ease of description, the following will first combine... Figure 2 It provides a detailed description of cloud service scenarios.
[0152] Figure 2 A schematic block diagram illustrating a cloud scenario applicable to embodiments of this application is shown. For example... Figure 2 As shown, the cloud scenario may include: cloud management platform 210, Internet 220, and client 230.
[0153] like Figure 2 As shown, the cloud management platform 210 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each cloud data center includes multiple servers, and each server includes cloud service resources to provide corresponding cloud services to tenants.
[0154] The cloud management platform 210 can be located in a cloud data center and provides access interfaces (such as user interfaces or application program interfaces, APIs). Tenants can use client 230 to remotely access the access interface to register a cloud account and password on the cloud management platform 210 and log in. After successful authentication of the cloud account and password on the cloud management platform 210, the tenant can further select and purchase virtual machines of specific specifications (processor, memory, disk) on the cloud management platform 210. After successful purchase, the cloud management platform 210 provides the remote login account and password for the purchased virtual machine, and client 230 can remotely log in to the virtual machine to install and run the tenant's applications. Therefore, tenants can create, manage, log in to, and operate virtual machines in the cloud data center through the cloud management platform 210. Virtual machines can also be referred to as Elastic Compute Service (ECS) or Elastic Instances (different cloud service providers may use different names).
[0155] It should be understood that cloud service tenants can be individuals, businesses, schools, hospitals, government agencies, etc.
[0156] The cloud management platform 210 includes, but is not limited to, a user console, compute management services, network management services, storage management services, authentication services, and image management services. The user console provides an interface or API for interaction with tenants. The compute management services manage servers running virtual machines and containers, as well as bare metal servers. The network management services manage network services (such as gateways and firewalls). The storage management services manage storage services (such as data bucket services). The authentication services manage tenant account passwords. The image management services manage virtual machine images. Tenants use client 230 and can log in to the cloud management platform 210 via the internet 220 to manage their rented cloud services.
[0157] As an example, this cloud service may include, but is not limited to, AI services. AI services and products in the cloud domain embody both the on-demand and purchase-based characteristics of cloud services and the abstract, diverse, and widely applicable characteristics of AI technology. AI services in the cloud domain include: Platform-as-a-Service (PaaS) type AI infrastructure development platform services.
[0158] It should be understood that the AI Infrastructure Development Platform service is a PaaS cloud service within the cloud management platform 210. It's a software platform provided to users (also known as tenants, AI developers, etc.) based on the abundant underlying resources and software capabilities of public cloud service providers. This platform assists users in building, training, and deploying AI models, as well as developing and deploying AI applications. In other words, public cloud service providers offer an AI Infrastructure Development Platform to tenants, leveraging their ample underlying resources and upper-layer AI algorithm capabilities. The AI development framework and various AI algorithms built into this platform allow tenants to quickly build and develop AI models or applications that meet their individual needs.
[0159] like Figure 3 As shown, Figure 3 A schematic diagram illustrating the interaction between a tenant and an AI basic development platform applicable to an embodiment of this application is shown. The interaction between the tenant and the AI basic development platform mainly includes: the tenant logging into the cloud management platform 210 through a webpage on the client 230, selecting and purchasing cloud services of the AI basic development platform on the cloud management platform 210, and after purchase, the tenant can carry out full-process AI development based on the functions provided by the AI basic development platform.
[0160] As an example, when tenants develop and train their AI models on an AI infrastructure development platform, they rely on the basic resources (primarily computing resources such as CPUs, GPUs, and NPUs) in the cloud service provider's data center. Therefore, when purchasing and using the AI infrastructure development platform, payment is mainly for the resources used. For instance, tenants need to prepay before using the AI infrastructure development platform. During prepayment, different resource specifications support different functions of the AI infrastructure development platform. Tenants mainly select the name and specifications of the resources they need based on the functions they require. Tenants can also choose the purchase duration. The cloud management platform 210 prices the package based on the user's selected resource name, specifications, and purchase duration. After purchasing the prepaid package, tenants can utilize the capabilities provided by the AI infrastructure development platform and the basic computing resources included in the prepaid package for building, training, and deploying AI models. When resource usage exceeds the current prepaid package limit, the cloud management platform 210 charges for the excess resources on a pay-as-you-go basis. In reality, the basic resources used by tenants on the AI infrastructure development platform are mainly virtualized computing resources, such as virtual machines and containers.
[0161] It should be understood that the sale of AI basic development platforms is actually a form of selling software capabilities together with hardware virtualization basic resources. Furthermore, the basic resources supporting any process in the AI basic development platform may be distributed across different physical devices. In other words, the hardware devices that actually execute a process are usually server clusters in the same data center or server clusters distributed across different data centers.
[0162] It should be understood that the system architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0163] In recent years, with the advancement of deep learning technology and the booming development of LLM, artificial intelligence has achieved revolutionary breakthroughs in fields such as natural language processing and logical reasoning tasks. Large language models such as GPT and Llama have demonstrated powerful generation and reasoning capabilities in numerous scenarios. To improve the capabilities of large language models in specific tasks or domains, fine-tuning of instructions can be performed on the large language models to make them more accurate and efficient in specific tasks or domains.
[0164] Fine-tuning of large language models requires instruction pairs. The quality or quantity of instruction pairs has a significant impact on the effectiveness of instruction fine-tuning.
[0165] In one possible implementation, to obtain a large number of instruction pairs, instruction pairs can be generated through a language model to construct an IT dataset. This method is highly efficient in generating instruction pairs.
[0166] However, the quality of instruction pairs generated by language models needs improvement. Furthermore, the quality of instruction pairs has a more significant impact on the effectiveness of instruction fine-tuning than their quantity.
[0167] In one possible implementation, to obtain high-quality instruction pairs, (input, output) pairs can be tagged by experienced experts to create instruction pairs. Alternatively, instruction pairs can be manually generated to ensure both quality and diversity.
[0168] However, creating high-quality instruction pairs through manual annotation is both time-consuming and labor-intensive. Obtaining instruction pairs is also costly.
[0169] In another possible implementation, to obtain high-quality instruction pairs, a large model can be used to filter through a large number of existing instruction pairs to obtain high-quality instruction pairs.
[0170] For example, one approach is to write a prompt based on scoring criteria (e.g., answer accuracy); use the prompt to call GPT-3.5-Turbo to score each of the 9,000 English instruction pairs, and filter the 9,000 instruction pairs based on the scoring results (e.g., the score value can be 1, 2, 3, 4, or 5; GPT-3.5-Turbo will delete instruction pairs with a score no higher than 4, and only output instruction pairs with a score of 5).
[0171] However, the method of using GPT to filter a large number of existing instruction pairs to obtain high-quality instruction pairs has drawbacks.
[0172] On the one hand, this method ignores the diversity of instruction pairs, resulting in poor diversity of the selected instruction pairs. On the other hand, experimental results show that GPT scores instruction pairs generated by GPT higher, demonstrating GPT's self-reinforcing bias, or in other words, the scoring model used for evaluation is not accurate or objective enough when evaluating instruction pairs generated by models within the same series. Furthermore, scoring locally stored instruction pairs using this method relies on the external GPT API, posing information risks and instability, making it unsuitable for local deployment in private domains with limited internet access. Moreover, the uncertainty introduced by frequent updates to large models and increasingly stringent blocking policies limits the application of this method in industrial deployments. Finally, this method only supports filtering English instruction pairs, making it difficult to support filtering instruction pairs in other languages.
[0173] Another approach is to use an open-source small model to perform unsupervised reasoning on each data point in the dataset and calculate the perplexity score (the perplexity score can reflect the quality of the data point to some extent); select the data points with low perplexity scores in the dataset as the filtered instruction pairs.
[0174] However, the method of using open-source small models to perform unsupervised inference on every data point in the dataset has drawbacks. This method only considers the quality of individual data points and ignores the overall diversity of instructions across the dataset, thus affecting the generalization ability of the large model fine-tuned based on the selected instructions.
[0175] While selecting a subset of high-quality instruction pairs from the complete dataset and fine-tuning the large model on this subset results in better performance compared to fine-tuning the large model on the complete dataset (because using low-quality instruction pairs in large model instruction fine-tuning reduces the model's instruction compliance ability), removing most instruction pairs compromises the integrity of the instruction pair dataset. This can lead to missing instructions in certain categories, reducing the instruction compliance ability of the subset-fine-tuned large model in these domains. For example, a high filtering ratio of code-related instruction pairs in the training dataset results in a relatively weaker ability of the fine-tuned large model to answer coding-related questions.
[0176] Furthermore, to reduce the time and labor costs of scoring, GPT preferences could be used as a substitute for human preferences (or, to score quality pairs using GPT, replacing human preferences with GPT's own preferences during the scoring process). However, GPT-4 exhibits systematic biases in its scoring, such as positional bias, redundancy bias, and self-reinforcing bias. This indicates that using GPT preferences as a substitute for human preferences results in inaccurate and subjective scores.
[0177] In view of this, embodiments of this application provide a method and apparatus for filtering data in order to obtain high-quality training data for fine-tuning instructions of a language model.
[0178] like Figure 4 As shown, Figure 4 A schematic diagram of a data filtering method 400 provided in an embodiment of this application is shown. The method 400 includes the following steps.
[0179] S410, according to the scoring parameters, score each of the multiple first training data to obtain a score result for each of the first training data. The scoring parameters include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security; each of the first training data includes instruction pairs in different languages.
[0180] S420, Based on the scoring result, determine n1 first training data from the multiple first training data, where n1 is a positive integer.
[0181] S430, based on the number of the plurality of first training data, cluster the plurality of first training data to obtain k1 classes, each class including at least one of the first training data, where k1 is an integer greater than 1.
[0182] S440, Based on the scoring result, determine n2 first training data from each class, where n2 is a positive integer.
[0183] S450, based on the n1 sets of first training data and k1 sets of the n2 sets of first training data, determine the second training data, which is used for fine-tuning the model's instructions.
[0184] Fine-tuning a language model using a large number of instruction-to-instruction pairs requires significant computational resources. Therefore, this method is limited in scenarios with scarce computational resources. To address this, a subset of training data is selected from a large dataset; this subset consists of instruction pairs used for fine-tuning the language model, thus reducing the computational resource requirements of instruction-to-instruction language model fine-tuning.
[0185] For example, by scoring and clustering multiple first training data, n1+k1*n2 first training data can be obtained; by processing the n1+k1*n2 first training data, second training data for fine-tuning the language model can be obtained.
[0186] The score obtained by scoring the first training data can be used to represent the quality of the first training data.
[0187] For example, the first training data with higher scores can be considered to be of better quality; the first training data with lower scores can be considered to be of poorer quality.
[0188] It is understandable that high or low ratings are relative concepts.
[0189] For example, the quality of the first training data #1 with a score of 4 is better than that of the first training data #2 with a score of 3; and the quality of the first training data #1 with a score of 4 is worse than that of the first training data #3 with a score of 5.
[0190] The scoring parameters can represent the dimensions used to evaluate the first training data. The scoring results can be obtained based on different scoring parameters.
[0191] For example, the first training data #1 has good relevance, but poor richness; the first training data #2 has poor humanization, but good accuracy; the scoring results of the first training data #1 and the first training data #2 can be the same.
[0192] Based on the method provided in the embodiments of this application, by filtering multiple first data to obtain n1+k1*n2 first training data, a small amount of high-quality, multilingual second training data for fine-tuning the language model using instructions can be obtained. The second training data helps save computational resources for fine-tuning the language model using instructions, and fine-tuning the language model using instructions helps improve the language model's ability to perform specific tasks or in specific scenarios.
[0193] Each set of first training data includes instruction pairs in different languages. This can be understood as the first training data including the results of translating an instruction pair into different languages.
[0194] When translating a pair of instructions into different languages, one can consider the cultural taboos and characteristics of the language users. The language users can be native speakers of that language.
[0195] For example, when translating instructions in language #1 into language #2, if people whose native language is #1 consider white to be a festive color, while people whose native language is #2 consider white to be an unlucky color, then a simple literal translation of instructions related to "white" may affect the rationality of the instruction pair.
[0196] Translations that do not consider cultural taboos and characteristics will affect the rationality of instruction pairs, and thus affect the performance of the large model obtained by fine-tuning the language model based on the instruction pairs.
[0197] Furthermore, by including instruction pairs in different languages in each initial training data set, compared to fine-tuning instructions using only single-language instruction pairs, the language model fine-tuned by the instructions can learn instructions from different cultural backgrounds and languages, which helps the language model fine-tuned by the instructions to better understand instruction pairs.
[0198] For example, the first training data includes instruction pairs for language #1 and instruction pairs for language #2. The large model obtained by fine-tuning instructions based on this first training data, compared to a large model obtained by fine-tuning instructions based only on single-language instruction pairs, improves the effectiveness of instruction fine-tuning because it learns features from multiple languages during the fine-tuning process.
[0199] In addition, by including instruction pairs in different languages in each initial training data set, the model obtained by instruction fine-tuning is more easily transferred to platforms / environments of languages that have not been trained on, compared to fine-tuning with instruction pairs in only a single language.
[0200] For example, the first training data includes instruction pairs for language #1 and instruction pairs for language #2. The large model obtained by fine-tuning instructions based on this first training data, although not fine-tuned using instruction pairs based on language #3, is less difficult to transfer to a model platform based on language #3 because it learns features from multiple languages during instruction fine-tuning, thus improving its capabilities across different languages.
[0201] Clustering is the process of dividing a dataset into different classes or clusters based on a specific criterion (such as distance), maximizing the similarity of data objects within the same cluster and maximizing the dissimilarity of data objects outside the same cluster. In other words, after clustering, data of the same class should be grouped together as much as possible, while data of different classes should be separated as much as possible.
[0202] Clustering is an unsupervised learning method. It can discover the inherent structure of data without relying on the labels on the training data. This characteristic makes clustering particularly suitable for scenarios where label acquisition is costly or data annotation is difficult. Furthermore, clustering automatically partitions data through similarity measures, revealing previously unknown natural groupings within the data (such as customer groups in market segmentation or implicit topics in text analysis). Clustering also effectively preserves the diversity of the training data.
[0203] Different clustering methods or different clustering parameters can yield different clustering results.
[0204] Based on the number of first training data points, the multiple first training data points can be clustered, which can be done using the K-means algorithm.
[0205] For example, if the number of initial training data sets is n, and the K-means algorithm is used for clustering, then k1 = (n / 2). 1 / 2 n is a positive integer.
[0206] Based on the method provided in the embodiments of this application, by clustering the multiple first training data and determining n2 first training data from each cluster, the diversity of the filtered data can be improved.
[0207] It is understandable that the value of n2 can be different or the same for different classes. For example, in the first class, n2 = 5; in the second class, n2 = 3; in the third class, n2 = 5... This application does not impose any restrictions on this.
[0208] If data with low scores in the overall dataset are directly discarded, the instruction pairs learned by the model during instruction fine-tuning will not be rich enough in terms of domain. This will result in the language model after instruction fine-tuning performing poorly in tasks outside the domain, having poor potential capabilities, or having a large gap between its potential capabilities and those of the model obtained by instruction fine-tuning based on the data before filtering.
[0209] Compared to directly discarding low-scoring data from the overall dataset, selecting k1*n2 initial training data points from clustering increases the diversity of the selected training data and reduces the potential performance gap of models fine-tuned based on this data. In other words, selecting n2 initial training data points from each cluster increases the diversity of the selected data. Higher diversity in the selected data leads to better performance of the model fine-tuned based on this data in low-computing-resource scenarios.
[0210] In one possible implementation, determining n1 first training data points from the plurality of first training data points based on the scoring result includes: determining the n1 first training data points with the highest scoring results from the plurality of first training data points based on the scoring result; and / or, determining n2 first training data points from each class based on the scoring result includes: determining the n2 first training data points with the highest scoring results from each class based on the scoring result.
[0211] For example, determining n1 first training data points from the plurality of first training data points based on the scoring results can be achieved by arranging the plurality of first training data points in descending order according to the scoring results; or by randomly (or according to some rule) determining n1 first training data points from the first M first training data points. M is an integer greater than or equal to n1.
[0212] For example, determining n1 first training data points from the plurality of first training data points based on the scoring results includes: arranging the plurality of first training data points in descending order according to the scoring results, and determining the top n1 first training data points. Determining the top n1 first training data points from the plurality of first training data points based on the scoring results can improve the quality of training data used for instruction fine-tuning.
[0213] For example, determining n² first training data points from each class based on the scoring results can be achieved by arranging the first training data points in each class from highest to lowest according to the scoring results; and then randomly (or according to some rule) determining n² first training data points from the first M' first training data points in each class. M' is an integer greater than or equal to n².
[0214] For example, determining n² first training data points from the first training data in each class based on the scoring results includes: arranging the first training data in each class from highest to lowest score based on the scoring results, and determining the top n² first training data points in the ranking. Determining the top n² first training data points from the first training data in each class based on the scoring results can improve the quality of the training data used for instruction fine-tuning.
[0215] Based on the method provided in the embodiments of this application, by determining the top n2 first training data from each class, it is possible to avoid determining n1 first training data based solely on the scoring results, which would result in the rejection of some first training data that are of good quality in the class but perform poorly in the dataset, thereby improving the diversity of the filtered data.
[0216] For example, n1 > n2. n1 > n2 helps to ensure the quality of the first training data obtained through screening.
[0217] It should be understood that the method 400 provided in this application embodiment can be executed by a single model or by multiple models with different functions (for example, the method 400 provided in this application embodiment is executed by a scoring model, a clustering model, and an evolutionary model; the scoring model is used to score the training data according to the scoring parameters, the clustering model is used to cluster according to the number of training data, and the evolutionary model is used to determine the second training data). This application does not impose any limitations on this.
[0218] In one possible implementation, method 400 may further include:
[0219] S460, based on the task to be processed after fine-tuning the model instructions, determines the scoring parameter.
[0220] Specifically, the scoring parameters can be determined based on the task that the fine-tuned model is used to process according to the instructions. Alternatively, the scoring parameters can be determined based on the characteristics of the task that the fine-tuned model is used to process according to the instructions.
[0221] For example, if the model fine-tuned by instructions is used for medical question answering, the scoring parameters can be correctness and humanization; or if the model fine-tuned by instructions is used for content retrieval, the scoring parameters can be one or more of correctness, richness, comprehensiveness, relevance, or information content; or if the model fine-tuned by instructions is used for logical reasoning, the scoring parameters can be one or more of correctness, comprehensiveness, fluency, or consistency.
[0222] Based on the method provided in this application, by fine-tuning the task to be processed according to the model instructions and determining the scoring parameters, training data more suitable for the needs of different tasks can be obtained. Compared with fixed scoring parameters, the selected training data can be more reasonable.
[0223] In one possible implementation, S450 specifically includes: performing evolutionary processing on the n1 sets of first training data and k1 sets of the n2 sets of first training data to obtain the second training data; the evolutionary processing includes at least one of the following: adding constraints; deepening understanding; concretizing; adding inference steps; complicating the input data; or generating new data based on the input data.
[0224] For example, evolutionary processing can be deep processing and / or breadth processing. Deep evolution aims to increase difficulty without changing the semantics of the original data. Deep evolution can specifically include five operations: adding constraints, deepening understanding, concretization, adding reasoning steps, and complicating the input data. Breadth evolution involves generating entirely new data based on given input data.
[0225] Evolutionary processing can increase the complexity of training data.
[0226] In one possible implementation, S450 specifically includes performing the evolutionary processing based on the cultural specificity of the language of the instruction pair.
[0227] For example, similar to the aforementioned translation, evolutionary processing can also take into account the cultural taboos and cultural characteristics of language users.
[0228] For example, users of language #1 use a certain animal as a common food, but users of language #2 consider using that animal as food taboo. Therefore, in the evolutionary processing of instruction pairs for language #2, content related to using that animal as food is not included.
[0229] Based on the method provided in the embodiments of this application, by performing this evolutionary processing according to the cultural specificity of the language of the instruction pair, the training data obtained through evolution can be made more reasonable.
[0230] In one possible implementation, method 400 may further include:
[0231] S470, delete duplicate data in the n1 first training data and k1 sets of the n2 first training data.
[0232] For example, method 400 may also include deleting duplicate parts from the n1+k1*n2 first training data, where the duplicate parts can be understood as duplicate data.
[0233] Based on the method provided in the embodiments of this application, redundant filtered data can be avoided by deleting duplicate data.
[0234] In one possible implementation, S410 specifically includes: inputting the plurality of first training data into a scoring model to obtain a scoring result for each of the first training data, wherein the number of model parameters of the scoring model is less than or equal to 1 billion.
[0235] For example, the model used for scoring is a small model (e.g., an SLM model). Small models are easier to deploy locally due to their small number of parameters, which helps with industrial implementation, improves the stability of method applications, and avoids the risks associated with external API access.
[0236] For example, the model used for scoring can be trained based on instructions annotated by experts. This small model can learn expert preferences during training, making the scoring results more reasonable, objective, and accurate.
[0237] The second training data obtained based on the above method 400 can be used to fine-tune the language model.
[0238] For example, this application provides a method for fine-tuning instructions. The method includes: fine-tuning a model based on second training data, which is determined based on n1 sets of first training data and k1 groups of n2 sets of first training data. The first training data includes instruction pairs in different languages, where n1 is a positive integer, n2 is a positive integer, and k1 is an integer greater than 1. The n1 sets of first training data are determined from multiple sets of first training data based on a scoring result, and the k1 groups of n2 sets of first training data are determined from k1 classes based on the scoring result. The scoring result is obtained by scoring each set of first training data in the multiple sets of first training data according to scoring parameters, which include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security. The k1 classes are obtained by clustering the multiple sets of first training data based on their number, and each class includes at least one set of first training data.
[0239] The language model is fine-tuned using the second training data obtained by the above method 400, which requires less computational resources and achieves better fine-tuning results. The model fine-tuned with the obtained instructions performs better on various tasks, is applicable to scenarios with different languages, and has stronger generalization ability.
[0240] The following text combines Figures 5 to 9 The possible implementations of the method 400 provided in the embodiments of this application are described in detail.
[0241] Thanks to contributions from the open-source community, a wealth of instruction fine-tuning data has emerged. Given the enormous resource allocation required for training and evaluating models, having an effective method for selecting high-quality instruction fine-tuning data is particularly important.
[0242] Method 400, also known as clustering and ranking (CaR), is an efficient method for selecting multilingual instruction pairs.
[0243] CaR consists of several stages: the first stage is to evaluate the quality of instruction pairs in the dataset and obtain a score for each instruction pair; the second stage is to cluster all instruction pairs; and the third stage is to perform evolutionary processing on the subset of data obtained by filtering the dataset in the first and second stages in the depth and / or breadth dimensions.
[0244] The dataset is an original dataset constructed based on expert translations. For example, instruction pairs representing the same meaning in the dataset have 19 different language representations. These 19 different language instruction pairs were translated by experts, who took into account the cultural specificities and taboos of the speakers of different languages.
[0245] In the embodiments of this application, the evaluation result can be interchanged with the scoring result and express the same meaning.
[0246] The method 400 provided in this application embodiment can be applied to: dataset selection and data quality optimization in the instruction fine-tuning data preparation stage of training multilingual large models.
[0247] like Figure 5 As shown, Figure 5 A schematic diagram of a data filtering method 500 provided in an embodiment of this application is shown.
[0248] The method 500 includes the following steps: constructing an instruction quality scoring (IQS) model, constructing a clustering model, constructing an evolutionary model, and filtering data based on the above IQS model, clustering model, evolutionary model, and method 400.
[0249] like Figure 6 As shown, Figure 6 A schematic diagram of a data filtering method 600 provided in an embodiment of this application is shown.
[0250] This application provides a simple and effective data selection method—CaR. CaR combines two dimensions: quality and diversity, and consists of three steps. First, a subset (containing n1 first training data points) is selected to ensure that the filtered data retains a large number of high-quality instructions. Then, a small number of high-quality instructions (consisting of n2 first training data points) are added from each cluster to enhance data diversity while maintaining instruction quality. Finally, evolutionary processing is used to further enhance the complexity of the filtered data to obtain second training data.
[0251] The three steps include: sorting, clustering, and evolution.
[0252] (1) Ranking: Each instruction pair in the entire dataset is evaluated using the IQS model (an IQS model is a pre-trained quality scoring model), and for each instruction pair... i Assign a score i The 'i' is used to identify different instructions or different scoring results; 'i' can be an integer.
[0253] (2) Clustering: The clustering model is used to divide all candidate instruction pairs into k1 clusters.
[0254] All instruction pairs are sorted according to the scoring results, and the top n1 pairs are selected; within each cluster, the top n2 pairs are selected according to the scoring results. By removing duplicate n1+k1*n2 pairs of instructions (which can also be understood as deleting the parts that are repeated with other data), a high-quality subset of data that maintains diversity is obtained.
[0255] (3) Evolution: Deepen and / or broaden the high-quality subset of data to further increase the complexity of the data in order to obtain a second training data.
[0256] In the implementation of CaR, steps (1) and (2) above can be repeated; or steps (1), (2) and (3) can be repeated.
[0257] For example, such as Figure 6 As shown, the original data in the original dataset is input into the scoring model to obtain the scoring result of each original data; based on the clustering model, the original data and the scoring results, k1*n2 original data are obtained; based on the original data and the scoring results, n1 original data are obtained; based on the evolution model and n1+k1*n2 original data, a filtered dataset is obtained, which includes the evolved data.
[0258] Method 600 is a possible implementation of Method 400. The IQS model is a possible implementation of the scoring model. The original data is a possible implementation of the first training data. The evolved data is a possible implementation of the second training data.
[0259] In the embodiments of this application, the order in which the IQS model is constructed, the clustering model is constructed, or the evolutionary model is constructed can be arbitrary, and this application does not impose any restrictions on this.
[0260] In the embodiments of this application, the IQS model and / or clustering model can be trained based on manually translated instruction pairs in 19 languages, and can be applied to instruction pairs in different languages.
[0261] The following text combines Figure 7 Describe in detail the construction of the IQS model.
[0262] like Figure 7 As shown, Figure 7 A schematic diagram of an IQS model provided in an embodiment of this application is shown.
[0263] To explore the instruction quality estimation (IQE) task, this application designs a framework suitable for training using expert preferences, based on the Comet framework.
[0264] To reduce the amount of training data and achieve high-quality training, the method provided in this application refines 3751 instruction pairs from the Alpaca_52k dataset. These 3571 instruction pairs may have been selected by linguistic experts, and can also be referred to as a preference dataset annotated by linguistic experts. These 3751 instruction pairs exhibit good fluency, accuracy, and semantic consistency between the questions and answers.
[0265] The Alpaca_52k dataset is constructed by Alpaca using GPT-3.
[0266] Unedited instructions and responses from GPT-3.5 were categorized as GPT-preference data, while expert-revised instructions were categorized as expert-preference data. To further optimize the linguistic expert-annotated preference dataset and enable the rating model to identify features / distinctions between GPT-preference and expert-preference data, 2541 instructions were selected from the 3751 instruction pairs and revised by experts. This revision resulted in more accurate, concise instructions, or in other words, instructions that better aligned with expert preferences. Furthermore, the edit distance between the expert revisions exceeded a small threshold, and the original and revised instruction pairs were significantly different.
[0267] The original instruction pair and the revised instruction pair are denoted as training instruction pairs. The training instruction pairs are randomly assigned to the training set, validation set, and test set in a ratio of 8:1:1.
[0268] To better utilize expert-revised paired annotations, such as Figure 7The left half (Model #1) uses the translation ranking model architecture in the Comet framework, employing instructions and inputs as anchors to train Model #1. During training, Model #1 reduces the semantic distance between better responses and expert-preferred data by pre-training the encoder, pooling layers, sentence embeddings, and triplet margin loss, while maximizing the distance between better responses and GPT-preferred data (or responses). The trained Model #1 achieves 72.44% accuracy on the test set. However, since the anchors are only references during training and do not participate in the learning process, the training of Model #1 fails to fully utilize the expert improvements to the input.
[0269] To better utilize expert improvements to the input section, such as Figure 7 The right half (model #2) uses the pre-trained XLM-RoBERTa large model in the Comet framework to concatenate the various parts of the instruction pair, training model #2. During the training of model #2, the parameters of model #2 are optimized through pre-trained encoder, pooling layer, feedforward, and mean-square error (MSE) loss. The trained model #2 can also be called the IQS model.
[0270] Table 1 shows the model performance (or the reasonableness of the model's scoring results for instruction pairs) on test set #1 or test set #2 using trained model #2, trained model #1, GPT-4, and GPT-3.5, respectively.
[0271] Table 1
[0272] Model #2 Model #1 GPT-4 GPT-3.5 Test set #1 84.25% 72.44% 63.19% 57.48% Test set #2 78.12% 45.00% 65.00% 56.25%
[0273] As shown in Table 1, Figure 7 The trained model #2 shows that it outperforms GPT-3.5 and GPT-4; and the trained model #2 outperforms the trained model #1.
[0274] Further analysis of the model's performance revealed that GPT-4 biased towards the original instructions (meaning instructions not revised by experts) in 62.2% of its error cases, indicating that even powerful GPT models often favor data with GPT bias. Furthermore, GPT-4 struggled to recognize subtle semantic changes made by expert revisions in 37.8% of its error cases, revealing its difficulty in identifying experts and subtle semantic changes through only minor adjustments. While GPT-4 showed high consistency with expert biases in most general tasks, its poor performance on expert-revised datasets highlights the subtle gap between expert biases and GPT biases.
[0275] The IQS model trained using the architecture shown in Model #2 helps the model better understand what constitutes "good" data as understood by humans.
[0276] The following text combines Figure 8 Provide a detailed description of clustering.
[0277] The trained model #2 is a possible implementation of the scoring model.
[0278] like Figure 8 As shown, Figure 8 This illustration shows a clustering method provided in an embodiment of this application.
[0279] To reduce the amount of training data while achieving high-quality training, the raw data can be categorized. In low-resource scenarios, a subset of data is selected from a large number of instruction pairs, thus identifying a limited number of instructions. Experiments show that mixing training data used to perform different tasks can enhance the performance of instruction-tuned LLMs across various capabilities. For example, by assigning task labels to each pair of instructions and retaining instruction pairs relevant to a wider range of tasks during the data selection process, cross-task instruction coordination can be promoted, improving the performance of models trained on the selected data.
[0280] To determine task labels for instruction pairs, manual annotation, classification models, or clustering models can be used. While manually labeled task labels are more accurate, manual annotation is labor-intensive and difficult to adapt well to all types of datasets. If we assume that instruction pairs used to perform the same task are semantically similar, we can allow classification models to learn the distribution of instruction pairs. However, classification models are less flexible when dealing with out-of-domain data.
[0281] To improve the versatility of the method, embodiments of this application maintain data diversity through clustering. Clustering algorithms can identify semantically similar instruction pairs and form clusters for different tasks. Furthermore, clustering enables the instruction-tuned model to efficiently adapt to different datasets when encountering instruction pairs from outside the domain, without needing to retrain from scratch, but simply by forming new clusters.
[0282] Regarding the clustering method, the K-means algorithm is used in the embodiments of this application.
[0283] First, a sentence transformation model is used to map sentences to a 384-dimensional dense vector space.
[0284] Subsequently, principal component analysis (PCA) was performed on the semantic features to reduce their dimensionality, retaining 95% of the semantic features' dimensions.
[0285] Finally, the number of clusters is set to k1 = (n / 2). 1 / 2 The 52k (52k = 52 * 1000 = 52000) instruction pairs were clustered into 161 clusters (classes).
[0286] By adjusting the number of instruction pairs within each cluster (class), the diversity of filtered instruction pairs is maintained.
[0287] like Figure 8 As shown, Figure 8 This illustrates one possible implementation of clustering. Using a clustering method, data with diagonal lines filled in one cluster, data with dots filled in another, and data without fill in a third. After clustering, n² data points can be selected from the diagonal-lined data, the dot-filled data, and the blank data. That is, k¹*n² data points can be obtained after clustering.
[0288] The following text combines Figure 9 Describe the evolution in detail.
[0289] like Figure 9 As shown, Figure 9 A schematic diagram illustrating an evolution provided by an embodiment of this application is shown.
[0290] In the technical process of instruction evolution, the first step is to understand how large-scale pre-trained language models modify given instructions through specific prompts, making them more complex and challenging. In this way, the model can not only rewrite existing instructions but also generate entirely new ones, which are comparable in complexity to the original instructions. Based on this discovery, the initial instruction dataset can be optimized through continuous iteration, increasing not only the difficulty of the instructions but also expanding their thematic coverage and diversity.
[0291] (1) Instruction Evolution Process
[0292] During instruction evolution, the instruction pool is first started with an initial instruction dataset (D(0)). In this embodiment, the initial instruction set is the high-quality subset obtained after the aforementioned sorting and clustering (the high-quality subset includes n1+k1*n2 first training data). Each round of evolution can extract previously unevolved instructions from the instruction pool and modify them through the instruction evolution module. Then, the success of the instruction evolution is checked through the instruction filter. If the evolution is successful, the new evolved instructions are placed in the instruction pool; if the evolution fails, the original instructions are retained in the instruction pool, waiting for the next evolution.
[0293] (2) Instruction evolver module
[0294] The instruction evolution module is a module based on a pre-trained large language model. It enables instruction evolution through specific prompts. The instruction evolution module has two main evolutionary modes: in-depth evolution and in-breadth evolution.
[0295] The goal of deep evolution is to make existing instructions more complex, increasing their difficulty. To achieve this, five different cues can be used: adding constraints, deepening the problem, specifying details, adding reasoning steps, and complicating the input. The core purpose of deep evolution is to make instructions "slightly harder" by adding constraint words, rather than making them overly verbose or incomprehensible, thus maintaining evolutionary balance. Deep evolution is also known as in-depth evolution.
[0296] The goal of breadth-based evolution is to expand the topic and skill coverage of existing instruction datasets. Since instruction datasets in the open-source domain are typically small and cover a limited range of topics and skills, breadth-based evolution addresses this problem by drawing inspiration from a given instruction to generate a completely new, rarer instruction. The new instruction will be within the same domain, but its rarity and difficulty will be greater than the original instruction.
[0297] (3) Generate response
[0298] When rewriting instructions using evolutionary cues with a large language model, the same model is used to generate responses corresponding to the rewritten instructions, and these responses are added to the instruction pool. To ensure fair comparison, if a response in D(0) is not generated by the same large language model, the response will be regenerated.
[0299] (4) Situations where evolution fails
[0300] During instruction evolution, several situations may be considered evolutionary failures: lack of information increment (the evolved instruction does not provide new information compared to the original instruction), difficulty in generating effective responses (if the model generates a short response and includes "sorry", it means that the model cannot effectively answer the instruction), low response quality (the generated response only contains punctuation marks or stop words, indicating insufficient content), or word duplication (the evolved instruction clearly duplicates specific words in the evolutionary prompt, such as "given prompt" or "rewrite prompt").
[0301] For example, such as Figure 9 As shown. The evolution includes: deep evolution of the initial data to obtain evolutionary data #1; breadth evolution of evolutionary data #1 to obtain evolutionary data #3; breadth evolution of the initial data to obtain evolutionary data #2; deep evolution of evolutionary data #2 to obtain evolutionary data #5, but the evolution of evolutionary data #5 fails; and breadth evolution of evolutionary data #2 to obtain evolutionary data #4. The evolutionary data includes evolutionary data #3 and evolutionary data #4.
[0302] Through continuous iteration and optimization, the evolutionary process ultimately generates a more complex and diverse instruction dataset, enhancing the language model fine-tuned by the instructions in this dataset to handle various complex tasks. This instruction dataset includes a second training dataset.
[0303] The sorting, clustering, and evolution methods described above allow for the filtering of the original dataset. The filtered data obtained through these processes can then be used to fine-tune the language model.
[0304] To compare AlpaCaR (AlpaCaR refers to a model that performs CaR to filter data) with other models that filter data, this embodiment generates a response for each instruction in each test set using a unified prompt and unified default parameter settings for the LLaMA-Factory. The evaluator then compares the responses generated by different models with human reference responses and selects the preferred response.
[0305] PandaLM, GPT-4, or humans as evaluators yielded relatively consistent assessment conclusions.
[0306] The evaluators can be three experts with an average of 12.57 years of experience, who can evaluate large language models (judge LLM) or independently evaluate them according to established evaluation criteria. The large language model (judge LLM) can be evaluated using either GPT-4Turbo or PandaLM. GPT-4Turbo uses prompts designed by vicuna as an alternative to manual response quality evaluation; however, GPT-4Turbo has limitations due to API dependency and inherent bias. PandaLM can be deployed locally to provide efficient LLM evaluation. Based on 300k samples (trained using GPT-3.5), PandaLM effectively reduces bias, achieving 88.3% of the evaluation capability of GPT-4. In most experiments, the evaluation results of PandaLM are used to measure the model's instruction-following ability; some experiments use both GPT-4 and human evaluation to measure the model's instruction-following ability.
[0307] The default parameter settings for LLaMA-Factory include: temperature = 0.95, top_p = 0.7, top_k = 50, no beam search, and a maximum token length limit of 512.
[0308] To avoid confusion caused by similarity in model and dataset naming, datasets are represented using the format "ModelName_DatasetSize". Four datasets were selected using the aforementioned method 400: Self-instruct_252, Vicuna_80, PandaLM_170, and CoachLM_150. This comparison method covers a wider range of instructions, reducing evaluation bias.
[0309] The evaluation of each sample is performed by the judge LLM. The judge LLM receives one instruction and two candidate responses and evaluates the candidate responses. The judge LLM marks the winning response based on the score; if the two responses have the same score, it is a tie. To avoid the potential bias of the judge LLM favoring responses from specific positions, two rounds of testing are conducted by swapping the order of the responses, and the final judgment is determined according to the following rules:
[0310] Win: Two wins, or one win and one draw; Loss: Two losses, or one loss and one draw; Draw: Two draws, or one win and one loss.
[0311] The winning percentage for the three types is calculated as follows:
[0312] 1. Winscore (WS): WS = 1 + (Number of wins - Number of losses) / Number of all test samples
[0313] 2. Winrate (WR): WR = Number of wins / Number of all test samples
[0314] 3. Quality Score (QS): QS = (Number of wins + Number of draws) / Total number of test samples
[0315] The performance of AlpaCaR is compared with baseline models (including Alpaca, Alpaca-PandaLM, Alpaca-cleaned, Alpagasus, or Vicuna). All baseline models are reproduced at a model parameter scale of 7B, and AlpaCaR's advantage at model parameter scales of 13B and 30B is demonstrated. B represents billions. The Alpaca model was fine-tuned on the LLaMA-7B model using the Alpaca_52k dataset.
[0316] This application compares two established baseline LLMs (Alpaca and Vicuna). They were fine-tuned using dialogues with 52,000 text instructions and 70,000 large language models, respectively. Furthermore, we explored three models that improve upon Alpaca: Alpaca-PandaLM and Alpaca-cleaned, which employ instruction enhancement methods, and Alpagasus, which employs instruction filtering methods. All models in Table 2 use the same hyperparameter settings.
[0317] Table 2
[0318]
[0319] As shown in Table 2, AlpaCaR outperforms Alpaca, Vicuna, Alpaca-PandaLM, Alpaca-cleaned, and Alpagasus at the 7B scale. Overall, AlpaCaR significantly improves performance compared to Alpaca at the 7B, 13B, and 30B scales, validating the effectiveness of the CaR method.
[0320] like Figure 10 As shown, Figure 10 This diagram illustrates the reliability of an IQE result provided by an embodiment of this application.
[0321] To verify whether the IQE results truly reflect the performance of the LLM after instruction fine-tuning, the correlation between the score given by the IQS model and the performance of the fine-tuned LLM on the test set was compared. For both the IQS and Comet models, the top 9k scores of each model were selected. The selected 9k data points were used to train / fine-tune the model respectively, and the average score of the three IT subsets was calculated. The results show that the IQS score and the performance of the fine-tuned model are generally consistent (or, the higher the IQS score, the better the performance of the fine-tuned model), indicating that the IQE results can roughly reflect the performance of the LLM after fine-tuning.
[0322] like Figure 11 As shown, Figure 11 This illustration shows a schematic diagram of the performance of a trained model provided in an embodiment of this application.
[0323] Based on the quality dimension, 52,000 instructions were ranked using scores obtained from the IQS model, and the top 1,000, top 2,000, and top 42,000 instructions were extracted for training LLaMA-7B. Compared to Alpaca as a baseline, the performance of the trained models shows that models trained on carefully selected data generally outperform those trained on the entire dataset.
[0324] like Figure 12 As shown, Figure 12 This diagram illustrates the reliability of an IQE result provided by an embodiment of this application.
[0325] Based on a trade-off between quantity and quality, the number of samples selected from each cluster after K-means clustering is compared. Experimental results show that the CaR method can effectively improve the performance of the fine-tuned model by selecting a small number of samples (maximum 5) from each cluster.
[0326] An ideal instruction fine-tuning dataset should have rich diversity. Considering the importance of data diversity, as shown in Table 3, by comparing the performance of the model with different numbers of clusters (classes) (the more clusters, the richer the data diversity), it can be determined that the model containing a more diverse instruction set performs better.
[0327] Table 3
[0328]
[0329] like Figure 13 As shown, Figure 13 A schematic diagram illustrating the reliability of a CaR method provided in an embodiment of this application is shown.
[0330] To verify the reliability of the CaR method, we compared the performance of different models with varying numbers of instruction pairs. Experimental results show that, compared to randomly selecting 10¹⁷ instruction pairs, the instruction pairs selected using the CaR method significantly improved model performance at scales of 7B, 13B, and 30B. Furthermore, we compared AlpaCaR with Alpaca using the GPT-4 evaluation framework, and the results show that AlpaCaR outperforms the baseline model in most cases, demonstrating the effectiveness of the CaR method.
[0331] To verify whether the CaR method can effectively represent human preferences, this application's embodiments establish detailed evaluation criteria covering seven aspects: fluency, relevance, correctness, consistency, satisfaction, information content, and security. The performance of AlpaCaR30B and Alpaca30B was compared on the Vicuna_80 test set. As shown in Table 4, human rating results indicate that AlpaCaR is at least comparable to Alpaca in all categories and is more favored by language experts in most cases.
[0332] Table 4
[0333] Dataset Win negative (lose) flat (tie) WS↑ Writing 8 1 1 1.700 Role-playing 5 0 5 1.500 Common sense 9 0 1 1.900 Fermi 7 2 1 1.500 Counterfactual 7 0 3 1.700 Programming (coding) 3 3 1 1.000 Mathematics 0 0 3 1.000 Generic 6 0 4 1.600 Knowledge 7 2 1 1.500 Total 52 8 20 1.550
[0334] This application provides a data screening method for multilingual instructions that is suitable for industrial applications, consistent with expert opinions, and maintains diversity: the CaR method. CaR employs a multi-stage process: first, instruction pairs are ranked using a high-precision (84.25%) scoring model consistent with expert preferences; second, clustering is used to maintain dataset diversity; and then the resulting n1+k1*n2 data points are further evolved. In our experiments, the CaR method selected only 1.96% of Alpaca's instruction fine-tuning data. However, the AlpaCaR model obtained through instruction fine-tuning outperformed Alpaca in the GPT-4 evaluation, showing an average performance improvement of 32.1% compared to Alpaca. Furthermore, the CaR method's data screening follows a consistent paradigm (or exhibits a certain degree of robustness), meaning that the CaR method is applicable and performs the aforementioned functions regardless of whether the model being fine-tuned has larger parameters (7B, 13B, 30B) or stronger capabilities (LLaMA1, LLaMA2, LLaMA3).
[0335] The scoring model described above can be a compact model with 550M parameters. Fine-tuning based on data selected using the CaR method significantly reduces training costs (e.g., training costs are only 11.2% of those based on fine-tuning using all initial training data). This enhances industrial deployability.
[0336] For details on fine-tuning instructions based on the filtered data obtained by the above method, please refer to relevant technologies, which will not be elaborated here.
[0337] The AlpaCaR mentioned above can also be deployed in the cloud.
[0338] The above text combined Figures 4 to 13 This application provides a detailed description of the solutions and possible implementations provided in its embodiments. The following section, in conjunction with... Figures 14 to 16 This application describes an apparatus for fine-tuning filtering data or instructions according to embodiments of the present application.
[0339] Figure 14 This is a schematic block diagram of a device 1000 provided in an embodiment of this application.
[0340] In one possible implementation, the device 1000 can be implemented by software, hardware, or a combination of both. The device 1000 provided in this application embodiment can implement the embodiments of this application. Figure 4 The method flow shown (or, the apparatus 1000 provided in the embodiments of this application can be implemented based on the embodiments of this application) Figure 4 The method flow shown obtains the second training data (instructions for fine-tuning the language model), and the device 1000 includes a processing module 1020.
[0341] The processing module 1020 is used to score each of the multiple first training data according to the scoring parameters to obtain a scoring result for each of the first training data. The scoring parameters include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security. Each of the first training data includes instruction pairs in different languages. Based on the scoring result, n1 first training data are determined from the multiple first training data, where n1 is a positive integer. Based on the number of the multiple first training data, the multiple first training data are clustered to obtain k1 classes, each class including at least one of the first training data, where k1 is an integer greater than 1. Based on the scoring result, n2 first training data are determined from each class, where n2 is a positive integer. Based on the n1 first training data and k1 groups of the n2 first training data, second training data is determined, which is used for instruction fine-tuning of the model.
[0342] In one possible implementation, the device 1000 further includes an output module 1010 for outputting second training data.
[0343] Alternatively, the processing module 1020 is used to fine-tune the model based on second training data, which is determined based on n1 first training data and k1 groups of n2 first training data. The first training data includes instruction pairs in different languages, where n1 is a positive integer, n2 is a positive integer, and k1 is an integer greater than 1. The n1 first training data are determined from multiple first training data based on scoring results, and the k1 groups of n2 first training data are determined from k1 classes based on the scoring results. The scoring results are obtained by scoring each of the multiple first training data according to scoring parameters, which include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security. The k1 classes are obtained by clustering the multiple first training data based on the number of data points, and each class includes at least one first training data point.
[0344] In one possible implementation, the device 1000 can be implemented by software, hardware, or a combination of both.
[0345] The device 1000 here may be embodied in the form of a functional module. The term "module" here can be implemented in software and / or hardware, without specific limitation.
[0346] For example, a processing module can be a processor or a processing unit; an output module can be a transceiver, a transceiver unit, or an output unit.
[0347] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. For instance, the implementation of output module 1010 in device 1000 will be described below. Similarly, the implementation of other modules in device 1000, such as processing module 1020, can refer to the implementation of output module 1010.
[0348] Output module 1010, as an example of a software functional unit, may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, output module 1010 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.
[0349] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.
[0350] As an example of a hardware functional unit, output module 1010 may include at least one computing device, such as a server. Alternatively, output module 1010 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.
[0351] The multiple computing devices included in the output module 1010 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the output module 1010 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the output module 1010 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0352] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0353] It should be noted that the above embodiments of the device 1000, when executing the above methods, are only illustrative examples of the division of functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device 1000 can be divided into different functional modules to complete all or part of the functions described above. For example, the output module 1010 can be used to execute any step in the above methods, and the processing module 1020 can be used to execute any step in the above methods. The steps implemented by the output module 1010 and the processing module 1020 can be specified as needed, and all the functions of the device 1000 can be realized by implementing different steps in the above methods through the output module 1010 and the processing module 1020 respectively.
[0354] Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.
[0355] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system is any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Furthermore, in some possible implementations, the computer system is a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.
[0356] The following is combined Figure 15 This application provides a detailed description of a computing device provided in an embodiment.
[0357] Figure 15 This is a schematic diagram of the architecture of a computing device 1100 provided in an embodiment of this application.
[0358] The computing device 1100 can be a server, a computer, or other device with computing capabilities. Figure 15 The computing device 1100 shown includes at least one processor 1110 and a memory 1120.
[0359] It should be understood that this application does not limit the number of processors and memories in the computing device 1100.
[0360] The processor 1110 executes the instructions in the memory 1120, causing the computing device 1100 to implement the method provided in this application. Alternatively, the processor 1110 executes the instructions in the memory 1120, causing the computing device 1100 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.
[0361] In some possible implementations, computing device 1100 also includes communication interface 1130. Communication interface 1130 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between computing device 1100 and other devices or communication networks.
[0362] In some possible implementations, the computing device 1100 also includes a system bus 1140, wherein the processor 1110, memory 1120, and communication interface 1130 are respectively connected to the system bus 1140. The processor 1110 can access the memory 1120 through the system bus 1140; for example, the processor 1110 can perform data read / write operations or code execution in the memory 1120 through the system bus 1140. The system bus 1140 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1140 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0363] In one possible implementation, the processor 1110 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in memory 1120 or cache 1116.
[0364] In some possible implementations, processor 1110 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, processor 1110 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor is a microprocessor, etc. For example, processor 1110 is a central processing unit (CPU).
[0365] In some possible implementations, each processor 1110 includes at least one processing unit 1112 and a memory control unit 1114.
[0366] In some possible implementations, the processing unit 1112, also referred to as the core, is the most important component of the processor. The processing unit 1112 is manufactured from single-crystal silicon using a specific production process. All calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, utilizing parallel computing capabilities to accelerate program execution. Various processing units have fixed logical structures; for example, a processing unit includes logical units such as a Level 1 cache, a Level 2 cache, an execution unit, an instruction-level unit, and a bus interface.
[0367] In one implementation example, the memory control unit 1114 controls the data interaction between the memory 1120 and the processing unit 1112. Specifically, the memory control unit 1114 receives memory access requests from the processing unit 1112 and controls access to memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).
[0368] In one implementation example, each memory control unit 1114 addresses the memory 1120 via the system bus. An arbitrator is configured in the system bus. Figure 15 (Not shown in the image), the arbitrator is responsible for handling and coordinating competing accesses by multiple processing units 1112.
[0369] In one implementation example, the processing unit 1112 and the memory control unit 1114 are connected via internal chip connection lines, such as address lines, thereby enabling communication between the processing unit 1112 and the memory control unit 1114.
[0370] In some possible implementations, each processor 1110 also includes a cache 1116, which is a buffer for data exchange (called a cache). When the processing unit 1112 needs to read data, it first looks for the required data in the cache. If the data is found, it is executed directly; otherwise, it is looked for in memory. Since the cache operates much faster than memory, its role is to help the processing unit 1112 run faster.
[0371] The memory 1120 provides runtime space for processes in the computing device 1100. For example, the memory 1120 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 1120. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 1120 stores data generated during the process's execution, such as intermediate data or process data, in the storage space corresponding to the process.
[0372] In some possible implementations, the memory is also called RAM, and its function is to temporarily store the data to be processed by the processor 1110, as well as the data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 1110 will load the data that needs to be processed into RAM for processing, and after the processing is completed, the processing unit 1112 will send the result out.
[0373] By way of example and not limitation, memory 1120 may be a non-transitory computer-readable storage medium, which may include volatile memory or non-volatile memory, or both. The non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory is random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory 1120 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0374] The above-described structure of the computing device 1100 is merely illustrative and is not intended to limit the application. The computing device 1100 in this application includes various hardware components found in existing computer systems. For example, the computing device 1100 may also include other storage devices besides the memory 1120, such as disk storage. Those skilled in the art should understand that the computing device 1100 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 1100 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the computing device 1100 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 15 All the devices shown.
[0375] Figure 16This is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application.
[0376] The computing device cluster includes at least one computing device. This computing device may be a server. In some embodiments, the computing device may also be a terminal device such as a desktop computer, laptop computer, or smartphone.
[0377] like Figure 16 As shown, the computing device cluster includes at least one computing device 1100. The memory 1120 of one or more computing devices 1100 in the computing device cluster may store the same instructions for performing the methods described above.
[0378] In some possible implementations, the memory 1120 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the above-described methods. In other words, a combination of one or more computing devices 1100 can jointly execute the instructions of the above-described methods.
[0379] It should be noted that the memory 1120 in different computing devices 1100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned device. That is, the instructions stored in the memory 1120 of different computing devices 1100 can implement the functions of one or more modules within the aforementioned device.
[0380] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.
[0381] For example, two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device.
[0382] It should be understood that the functions of computing device 1100A can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.
[0383] This application also provides a system for filtering data or fine-tuning instructions, which may include the above-described device 1000 for filtering data or fine-tuning instructions.
[0384] This application also provides a computer program product containing instructions, which may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When run on a computing device, it causes the computing device to perform the methods provided above, or causes the computing device to perform the functions of the apparatus provided above.
[0385] This application also provides a computer-readable storage medium, which can be any available medium capable of being stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method provided above.
[0386] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0387] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0388] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0389] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0390] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0391] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0392] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0393] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for filtering data, characterized in that, The method includes: According to the scoring parameters, each of the first training data in the plurality of first training data is scored to obtain a scoring result for each first training data. The scoring parameters include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security; each first training data includes instruction pairs in different languages. Based on the scoring results, n1 first training data points are determined from the plurality of first training data points, where n1 is a positive integer; Based on the number of the plurality of first training data, the plurality of first training data are clustered to obtain k1 classes, each class including at least one first training data, where k1 is an integer greater than 1; Based on the scoring results, n2 first training data points are determined from each class, where n2 is a positive integer; Based on the n1 sets of first training data and the k1 sets of n2 sets of first training data, second training data is determined, and the second training data is used for fine-tuning the model's instructions.
2. The method according to claim 1, characterized in that, The method further includes: The scoring parameters are determined based on the task to be processed after fine-tuning the model instructions.
3. The method according to claim 1 or 2, characterized in that, The step of determining the second training data based on the n1 sets of first training data and k1 sets of the n2 sets of first training data includes: The n1 sets of first training data and k1 sets of n2 sets of first training data are subjected to evolutionary processing to obtain the second training data; The evolutionary process includes at least one of the following: Add constraints; To deepen understanding; concretize; Add reasoning steps; To complicate the input data; or, New data is generated based on the input data.
4. The method according to claim 3, characterized in that, The step of determining the second training data based on the n1 sets of first training data and k1 sets of the n2 sets of first training data further includes: The evolutionary processing is performed based on the cultural specificity of the language of the instruction pair.
5. The method according to any one of claims 1 to 4, characterized in that, The step of determining n1 first training data points from the plurality of first training data points based on the scoring results includes: determining the n1 first training data points with the highest scoring results from the plurality of first training data points based on the scoring results; and / or, Based on the scoring results, n2 first training data are determined from each class, including: based on the scoring results, determining the n2 first training data with the highest scoring results from each class.
6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Delete duplicate data from the n1 first training data sets and the k1 sets of the n2 first training data sets.
7. The method according to any one of claims 1 to 6, characterized in that, The step of scoring each of the multiple first training data points according to the scoring parameters to obtain a scoring result for each of the first training data points includes: The multiple first training data are input into the scoring model to obtain a scoring result for each of the first training data, wherein the number of model parameters of the scoring model is less than or equal to 1 billion.
8. A method for fine-tuning instructions, characterized in that, include: The model is fine-tuned based on the second training data, which is determined from n1 sets of first training data and k1 sets of n2 sets of first training data. The first training data includes instruction pairs in different languages, where n1 is a positive integer, n2 is a positive integer, and k1 is an integer greater than 1. Wherein, the n1 first training data points are determined from multiple first training data points based on the scoring results, and the k1 groups of n2 first training data points are determined from k1 classes based on the scoring results. The scoring result is obtained by scoring each of the plurality of first training data according to the scoring parameters, wherein the scoring parameters include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security. The k1 classes are obtained by clustering the plurality of first training data based on the number of the plurality of first training data, and each class includes at least one first training data.
9. A device for filtering data, characterized in that, include: A processing unit is configured to score each of the multiple first training data sets according to scoring parameters to obtain a scoring result for each of the first training data sets. The scoring parameters include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security. Each of the first training data sets includes instruction pairs in different languages. The processing unit is further configured to determine n1 first training data from the plurality of first training data based on the scoring result, where n1 is a positive integer; The processing unit is further configured to cluster the plurality of first training data according to the number of the plurality of first training data to obtain k1 classes, each class including at least one first training data, where k1 is an integer greater than 1; The processing unit is further configured to determine n2 first training data from each class based on the scoring results, where n2 is a positive integer; The processing unit is further configured to determine second training data based on the n1 sets of first training data and k1 sets of the n2 sets of first training data, wherein the second training data is used for fine-tuning of the model's instructions.
10. A device for fine-tuning instructions, characterized in that, include: The processing unit is used to fine-tune the model based on second training data, which is determined based on n1 sets of first training data and k1 sets of n2 sets of first training data. The first training data includes instruction pairs in different languages, where n1 is a positive integer, n2 is a positive integer, and k1 is an integer greater than 1. Wherein, the n1 first training data points are determined from multiple first training data points based on the scoring results, and the k1 groups of n2 first training data points are determined from k1 classes based on the scoring results. The scoring result is obtained by scoring each of the plurality of first training data according to the scoring parameters, wherein the scoring parameters include at least one of the following: correctness, richness, comprehensiveness, humanization, fluency, relevance, consistency, satisfaction, information content, or security. The k1 classes are obtained by clustering the plurality of first training data based on the number of the plurality of first training data, and each class includes at least one first training data.
11. A chip, characterized in that, The chip is used to perform the method as described in any one of claims 1 to 7, or the chip is used to perform the method as described in claim 8.
12. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 7, or the method as described in claim 8.
13. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1 to 7, or the method as described in claim 8.
14. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 7, or the method as described in claim 8.
