Sample data set construction method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a sample data set construction method and device, equipment and a storage medium. The method is applied to the technical field of machine learning, a question set corresponding to a target model is determined, a sample data set corresponding to the target model is determined based on the question set, the sample data set comprises a plurality of difficult questions and correct answers corresponding to the difficult questions, and the difficult questions are questions wrongly answered by the target model; performing clustering analysis processing on the problem set to obtain a plurality of first problem classes corresponding to the problem set; determining a candidate problem set of the target model based on the plurality of first problem classes and the problem set; and the target sample data set of the target model is determined based on the candidate problem set and the sample data set, so that the construction cost of the sample data set is reduced, and the efficiency and the accuracy are improved.

Description

Technical Field

[0001] The present application belongs to the field of machine learning technology, and in particular, relates to a method, device, equipment and storage medium for constructing a sample data set. Background Art

[0002] Machine learning focuses on algorithms and statistical models, allowing computer systems to use data to continuously improve performance. Model training usually uses unsupervised learning for pre-training, learning common features through a large amount of unlabeled data. After that, supervised learning is used for fine-tuning to adapt the model to specific tasks or fields.

[0003] There are two main strategies for constructing supervised fine-tuning samples for models: manual data labeling and data synthesis. Manual data labeling involves professionals writing questions and answers based on existing data sets or specific task scenarios to ensure the accuracy of samples. Data synthesis uses models to automatically generate samples.

[0004] However, manual data annotation requires experts or skilled personnel to manually create high-quality data samples, which is costly and inefficient. The data synthesized by the data synthesis method deviates from the real data, affecting the accuracy and generalization ability of the model. Summary of the invention

[0005] The present application provides a method, device, equipment and storage medium for constructing a sample data set, so as to solve the defects of high cost, low efficiency and poor accuracy in constructing a sample data set in the prior art.

[0006] In a first aspect, the present application provides a method for constructing a sample data set, the method comprising:

[0007] Determine a question set corresponding to the target model, and based on the question set, determine a sample data set corresponding to the target model, wherein the sample data set includes a plurality of difficult questions and a correct answer corresponding to each difficult question, wherein the difficult question is a question that the target model answers incorrectly;

[0008] Performing cluster analysis on the question set to obtain a plurality of first question classes corresponding to the question set;

[0009] Determining a candidate question set for the target model based on the plurality of first question classes and the question set;

[0010] Based on the candidate question set and the sample data set, a target sample data set of the target model is determined.

[0011] Optionally, determining a sample data set corresponding to the target model based on the problem set includes:

[0012] Inputting a plurality of questions in the question set into the target model and a plurality of other models to obtain a first answer output by the target model and a second answer output by each of the other models;

[0013] determining the plurality of difficult questions from the question set based on first answers to the plurality of questions;

[0014] For any one of the multiple difficult questions, determine a correct answer corresponding to the difficult question from the multiple second answers, wherein the correct answer is the answer with the highest reply matching degree among the multiple second answers;

[0015] The multiple difficult questions and the correct answer corresponding to each difficult question are used as a sample data set corresponding to the target model.

[0016] Optionally, determining a candidate question set of the target model based on the multiple first question categories and the question set includes:

[0017] Determine the accuracy of the target model relative to each first question class according to the first answers to the multiple questions and the multiple first question classes, and take at least one first question class whose accuracy meets a first preset condition as a difficult question class;

[0018] The candidate question set is determined based on the at least one difficult question class.

[0019] Optionally, determining the candidate question set based on the at least one difficult question class includes:

[0020] Based on at least one difficult problem class, generating at least one second problem class, where the second problem class is an associated class of the difficult problem class;

[0021] A plurality of candidate difficult questions corresponding to the at least one difficult question class and the at least one second question class are generated, and the plurality of candidate difficult questions are used as the candidate question set.

[0022] Optionally, determining a target sample data set of the target model based on the candidate question set and the sample data set includes:

[0023] For any one of the plurality of candidate difficult questions, using an answer verification model to determine the correctness of the answer of the target model relative to the candidate difficult question, wherein the correctness of the answer is used to indicate the probability that the target model can correctly answer the candidate difficult question;

[0024] Deleting at least one candidate difficult question whose answer accuracy rate satisfies a second preset condition from the candidate question set among the multiple candidate difficult questions, to obtain an updated candidate question set;

[0025] In the case where there is a candidate difficult question in the updated candidate question set, determining a correct answer to the corresponding candidate difficult question;

[0026] The candidate difficult questions and the correct answers to the candidate difficult questions are added to the sample data set to obtain a target sample data set for the target model.

[0027] Optionally, the method further includes:

[0028] When there is no candidate difficult problem in the updated candidate problem set, the sample data set is used as the target sample data set.

[0029] Optionally, the method further includes:

[0030] When the number of difficult questions in the sample data set does not reach a preset number, the sample data set is used as a question set corresponding to the target model, and based on the question set, the target sample data set corresponding to the target model is re-iterated;

[0031] When the number of difficult problems in the sample data set reaches a preset number, the iteration of the sample data set is stopped, and the sample data set corresponding to the last iteration is determined as the target sample data set corresponding to the target model.

[0032] In a second aspect, the present application provides a device for constructing a sample data set, the device comprising:

[0033] A determination module, used to determine a question set corresponding to a target model, and based on the question set, determine a sample data set corresponding to the target model, wherein the sample data set includes a plurality of difficult questions and a correct answer corresponding to each difficult question, wherein the difficult question is a question that the target model answers incorrectly;

[0034] A processing module, configured to perform cluster analysis on the question set to obtain a plurality of first question classes corresponding to the question set;

[0035] The determination module is further configured to determine a candidate question set of the target model based on the multiple first question categories and the question set;

[0036] The processing module is further configured to determine a target sample data set of the target model based on the candidate question set and the sample data set.

[0037] Optionally, the processing module is further used to input multiple questions in the question set into the target model and multiple other models to obtain a first answer output by the target model and a second answer output by each of the other models;

[0038] The determination module is further configured to determine the plurality of difficult questions from the question set according to the first answers to the plurality of questions;

[0039] The determination module is further configured to determine, for any one of the multiple difficult questions, a correct answer corresponding to the difficult question from the multiple second answers, wherein the correct answer is the answer with the highest reply matching degree among the multiple second answers;

[0040] The processing module is further configured to use the multiple difficult questions and the correct answer corresponding to each difficult question as a sample data set corresponding to the target model.

[0041] Optionally, the determination module is further used to determine the accuracy of the target model relative to each first question class based on the first answers to the multiple questions and the multiple first question classes;

[0042] The processing module is further configured to take at least one first problem class whose accuracy meets a first preset condition as a difficult problem class;

[0043] The determination module is further configured to determine the candidate question set based on the at least one difficult question class.

[0044] Optionally, the processing module is further used to generate at least one second problem class based on at least one difficult problem class, where the second problem class is an associated class of the difficult problem class;

[0045] The processing module is further configured to generate a plurality of candidate difficult problems corresponding to the at least one difficult problem class and the at least one second problem class, and use the plurality of candidate difficult problems as the candidate problem set.

[0046] Optionally, the processing module is further used to determine, for any one of the multiple candidate difficult questions, using an answer verification model to determine the answer accuracy of the target model relative to the candidate difficult question, wherein the answer accuracy is used to indicate the probability that the target model can correctly answer the candidate difficult question;

[0047] The processing module is further configured to delete at least one candidate difficult question whose answer accuracy rate satisfies a second preset condition from the candidate question set, so as to obtain an updated candidate question set;

[0048] The determination module is further configured to determine the correct answer to the corresponding difficult candidate question if there is a difficult candidate question in the updated candidate question set;

[0049] The processing module is further configured to add the candidate difficult questions and the correct answers to the candidate difficult questions to the sample data set to obtain a target sample data set of the target model.

[0050] Optionally, the processing module is further configured to use the sample data set as the target sample data set when there is no candidate difficult problem in the updated candidate problem set.

[0051] Optionally, the processing module is further configured to, when the number of difficult questions in the sample data set does not reach a preset number, use the sample data set as a question set corresponding to the target model, and re-iterate the target sample data set corresponding to the target model based on the question set;

[0052] The processing module is further configured to stop the iteration of the sample data set when the number of difficult questions in the sample data set reaches a preset number;

[0053] The determination module is further configured to determine the sample data set corresponding to the last iteration as the target sample data set corresponding to the target model.

[0054] In a third aspect, the present application provides a device for constructing a sample data set, including:

[0055] Memory;

[0056] processor;

[0057] Wherein, the memory stores computer-executable instructions;

[0058] The processor executes the computer-executable instructions stored in the memory to implement the method for constructing a sample data set as described in the first aspect and various possible implementations of the first aspect.

[0059] In a fourth aspect, the present application provides a computer storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method for constructing a sample data set as described in the first aspect and various possible implementations of the first aspect.

[0060] The present application provides a method, device, equipment and storage medium for constructing a sample data set. The method determines a question set corresponding to a target model, and based on the question set, determines a first sample data set corresponding to the target model, wherein the first sample data set includes multiple difficult questions and the correct answer corresponding to each difficult question, wherein the difficult question is a question that the target model answers incorrectly; performs cluster analysis on the question set to obtain multiple first question classes corresponding to the question set; determines a candidate question set for the target model based on the multiple first question classes and the question set; generates a target sample data set for the target model based on the candidate question set and the first sample data set, thereby reducing the sample data set construction cost, improving efficiency and improving accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0062] Figure 1 Schematic diagram of the process of constructing the sample data set provided in this application Figure 1 ;

[0063] Figure 2 Schematic diagram of the process of constructing the sample data set provided in this application Figure 2 ;

[0064] Figure 3 Schematic diagram of the process of constructing the sample data set provided in this application Figure 3 ;

[0065] Figure 4 A schematic diagram of the structure of a device for constructing a sample data set provided in this application;

[0066] Figure 5 A schematic diagram of the structure of a device for constructing a sample data set provided in this application.

[0067] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0070] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0071] Machine learning focuses on developing algorithms and statistical models that enable computer systems to continuously optimize their performance through data. In this process, the model is often first pre-trained with the help of unsupervised learning methods. At this time, the model will use a large amount of unlabeled data to learn common feature representations. Then, based on the initial mastery of a broad knowledge base, the model will enter the fine-tuning stage of supervised learning. This stage is carried out by using labeled data sets. The purpose is to enable the model to adapt and optimize for specific tasks or fields, thereby improving its performance in specific application scenarios.

[0072] There are two main strategies for constructing supervised fine-tuning samples for models: manual data labeling and data synthesis. Manual data labeling involves professionals writing questions and answers based on existing data sets or specific task scenarios to ensure the accuracy of samples. Data synthesis uses models to automatically generate samples.

[0073] Manual data annotation methods rely on experts or well-trained employees to manually create accurate data samples, which usually leads to high costs and low work efficiency. At the same time, although data synthesis technology can quickly generate a large amount of data, the synthesized data may deviate from the actual situation, which will reduce the prediction accuracy of the model and its application ability on unknown data.

[0074] In response to the above problems, the present application proposes a method for constructing a sample data set, which constructs a first sample data set by analyzing the questions that the target model answers incorrectly, i.e., difficult questions, and ensures that each question is accompanied by a correct answer. By clustering these questions, multiple first question classes are formed, and then a candidate question set is determined. Finally, the candidate question set is combined with the first sample data set to generate the target sample data set required by the target model. This reduces the cost of constructing the sample data set and improves construction efficiency and accuracy.

[0075] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0076] Figure 1 A schematic diagram of a method for constructing a sample data set provided in an application example Figure 1 .like Figure 1 As shown, the method for constructing a sample data set provided in this embodiment includes:

[0077] S101. Determine a problem set corresponding to a target model, and based on the problem set, determine a sample data set corresponding to the target model.

[0078] Among them, determine the specific field or scenario of the target model application, and determine the question set corresponding to the target model from existing databases, historical conversation records, public data sets and other sources. Input the collected questions into the target model, obtain the output results of the model, compare the output results of the model with the known standard answers or the correct answers provided by other models, and mark the questions that the target model answers incorrectly. Questions that the target model cannot answer correctly are defined as difficult questions, and a sample data set corresponding to the target model is obtained. The sample data set contains multiple difficult questions and the correct answers corresponding to each difficult question, wherein the difficult questions are questions that the target model answers incorrectly.

[0079] One possible implementation method is to determine the specific application field of the target model, obtain the corresponding problem set of the target model from the existing database or manually construct it. For each problem in the problem set, obtain the answer output by the target model and other models, mark the questions that the target model cannot answer correctly, and define them as difficult questions. Difficult questions and their corresponding correct answers are combined into samples to obtain the sample data set corresponding to the target model.

[0080] Determine the problem set corresponding to the target model, and based on the problem set, determine the sample data set corresponding to the target model. It can optimize the weaknesses of the target model, thereby improving the accuracy and reliability of the model in practical applications.

[0081] S102: Perform cluster analysis on the question set to obtain a plurality of first question classes corresponding to the question set.

[0082] Among them, select a suitable clustering algorithm according to the characteristics of the problem, such as hierarchical clustering, DBScan, spectral clustering, topic clustering LDA, SOM, Agglomerative, etc. The text similarity algorithm required by the clustering algorithm can be based on word matching algorithms such as BM25 and statistical translation models, or based on the similarity of word embedding vectors generated by Word2Vec or pre-trained models such as Bert, T5, etc. Use the selected clustering algorithm to perform cluster analysis on the question set, assign each question in the question set to different classes according to its characteristics, and obtain multiple first question classes corresponding to the question set.

[0083] By clustering the problem set, problems can be systematically identified and classified, thereby better supporting target model training and optimization and improving the practicality and efficiency of the model.

[0084] S103: Determine a candidate question set of the target model based on multiple first question categories and question sets.

[0085] Among them, based on multiple first question categories and question sets, question categories similar to the multiple first question categories and question sets and belonging to the same target field are generated, and these categories and existing question categories with high answer error rates together constitute a candidate difficult question category set. For each category in the candidate difficult question category set, a new question corresponding to each category is generated, and these new questions constitute the candidate question set of the target model.

[0086] By screening out the categories of questions that are difficult for the target model to answer, more difficult questions are generated, thereby improving the accuracy of the target model in a targeted manner.

[0087] S104: Determine a target sample data set of a target model based on the candidate question set and the sample data set.

[0088] Among them, for any candidate difficult question in the candidate difficult question set, the answer verification model is used to determine the answer accuracy of the target model relative to the candidate difficult question; those questions that the target model indicated by the target model answer verification model can correctly answer are removed from the candidate difficult question set, so as to obtain an updated candidate difficult question set.

[0089] If there are still remaining difficult questions in the updated candidate difficult question set, it is necessary to determine the correct answers to these difficult questions, and add these difficult questions and their corresponding correct answers to the sample data set to obtain the target sample data set.

[0090] By screening out questions that are difficult for the target model to answer correctly, a more challenging sample set is constructed, which helps to significantly improve the performance of the model. Generating specific high-value samples based on the weaknesses of the target model is more accurate than general data synthesis methods.

[0091] This embodiment proposes a method for constructing a sample data set. The method determines a question set corresponding to a target model, and based on the question set, determines a sample data set corresponding to the target model, wherein the sample data set includes multiple difficult questions and correct answers corresponding to each difficult question, wherein the difficult questions are questions that the target model answers incorrectly; performs cluster analysis on the question set to obtain multiple first question classes corresponding to the question set; determines a candidate question set for the target model based on the multiple first question classes and the question set; determines a target sample data set for the target model based on the candidate question set and the sample data set, thereby reducing the sample data set construction cost, improving efficiency, and improving accuracy.

[0092] Figure 2 This is a flow chart of a method for constructing a sample data set provided in an embodiment of the present application. Figure 2 This embodiment is in Figure 1 Based on the embodiment, the method for constructing the sample data set is described in detail. Figure 2 As shown, the method for constructing a sample data set provided in this embodiment includes:

[0093] S201. Determine a question set corresponding to a target model, input multiple questions in the question set into the target model and multiple other models, and obtain a first answer output by the target model and a second answer output by each other model.

[0094] Among them, a set of questions is selected for the target model, that is, the set of questions corresponding to the target model is determined, and these questions are used to test and optimize the target model. Each question in the set of questions is input into the target model and multiple other models to obtain the first answer of the target model to these questions, and also obtain the second answer of other models to these questions. The first answer and the second answer of the target model and multiple other models will be used for subsequent comparative analysis and sample set construction of the target model.

[0095] By comparing the performance of the target model with multiple other models on the same question, the target model's capability boundary can be evaluated. The high-quality answers generated by other models can be used to supplement or correct the shortcomings of the target model. In particular, when the target model's answers to certain questions are inaccurate, the answers of the better-performing models can be used to correct the answers of the target model.

[0096] S202: Determine a plurality of difficult questions from a question set according to first answers to the plurality of questions.

[0097] Among them, according to the first answers of multiple questions, multiple difficult questions are determined from the question set, and the questions that the target model answers with a high error rate are considered to be difficult questions. For example, in mathematical comparison questions, if the target model often makes mistakes when comparing the sizes of 1.3 and 1.13, this question is considered to be a difficult question, and the difficult questions are aggregated to obtain multiple difficult questions.

[0098] By identifying multiple difficult questions that the target model answers with high error rates, you can focus on the questions that are difficult for the target model to answer in the subsequent construction of data sample sets for the target model, making sample set construction more accurate and efficient, and avoiding wasting resources on problems that have already been mastered.

[0099] S203: For any one of the multiple difficult questions, determine a correct answer corresponding to the difficult question from the multiple second answers.

[0100] For any one of the multiple difficult questions, answers given by multiple other models are collected, and these answers constitute multiple second answers. The matching degree between the multiple second answers and the correct answer is evaluated, for example, the completeness, logic, and relevance of the answers to the question are evaluated. From the multiple second answers, the answer with the highest matching degree with the correct answer is selected as the correct answer corresponding to the difficult question.

[0101] For any one of the multiple difficult questions, the correct answer corresponding to the difficult question is determined from the multiple second answers, and a high-quality and high-matching correct answer can be determined for each difficult question. These correct answers will be used to train and optimize the target model to improve its performance when dealing with similar problems.

[0102] S204: Use the multiple difficult questions and the correct answer corresponding to each difficult question as a sample data set corresponding to the target model.

[0103] For each difficult question, the correct answer corresponding to each difficult question is determined, and the correct answer is completed by comparing the answers of different models and selecting the answer with the highest matching degree. Each difficult question and its corresponding correct answer are paired to obtain a sample data set corresponding to the target model.

[0104] Multiple difficult questions and the correct answers corresponding to each difficult question are used as sample data sets corresponding to the target model. The sample data sets corresponding to the target model will contain problem areas that need to be improved by the target model, providing opportunities for the target model to learn and improve.

[0105] S205: Perform cluster analysis on the question set to obtain a plurality of first question classes corresponding to the question set.

[0106] Step S205 is the same as step S102 and will not be described again.

[0107] S206: Determine a candidate question set of the target model based on the multiple first question categories and question sets.

[0108] Step S206 is the same as step S103 and will not be described again.

[0109] S207. For any one of the multiple candidate difficult questions, use the answer verification model to determine the answer accuracy of the target model relative to the candidate difficult question.

[0110] Among them, an answer verification model is designed and constructed to evaluate the accuracy of the answer generated by the target model relative to the correct answer. For any candidate difficult question among multiple candidate difficult questions, the answer verification model is used to evaluate the answer accuracy of the answer generated by the target model for each candidate difficult question. The answer accuracy is used to indicate the probability that the target model can correctly answer the candidate difficult question.

[0111] Adopting the answer verification model to determine the target model's answer accuracy relative to the candidate difficult questions can accurately evaluate the target model's performance in handling the candidate difficult questions and provide guidance for the improvement of the target model.

[0112] S208: Delete at least one candidate difficult question whose answer accuracy rate meets the second preset condition from the candidate question set among the multiple candidate difficult questions to obtain an updated candidate question set.

[0113] The specific content of the second preset condition is determined, for example, a threshold value of the correct answer rate. The second preset condition is used to evaluate whether the performance of the target model on the candidate difficult questions is good enough so that the questions that the target model can correctly answer can be removed from the candidate difficult set.

[0114] All candidate difficult questions whose answer accuracy meets the second preset condition are deleted from the candidate question set, and the remaining questions after the deletion are formed into a new candidate question set to obtain an updated candidate question set.

[0115] Updating the candidate problem set can effectively determine which problems are difficult for the target model and retain them in the candidate difficult problem set for subsequent training, thereby helping the target model to continuously overcome its shortcomings and improve its performance in specific areas.

[0116] S209: When there is a candidate difficult question in the updated candidate question set, determine a correct answer to the corresponding candidate difficult question.

[0117] S210: Add the candidate difficult questions and the correct answers to the candidate difficult questions to the sample data set to obtain a target sample data set of the target model.

[0118] Among them, after preliminary screening, those questions whose predicted target model answers have a higher accuracy rate are removed from the candidate difficult question set, leaving those questions with a lower target model prediction accuracy rate, and retaining the truly difficult questions as the updated candidate question set.

[0119] When there are candidate difficult questions in the updated candidate question set, for any difficult question in the candidate difficult questions, a most matching answer is selected from the output results of multiple other models as the correct answer to the candidate difficult question.

[0120] The updated candidate difficult questions and their correct answers are combined into new samples, which will be added to the sample dataset to obtain the target sample dataset of the target model. The target sample dataset will be used to further train the target model to facilitate the improvement of the target model on the problems where it performed poorly before.

[0121] By determining the target sample data set, we ensure that the problems in the target sample data set are challenging for the target model, reduce the training time and resources of the target model on the problems that have been mastered, and focus on improving the accuracy of the target model on difficult problems.

[0122] A possible implementation method is to use the sample data set as the target sample data set when there is no candidate difficult problem in the updated candidate problem set.

[0123] A possible implementation method is, when the number of difficult questions in the sample data set does not reach a preset number, the sample data set is used as the question set corresponding to the target model, and based on the question set, the target sample data set corresponding to the target model is re-iterated;

[0124] When the number of difficult problems in the sample data set reaches a preset number, the iteration of the sample data set is stopped, and the sample data set corresponding to the last iteration is determined as the target sample data set corresponding to the target model.

[0125] Wherein, when the number of difficult problems in the sample data set does not reach the preset number, the sample data set is used as the problem set corresponding to the target model, and the problem set is used as the basis for the next round of iteration, the sample data set corresponding to the target model is re-determined, and the problem set is clustered and analyzed to obtain multiple first problem classes corresponding to the problem set; based on the multiple first problem classes and the problem set, the candidate problem set of the target model is re-determined; based on the candidate problem set and the sample data set, the target sample data set of the target model is re-determined. In this way, the target model will be more focused on difficult problems in iteration after iteration. When the number of difficult problems in the sample data set reaches the preset number, the sample data set corresponding to the target model will no longer be further iterated. This means that the target model has been able to identify enough difficult problems on the current sample data set. The sample data set obtained in the last iteration will be determined as the final target sample data set, and the target sample data set will eventually be used for model training or evaluation of the target model.

[0126] By iteratively training the target model using the target sample data set, it can be ensured that the target model can be continuously optimized on problems in a specific field until the expected performance level is reached. This method not only improves the efficiency of training, but also ensures that the performance of the target model on difficult problems is significantly improved.

[0127] This embodiment proposes a method for constructing a sample data set. The method determines a question set of a target model and inputs it into the target model and other models; based on the answers of the target model, finds out the questions that are answered incorrectly as difficult questions, selects the most matching answers from the answers of other models as correct answers, and forms a first sample data set with the difficult questions and their correct answers; clusters the question set, divides the question categories, calculates the accuracy of the target model in each question category, and marks the question categories with low accuracy as difficult question categories; selects candidate questions based on the difficult question categories, evaluates the accuracy of the target model's answers to the candidate questions, removes candidate questions with low accuracy, and updates the candidate question set; adds the remaining candidate questions and their correct answers to the sample data set to form a target sample data set, thereby reducing the sample construction cost and improving efficiency and accuracy.

[0128] Figure 3 This is a flow chart of a method for constructing a sample data set provided in an embodiment of the present application. Figure 3 This embodiment is in Figure 1 Based on the embodiment, a possible implementation method for determining a candidate question set of a target model based on multiple first question types and question sets is described in detail. Figure 3 As shown, the method comprises:

[0129] S301. Determine the accuracy of a target model relative to each first question class based on first answers to multiple questions and multiple first question classes.

[0130] Among them, a cluster analysis is performed on the question set to obtain multiple first question classes corresponding to the question set, that is, what types of questions are there in the question set, and based on the first answer output by the target model and multiple second answers output by multiple other models, the accuracy of the target model for each of the multiple first question classes is determined.

[0131] Calculate the accuracy of the target model's answers for each question category, that is, the proportion of correct answers given by the target model in each category. Filter out question categories that are difficult for the target model to answer, so as to facilitate the subsequent generation of more challenging sample sets.

[0132] S302: taking at least one first question class whose accuracy meets a first preset condition as a difficult question class.

[0133] A preset condition is set as a threshold of the accuracy rate, for example, the accuracy rate is lower than a certain percentage value. According to the accuracy rate, at least one first question class with an accuracy rate lower than the preset condition is selected as a difficult question class. The difficult question class indicates that the target model performs relatively poorly in answering such questions.

[0134] S303: Generate at least one second problem class based on at least one difficult problem class, where the second problem class is an associated class of the difficult problem class.

[0135] Based on at least one difficult problem class, at least one second problem class similar to the at least one difficult problem class is generated using the model. The second problem class belongs to the same field as the difficult problem class and has similar difficulty and challenge to the target model.

[0136] S304: Generate a plurality of candidate difficult questions corresponding to at least one difficult question class and at least one second question class, and use the plurality of candidate difficult questions as a candidate question set.

[0137] Among them, at least one difficult problem class and at least one second problem class together form a candidate difficult problem class set. For any class in the candidate difficult problem class set, the model is used to generate multiple new difficult problems belonging to the class, and these new difficult problems form a new candidate problem set for subsequent training of the target model.

[0138] Customize sample sets according to the specific weak problem categories of the target model to improve training results. Generate new candidate difficult problems iteratively to continuously push the learning boundaries of the target model and expand its capabilities.

[0139] This embodiment proposes a method for constructing a sample data set, which calculates the accuracy of a target model on different problem categories and identifies difficult problem categories that meet preset conditions; for each identified difficult problem category, determines a second problem category related to it; generates candidate difficult problems from the difficult problem category and the associated categories to form a candidate problem set, and improves the accuracy of the model in a targeted manner by accurately locating the deficiencies of the model in specific problem categories and expanding the problem range through association analysis.

[0140] Figure 4 A schematic diagram of a structure of a sample data set construction device provided in this application, such as Figure 4 As shown, the sample data set construction device 400 provided in this embodiment includes:

[0141] A determination module 401 is used to determine a question set corresponding to a target model, and based on the question set, determine a sample data set corresponding to the target model, wherein the sample data set includes a plurality of difficult questions and a correct answer corresponding to each difficult question, wherein the difficult question is a question that the target model answers incorrectly;

[0142] A processing module 402 is used to perform cluster analysis on the question set to obtain a plurality of first question classes corresponding to the question set;

[0143] The determination module 401 is further configured to determine a candidate question set of the target model based on the multiple first question types and the question set;

[0144] The processing module 402 is further configured to determine a target sample data set of the target model based on the candidate question set and the sample data set.

[0145] Optionally, the processing module 402 is further configured to input multiple questions in the question set into the target model and multiple other models to obtain a first answer output by the target model and a second answer output by each of the other models;

[0146] The determination module 401 is further configured to determine the plurality of difficult questions from the question set according to the first answers to the plurality of questions;

[0147] The determination module 401 is further configured to determine, for any one of the multiple difficult questions, a correct answer corresponding to the difficult question from the multiple second answers, wherein the correct answer is the answer with the highest reply matching degree among the multiple second answers;

[0148] The processing module 402 is further configured to use the multiple difficult questions and the correct answer corresponding to each difficult question as a sample data set corresponding to the target model.

[0149] Optionally, the determination module 401 is further configured to determine the accuracy of the target model relative to each first question class according to the first answers to the multiple questions and the multiple first question classes;

[0150] The processing module 402 is further configured to take at least one first problem class whose accuracy meets the first preset condition as a difficult problem class;

[0151] The determination module 401 is further configured to determine the candidate question set based on the at least one difficult question class.

[0152] Optionally, the processing module is further used to generate at least one second problem class based on at least one difficult problem class, where the second problem class is an associated class of the difficult problem class;

[0153] The processing module 402 is further configured to generate a plurality of candidate difficult problems corresponding to the at least one difficult problem class and the at least one second problem class, and use the plurality of candidate difficult problems as the candidate problem set.

[0154] Optionally, the processing module 402 is further configured to determine, for any one of the multiple candidate difficult questions, a correct answer rate of the target model relative to the candidate difficult question using an answer verification model, wherein the correct answer rate is used to indicate a probability that the target model can correctly answer the candidate difficult question;

[0155] The processing module 402 is further configured to delete at least one candidate difficult question whose answer accuracy rate satisfies a second preset condition from the candidate question set, to obtain an updated candidate question set;

[0156] The determination module 401 is further configured to determine the correct answer to the corresponding difficult candidate question if there is a difficult candidate question in the updated candidate question set;

[0157] The processing module 402 is further configured to add the candidate difficult questions and the correct answers to the candidate difficult questions to the sample data set to obtain a target sample data set of the target model.

[0158] Optionally, the processing module 402 is further configured to use the sample data set as the target sample data set when there is no candidate difficult problem in the updated candidate problem set.

[0159] Optionally, the processing module 402 is further configured to, when the number of difficult questions in the sample data set does not reach a preset number, use the sample data set as a question set corresponding to the target model, and re-iterate the target sample data set corresponding to the target model based on the question set;

[0160] The processing module 402 is further configured to stop the iteration of the sample data set when the number of difficult questions in the sample data set reaches a preset number;

[0161] The determination module 401 is further configured to determine the sample data set corresponding to the last iteration as the target sample data set corresponding to the target model.

[0162] The device for constructing a sample data set provided in this embodiment can execute the method for constructing a sample data set provided in the above method embodiment. The implementation principle and technical effect are similar and will not be described in detail in this embodiment.

[0163] Figure 5 This is a schematic diagram of a sample data set construction device provided in this application. Figure 5 As shown, the present application provides a device for constructing a sample data set. The device 500 for constructing a sample data set includes: a receiver 501 , a transmitter 502 , a processor 503 and a memory 504 .

[0164] Receiver 501, used for receiving instructions and data;

[0165] A transmitter 502, used for sending instructions and data;

[0166] Memory 504, used to store computer-executable instructions;

[0167] The processor 503 is used to execute the computer-executable instructions stored in the memory 504 to implement the various steps performed by the method for constructing a sample data set in the above embodiment. For details, please refer to the relevant description in the above embodiment of the method for constructing a sample data set.

[0168] Optionally, the memory 504 may be independent or integrated with the processor 503 .

[0169] When the memory 504 is independently provided, the electronic device further includes a bus for connecting the memory 504 and the processor 503 .

[0170] The device for constructing a sample data set provided in this embodiment can execute the method for constructing a sample data set provided in the above method embodiment. The implementation principle and technical effect are similar and will not be described in detail in this embodiment.

[0171] The present application also provides a computer storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, a method for constructing a sample data set as executed by the sample data set construction device described above is implemented.

[0172] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0173] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0174] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for constructing a sample data set, characterized in that: The method comprises: Determine a question set corresponding to the target model, and based on the question set, determine a sample data set corresponding to the target model, wherein the sample data set includes a plurality of difficult questions and a correct answer corresponding to each difficult question, wherein the difficult question is a question that the target model answers incorrectly; Performing cluster analysis on the question set to obtain a plurality of first question classes corresponding to the question set; Determining a candidate question set for the target model based on the plurality of first question classes and the question set; Based on the candidate question set and the sample data set, a target sample data set of the target model is determined.

2. The method according to claim 1, characterized in that The step of determining the sample data set corresponding to the target model based on the problem set includes: Inputting a plurality of questions in the question set into the target model and a plurality of other models to obtain a first answer output by the target model and a second answer output by each of the other models; determining the plurality of difficult questions from the question set based on first answers to the plurality of questions; For any one of the multiple difficult questions, determine a correct answer corresponding to the difficult question from the multiple second answers, wherein the correct answer is the answer with the highest reply matching degree among the multiple second answers; The multiple difficult questions and the correct answer corresponding to each difficult question are used as a sample data set corresponding to the target model.

3. The method according to claim 2, characterized in that The step of determining a candidate question set of the target model based on the plurality of first question types and the question set comprises: Determine, according to the first answers to the plurality of questions and the plurality of first question classes, the accuracy of the target model relative to each first question class, and take at least one first question class whose accuracy meets a first preset condition as a difficult question class; The candidate question set is determined based on the at least one difficult question class.

4. The method according to claim 3, characterized in that: The step of determining the candidate problem set based on the at least one difficult problem class comprises: Based on at least one difficult problem class, generating at least one second problem class, where the second problem class is an associated class of the difficult problem class; A plurality of candidate difficult questions corresponding to the at least one difficult question class and the at least one second question class are generated, and the plurality of candidate difficult questions are used as the candidate question set.

5. The method according to claim 4, characterized in that: The step of determining a target sample data set of the target model based on the candidate question set and the sample data set includes: For any one of the plurality of candidate difficult questions, using an answer verification model to determine the correctness of the answer of the target model relative to the candidate difficult question, the correctness of the answer being used to indicate the probability that the target model can correctly answer the candidate difficult question; Deleting at least one candidate difficult question whose answer accuracy rate satisfies a second preset condition from the candidate question set among the multiple candidate difficult questions, to obtain an updated candidate question set; In the case where there is a candidate difficult question in the updated candidate question set, determining a correct answer to the corresponding candidate difficult question; The candidate difficult questions and the correct answers to the candidate difficult questions are added to the sample data set to obtain a target sample data set for the target model.

6. The method according to claim 4, characterized in that The method further comprises: When there is no candidate difficult problem in the updated candidate problem set, the sample data set is used as the target sample data set.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When the number of difficult questions in the sample data set does not reach a preset number, the sample data set is used as a question set corresponding to the target model, and based on the question set, the target sample data set corresponding to the target model is re-iterated; When the number of difficult problems in the sample data set reaches a preset number, the iteration of the sample data set is stopped, and the sample data set corresponding to the last iteration is determined as the target sample data set corresponding to the target model.

8. A device for constructing a sample data set, characterized in that: The device comprises: A determination module, used to determine a question set corresponding to a target model, and based on the question set, determine a sample data set corresponding to the target model, wherein the sample data set includes a plurality of difficult questions and a correct answer corresponding to each difficult question, wherein the difficult question is a question that the target model answers incorrectly; A processing module, configured to perform cluster analysis on the question set to obtain a plurality of first question classes corresponding to the question set; The determination module is further configured to determine a candidate question set of the target model based on the multiple first question categories and the question set; The processing module is further configured to determine a target sample data set of the target model based on the candidate question set and the sample data set.

9. A device for constructing a sample data set, characterized in that: include: Memory; processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method for constructing a sample data set according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The computer storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method for constructing a sample data set according to any one of claims 1 to 7.

Images