Data set adjustment method and device, equipment and medium

Abstract

The invention discloses a data set adjustment method and device, equipment and a medium. The method comprises the following steps: extracting a target number of candidate sets from an original data set; for each candidate set, performing addition and deletion adjustment on data in the candidate set to generate an alternative set corresponding to the candidate set; obtaining the accuracy of at least one test model for each alternative set; screening the target number of alternative sets according to the accuracy rate of each alternative set, and updating the alternative sets; and determining a target data set according to the accuracy of each candidate set. According to the embodiment of the invention, the evaluation efficiency of the data set can be improved while the representativeness of the data set is considered.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a data set adjustment method, apparatus, device and medium. Background Art

[0002] Large language models represent a groundbreaking development in AI technology, demonstrating groundbreaking performance across multiple capabilities. The evaluation of large language models relies on large-scale, cross-domain evaluation datasets to fully demonstrate their performance in various scenarios.

[0003] However, in the current large language model evaluation, the operation of large language models requires high computing power costs and the evaluation time is long, which will reduce the efficiency of large language model analysis iteration. Summary of the Invention

[0004] The present invention provides a data set adjustment method, apparatus, device and medium, which can improve the evaluation efficiency of the data set while taking into account the representativeness of the data set.

[0005] In a first aspect, an embodiment of the present invention provides a method for adjusting a data set, the method comprising:

[0006] Extract the target number of candidate sets from the original dataset;

[0007] For each candidate set, adding, deleting, and adjusting the data in the candidate set to generate an alternative set corresponding to the candidate set;

[0008] Obtaining the accuracy of at least one test model for each of the candidate sets;

[0009] According to the accuracy of each candidate set, the target number of candidate sets is screened and the candidate set is updated;

[0010] According to the accuracy of each candidate set, a target data set is determined. In a second aspect, an embodiment of the present invention further provides a data set adjustment device, which includes:

[0011] The candidate set extraction module is used to extract the target number of candidate sets from the original data set;

[0012] An alternative set generation module is used to add, delete and adjust the data in the candidate set for each candidate set to generate an alternative set corresponding to the candidate set;

[0013] an accuracy calculation module, configured to obtain the accuracy of at least one test model for each of the candidate sets;

[0014] A candidate set screening module is used to screen the target number of candidate sets according to the accuracy of each candidate set and update the candidate set;

[0015] The data set adjustment module is used to determine the target data set according to the accuracy of each candidate set.

[0016] In a third aspect, an embodiment of the present invention further provides a data set adjustment device, the data set adjustment device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the data set adjustment method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions, which are used to enable a processor to implement the data set adjustment method according to any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention can reduce a large amount of evaluation data and improve the evaluation efficiency of the evaluation data set by extracting a smaller number of candidate sets from the original data set and screening out the target data set therefrom. By randomly deleting the candidate sets, generating alternative sets, and calculating the accuracy corresponding to each alternative set processed by the test model, and screening out new candidate sets from multiple alternative sets, it can be ensured that the accuracy of the screened data set is high, so that a smaller data set can obtain an evaluation effect with a higher accuracy, making the data set more representative, and achieving the goal of improving the evaluation efficiency while making the evaluation results reliable.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a data set adjustment method provided according to an embodiment of the present invention;

[0025] Figure 2is a flow chart of a data set adjustment method provided according to an embodiment of the present invention;

[0026] Figure 3 is a structural diagram of a data set adjustment device provided according to an embodiment of the present invention;

[0027] Figure 4 It is a structural diagram of a data set adjustment device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] In the technical solutions of the embodiments of the present invention, the acquisition, storage and application of driving trajectory points, etc., all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] Figure 1 This is a flow chart of a dataset adjustment method provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the case of dataset adjustment. The method can be executed by a dataset adjustment device, which can be implemented in the form of hardware and / or software.

[0032] See also Figure 1 The dataset adjustment method shown includes:

[0033] S101. Extract a target number of candidate sets from the original data set.

[0034] The original dataset refers to a large-scale collection of data covering a wide range of domains. This dataset is used to test the large language model. Specifically, it can test the accuracy, speed, and completeness of the output of the large language model. The target number can be set as needed, for example, 10. A candidate set refers to a set with a smaller amount of data than the original dataset. The data size of different candidate sets can be the same or different. The original dataset contains much more data than the target number.

[0035] In some embodiments, a random number of data may be randomly sampled sequentially from the original dataset, and each sampled data may be used to generate a candidate set. Alternatively, the original dataset may be divided into multiple datasets, each dataset including a random number of data, and at least one candidate set may be randomly sampled from the multiple datasets.

[0036] S102: For each candidate set, add, delete, and adjust the data in the candidate set to generate an alternative set corresponding to the candidate set.

[0037] Adjusting the data in the candidate set by adding, deleting, or deleting data can refer to adding and / or deleting any number of arbitrary data in the candidate set. The data to be added, deleted, or adjusted is randomly determined, and the number of data to be added, deleted, or adjusted is at least one. The candidate set retains at least one data item in the candidate set, i.e., the intersection between the candidate set and the candidate set is non-empty. One candidate set is generated for each candidate set. Multiple candidate sets are generated for each candidate set. In some embodiments, the number of data items to be added, deleted, or adjusted is at least one.

[0038] In an optional embodiment, the adding, deleting and adjusting the data in the candidate set includes at least one of the following: randomly deleting data in the candidate set; and randomly adding data in the candidate set, wherein the added data is different from the data in the candidate set.

[0039] A random number of data points are randomly selected from the candidate set and deleted, resulting in a candidate set as the candidate set. A random number of data points are randomly generated and added to the candidate set to obtain a candidate set, which is also the candidate set. A random number of data points are randomly selected from the candidate set and deleted, while a random number of data points are randomly obtained and added to the candidate set to obtain a candidate set as the candidate set. The obtained data can be randomly selected from the original dataset.

[0040] In some embodiments, at least one piece of data is randomly deleted from the candidate set, and the deleted candidate set is used as an alternative set. At least one piece of data that is completely different from or not completely identical to the previously selected data is randomly deleted from the candidate set, and the deleted candidate set is used as an alternative set. This process is repeated multiple times to obtain multiple alternative sets. The number of data items included in the alternative set can be limited to be greater than or equal to a lower limit, for example, a lower limit of 10.

[0041] In some embodiments, at least one piece of data is randomly acquired, and the acquired data is different from any data in the candidate set, and the generated data are different from each other. The acquired data is added to the candidate set to obtain an alternative set. At least one piece of data that is completely different or not completely identical to the aforementioned data is randomly acquired, and the acquired data is different from any data in the candidate set, and the acquired data are different from each other. The acquired data is added to the candidate set to obtain an alternative set. This process is repeated multiple times to obtain multiple alternative sets. The number of data included in the alternative set can be limited to less than or equal to an upper limit, for example, an upper limit of 100.

[0042] In some embodiments, at least one piece of data may be randomly deleted from the candidate set, and at least one randomly obtained and different piece of data may be added to obtain an alternative set.

[0043] In which, a piece of data is a question, and random addition and deletion of data can refer to random addition and deletion of questions. In an example, for the i-th candidate set generated at the t-th time or the t-th round At the t+1th moment or the t+1th round, generate 5 new candidate sets Each new alternative set is created by Generated by random addition and deletion operations. The addition and deletion operations are as follows:

[0044] Among them, the deletion operation: Randomly delete a question and ask

[0045] Add operation: A non-repeated question is randomly added to the question.

[0046] It can be seen that by adding and / or deleting data in the candidate set to generate the alternative set, random sampling of the candidate set can be achieved, so that the obtained alternative set has randomness.

[0047] S103: Obtain the accuracy of at least one test model for each of the candidate sets.

[0048] Each test model can be a generative model of a different structure, such as a large language model of a different structure. Each test model needs to have the same function as the large language model to be tested. A candidate set is input into the test model to obtain the accuracy of the test model for the candidate set. The number of test models is as large as possible. For example, the number of test models is greater than or equal to 50, the number of test models is greater than or equal to 100, etc.

[0049] S104 : Filter the target number of candidate sets according to the accuracy of each candidate set, and update the candidate set.

[0050] A target number of datasets can be selected from multiple candidate sets based on the accuracy of each test model for each candidate set and used as new candidate sets. Alternatively, a target number of datasets can be selected from multiple candidate sets and candidate sets based on the accuracy of each test model for each candidate set and the accuracy of each test model for each candidate set and used as new candidate sets. The new candidate set can replace the previous candidate set or be added to the previous candidate set.

[0051] In one example, the candidate set includes 10 datasets: A1, A2, ..., A10. Based on these 10 datasets, at least one candidate set is added or deleted to obtain Bn, where n ranges from 10 to 20. For example, based on the 10 candidate sets A1-A10, candidate sets B1, B2, ..., B10 are generated. From the 20 datasets A1-A10 and B1-B10, 10 datasets are selected as new candidate sets. These new candidate sets are then added or deleted to obtain new candidate sets, which are then filtered again, achieving multiple iterations of the candidate set.

[0052] S105: Determine the target data set according to the accuracy of each candidate set.

[0053] Based on the accuracy of the candidate sets, at least one candidate set with the highest accuracy is selected as the target dataset. The target dataset is used as input to the large language model to be tested to test the performance of the large language model.

[0054] In some embodiments, the process can be repeated to add, delete, and adjust the data in each candidate set to generate an alternative set. A target number of new candidate sets can then be selected from the candidate and alternative sets, implementing multiple iterations of candidate set selection. When the number of iterations reaches a preset threshold, the target dataset is determined based on the accuracy of each candidate set.

[0055] In some embodiments, the target data set may be determined from a candidate set screened from a plurality of candidate sets and a plurality of alternative sets.

[0056] In an optional embodiment, after determining the target data set based on the accuracy of each candidate set, the method further includes: testing the large language model based on the target data set; wherein the large language model is different from each test model.

[0057] The target dataset is input into the large language model for processing, resulting in the output of the large language model. Typically, the large language model to be evaluated differs from the test models. Based on the target dataset and the output, the performance test results of the large language model are determined. Based on the performance test results, it is determined whether the large language model should be iteratively optimized or whether the large language model can be released for application.

[0058] In one example, the performance test result is accuracy. When the accuracy is greater than or equal to a preset accuracy threshold, it is determined that the large language model does not require iterative optimization and the application can be released. When the accuracy is less than or equal to the preset accuracy threshold, it is determined that the large language model requires iterative optimization and the application cannot be released.

[0059] It can be seen that by using a small-scale target dataset with high representativeness and comparative significance to test the large language model, the amount of data processed in the test of the large language model is reduced, the test efficiency is improved, and the test results of the large language model are as reliable as the original dataset, thereby improving the accuracy and reliability of the test results.

[0060] In an optional embodiment, the original data set includes: multiple questions.

[0061] A piece of data is the natural language content of a question, and the original dataset contains a large number of questions. Adding, deleting, and adjusting data is equivalent to adding, deleting, and adjusting questions. The large language model outputs the natural language text of the answer to the question corresponding to a piece of data.

[0062] In one example, the raw data includes the Massive Multitask Language Understanding (MMLU) dataset.

[0063] In reality, the MMLU test contains 14,042 evaluation questions, and evaluating a large speech model on the MMLU typically takes 3-4 hours. Experiments show that using the dataset adjustment method proposed in this embodiment of the present invention, a target dataset containing 100 questions can be selected to test a large language model. This method can maximize the restoration of the large language model's MMLU evaluation score, achieving an average standard error of 1.61 and an average error of 2.17, while significantly reducing the test time to less than 3-4 minutes.

[0064] It can be seen that by configuring data as questions, the language understanding and question-answering functions of the large language model can be efficiently tested, avoiding the lengthy testing process, improving the efficiency of the performance evaluation of the large language model, and enabling the optimization of the large language model to be completed more quickly.

[0065] The technical solution of the embodiment of the present invention can reduce a large amount of evaluation data and improve the evaluation efficiency of the evaluation data set by extracting a smaller number of candidate sets from the original data set and screening out the target data set therefrom. By randomly deleting the candidate sets, generating alternative sets, and calculating the accuracy corresponding to each alternative set processed by the test model, and screening out new candidate sets from multiple alternative sets, it can be ensured that the accuracy of the screened data set is high, so that a smaller data set can obtain an evaluation effect with a higher accuracy, making the data set more representative, and achieving the goal of improving the evaluation efficiency while making the evaluation results reliable.

[0066] Figure 2 A flowchart of a dataset adjustment method provided by an embodiment of the present invention. Based on the above embodiment, the embodiment of the present invention refines "screening the target number of alternative sets based on the accuracy of each alternative set and updating the candidate set" to "obtaining the accuracy of each test model for the original dataset; calculating the accuracy difference of each test model for each alternative set based on the accuracy of each test model for each alternative set and the accuracy of each test model for the original dataset; calculating the accuracy variance of each alternative set based on the accuracy variance of each test model for each alternative set; and screening the target number of alternative sets based on the accuracy variance of each alternative set and updating the candidate set."

[0067] It should be noted that for parts not described in detail in the embodiments of the present invention, reference may be made to the descriptions of other embodiments.

[0068] See also Figure 2 The dataset adjustment method shown includes:

[0069] S201: Extract a target amount of data from the original data set multiple times to generate at least one candidate set.

[0070] S202: For each candidate set, add, delete, and adjust the data in the candidate set to generate an alternative set corresponding to the candidate set.

[0071] S203: Obtain the accuracy of at least one test model for each of the candidate sets.

[0072] Among them, an alternative set is input into a test model for processing, and the output result of the test model for the alternative set is obtained. Based on the output result, the accuracy is calculated to obtain the accuracy of the test model for the alternative set. Each alternative set is input into each test model for processing, and the accuracy of each test model for each alternative set is calculated. Alternatively, the accuracy of each test model for each data in the original data set can be obtained in advance. When calculating the accuracy of a test model for an alternative set, the accuracy of the test model for each data in the alternative set can be obtained, and the accuracy of the alternative set can be obtained by statistics. In some embodiments, the average accuracy of the test model for all data in the alternative set can be calculated as the accuracy of the test model for the alternative set.

[0073] The test model is a large language model, and the candidate set includes multiple questions. The accuracy of the test model for this candidate set can refer to the accuracy of the answers output by the test model for the candidate set. The accuracy of the output answers can be calculated based on the similarity between the output answers and the standard answers to the corresponding questions. The accuracy can be calculated for each question and then statistically calculated to obtain the accuracy of the candidate set.

[0074] S204: Obtain the accuracy of each test model for the original data set.

[0075] The original data set is input into the test model to obtain the accuracy of the test model for the original data set. The accuracy of the test model for each data item in the original data set can be calculated by calculating the mean based on the pre-calculated accuracy of the test model for the original data set, and then the accuracy of the test model for the original data set can be obtained by statistics.

[0076] S205 , calculating the accuracy difference of each test model for each candidate set based on the accuracy of each test model for each candidate set and the accuracy of each test model for the original data set.

[0077] Among them, the accuracy difference of the alternative set is used to measure the difference between the accuracy of the same test model for the alternative set and the accuracy of the original dataset, which is equivalent to describing the representativeness of the alternative set relative to the original dataset.

[0078] S206 , calculating the accuracy variance of each candidate set according to the accuracy difference of each test model for each candidate set.

[0079] Among them, the accuracy variance is used to describe the degree of dispersion of the accuracy differences of different test models for the same candidate set.

[0080] S207 : Filter the target number of candidate sets according to the accuracy variance of each candidate set, and update the candidate set.

[0081] Among them, the target number of candidate sets with the smallest accuracy variance can be selected to update the candidate set.

[0082] S208: Determine the target data set according to the accuracy of each candidate set.

[0083] In an optional embodiment, the calculating the accuracy variance of each alternative set based on the accuracy difference of each test model for each alternative set includes: for each alternative set, calculating the accuracy variance of each alternative set based on the accuracy difference of each test model for the alternative set and the number of test models.

[0084] For each candidate set, the sum of the squares of the accuracy differences of each test model for that candidate set is calculated, the ratio between the sum of squares and the number of test models is calculated, and the square root of the ratio is calculated to obtain the accuracy variance. The accuracy variance can be calculated based on the following formula:

[0085]

[0086] Where M is the number of test models. ΔAcc is the absolute value of the accuracy difference. t is the tth iteration. k is the kth candidate set. j is the jth test model. i is the ith candidate set. The variance of the accuracy of the kth candidate set generated for the i-th candidate set in round t+1. The absolute value of the accuracy difference between the kth candidate set generated by the jth test model for the i-th candidate set in the t+1th round.

[0087] It can be seen that by calculating the accuracy difference of the alternative sets and the number of test models, the accuracy variance of multiple test models for the same alternative set is calculated. The target data set is screened based on the discrete degree of the accuracy difference of multiple test models for the same alternative set. This can reduce the inaccuracy of iteratively updating the target data set with a single test model, thereby improving the orderliness and guidance of the selection of the target data set.

[0088] In an optional embodiment, determining the target data set according to the accuracy of each candidate set includes: obtaining the update number of the candidate set; and filtering the target data set from the current multiple candidate sets when the update number meets the iteration number condition.

[0089] The number of updates may refer to the number of times the candidate set is updated based on the selected candidate set. The number of updates is equivalent to the number of iterations of the candidate set. The iteration number condition is used to determine whether the number of updates is greater than or equal to the number threshold. When the number of updates is greater than or equal to the number threshold, the number of updates is determined to meet the iteration number condition. When the number of updates is less than the number threshold, the number of updates is determined to not meet the iteration number condition. The iteration number condition can be determined based on the domain to which the data in the original dataset belongs.

[0090] The current multiple candidate sets can be the latest candidate sets. The top n candidate sets with the smallest accuracy variance are selected from the candidate sets as the target dataset. Here, n can be determined based on the scenario of the problem, for example, n is 1.

[0091] It can be seen that by using the number of updates of the candidate set as the basis for stopping iteration, the iteration process can be controlled to avoid too many redundant iterations and waste of resources, as well as too few iterations and reduce the representativeness of the target dataset.

[0092] The embodiment of the present invention calculates the accuracy difference between the accuracy of each alternative set of each test model and the accuracy of the original data set, and calculates the accuracy variance of each alternative set based on the accuracy difference of each alternative set, thereby screening out a target number of alternative sets based on the accuracy variance, updating the candidate set, and screening out alternative sets that are close to the test results of the original data set, thereby increasing the representativeness of the screened alternative sets.

[0093] In one example, a set of test models M and an original dataset D are obtained. Ten initial candidate sets are randomly selected from the original dataset, S i represents the i-th candidate set, |S i |=10. The number threshold is T, which is equivalent to performing T iterations, with the initial time t=0. In each iteration, for each candidate set Generate 5 new alternative sets Each new alternative set is created by Generated by random addition and deletion operations. The addition and deletion operations are as follows:

[0094] Delete operation: From Randomly delete a question and ask

[0095] Add operation: A non-repeated question is randomly added to the question.

[0096] For each generated alternative set Calculate the accuracy of each test model on the candidate set The accuracy of each test model on the original data set Acc(D) and the absolute value of the difference between the two Then calculate the RMSE value corresponding to the alternative set:

[0097]

[0098] In each iteration, 10 candidate sets with the smallest corresponding RMSE values ​​generated in this iteration are selected as new candidate sets.

[0099] After T iterations, the candidate set with the minimum RMSE value is output as the target dataset.

[0100] The embodiment of the present invention can achieve the effect of random sampling through flexible dynamic addition and deletion operations and iterative optimization of candidate sets; by comparing the results obtained with the original data set, a candidate data alternative set with greater comparative significance and reference value is obtained; the root mean square error (RSEM) is used to screen the target data set, which is more scientific than the alternative set obtained by completely random sampling, and can find the candidate set that meets the requirements more quickly under certain guidance, thereby achieving the goal of miniaturization of the evaluation set, making the final target data set scientific and representative, thereby shortening the evaluation time, reducing the computing cost, and achieving high efficiency, scientificity and scalability.

[0101] Figure 3 This is a schematic diagram of the structure of a data set adjustment device provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the case of data set adjustment. The device can execute a data set adjustment method and can be implemented in the form of hardware and / or software.

[0102] See also Figure 3 The data set adjustment device shown includes: a candidate set extraction module 301, an alternative set generation module 302, an accuracy calculation module 303, a candidate set screening module 304 and a data set adjustment module 305, wherein,

[0103] The candidate set extraction module 301 is used to extract a target number of candidate sets from the original data set;

[0104] The candidate set generation module 302 is used to add, delete and adjust the data in the candidate set for each candidate set to generate a candidate set corresponding to the candidate set;

[0105] An accuracy calculation module 303 is used to obtain the accuracy of at least one test model for each candidate set;

[0106] The candidate set screening module 304 is configured to screen the target number of candidate sets according to the accuracy of each candidate set and update the candidate set;

[0107] The data set adjustment module 305 is configured to determine a target data set according to the accuracy of each candidate set.

[0108] The technical solution of the embodiment of the present invention can reduce a large amount of evaluation data and improve the evaluation efficiency of the evaluation data set by extracting a smaller number of candidate sets from the original data set and screening out the target data set therefrom. By randomly deleting the candidate sets, generating alternative sets, and calculating the accuracy corresponding to each alternative set processed by the test model, and screening out new candidate sets from multiple alternative sets, it can be ensured that the accuracy of the screened data set is high, so that a smaller data set can obtain an evaluation effect with a higher accuracy, making the data set more representative, and achieving the goal of improving the evaluation efficiency while making the evaluation results reliable.

[0109] Optionally, the candidate set generation module 302 is specifically configured to perform at least one of the following:

[0110] Randomly delete data from the candidate set; and

[0111] Data is randomly added to the candidate set, wherein the added data is different from each data in the candidate set.

[0112] Optionally, the candidate set screening module 304 is specifically configured to:

[0113] Obtaining the accuracy of each test model for the original data set;

[0114] Calculating the accuracy difference of each test model for each candidate set according to the accuracy of each test model for each candidate set and the accuracy of each test model for the original data set;

[0115] Calculating the accuracy variance of each candidate set according to the accuracy difference of each test model for each candidate set;

[0116] According to the accuracy variance of each candidate set, the target number of candidate sets is screened, and the candidate set is updated.

[0117] Optionally, the candidate set screening module 304 is specifically configured to:

[0118] For each candidate set, the accuracy variance of each candidate set is calculated according to the accuracy difference of each test model for the candidate set and the number of test models.

[0119] Optionally, the data set adjustment module 305 is specifically configured to:

[0120] Get the update count of the candidate set;

[0121] When the number of updates meets the iteration number condition, the target data set is filtered from the current multiple candidate sets.

[0122] Optionally, the data set adjustment device further includes:

[0123] The model testing module is used to test the large language model based on the target data set after determining the target data set according to the accuracy of each candidate set; wherein the large language model is different from each test model.

[0124] Optionally, the original data set includes: multiple questions.

[0125] The data set adjustment device provided in the embodiment of the present invention can execute the data set adjustment method provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the data set adjustment method.

[0126] Example 4

[0127] Figure 4 FIG. 4 is a schematic structural diagram of a data set adjustment device 400 that can be used to implement an embodiment of the present invention.

[0128] like Figure 4 As shown, data set adjustment device 400 includes at least one processor 401 and a memory, such as a read-only memory (ROM) 402 or a random access memory (RAM) 403, communicatively connected to the at least one processor 401. The memory stores a computer program executable by the at least one processor. Processor 401 can perform various appropriate actions and processes based on the computer program stored in ROM 402 or loaded from storage unit 408 into RAM 403. RAM 403 can also store various programs and data required for the operation of data set adjustment device 400. Processor 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0129] Multiple components in the dataset adjustment device 400 are connected to an I / O interface 405, including an input unit 406, such as a keyboard and mouse; an output unit 407, such as various types of displays and speakers; a storage unit 408, such as a magnetic disk and optical disk; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the dataset adjustment device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0130] Processor 401 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. Processor 501 executes the various methods and processes described above, such as the dataset adjustment method.

[0131] In some embodiments, the dataset adjustment method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the dataset adjustment device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by processor 401, one or more steps of the dataset adjustment method described above may be performed. Alternatively, in other embodiments, processor 401 may be configured to perform the dataset adjustment method in any other appropriate manner (e.g., via firmware).

[0132] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] To provide for interaction with a user, the systems and techniques described herein can be implemented on an operation detection device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the dataset adjustment device. Other types of devices can also be used to provide for interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0136] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0137] A computing system may include clients and servers. The clients and servers are generally remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS (Virtual Private Server) services.

[0138] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0139] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data set adjustment method, characterized in that: The method comprises: Extract the target number of candidate sets from the original dataset; For each candidate set, adding, deleting, and adjusting the data in the candidate set to generate an alternative set corresponding to the candidate set; Obtaining the accuracy of at least one test model for each of the candidate sets; According to the accuracy of each candidate set, the target number of candidate sets is screened and the candidate set is updated; The target data set is determined according to the accuracy of each candidate set.

2. The method according to claim 1, characterized in that Adding, deleting, and adjusting the data in the candidate set includes at least one of the following: Randomly delete data from the candidate set; and Data is randomly added to the candidate set, wherein the added data is different from each data in the candidate set.

3. The method according to claim 1, characterized in that The step of screening the target number of candidate sets according to the accuracy of each candidate set and updating the candidate set includes: Obtaining the accuracy of each test model for the original data set; Calculating the accuracy difference of each test model for each candidate set according to the accuracy of each test model for each candidate set and the accuracy of each test model for the original data set; Calculating the accuracy variance of each candidate set according to the accuracy difference of each test model for each candidate set; According to the accuracy variance of each candidate set, the target number of candidate sets is screened, and the candidate set is updated.

4. The method according to claim 3, characterized in that Calculating the accuracy variance of each candidate set according to the accuracy difference of each test model for each candidate set includes: For each candidate set, the accuracy variance of each candidate set is calculated according to the accuracy difference of each test model for the candidate set and the number of test models.

5. The method according to claim 1, wherein Determining the target data set according to the accuracy of each candidate set includes: Get the update count of the candidate set; When the number of updates meets the iteration number condition, the target data set is filtered from the current multiple candidate sets.

6. The method according to claim 1, characterized in that After determining the target dataset based on the accuracy of each candidate dataset, the method further includes: Based on the target data set, a large language model is tested; wherein the large language model is different from each of the test models.

7. The method according to claim 6, characterized in that The original data set includes: multiple questions.

8. A data set adjustment device, characterized in that: The device comprises: The candidate set extraction module is used to extract the target number of candidate sets from the original data set; An alternative set generation module is used to add, delete and adjust the data in the candidate set for each candidate set to generate an alternative set corresponding to the candidate set; an accuracy calculation module, configured to obtain the accuracy of at least one test model for each of the candidate sets; A candidate set screening module is used to screen the target number of candidate sets according to the accuracy of each candidate set and update the candidate set; The data set adjustment module is used to determine the target data set according to the accuracy of each candidate set.

9. A data set adjustment device, characterized in that: The data set adjustment device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the dataset adjustment method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data set adjustment method according to any one of claims 1 to 7 when executed.

Images