Training data validity evaluation method, system and device

Abstract

The invention provides a training data validity evaluation method, system and device, and the method comprises the steps: obtaining a training set which is obtained through uniform downsampling from to-be-evaluated data; obtaining a test set, wherein the test set comprises at least one benchmark test set and at least one associated test set; training a probe model by using the training set; testing the probe model by using the test set, and recording test indexes; making an observation diagram according to the test indexes, wherein the step of making the observation diagram comprises: establishing a rectangular coordinate system by taking the test index of the benchmark test set as a horizontal axis and the test index of the associated test set as a longitudinal axis; drawing key points in the rectangular coordinate system according to the test indexes; and performing validity evaluation on the to-be-evaluated data according to the observation map. The method has the advantages that the data validity evaluation can be quickly given with very low computing power, the model research and development iteration efficiency can be improved, and the gain is more obvious especially for a leading-edge complex model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, system and device for evaluating the effectiveness of training data. Background Art

[0002] With the development of Internet technology, artificial intelligence has become an important part of people's daily life. As artificial intelligence business rises, users' requirements for the accuracy of artificial intelligence models are also increasing. The increasing accuracy requirements have led to a rapid expansion of model size and training data volume.

[0003] The current difficulties in training a high-quality artificial intelligence model include: due to the large size of the model and the amount of training data, the computing power and time cost consumed by a single experiment are very high. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method, system and device for evaluating the effectiveness of training data. Through a small-scale probe model, it is possible to directly evaluate the effectiveness of training data. It has the obvious advantages of low computing power consumption, low time cost, accurate evaluation results and strong robustness.

[0005] In a first aspect, the present invention provides a method for evaluating the effectiveness of training data, comprising:

[0006] Obtaining a training set, wherein the training set is obtained by uniformly downsampling the data to be evaluated;

[0007] Acquire a test set, wherein the test set includes at least one benchmark test set and at least one associated test set;

[0008] training a probe model using the training set;

[0009] Testing the probe model using the test set and recording the test indicators;

[0010] Drawing an observation graph according to the test index, wherein drawing the observation graph comprises: establishing a rectangular coordinate system with the test index of the benchmark test set as the horizontal axis and the test index of the associated test set as the vertical axis; drawing key points in the rectangular coordinate system according to the test index;

[0011] The validity of the data to be evaluated is evaluated based on the observation graph.

[0012] Furthermore, the training of a probe model includes: updating the probe model weights by back propagation and gradient descent method.

[0013] Furthermore, the updating of the probe model weights by back propagation and gradient descent method includes: first converting the training corpus into tokens, sending them into the probe model for forward propagation to obtain output tokens, then calculating the cross entropy loss of the output tokens, and calculating the gradient by back propagation; after obtaining the gradient of the probe model, applying the gradient descent algorithm to update the probe model weights.

[0014] Further, the probe model is tested using the test data and the indicator calculation method, and recording the test indicators includes: recording the test indicators of the probe model in the test set in at least 2 convergence stages.

[0015] Furthermore, key points are drawn in the rectangular coordinate system according to the test indicators, including: reading the test indicator composition coordinates corresponding to the horizontal and vertical axes of an optimization stage, and drawing a key point; traversing all convergence stages, and drawing other key points.

[0016] Furthermore, the evaluation of the effectiveness of the training data based on the observation graph includes: fitting the key points with a curve; if the test index of the associated test set is as small as possible, the lower the position of the curve is, the better the effectiveness of the training data; if the test index of the associated test set is as large as possible, the higher the position of the curve is, the better the effectiveness of the training data.

[0017] In a second aspect, the present invention also provides a training data effectiveness evaluation system, the system comprising an acquisition module, a training module, a testing module and an evaluation module;

[0018] The acquisition module is used to acquire a training set and a test set; wherein the training set is obtained by uniformly downsampling the data to be evaluated, and the test set includes at least one benchmark test set and at least one associated test set;

[0019] The training module is used to train a probe model using the training set;

[0020] The test module is used to test the probe model using the test set and record the test indicators; and to make an observation diagram according to the test indicators;

[0021] The evaluation module is used to evaluate the effectiveness of the data to be evaluated based on the observation graph.

[0022] In a third aspect, the present invention further provides a training data validity assessment device, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the training data validity assessment method is implemented.

[0023] In a fourth aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the method for evaluating the effectiveness of training data is implemented.

[0024] In a fifth aspect, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the training data validity evaluation method.

[0025] The advantages of the present invention are that it can quickly provide data validity evaluation with very low computing power, which can greatly improve the efficiency of model research and development iteration, especially for cutting-edge complex models, the gain is more obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0027] Figure 1 A flowchart of a training data validity evaluation method provided in Embodiment 1 of the present invention;

[0028] Figure 2 A schematic diagram of the probe model training process provided in Example 6 of the present invention;

[0029] Figure 3 A schematic diagram of a process of drawing key points in the rectangular coordinate system provided in Embodiment 8 of the present invention;

[0030] Figure 4 This is an example of an observation diagram provided by Embodiment 8 of the present invention;

[0031] Figure 5 This is an example of an observation diagram provided in Embodiment 9 of the present invention;

[0032] Figure 6 A schematic diagram of a process for evaluating the effectiveness of data to be evaluated provided in Embodiment 9 of the present invention;

[0033] Figure 7 A structural diagram of a training data validity evaluation device provided by the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0036] Embodiment 1:

[0037] In this embodiment, a method for evaluating the effectiveness of training data is provided. Figure 1 is a flowchart of a method for evaluating the effectiveness of training data according to an embodiment of the present invention, the process comprising the following steps:

[0038] Step S101, obtaining a training set, wherein the training set is obtained by uniformly downsampling the data to be evaluated;

[0039] Step S102, obtaining a test set, wherein the test set includes at least one benchmark test set and at least one associated test set;

[0040] Step S103, training a probe model using the training set;

[0041] Step S104, testing the probe model using the test set and recording the test indicators;

[0042] Step S105, making an observation diagram according to the test indicators, the making of the observation diagram comprising: establishing a rectangular coordinate system with the test indicators of the benchmark test set as the horizontal axis and the test indicators of the associated test set as the vertical axis; drawing key points in the rectangular coordinate system according to the test indicators;

[0043] Step S106: evaluating the effectiveness of the data to be evaluated based on the observation graph.

[0044] In the above steps, there is no restriction on the form of the training set as long as it meets the business requirements. If you want to train a large language model, the training set can be in the form of natural language, mathematical symbols, codes and other texts; if you want to train a visual model, the training data can be in the form of pictures, classification labels of pictures, target detection labels of pictures, segmentation labels of pictures, etc.; if you want to train a multimodal question-answering model, the training set can be in the form of pictures or videos and the corresponding question-answer texts. This means that this method can adapt to a variety of test data and model algorithm types. The training set is obtained by uniformly downsampling the data to be evaluated, which not only maintains the data distribution of the test data, but also reduces the number of samples contained in the training set, thereby improving the operating efficiency of the method.

[0045] The test set is mainly obtained according to the requirements of downstream tasks. Similar to the training set, if you train a large language model, the test set can be common sense questions, math questions, code questions, etc. The common sense questions and math questions can use the accuracy rate as the evaluation indicator, and the code questions can use the successful operation ratio as the evaluation indicator; if you want to train a visual model, the test set is also obtained according to the required downstream task type, such as image classification test set, target detection test set, image segmentation test set; if you want to train a multimodal problem of text and images, the test set can be the text-related tasks mentioned above, the image-related tasks mentioned above, or the question-answering understanding, image generation, video understanding, video generation and other tasks combined with text and images. For models required by other businesses, the test set can also be obtained according to the business type and downstream task requirements.

[0046] In the above steps, the training set is a small part uniformly sampled from the complete training data. In addition, the probe model is also a very small model. Therefore, it can be trained in a very short time with very low computing power, thereby ensuring the efficiency of the method. The observation graph is drawn by combining the test indicators of multiple test sets, which can simultaneously reflect the response of the training set data to multiple downstream tasks, thereby ensuring the reliability of the method.

[0047] After determining the business type and downstream tasks, the construction of the test set can be very flexible, and 4 optional methods are provided here. For a complex multi-task model, there are usually some relatively basic downstream tasks that can be used as benchmark test sets, and the remaining more specialized downstream tasks can be used as associated test sets.

[0048] Embodiment 2:

[0049] In a preferred embodiment, the business requirement is a general large language model. For a large language model, its benchmark capability can be defined as natural language writing capability, so that web page text data can be collected as a test set, and the perplexity (PPL) of the text is used as an indicator calculation method. The associated test sets can include MMLU (knowledge test questions, accuracy index), HumanEval (code test set, code running index, sample passing index, etc.), gsm8k (mathematical ability, accuracy index), etc.

[0050] Embodiment three:

[0051] In another preferred embodiment, the business requirement is a general large language model, whose benchmark capabilities can be defined as Chinese writing capabilities and English writing capabilities, so that two benchmark test sets can be collected: 1) Collect Chinese web page data and use text perplexity as an indicator calculation method; 2) Collect English web page data and use text perplexity as an indicator calculation method. The construction of the associated test set can be the same as in the second embodiment, and some downstream tasks can also be added or subtracted.

[0052] Embodiment 4:

[0053] In another preferred embodiment, the business requirement is a visual model, and its benchmark capability can be set as image classification capability, so that rich images and image category information can be collected as test sets, and the accuracy is used as the indicator calculation method. The associated test set can be an object detection test set, which uses mAP (mean average precision) as the indicator calculation method, and a segmentation test set, which uses pixel accuracy as the indicator calculation method.

[0054] Embodiment five:

[0055] In another preferred embodiment, the business requirement is a multimodal question-answering model, whose benchmark capabilities can be defined as natural language writing capabilities and image classification capabilities (using text generation to make category predictions), thereby collecting two benchmark test sets: 1) web page data, using text perplexity as an indicator calculation method; 2) image categories represented by images and text, using accuracy, text edit distance, etc. as indicators. The corresponding associated test sets can be picture-text question-answering, using key point recall as an indicator calculation method, and text-generated images, using CLIP Score as an indicator calculation method.

[0056] After determining the training set and test set, it is necessary to use the probe model to observe some indicators. The training of the probe model is actually based on the model structure and algorithm. At present, the optimization algorithms of the most complex models are mainly based on gradient-based algorithms.

[0057] Embodiment six:

[0058] In a preferred embodiment, the step S103 described above trains a probe model, specifically by applying back propagation and gradient descent algorithms to update the model weights multiple times until convergence. The process is as follows: Figure 2 As shown. Taking the large language model optimization of Example 2 as an example, the training corpus can be first converted into tokens, sent to the probe model for forward propagation, and the output tokens can be obtained. Then the cross entropy loss of the output tokens is calculated, and the gradient is calculated by back propagation. After obtaining the gradient of the probe model, the gradient descent algorithm is applied to update the probe model weights. This cycle is repeated until all the training set data is exhausted.

[0059] If the probe model is other algorithms such as clustering, the optimization strategy corresponding to the algorithm is applied, which has no impact on the implementation of this method. After determining the probe model optimization strategy, it is necessary to evaluate the probe models with different optimization degrees.

[0060] Embodiment seven:

[0061] Taking the situation of Example 6 as an example, in another preferred embodiment, in the above step S104, testing the probe model using test data and indicator calculation method includes: recording the test indicators of the probe model in at least 2 convergence stages in the test set. Specifically, it can be: every time 1 / 10 of the training set is consumed, it can be considered that the probe model has reached a new optimization stage, so the probe model in this new optimization stage is tested on the test set, and 10 sets of test indicators can be obtained.

[0062] The definition of the optimization stage can be various. In addition to the definition in Example 7, it can also be considered that a new optimization stage has been reached after each training period, or that a new optimization stage has been reached after each probe model weight is updated a certain number of times. Depending on the specific implementation of the optimization stage, more or less than 10 groups of test indicators may be obtained in the end, which does not affect the implementation of this method.

[0063] Embodiment eight:

[0064] In a preferred embodiment, it is assumed that the benchmark test set configuration is as described in Example 3, the associated test set configuration is as described in Example 2, and the probe model training and testing are as described in Example 7. Then this experimental configuration contains 2 benchmark test sets and 3 associated test sets, and the combination of benchmark test sets and associated test sets is 6 in total. By establishing a rectangular coordinate system with the benchmark test set test index as the horizontal axis and the associated test set test index as the vertical axis as described in the above step S105, 6 different rectangular coordinate systems can be constructed. As shown in the process Figure 3 As shown, the step S105 of drawing key points in the rectangular coordinate system according to the test index includes:

[0065] S301, reading the test index corresponding to the horizontal axis and the vertical axis of an optimization stage to form a coordinate, and drawing a key point;

[0066] S302, traverse all optimization stages and draw other key points.

[0067] Specifically, the first set of test indicators in Example 7 can be read, and a key point can be drawn in each of the six coordinate systems. Then, the other nine sets of test indicators can be traversed, and key points can be drawn in the same way in the six coordinate systems. Finally, each coordinate system contains 10 key points to form an observation diagram. Figure 4 shown.

[0068] Embodiment nine:

[0069] In a preferred embodiment, the test set configuration is fixed, including one benchmark test set web text and one associated test set humaneval. If there are two test data, and the observation diagram described in step S105 above has been drawn, for example Figure 5 As shown in the figure. The dots are drawn for the data to be tested 1, and the square dots are drawn for the data to be tested 2. Figure 6 As shown, the above step S106 of evaluating the effectiveness of the data to be evaluated according to the observation graph includes:

[0070] S601, fitting the key points with a curve;

[0071] S602, if the test index of the associated test set is as small as possible, the lower the curve position is, the better the validity of the training data is; if the test index of the associated test set is as large as possible, the higher the curve position is, the better the validity of the training data is.

[0072] Specifically, the points corresponding to the measured data 1 can be fitted with a straight line 1 ( Figure 5 The square points corresponding to the measured data 2 can be fitted into a straight line 2 ( Figure 5 As shown by the dashed line in the figure, we can see that the dashed line is below the solid line, so we can make a decision: when the test index of the associated test set is as small as possible, the validity of data 2 is higher than that of data 1; when the test index of the associated test set is as large as possible, the validity of data 1 is higher than that of data 2.

[0073] Embodiment 9 only takes one benchmark test set and one associated test set as examples. When the test set configuration is richer, such as the test set of embodiment 8, multiple subgraphs will appear in the observation graph. In each task, the test data 1 and the test data 2 may have their own advantages and disadvantages, and the winning ratio or the importance ranking of the task can be used to make a choice.

[0074] Corresponding to the embodiment of the aforementioned training data validity evaluation method, the present invention also provides an embodiment of a training data validity evaluation system. The system includes an acquisition module, a training module, a test module, and an evaluation module; the specific process of each module implementation can refer to the specific steps of the embodiment of the aforementioned training data validity evaluation method.

[0075] The acquisition module is used to acquire a training set and a test set; wherein the training set is obtained by uniformly downsampling the data to be evaluated, and the test set includes at least one benchmark test set and at least one associated test set;

[0076] The training module is used to train a probe model using the training set;

[0077] The test module is used to test the probe model using the test set and record the test indicators; and to make an observation diagram according to the test indicators;

[0078] The evaluation module is used to evaluate the effectiveness of the data to be evaluated based on the observation graph.

[0079] Corresponding to the above-mentioned embodiment of the training data validity evaluation method, the present invention also provides an embodiment of a training data validity evaluation device.

[0080] See also Figure 7 A training data validity evaluation device provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the training data validity evaluation method in the above embodiment.

[0081] An embodiment of a training data validity evaluation device provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 7 As shown, it is a hardware structure diagram of a training data validity evaluation device provided by the present invention, in which any device with data processing capability is located, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0082] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0083] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0084] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the training data validity evaluation method in the above embodiment.

[0085] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0086] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the training data validity evaluation method is implemented.

[0087] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0088] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for evaluating the effectiveness of training data, characterized in that: include: Obtaining a training set, wherein the training set is obtained by uniformly downsampling the data to be evaluated; Acquire a test set, wherein the test set includes at least one benchmark test set and at least one associated test set; training a probe model using the training set; Testing the probe model using the test set and recording the test indicators; Drawing an observation graph according to the test index, wherein drawing the observation graph comprises: establishing a rectangular coordinate system with the test index of the benchmark test set as the horizontal axis and the test index of the associated test set as the vertical axis; drawing key points in the rectangular coordinate system according to the test index; The validity of the data to be evaluated is evaluated based on the observation graph.

2. The training data validity evaluation method according to claim 1, characterized in that: The training of a probe model includes: updating the probe model weights by back propagation and gradient descent method.

3. The training data validity evaluation method according to claim 2, characterized in that: The method of updating the probe model weights by back propagation and gradient descent includes: first converting the training corpus into tokens, sending them into the probe model for forward propagation to obtain output tokens, then calculating the cross entropy loss of the output tokens, and calculating the gradient by back propagation; after obtaining the gradient of the probe model, applying the gradient descent algorithm to update the probe model weights.

4. The training data validity evaluation method according to claim 1, characterized in that: The probe model is tested using the test data and the indicator calculation method, and the recording of the test indicators includes: recording the test indicators of the probe model in at least 2 convergence stages in the test set.

5. The training data validity evaluation method according to claim 4, characterized in that: Drawing key points in the rectangular coordinate system according to the test indicators includes: reading the test indicator composition coordinates corresponding to the horizontal axis and the vertical axis of an optimization stage, and drawing a key point; traversing all convergence stages, and drawing other key points.

6. The training data validity evaluation method according to claim 5, characterized in that: The evaluation of the effectiveness of the training data based on the observation graph includes: fitting the key points with a curve; if the test index of the associated test set is as small as possible, the lower the position of the curve is, the better the effectiveness of the training data is; if the test index of the associated test set is as large as possible, the higher the position of the curve is, the better the effectiveness of the training data is.

7. A training data validity evaluation system for implementing the method according to any one of claims 1 to 6, characterized in that: The system includes an acquisition module, a training module, a testing module and an evaluation module; The acquisition module is used to acquire a training set and a test set; wherein the training set is obtained by uniformly downsampling the data to be evaluated, and the test set includes at least one benchmark test set and at least one associated test set; The training module is used to train a probe model using the training set; The test module is used to test the probe model using the test set and record the test indicators; and to make an observation diagram according to the test indicators; The evaluation module is used to evaluate the effectiveness of the data to be evaluated based on the observation graph.

8. A training data validity evaluation device, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the training data validity evaluation method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the training data validity evaluation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the training data validity evaluation method according to any one of claims 1 to 6 is implemented.

Images