A training data validity evaluation method, system and device

Abstract

The application provides a kind of training data validity evaluation method, system and device, the method comprises: obtaining training set, the training set is obtained from the uniform down sampling of to be evaluated data;Obtain test set, the test set includes at least 1 reference test set and at least 1 associated test set;With the training set trains a probe model;With the test set tests the probe model, records test index;According to the test index makes observation chart, the observation chart includes: with reference test set test index as horizontal axis, associated test set test index as vertical axis establishes rectangular coordinate system;According to the test index draws key point in the rectangular coordinate system;According to the observation chart makes effectiveness evaluation to to be evaluated data.The advantage of the application is that: can use very low computing power, quickly gives data effectiveness evaluation, can improve model development iteration efficiency, especially for the complex model of front, gain is more obvious.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, system, and apparatus for evaluating the effectiveness of training data. Background Technology

[0002] With the development of internet technology, artificial intelligence has become an important part of people's daily lives. As AI businesses flourish, users' demands for the accuracy of AI models are also constantly increasing. These ever-increasing accuracy requirements are driving a rapid expansion in both model size and training data volume.

[0003] The current challenges in training a high-quality artificial intelligence model include: due to the large size of the model and the amount of training data, the computational and time costs consumed in a single experiment are very high. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method, system, and apparatus for evaluating the effectiveness of training data. By using a small-scale probe model, it can directly evaluate the effectiveness of training data, offering significant advantages such as low computational power consumption, low time cost, accurate evaluation results, and strong robustness.

[0005] In a first aspect, the present invention provides a method for evaluating the validity of training data, comprising:

[0006] Obtain a training set, which is obtained by uniformly downsampling the data to be evaluated;

[0007] Obtain a test set, which includes at least one benchmark test set and at least one associated test set;

[0008] Train a probe model using the training set;

[0009] The probe model was tested using the test set, and the test metrics were recorded.

[0010] Based on the test indicators, an observation plot is created, which includes: establishing a rectangular coordinate system with the benchmark test set test indicators as the horizontal axis and the related test set test indicators as the vertical axis; and drawing key points in the rectangular coordinate system based on the test indicators.

[0011] The validity of the data to be evaluated is assessed based on the observation graph.

[0012] Furthermore, training a probe model includes updating the probe model weights through backpropagation and gradient descent.

[0013] Furthermore, updating the probe model weights using backpropagation and gradient descent includes: first, converting the training corpus into tokens, feeding them into the probe model for forward propagation to obtain output tokens, then calculating the cross-entropy loss of the output tokens, and calculating the gradient using backpropagation; after obtaining the gradient of the probe model, applying the gradient descent algorithm to update the probe model weights.

[0014] Furthermore, the probe model is tested using test data and index calculation methods. The test indexes recorded include: the test indexes of the probe model in the test set for at least two convergence phases.

[0015] Furthermore, drawing key points in the Cartesian coordinate system based on the test indicators includes: reading the coordinates of the test indicators corresponding to the horizontal and vertical axes of an optimization stage and drawing a key point; traversing all convergence stages and drawing other key points.

[0016] Furthermore, the evaluation of the effectiveness of the training data based on the observation map includes: fitting the key points with a curve; if the correlation test set test index is smaller the better, then the lower the position of the curve, the better the effectiveness of the training data; if the correlation test set test index is larger the better, then the higher the position of the curve, the better the effectiveness of the training data.

[0017] Secondly, the present invention also provides a training data validity evaluation system, which includes an acquisition module, a training module, a testing module and an evaluation module;

[0018] The acquisition module is used to acquire a training set and a test set; wherein the training set is obtained by uniform downsampling from the data to be evaluated, and the test set includes at least one benchmark test set and at least one associated test set;

[0019] The training module is used to train a probe model using the training set;

[0020] The testing module is used to test the probe model using the test set, record test indicators, and generate observation graphs based on the test indicators.

[0021] The evaluation module is used to evaluate the effectiveness of the data to be evaluated based on the observation graph.

[0022] Thirdly, the present invention also provides a training data validity evaluation device, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the training data validity evaluation method.

[0023] Fourthly, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the training data validity evaluation method.

[0024] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the training data validity evaluation method.

[0025] The advantages of this invention are: it can quickly provide data validity assessment with very low computing power, which can greatly improve the efficiency of model development and iteration, especially for cutting-edge complex models, the gains are more obvious. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating a training data validity evaluation method provided in Embodiment 1 of the present invention;

[0028] Figure 2 This is a schematic diagram of the probe model training process provided in Embodiment Six of the present invention;

[0029] Figure 3 This is a schematic diagram of the process of drawing key points in the rectangular coordinate system according to Embodiment 8 of the present invention;

[0030] Figure 4 This is an example of an observation diagram provided in Embodiment 8 of the present invention;

[0031] Figure 5 This is an example of an observation diagram provided in Embodiment Nine of the present invention;

[0032] Figure 6 This is a schematic diagram of the validity evaluation process for the data to be evaluated provided in Embodiment 9 of the present invention;

[0033] Figure 7 This is a structural diagram of a training data validity evaluation device provided by the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0036] Example 1:

[0037] This embodiment provides a method for evaluating the validity of training data. Figure 1 This is a flowchart of a training data validity evaluation method according to an embodiment of the present invention, the process including the following steps:

[0038] Step S101: Obtain the training set, which is obtained by uniform downsampling from the data to be evaluated;

[0039] Step S102: Obtain a test set, which includes at least one benchmark test set and at least one associated test set;

[0040] Step S103: Train a probe model using the training set;

[0041] Step S104: Test the probe model using the test set and record the test metrics;

[0042] Step S105: Create an observation plot based on the test indicators. Creating the observation plot includes: establishing a rectangular coordinate system with the benchmark test set test indicators as the horizontal axis and the related test set test indicators as the vertical axis; and drawing key points in the rectangular coordinate system based on the test indicators.

[0043] Step S106: Evaluate the validity of the data to be evaluated based on the observation map.

[0044] In the above steps, the form of the training set is not limited, as long as it meets the business requirements. If you want to train a large language model, the training set can be in the form of natural language, mathematical symbols, code, or other text; if you want to train a visual model, the training data can be in the form of images, image classification labels, image object detection labels, image segmentation labels, etc.; if you want to train a multimodal question-answering model, the training set can be in the form of images or videos and corresponding question-answer pairs. This means that this method can adapt to diverse test data and model algorithm types. The training set is obtained by uniformly downsampling from the data to be evaluated, which maintains the data distribution of the test data while reducing the number of samples in the training set, thus improving the efficiency of the method.

[0045] The test set is primarily obtained based on the requirements of downstream tasks. Similar to the training set, if training a large language model, the test set can include common sense questions, math problems, coding problems, etc. Accuracy rate can be used as the evaluation metric for common sense and math problems, while the success rate can be used as the evaluation metric for coding problems. If training a visual model, the test set is also obtained based on the required downstream task type, such as an image classification test set, an object detection test set, or an image segmentation test set. If training a text-image multimodal problem, the test set can be the text-related tasks mentioned above, the image-related tasks mentioned above, or tasks combining text and images such as question-answering understanding, image generation, video understanding, and video generation. For other models required by other businesses, the test set can be obtained similarly based on the business type and downstream task requirements.

[0046] In the above steps, the training set is a small portion uniformly sampled from the complete training data, and the probe model is also a very small model. Therefore, it can be trained with very low computing power in a very short time, ensuring the efficiency of the method. The observation graph is drawn by combining test metrics from multiple test sets, which can simultaneously reflect the response of the training set data to multiple downstream tasks, ensuring the reliability of the method.

[0047] Once the business type and downstream tasks are determined, the construction of the test set can be very flexible. Here are four options. For a complex multi-task model, there are usually some basic downstream tasks that can be used as a benchmark test set, while the other more specialized downstream tasks can be used as a correlation test set.

[0048] Example 2:

[0049] In a preferred embodiment, the business requirement is a general large language model. For a large language model, its baseline capability can be defined as natural language writing ability. Therefore, web page text data can be collected as a test set, and the text perplexity (PPL) is used as the metric calculation method. The associated test set can include multiple methods such as MMLU (knowledge-based test questions, accuracy metric), HumanEval (code test set, code execution metrics, sample pass metrics, etc.), and gsm8k (mathematical ability, accuracy metric).

[0050] Example 3:

[0051] In another preferred embodiment, the business requirement is a general large language model, whose benchmark capabilities can be defined as Chinese writing ability and English writing ability. Therefore, two benchmark test sets can be collected: 1) Chinese webpage data, calculated using text perplexity as the indicator; 2) English webpage data, calculated using text perplexity as the indicator. The construction of the associated test sets can be the same as in Embodiment 2, or some downstream tasks can be added or removed.

[0052] Example 4:

[0053] In another preferred embodiment, the business requirement is a visual model whose baseline capability can be defined as image classification capability. This allows for the collection of rich image and image category information as a test set, with accuracy as the metric. The associated test set can be an object detection test set, calculated using mAP (mean average precision), or a segmentation test set, calculated using pixel accuracy.

[0054] Example 5:

[0055] In another preferred embodiment, the business requirement is a multimodal question-answering model, whose baseline capabilities can be defined as natural language text processing capability and image classification capability (using generated text for category prediction). This involves collecting two benchmark test sets: 1) web page data, calculated using text perplexity as the metric; and 2) image and text representations of image categories, calculated using accuracy, text edit distance, etc. The corresponding association test set could be image-text question answering, calculated using keypoint recall as the metric, or text-to-image generation, calculated using CLIP Score as the metric.

[0056] After determining the training and test sets, a probe model is needed to observe some metrics. The training of the probe model depends on the model structure and algorithm; currently, the optimization algorithms for the most complex models are mainly gradient-based.

[0057] Example 6:

[0058] In a preferred embodiment, step S103 above, which involves training a probe model, specifically applies backpropagation and gradient descent algorithms to update the model weights multiple times until convergence, as follows: Figure 2 As shown. Taking the large language model optimization in Example 2 as an example, the training corpus can first be converted into tokens, fed into the probe model for forward propagation to obtain output tokens, then the cross-entropy loss of the output tokens can be calculated, and the gradient can be calculated using backpropagation. After obtaining the gradient of the probe model, the gradient descent algorithm is applied to update the probe model weights. This process is repeated until all training set data has been processed.

[0059] If the probe model uses clustering or other algorithms, the corresponding optimization strategy should be applied; this will not affect the implementation of this method. After determining the probe model optimization strategy, probe models with different optimization levels need to be evaluated.

[0060] Example 7:

[0061] Taking Example 6 as an example, in another preferred embodiment, step S104 above, testing the probe model using test data and index calculation methods, includes: recording the test indices of the probe model on the test set for at least two convergence phases. Specifically, this can be achieved by: consuming 1 / 10 of the training set, which indicates that the probe model has reached a new optimization phase. Then, the probe model in this new optimization phase is tested on the test set, yielding 10 sets of test indices.

[0062] The definition of the optimization phase can vary. In addition to the definition in Example 7, it can also be defined as reaching a new optimization phase after a certain training period, or after the probe model weights are updated a certain number of times. Depending on the specific implementation of the optimization phase, more or fewer sets of test metrics may be obtained, but this does not affect the implementation of this method.

[0063] Example 8:

[0064] In a preferred embodiment, assuming the benchmark test set configuration is as described in Embodiment 3, the correlation test set configuration is as described in Embodiment 2, and the probe model training and testing are as described in Embodiment 7, then this experimental configuration includes 2 benchmark test sets and 3 correlation test sets. There are a total of 6 possible pairwise combinations of the benchmark test sets and correlation test sets. By establishing a Cartesian coordinate system with the benchmark test set metrics as the horizontal axis and the correlation test set metrics as the vertical axis as described in step S105 above, 6 different Cartesian coordinate systems can be constructed. (See flowchart...) Figure 3 As shown, step S105 above, which involves drawing key points in the Cartesian coordinate system according to the test index, includes:

[0065] S301, read the test indicators corresponding to the horizontal and vertical axes of an optimization stage to form a coordinate system and draw a key point;

[0066] S302, traverse all optimization stages and draw other key points.

[0067] Specifically, the first set of test indicators in Example 7 can be read, and a key point can be drawn in each of the six coordinate systems. Then, the other nine sets of test indicators can be traversed, and key points can be drawn in the same way in the six coordinate systems. Ultimately, each coordinate system contains 10 key points, forming an observation map. An example of an observation map is shown below. Figure 4 As shown.

[0068] Example 9:

[0069] In a preferred embodiment, the test set configuration is fixed, containing one benchmark test set (web page text) and one association test set (humaneval). If there are two datasets to be tested, and the observation plot described in step S105 above has already been plotted, for example... Figure 5 As shown. The circles represent data point 1, and the squares represent data point 2. See the flowchart. Figure 6 As shown, step S106 above, which involves evaluating the validity of the data to be evaluated based on the observation map, includes:

[0070] S601, Fit the key points with a curve;

[0071] S602, if the correlation test set test metric is smaller the better, then the lower the position of the curve, the better the effectiveness of the training data; if the correlation test set test metric is larger the better, then the higher the position of the curve, the better the effectiveness of the training data.

[0072] Specifically, the circle corresponding to the measured data 1 can be fitted with a straight line 1. Figure 5 As shown by the solid line in the middle, the square points corresponding to the measured data 2 can be fitted to the straight line 2 ( Figure 5 (As shown by the dashed line). We can see that the dashed line is below the solid line, so we can make the following decision: when the association test set metric is set to smaller is better, data 2 is more effective than data 1; when the association test set metric is set to larger is better, data 1 is more effective than data 2.

[0073] Example 9 uses only one benchmark test set and one associated test set as an example. When the test set configuration is richer, such as the test set in Example 8, multiple subgraphs will appear in the observation graph. In each task, test data 1 and test data 2 may have their own advantages and disadvantages. The choice can be made by ranking them by the winning ratio or the importance of the task.

[0074] Corresponding to the aforementioned embodiments of the training data validity evaluation method, this invention also provides an embodiment of a training data validity evaluation system. This system includes an acquisition module, a training module, a testing module, and an evaluation module; for the specific implementation process of each module, please refer to the specific steps of the aforementioned embodiments of the training data validity evaluation method.

[0075] The acquisition module is used to acquire a training set and a test set; wherein the training set is obtained by uniform downsampling from the data to be evaluated, and the test set includes at least one benchmark test set and at least one associated test set;

[0076] The training module is used to train a probe model using the training set;

[0077] The testing module is used to test the probe model using the test set, record test indicators, and generate observation graphs based on the test indicators.

[0078] The evaluation module is used to evaluate the effectiveness of the data to be evaluated based on the observation graph.

[0079] Corresponding to the aforementioned embodiments of the training data validity evaluation method, the present invention also provides an embodiment of a training data validity evaluation device.

[0080] See Figure 7 The present invention provides a training data validity evaluation device, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement the training data validity evaluation method in the above embodiment.

[0081] An embodiment of the training data validity evaluation device provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of any device with data processing capabilities, where the training data validity evaluation device provided by the present invention is located. (Except for...) Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0082] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0083] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0084] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the training data validity evaluation method described in the above embodiments.

[0085] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0086] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the training data validity evaluation method.

[0087] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0088] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A training data validity evaluation method, characterized in that, The method comprises: obtaining a training set, which is obtained by uniformly down-sampling from the to-be-evaluated data; obtaining a test set, which comprises at least one benchmark test set and at least one correlation test set; training a probe model using the training set; testing the probe model using the test set and recording test indicators; plotting an observation graph according to the test indicators, which comprises: establishing a rectangular coordinate system with the benchmark test indicators as the horizontal axis and the correlation test indicators as the vertical axis; and plotting key points in the rectangular coordinate system according to the test indicators; evaluating the effectiveness of the to-be-evaluated data according to the observation graph.

2. The training data validity evaluation method of claim 1, wherein, The training of the probe model comprises: updating the weights of the probe model by back propagation and gradient descent method.

3. The training data validity evaluation method of claim 2, wherein, The updating of the weights of the probe model by back propagation and gradient descent method comprises: first converting the training corpus into tokens, feeding them into the probe model for forward propagation to obtain output tokens, then calculating the cross-entropy loss of the output tokens, and calculating the gradient by back propagation; after obtaining the gradient of the probe model, applying the gradient descent algorithm to update the weights of the probe model.

4. The training data validity evaluation method of claim 1, wherein, The testing of the probe model using the test data and indicator calculation method comprises: recording the test indicators of the probe model in at least two convergence stages.

5. The training data validity evaluation method of claim 4, wherein, The plotting of key points in the rectangular coordinate system according to the test indicators comprises: reading the test indicators corresponding to the horizontal axis and the vertical axis of an optimization stage to form coordinates, and plotting a key point; and traversing all convergence stages to plot other key points.

6. The training data validity evaluation method of claim 5, wherein, The evaluation of the effectiveness of the to-be-evaluated data according to the observation graph comprises: fitting the key points with a curve; if the correlation test indicators are the smaller the better, the lower the position of the curve, the better the effectiveness of the training data; if the correlation test indicators are the larger the better, the higher the position of the curve, the better the effectiveness of the training data.

7. A training data validity assessment system implementing the method of any one of claims 1-6, characterized by The system comprises an obtaining module, a training module, a testing module, and an evaluation module. The obtaining module is configured to obtain a training set and a test set; wherein the training set is obtained by uniformly down-sampling from to-be-evaluated data, and the test set comprises at least one benchmark test set and at least one correlation test set. The training module is configured to train a probe model using the training set. The testing module is configured to test the probe model using the test set, record test indicators, and plot an observation graph according to the test indicators. The evaluation module is configured to evaluate the effectiveness of the to-be-evaluated data according to the observation graph.

8. A training data validity evaluation apparatus comprising a memory and one or more processors, the memory having stored therein executable code to cause the one or more processors to perform operations comprising: The processor executes the executable code to implement the training data effectiveness evaluation method of any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the training data effectiveness evaluation method of any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the training data effectiveness evaluation method of any one of claims 1-6.

Images