Machine learning dataset evaluation support device and machine learning dataset evaluation support method

Abstract

To configure a sophisticated learning data set that avoids inappropriate learning or lack of learning.SOLUTION: A machine learning data set evaluation support device 100 is configured to include: a learning data set holding unit 116 that holds a learning data set and label information used for supervised machine learning; an inference execution unit 111 that inputs a predetermined evaluation data set to a plurality of learned AI models and obtains a plurality of inference results; a difference analysis unit 113 that analyzes the difference between the inference results from a confidence degree of the inference result and a stability degree of the inference result in each of the plurality of learned AI models; a data evaluation unit 114 that evaluates the evaluation data set based on the analysis results of the difference; and a learning data set editing unit 115 that edits the learning data set and the label information based on the evaluation result of the evaluation data set.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a machine learning dataset evaluation support device and a machine learning dataset evaluation support method. [Background technology]

[0002] Machine learning technology is being introduced in a wide range of fields, such as monitoring and controlling various devices, predicting failures, and recognizing objects in autonomous driving, by utilizing various types of big data such as IoT data. In machine learning technology in such fields, supervised machine learning is useful because it has high learning accuracy and fast learning speed because correct answer data is provided by humans.

[0003] In supervised machine learning, data is collected from the real world, and the expected output value when that data is input into an AI model, that is, a learning dataset (training data and test data) is created with a correct answer label. Of these, the training data is used as teacher data to teach the AI ​​model, and the accuracy of the trained AI model is evaluated using the test data.

[0004] As a conventional technology related to such machine learning, for example, in light of the issue that it is difficult to perform appropriate data cleansing processing, a training data generation device has been proposed for generating training data suitable for machine learning algorithms from input data (see Patent Document 1).

[0005] This technology is a training data generation device having a processing means that performs a cleansing process on input data to generate training data, and a generation means that is provided with a machine learning algorithm and generates a learning model based on the training data using the machine learning algorithm, wherein the generation means is provided with a plurality of machine learning algorithms of different types as the machine learning algorithm, and the processing means performs a first cleansing process on the input data in correspondence with each of the plurality of machine learning algorithms. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-38514 A Summary of the Invention [Problem to be solved by the invention]

[0007] The desired requirements for the above-mentioned training dataset are as follows: the training dataset must cover the expected input data, a robust (stable against small deviations in the input values) AI model must be constructed, and the labels assigned to the training data must be valid.

[0008] However, with a training dataset collected naively without any special ingenuity, it is difficult to control the progress of learning. As a result, unintended learning may occur. For example, this includes cases where there is insufficient training data, where training data with different correct labels are inadvertently close to each other, and where features different from the learning intent are dominant.

[0009] On the other hand, according to the conventional technology, the cleansing of the learning data is performed based on the characteristics of the machine learning algorithm. It is not possible to refine the training data by providing feedback.

[0010] It is therefore an object of the present invention to provide a technique that allows for the construction of sophisticated training data sets that avoid inappropriate training and under-training. [Means for solving the problem]

[0011] The machine learning dataset evaluation support device of the present invention that solves the above-mentioned problems is characterized in that it comprises a learning dataset holding unit that holds a learning dataset and label information used for supervised machine learning, an inference execution unit that inputs a predetermined evaluation dataset to a plurality of trained AI models and obtains a plurality of inference results, a difference analysis unit that analyzes the differences between the plurality of inference results from the confidence level of the inference result and the stability of the inference result in each of the plurality of trained AI models, a data evaluation unit that evaluates the evaluation dataset based on the analysis result of the difference, and a learning dataset editing unit that edits the learning dataset and label information held in the learning dataset holding unit based on the evaluation result of the evaluation dataset.

[0012] In addition, the machine learning dataset evaluation support method of the present invention is characterized in that an information processing device executes the following processes: storing a learning dataset and label information used for supervised machine learning in a storage device, inputting a predetermined evaluation dataset into a plurality of trained AI models to obtain a plurality of inference results, analyzing the differences between the plurality of inference results from the confidence of the inference result and the stability of the inference result in each of the plurality of trained AI models, evaluating the evaluation dataset based on the analysis results of the differences, and editing the learning dataset and label information stored in the storage device based on the evaluation results of the evaluation dataset. Effect of the Invention

[0013] The present invention allows for the construction of sophisticated training data sets that avoid inappropriate training and under-training. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is a block diagram showing an example of a functional configuration of a learning dataset evaluation support device according to the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the learning dataset evaluation support device of the present embodiment. [Diagram 3]FIG. 1 is a diagram showing an example of a flow of a learning dataset evaluation support method according to the present embodiment. [Figure 4] FIG. 1 is a flow diagram of a process for obtaining a trained AI model by varying a learning data set in this embodiment. [Diagram 5] FIG. 1 is a flow diagram of a process for obtaining a trained AI model by changing a model structure in this embodiment. [Figure 6] FIG. 1 is a flow diagram of an inference execution process in this embodiment, which is a flow diagram of a process for sequentially executing inferences of a trained AI model. [Figure 7] FIG. 1 is a flow diagram of an inference execution process in this embodiment, and is a flow diagram of a process of executing inferences of a trained AI model in parallel. [Figure 8] FIG. 1 is a flow diagram of an inference execution process in this embodiment, and is a flow diagram of a process using an AI model in which the model structure is changed during inference. [Figure 9] FIG. 11 is a flow diagram of an inference result difference analysis process in this embodiment. [Figure 10] 11 is a diagram showing an example of a method for calculating the certainty and stability of an inference result in this embodiment. FIG. [Figure 11] 11A and 11B are diagrams illustrating an example of an analysis result of a difference analysis unit in the present embodiment. [Figure 12] FIG. 4 is a flow diagram of a data evaluation process in the present embodiment. [Figure 13] 11 is a diagram illustrating an example of a threshold setting process performed by a data evaluation unit in the present embodiment. FIG. [Figure 14] FIG. 11 is a flow diagram of a learning dataset editing process in this embodiment. [Figure 15] FIG. 11 is a diagram showing an example of a method for determining similar data performed by a learning dataset editing processing unit in this embodiment. [Figure 16] FIG. 4 is a diagram illustrating an example of a method for generating similar data in the present embodiment. [Figure 17] FIG. 2 is a block diagram showing a functional configuration of a learning dataset evaluation support device in the first embodiment. [Figure 18]1A to 1C are diagrams illustrating examples of evaluation data and examples of inference results and difference analysis results in the first embodiment. [Figure 19] 11A and 11B are diagrams illustrating an example of a high confidence threshold, an example of a low confidence threshold, and an example of a low stability threshold in the first embodiment. [Figure 20] FIG. 4 is a diagram showing an example of an evaluation result of evaluation data in the first embodiment. [Figure 21] FIG. 11 is a diagram illustrating an example of a learning dataset editing process in the first embodiment, showing a case where the label of the learning data matches the inference result. [Figure 22] FIG. 13 is a diagram illustrating an example of the learning dataset editing process in the first embodiment, which illustrates a case where the label of the learning data does not match the inference result. [Diagram 23] FIG. 11 is a diagram illustrating an example of a learning dataset editing process in the first embodiment. [Figure 24] FIG. 11 is a diagram illustrating an example of a learning dataset editing process in the first embodiment. [Diagram 25] FIG. 11 is a block diagram showing a functional configuration of a learning dataset evaluation support device in a second embodiment. [Figure 26] FIG. 11 is a diagram showing an example of displaying the confidence level of an inference result in the second embodiment. [Figure 27] FIG. 11 is a diagram showing an example of displaying the stability of an inference result in the second embodiment. [Figure 28] 13 is a diagram showing an example of a high confidence threshold, an example of a low confidence threshold, and an example of a low stability threshold in the second embodiment. FIG. [Figure 29] FIG. 11 is a diagram showing an example of an evaluation result of evaluation data in Example 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] <Example of the configuration of a machine learning dataset evaluation support device> Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a diagram showing an example of the functional configuration of a machine learning dataset evaluation support device 100 according to this embodiment. The machine learning dataset evaluation support device 100 shown in Fig. 1 is a computer that enables the configuration of a sophisticated training dataset that avoids inappropriate learning or insufficient learning.

[0016] As shown in FIG. 1, the machine learning dataset evaluation support device 100 of this embodiment has an evaluation dataset holding unit 110, an inference execution unit 111, a trained AI model holding unit 112, a difference analysis unit 113, a data evaluation unit 114, a learning dataset editing unit 115, and a learning dataset holding unit 116.

[0017] Among these, the evaluation dataset holding unit 110 holds an evaluation dataset 125. Furthermore, the trained AI model holding unit 112 holds a trained AI model 118. Furthermore, the training dataset holding unit 116 holds a training dataset 126 and label information 127.

[0018] The learning data set 126 is, for example, training data collected from the real world. The label information 127 is information on the output, that is, the correct label, that is expected when the learning data set, which is training data, is input to the AI ​​model.

[0019] Moreover, the evaluation dataset 125 is to be input to each of the AI ​​models that have progressed in learning, that is, the trained AI models 118. In this case, multiple inference results are obtained from the trained AI models 118.

[0020] Specifically, the machine learning dataset evaluation support device 100 can be assumed to be a server device, a personal computer, or the like.

[0021] <Hardware configuration> Moreover, the hardware configuration of the machine learning dataset evaluation support device 100 of this embodiment is as shown in FIG.

[0022] That is, the machine learning dataset evaluation support device 100 includes a storage device 101 , a memory 103 , a calculation device 104 , an input device 105 , an output device 106 , and a communication device 107 .

[0023] Of these, the storage device 101 is configured with an appropriate non-volatile storage element such as an SSD (Solid State Drive) or a hard disk drive.

[0024] The memory 103 is composed of a volatile storage element such as a RAM.

[0025] The arithmetic unit 104 is a CPU that reads out the program 102 stored in the storage unit 101 into the memory 103 and executes the program, controls the device itself, and performs various types of judgment, calculation, and control processing.

[0026] The input device 105 is a device such as a keyboard, a mouse, or a microphone that accepts key input or voice input from the user.

[0027] The output device 106 is a device such as a display or a speaker that outputs the results of processing by the arithmetic unit 104 .

[0028] The communication device 107 is assumed to be a network interface card or the like that is connected to an appropriate network and handles communication processing with a user terminal, etc. However, when the machine learning dataset evaluation support device 100 is a standalone machine, the communication device 107 can be omitted.

[0029] In addition to the program 102 for implementing functions required for the machine learning dataset evaluation support device of this embodiment, at least an evaluation dataset 125 in the evaluation dataset holding unit 110, a training dataset 126 and label information 127 in the training dataset holding unit 115, and a trained AI model 118 in the trained AI model holding unit 112 are stored in the storage device 101. However, details of these will be described later.

[0030] Furthermore, when the program 102 is executed by the computing device 104, it implements the functions of an inference execution unit 111, a difference analysis unit 113, a data evaluation unit 114, and a learning dataset editing unit 115.

[0031] Of these, the inference execution unit 111 inputs the evaluation dataset held in the evaluation dataset holding unit 110 to multiple trained AI models 118 and obtains multiple inference results.

[0032] Note that the inference execution unit 111 may use, as the multiple trained AI models 118, AI models that have been trained by changing the learning data set or the model structure during learning.

[0033] Furthermore, the inference execution unit 111 may use, as the multiple trained AI models, trained AI models whose model structures have been changed during inference.

[0034] Furthermore, the inference execution unit 111 may sequentially execute a plurality of trained AI models to obtain a plurality of inference results.

[0035] Furthermore, the inference execution unit 111 may execute multiple trained AI models in parallel to obtain multiple inference results simultaneously.

[0036] In addition, the difference analysis unit 113 analyzes the differences between the multiple inference results obtained by the above-mentioned inference execution unit 111 based on the confidence level and stability of the inference results in each of the multiple trained AI models 118.

[0037] Moreover, the data evaluation section 114 evaluates the evaluation data set 125 based on the above-mentioned difference analysis result obtained by the difference analysis section 113 .

[0038] Furthermore, the learning dataset editing unit 115 edits the learning dataset 126 and label information 127 stored in the learning dataset storage unit 116 based on the evaluation result of the evaluation dataset 125 .

[0039] In addition, the above-mentioned learning dataset editing unit 115 may delete from the learning dataset 126 learning data that is similar to evaluation data in which the confidence level of the inference result by the inference execution unit 111 is greater than a predetermined threshold value and has a correct answer label that differs from the inference result of the data to be evaluated.

[0040] Furthermore, the learning dataset editing unit 115 may delete from the learning dataset 126 learning data that is similar to evaluation data for which the confidence level of the inference result by the inference execution unit 111 is smaller than a predetermined threshold value.

[0041] In addition, the learning dataset editing unit 115 may generate learning data similar to evaluation data in which the confidence level of the inference result by the inference execution unit 111 is greater than a predetermined threshold and the stability is less than a predetermined threshold, and add the learning data to the learning dataset 126.

[0042] 25 instead of Fig. 1, the machine learning dataset evaluation support device 100 also has an analysis result display unit 117 in addition to the above. This analysis result display unit 117 displays the analysis results from the difference analysis unit 113 on the output device 106.

[0043] <Flow example> The actual procedure of the machine learning dataset evaluation support method in this embodiment will be described below with reference to the drawings. Various operations corresponding to the machine learning dataset evaluation support method described below are realized by a program that the machine learning dataset evaluation support device 100 reads into a memory or the like and executes. This program is composed of codes for performing the various operations described below.

[0044] 3 is a diagram showing an example of a flow of the machine learning dataset evaluation support method in this embodiment. In this case, the machine learning dataset evaluation support device 100 acquires a trained AI model 118 (301). This trained AI model 118 is stored in advance in the trained AI model storage unit 112 of the storage device 101.

[0045] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 inputs one piece of evaluation data from the evaluation dataset 125 in the evaluation dataset holding unit 110 to the trained AI model 118 obtained in step 301, and executes inference processing (302). Details of this inference processing will be described later with reference to Figs. 4 to 8.

[0046] Next, the difference analysis unit 113 of the machine learning dataset evaluation support device 100 analyzes the confidence and stability of the result of the inference process in step 302, and records this in, for example, the memory 103 (303).

[0047] Next, the machine learning dataset evaluation support device 100 determines whether the processing target in the steps up to this point is the last evaluation data among the unprocessed evaluation data included in the evaluation dataset 125 (304).

[0048] As a result of the above-mentioned determination, if it is not the last evaluation data (304: NO), the machine learning dataset evaluation support device 100 returns the process to step 302.

[0049] On the other hand, if the result of the above-mentioned determination is that the evaluation data is the last one (304: YES), the data evaluation unit 114 of the machine learning dataset evaluation support device 100 evaluates the evaluation data and records the result in, for example, the memory 103 (305). Details of the processing by the data evaluation unit 114 will be described later with reference to FIG. 12 etc.

[0050] Furthermore, the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 edits (306) the learning dataset 126 and the label information 127 in the learning dataset storage unit 116, and ends this flow. Details of the processing by this learning dataset editing unit 115 will be described later with reference to Fig. 14 etc. Through the processing up to this point, a refined learning dataset that avoids inappropriate learning and insufficient learning has been generated.

[0051] <Example 1 of the inference execution flow> Next, FIG. 4 shows an example of a process flow in which the inference execution unit 111 of this embodiment generates a plurality of AI models by learning using a learning data set obtained by varying the learning data set.

[0052] In this case, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires the pre-training AI model from, for example, the input device 105 (401).

[0053] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 sets the argument i to 1 (402).

[0054] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires a plurality of training data from the training dataset 126 in the training dataset storage unit 115 (403).

[0055] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires the labels of the training data obtained in the above-mentioned step 403 from the label information 127 in the training dataset storage unit 115 (404).

[0056] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 trains the pre-learning AI model obtained in step 401 using the learning data and labels obtained in steps 403 and 404 as teacher data (405).

[0057] In addition, the inference execution unit 111 of the machine learning dataset evaluation support device 100 stores the trained AI model 118 obtained by training in step 405 in the trained AI model holding unit 112 (406).

[0058] Here, the inference execution unit 111 of the machine learning dataset evaluation support device 100 calculates the value of the argument i as The counter is incremented (407) and the process proceeds to step 408.

[0059] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 determines whether the above-mentioned argument i exceeds a predetermined value T (408).

[0060] If the result of the above determination is that the value of argument i does not exceed T (408: NO), the inference execution unit 111 returns the process to step 403. When returning to step 403, in addition to setting unprocessed learning data included in the learning dataset 126 as the next learning target, it is also possible to add, change, or delete learning data according to a predetermined rule or a user operation and set it as the learning target.

[0061] On the other hand, if the result of the above determination is that the value of argument i has exceeded T (408: YES), the inference execution unit 111 ends this flow.

[0062] <Example 2 of the inference execution flow> Next, FIG. 5 shows an example of a process flow in which the inference execution unit 111 of this embodiment changes the model structure and acquires a trained AI model.

[0063] In this case, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires a pre-training AI model from, for example, the input device 105 (501).

[0064] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 sets the argument i to 1 (502).

[0065] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires a plurality of training data from the training dataset 126 in the training dataset storage unit 115 (503).

[0066] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires the labels of the training data obtained in the above-mentioned step 503 from the label information 127 in the training dataset storage unit 115 (504).

[0067] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 modifies (505) the structure of the pre-learning AI model obtained in step 501. This modification of the structure can be, for example, randomly deleting one of the nodes in a predetermined layer (e.g., a certain layer among the intermediate layers) of the neural network constituting the AI ​​model, or deleting one edge between the node and a predetermined node in another layer.

[0068] In addition, the inference execution unit 111 of the machine learning dataset evaluation support device 100 trains the pre-learning AI model whose structure has been modified in step 505 by providing the learning data and labels obtained in steps 503 and 504 as teacher data (506).

[0069] In addition, the inference execution unit 111 of the machine learning dataset evaluation support device 100 stores the trained AI model 118 obtained by training in step 506 in the trained AI model holding unit 112 (507).

[0070] Here, the inference execution unit 111 of the machine learning dataset evaluation support device 100 increments the value of the argument i (507) and transitions to step 508.

[0071] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 calculates the above-mentioned argument i It is determined whether or not the value T exceeds a predetermined value T (509).

[0072] If the result of the above determination is that the value of argument i does not exceed T (509: NO), the inference execution unit 111 returns the process to step 505. On the other hand, if the result of the above determination is that the value of argument i exceeds T (509: YES), the inference execution unit 111 ends this flow.

[0073] <Example 3 of the inference execution flow> Next, FIG. 6 shows a flow of an inference execution process by the inference execution unit 111 of this embodiment, which is an example of a process flow for sequentially executing inferences of a trained AI model.

[0074] In this case, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires one piece of evaluation data from the evaluation dataset 125 in the evaluation dataset holding unit 110 (601).

[0075] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires T trained AI models 118 from the trained AI model storage unit 112 (602).

[0076] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 sets the value of the argument i to 1 (603).

[0077] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 inputs the evaluation data obtained in step 601 into the k-th trained AI model 118 and executes inference (604).

[0078] Furthermore, the inference execution unit 111 of the machine learning dataset evaluation support device 100 stores the result of the inference in the above-mentioned step 605 in, for example, the memory 103 (605).

[0079] Furthermore, the inference execution unit 111 of the machine learning dataset evaluation support device 100 increments the value of the argument i by 1 (606).

[0080] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 determines whether the value of the above-mentioned argument i has exceeded T (607).

[0081] If the result of the above determination is that the value of argument i does not exceed T ( 607 : NO), the inference execution unit 111 returns the process to step 604 .

[0082] On the other hand, if the result of the above determination is that the value of argument i has exceeded T (607: YES), the inference execution unit 111 ends this flow.

[0083] <Example 4 of the inference execution flow> Next, FIG. 7 shows a flow of an inference execution process in this embodiment, which is an example of a process flow for executing inferences of trained AI models in parallel.

[0084] In this case, the machine learning dataset evaluation support device 100 acquires one piece of evaluation data from the evaluation dataset storage unit 110 (701).

[0085] Next, the machine learning dataset evaluation support device 100 acquires T trained AI models 118 from the trained AI model storage unit 112 (702).

[0086] Next, the machine learning dataset evaluation support device 100 evaluates the T learning The completed AI model 118 is deployed to the inference execution unit 111 (703).

[0087] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 inputs the evaluation data obtained in step 701 into the T trained AI models 118 developed in step 703 and executes inference (704).

[0088] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 stores the result of the inference in step 704, for example, in the memory 103 (705), and ends this flow.

[0089] <Example 5 of the inference execution flow> Next, FIG. 8 shows a flow of an inference execution process in this embodiment, which is an example of a process flow using an AI model in which the model structure is changed during inference.

[0090] In this case, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires one piece of evaluation data from the evaluation dataset holding unit 110 (801).

[0091] In addition, the inference execution unit 111 of the machine learning dataset evaluation support device 100 acquires the trained AI model 118 from the trained AI model storage unit 112 (802).

[0092] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 sets the value of the argument i to 1 (803).

[0093] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 inactivates (804) some of the components of the trained AI model 118 obtained in step 802. In this case, the components are, for example, nodes and edges that make up a neural network, and partial inactivation can be assumed to be a measure to set the weight value of a node to zero or to delete an edge.

[0094] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 inputs the evaluation data obtained in step 801 into the trained AI model 118 that was partially deactivated in step 804, and executes inference (805).

[0095] Furthermore, the inference execution unit 111 of the machine learning dataset evaluation support device 100 stores the result of the inference in step 805 in, for example, the memory 103 (806).

[0096] Next, the inference execution unit 111 of the machine learning dataset evaluation support device 100 increments the value of argument i (807), and determines whether the value of argument i has exceeded a predetermined value of T (808).

[0097] If the result of the above determination is that the value of argument i does not exceed T ( 808 : NO), the inference execution unit 111 returns the process to step 804 .

[0098] On the other hand, if the result of the above determination is that the value of argument i has exceeded T (808: YES), the inference execution unit 111 ends this flow.

[0099] <Example of flow of difference analysis part> 9 shows an example of a flow of an inference result difference analysis process in the difference analysis unit 113 of this embodiment. Also, Fig. 10 shows an example of a method of calculating the confidence and stability of an inference result in this embodiment.

[0100] In this case, the difference analysis unit 113 of the machine learning dataset evaluation support device 100 acquires K inference results by the inference execution unit 111 from the memory 103 (901).

[0101] Furthermore, the difference analysis unit 113 of the machine learning dataset evaluation support device 100 calculates the confidence level of the inference result obtained in step 901, and stores the confidence level in, for example, the memory 103 (902).

[0102] Next, the machine learning dataset evaluation support device 100 calculates the stability of the inference result obtained in step 901, and stores it, for example, in the memory 103 (903), and ends this flow.

[0103] As a method of calculating the confidence and stability by the above-mentioned difference analysis unit 113, for example, as shown in FIG. 10, if the inference result of the classification problem is expressed as formula 1201 and the average value of the probability of the inference results of T trained AI models 118 is expressed as formula 1202, the confidence can be calculated by formula 1203 and the stability can be calculated by formula 1204.

[0104] Similarly, if the inference result of the regression problem is expressed as equation 1205 and the average probability of the inference results of T trained AI models 118 is expressed as equation 1206, the confidence level can be calculated using equation 1207 and the stability level can be calculated using equation 1208.

[0105] The results of the confidence and stability calculated as described above can be output, for example, as shown in Fig. 11. In the confidence distribution chart 1401, each point represents the average value of the probability that a plurality of inference results for one evaluation data will be determined to be class k and the confidence of the inference result. Also, in the stability distribution chart 1402, each point represents the average value of the probability that a plurality of inference results for one evaluation data will be determined to be class k and the stability of the inference result.

[0106] <Example of data evaluation flow> 12 shows an example of a flow of the data evaluation unit 114 in this embodiment. In this case, the data evaluation unit 114 of the machine learning dataset evaluation support device 100 sets a high confidence threshold, a low confidence threshold, and a low stability threshold (1001).

[0107] Possible setting methods in this case are, for example, one that uses a pre-specified default value, one that is mechanically set so that the evaluation data to be classified is a fixed ratio, one that is mechanically set so that areas with high data density are separated from areas with low data density, and one that is set by an operator with reference to the difference analysis results (see FIG. 11).

[0108] As a specific conceptual example of threshold setting, as shown in the distribution diagram 1401 in Figure 13, the data around "1" has a high degree of confidence that the inference result is "class k", and the data around "2" has a high degree of confidence that the inference result is "not class k", so a high confidence threshold is set to cover the data around "1" and "2".

[0109] Furthermore, as shown in distribution diagram 1402, in the case where data with low confidence in the inference result is unstable and reliable recognition is difficult, a low confidence threshold where the high confidence threshold is greater than the low confidence threshold is set based on the distribution of data with low confidence.

[0110] Also, as shown in distribution diagram 1403, in the case of a situation where the data with low stability of the inference result is likely to stably converge to the inference result through learning, a low stability threshold is set based on the distribution of the low stability data.

[0111] In addition, the data evaluation unit 114 of the machine learning dataset evaluation support device 100 extracts and records evaluation data whose confidence level is equal to or higher than the high confidence threshold (set in step 1001) ( 1002).

[0112] Next, the data evaluation unit 114 of the machine learning dataset evaluation support device 100 extracts and records evaluation data whose confidence level is less than the low confidence level threshold (set in step 1001) (1003).

[0113] Next, the data evaluation unit 114 of the machine learning dataset evaluation support device 100 extracts and records (1004) evaluation data whose stability is less than the low stability threshold (set in step 1001), and ends this flow.

[0114] <Example of the learning dataset editing flow> 14 shows an example of a processing flow in the learning dataset editing unit 115 in this embodiment. In this case, the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 acquires evaluation data from the evaluation dataset holding unit 110 (1101).

[0115] Next, the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 determines whether the evaluation data indicates low confidence (1102).

[0116] If the result of the above-mentioned judgment indicates that the evaluation data indicates low confidence (1102: YES), the learning dataset editing unit 115 obtains all learning data similar to the evaluation data obtained in step 1101 from the learning dataset 126 in the learning dataset storage unit 116 (1109).

[0117] Furthermore, the learning dataset editing unit 115 acquires (1110) the label of the learning data obtained in step 1109 from the label information 127 in the learning dataset storage unit 116, and transitions the process to step 1111.

[0118] On the other hand, if the result of the above-mentioned judgment is that the evaluation data does not indicate low confidence (1102: NO), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 judges whether the evaluation data indicates high confidence (1103).

[0119] If the result of the above-mentioned determination is that the evaluation data does not indicate high confidence (1103: NO), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 transitions the process to 1116.

[0120] On the other hand, if the result of the above-mentioned judgment is that the evaluation data indicates high confidence (1103: YES), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 judges whether the evaluation data indicates low stability (1104).

[0121] If the result of the above-mentioned judgment is that the evaluation data does not indicate low stability (1104: NO), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 obtains all learning data similar to the evaluation data from the learning dataset 126 in the learning dataset storage unit 116 (1105).

[0122] As a method for determining the above-mentioned "similarity," for example, a method using formulas 1501 and 1502 shown in Fig. 15 can be assumed. Alternatively, as shown in Fig. 16, a configuration 1503 can be assumed in which an AI model that learns similar data and its similarity as learning data is prepared and the similarity is determined.

[0123] Next, the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 obtains the labels of the learning data obtained in step 1105 from the label information 127 in the learning dataset storage unit 16 (1106).

[0124] Furthermore, the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 compares the label acquired in step 1106 with the inference result of the evaluation data obtained in step 1101 (1107).

[0125] If the comparison result shows that the label and the inference result match (1108: YES), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 transitions the process to step 1116.

[0126] On the other hand, if the result of the above-mentioned judgment is that the label and the inference result do not match (1108: NO), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 deletes the learning data from the learning dataset 126 in the learning dataset storage unit 116 (1111).

[0127] In addition, the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 deletes the label of the learning data from the label information 127 of the learning dataset storage unit 116 (1112) and transitions the process to step 1116.

[0128] In such a case where the confidence and stability of the evaluation data are high, if the label of the training data similar to the evaluation data does not match the inference result of the evaluation data, the training data or label can be deleted, thereby making it possible to eliminate the training data with an inappropriate correct label (see FIG. 22). Note that the training dataset editing unit 115 may display information on the training data and labels to be deleted on the output device 106.

[0129] On the other hand, in cases where the confidence level in the evaluation data is low, it is possible to eliminate inadvertent proximity of learning data with different correct labels, and to eliminate learning data in which features different from the learning intention are dominant (see FIG. 23). Note that the learning dataset editing unit 115 may display information on the learning data and labels to be deleted on the output device 106.

[0130] In addition, in cases where the confidence level of the evaluation data is high and the stability level is low, it is possible to supplement the missing training data (see FIG. 24). Note that the training dataset editing unit 115 may display information on the supplemented training data on the output device 106.

[0131] Here, if the result of the judgment in the above-mentioned process 1104 is that the evaluation data indicates low stability (1104: YES), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 generates data similar to the evaluation data (1113).

[0132] As shown in FIG. 16, possible methods for generating similar data include method 1601, which generates data by slightly changing the features of the original data, method 1602, which generates data by slightly rotating, deforming, or discoloring the original data, and method 1603, which generates data using a data generation neural network.

[0133] Furthermore, the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 adds the learning data generated in step 1113 to the learning dataset 126 in the learning dataset storage unit 116 (1114).

[0134] Next, the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 Then, the label related to the learning data generated in step 1113 is added to the label information 127 in the learning data set storage unit 116 (1115), and the process proceeds to step 1116.

[0135] In addition, the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 determines whether any unprocessed evaluation data remains among the evaluation dataset 125 (1116), and if there is any unprocessed evaluation data remaining, i.e., the evaluation data to be processed in the current flow is not the last evaluation data (1116: NO), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 returns the process to 1101.

[0136] On the other hand, if the result of the above-mentioned judgment is that there is nothing left unprocessed, that is, the evaluation data to be processed in the current flow is the last evaluation data (1116: YES), the learning dataset editing unit 115 of the machine learning dataset evaluation support device 100 terminates this flow.

[0137] <Specific example: Example 1> Next, the machine learning dataset evaluation support technology of the present embodiment will be described along with a more specific example. Fig. 17 is a block diagram showing a functional configuration of a machine learning dataset evaluation support device 100 in the first embodiment.

[0138] The basic configuration is the same as that shown in Fig. 1, except that the AI ​​model held in trained AI model holding unit 112 in inference execution unit 111 is sign identification AI model 118A. That is, the trained AI model in this case is one that identifies road signs from images of the road signs.

[0139] 18 shows examples of four pieces of evaluation data 1801 (1 to 4) related to a road sign showing the numerical value "60", its inference result 1802, and difference analysis result 1803. Also, threshold values ​​1901 to 1903 related to such difference analysis results are shown in FIG.

[0140] Moreover, the evaluation results 2001 for each evaluation data are shown in FIG. 20. The evaluation result of evaluation data 1 is "confidence is equal to or greater than the high confidence threshold, stability is equal to or greater than the low stability threshold, so confidence is high and stability is also high." The evaluation result of evaluation data 2 is "confidence is less than the low confidence threshold, so confidence is low." The evaluation result of evaluation data 3 is "stability is less than the low stability threshold, so stability is low." Moreover, the evaluation result of evaluation data 4 is "confidence is equal to or greater than the high confidence threshold, stability is equal to or greater than the low stability threshold, so confidence is high and stability is also high."

[0141] Taking these evaluation results into consideration, for evaluation data 1, as shown in Fig. 21, similar data whose labels match the inference results are retained. For evaluation data 4, as shown in Fig. 22, similar data whose labels do not match the inference results are deleted together with their labels. For evaluation data 2, as shown in Fig. 23, learning data is deleted together with the labels. For evaluation data 3, as shown in Fig. 24, similar data is added.

[0142] <Example: Example 2> It is also possible to assume a form in which the machine learning dataset evaluation support device 100 further includes an analysis result display unit 117 as shown in Fig. 25. In this case, the analysis result display unit 117 causes the output device 106 to display distribution diagrams 2601 and 2602 indicating the confidence level of the inference result as shown in Fig. 26. Similarly, the analysis result display unit 117 causes the output device 106 to display distribution diagrams 2701 and 2702 indicating the stability of the inference result as shown in Fig. 27.

[0143] 26 and 27. As shown in FIG. 28, the analysis result display unit 117 displays the inference results shown in FIG. Based on the results, the output device 106 displays the threshold values ​​2801 to 2803 for the high confidence level, the low confidence level, and the low stability level.

[0144] Moreover, the analysis result display unit 117 causes the output device 106 to display the evaluation result 2901 of the evaluation data, as shown in FIG.

[0145] Although the best mode for carrying out the present invention has been specifically described above, the present invention is not limited to this, and various modifications are possible without departing from the spirit and scope of the present invention.

[0146] According to this embodiment, by analyzing the inference characteristics of a trained AI model from the confidence and stability, it is possible to configure a training data set in which data that causes inappropriate training is removed and data with insufficient training is added. As a result, it is possible to configure a refined training data set that avoids inappropriate training and insufficient training.

[0147] The description of this specification makes at least the following clear: That is, the machine learning dataset evaluation support device of this embodiment may further execute a process of displaying the analysis result on an output device.

[0148] This makes it possible to clearly show the differences between multiple inference results to the user in terms of the degree of certainty and the stability of the inference results.

[0149] In addition, in the machine learning dataset evaluation support device of this embodiment, the inference execution unit may use AI models that have been trained by changing the learning dataset or model structure during learning as the multiple trained AI models.

[0150] This makes it possible to efficiently obtain inference results that take into account fluctuations in the training data set and model structure.

[0151] In addition, in the machine learning dataset evaluation support device of this embodiment, the inference execution unit may use trained AI models whose model structure has been changed during inference as the multiple trained AI models.

[0152] This makes it possible to reduce the number of models by sequentially changing one model, and to improve the accuracy of the analysis of the inference results by reflecting the history of the model's inference results in the model changes. This in turn makes it possible to construct a more sophisticated training data set that avoids inappropriate learning and insufficient learning.

[0153] In addition, in the machine learning dataset evaluation support device of this embodiment, the inference execution unit may be configured to sequentially execute the multiple trained AI models to obtain multiple inference results.

[0154] This allows for a reduction in the computational resources required to execute inference since only one model is executed at a time.

[0155] In addition, in the machine learning dataset evaluation support device of this embodiment, the inference execution unit may execute the multiple trained AI models in parallel to obtain multiple inference results simultaneously.

[0156] This makes it possible to shorten processing time by obtaining multiple inference results simultaneously.

[0157] In addition, in the machine learning dataset evaluation support device of this embodiment, the learning dataset editing unit may further execute a process of deleting from the learning dataset, learning data that is similar to evaluation data in which the confidence level of the inference result is greater than a predetermined threshold and has a correct answer label that differs from the inference result of the data to be evaluated.

[0158] This makes it possible to eliminate training data with inappropriate correct labels.

[0159] In addition, in the machine learning dataset evaluation support device of this embodiment, the learning dataset editing unit may further perform a process of deleting, from the learning dataset, learning data that is similar to evaluation data whose confidence level of the inference result is smaller than a predetermined threshold value.

[0160] This makes it possible to eliminate inadvertent proximity of training data with different correct labels, and to eliminate training data in which features different from the learning intent predominate.

[0161] In addition, in the machine learning dataset evaluation support device of this embodiment, the learning dataset editing unit may further perform a process of generating learning data similar to evaluation data in which the confidence level of the inference result is greater than a predetermined threshold and the stability is less than a predetermined threshold, and adding the learning data to the learning dataset.

[0162] This makes it possible to supplement missing learning data. [Explanation of symbols]

[0163] 100 Machine learning dataset evaluation support device 101 Storage device 102 Programs 103 Memory 104 Arithmetic unit 105 Input Device 106 Output Device 107 Communication Equipment 110 Evaluation Data Set Storage Unit 111 Inference execution unit 112 Trained AI model storage unit 113 Difference Analysis Department 114 Data Evaluation Department 115 Learning Dataset Editorial Department 116 Learning Data Set Storage Unit 117 Analysis result display section 118 trained AI models 125 Evaluation Dataset 126 Training Data Set 127 Label Information

Claims

1. A learning dataset storage unit that stores a learning dataset and label information used in supervised machine learning; An inference execution unit that inputs a predetermined evaluation data set into a plurality of trained AI models to obtain a plurality of inference results; A difference analysis unit that analyzes the differences between the plurality of inference results from the confidence level of the inference result and the stability of the inference result in each of the plurality of trained AI models; a data evaluation unit that evaluates the evaluation data set based on a result of the difference analysis; a learning dataset editing unit that edits the learning dataset and label information stored in the learning dataset storage unit based on an evaluation result of the evaluation dataset; A machine learning dataset evaluation support device comprising:

2. The machine learning dataset evaluation support device according to claim 1 , further comprising a process for displaying the analysis results on an output device.

3. The machine learning dataset evaluation support device according to claim 1, characterized in that the inference execution unit uses AI models trained by changing a learning dataset or a model structure during learning as the multiple trained AI models.

4. The machine learning dataset evaluation support device according to claim 1, characterized in that the inference execution unit uses trained AI models whose model structure has been changed during inference as the multiple trained AI models.

5. The machine learning dataset evaluation support device according to claim 1 , characterized in that the inference execution unit sequentially executes the multiple trained AI models to obtain multiple inference results.

6. The machine learning dataset evaluation support device according to claim 1 , characterized in that the inference execution unit executes the multiple trained AI models in parallel to obtain multiple inference results simultaneously.

7. 2. The machine learning dataset evaluation support device according to claim 1, wherein the learning dataset editing unit further executes a process of deleting from the learning dataset, learning data that is similar to evaluation data in which the confidence level of the inference result is greater than a predetermined threshold and has a correct answer label that differs from the inference result of the data to be evaluated.

8. The machine learning dataset evaluation support device according to claim 1, characterized in that the learning dataset editing unit further executes a process of deleting, from the learning dataset, learning data that is similar to evaluation data in which the confidence level of the inference result is smaller than a predetermined threshold value.

9. 2. The machine learning dataset evaluation support device according to claim 1, wherein the learning dataset editing unit further executes a process of generating learning data similar to evaluation data in which the confidence level of the inference result is greater than a predetermined threshold and the stability is less than a predetermined threshold, and adding the learning data to the learning dataset.

10. An information processing device, A learning dataset and label information used in supervised machine learning are stored in a storage device; A process of inputting a predetermined evaluation data set into a plurality of trained AI models to obtain a plurality of inference results, and calculating the difference between the plurality of inference results for each of the plurality of trained AI models. a process of analyzing the inference result based on a degree of confidence and a degree of stability of the inference result in the difference, a process of evaluating the evaluation dataset based on the result of the difference analysis, and a process of editing the learning dataset and label information stored in the storage device based on the evaluation result of the evaluation dataset, A method for supporting evaluation of machine learning datasets, comprising:

Images