Dataset generation method, dataset generation program, and dataset generation device

Abstract

This invention provides a dataset generation method, a dataset generation program, and a dataset generation device that set up training and evaluation datasets for generating a trained model that suppresses the decline in prediction accuracy, even when the number of datasets is small. [Solution] The dataset generation method includes a feature extraction step S102 for extracting features from datasets classified into classification classes that exhibit different characteristics from each other; a classification step S103 for classifying datasets belonging to each classification class into a plurality of subclassification classes based on the features; a splitting step S104 for splitting datasets belonging to each subclassification class into a subtraining dataset and a subevaluation dataset; and an output step S106 for outputting the subtraining dataset and the subevaluation dataset.

Description

Technical Field

[0001] The present invention relates to a dataset generation method, a dataset generation program, and a dataset generation device.

Background Art

[0002] Conventionally, when constructing a learned model, a dataset including a plurality of pairs of data and teacher labels is divided into a learning dataset and an evaluation dataset, and learning is performed using the learning dataset, and the learned model is evaluated using the evaluation dataset, thereby generating a learned model (see, for example, Patent Document 1). The learning dataset and the evaluation dataset are, for example, randomly divided or extracted from the dataset, or divided or extracted based on features.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

[0005] Incidentally, when the number of data points (samples) is small, it is difficult to uniformly divide the data into training and evaluation datasets, and it is known that the classification accuracy (generalization) when predicting with new data outside the dataset decreases compared to the accuracy when classifying with the evaluation data using the trained model (see, for example, Non-Patent Documents 1-3). Non-Patent Documents 1-3 specifically report that when randomly splitting, if the number of data points is small, the distribution of the training dataset and the evaluation dataset may differ, for example, when data similar to the training dataset does not exist but exists only in the evaluation dataset, and the classification accuracy of the trained model trained using this dataset decreases on new data. For this reason, there has been a need to set up training and evaluation datasets that can generate a trained model that suppresses the decrease in classification accuracy on new data, even when the number of data points is small.

[0006] The present invention has been made in view of the above, and aims to provide a dataset generation method, a dataset generation program, and a dataset generation device that can set up training and evaluation datasets that can generate a trained model that suppresses the decrease in classification accuracy for new data, even when the amount of data is small. [Means for solving the problem]

[0007] To solve the above-mentioned problems and achieve the objective, the present invention provides a dataset generation method in which a computer generates a training dataset for generating a trained model and an evaluation dataset for evaluating the trained model from a dataset, comprising: a feature extraction step of extracting features from datasets classified into classification classes that exhibit different characteristics from each other; a classification step of classifying datasets belonging to each classification class into a plurality of subclassification classes based on the features; a splitting step of splitting datasets belonging to each subclassification class into a subtraining dataset and a subevaluation dataset; and an output step of outputting the subtraining dataset and the subevaluation dataset.

[0008] Furthermore, in the dataset generation method according to the present invention, the dataset is classified into a plurality of classification classes based on its features.

[0009] Furthermore, the dataset generation method according to the present invention further includes an integration step in which all of the sub-training datasets are integrated and set as a training dataset, and all of the sub-evaluation datasets are integrated and set as an evaluation dataset, and the output step outputs the integrated training dataset and the evaluation dataset.

[0010] Furthermore, in the dataset generation method according to the present invention, the division step involves calculating the centroid from the feature distribution of the dataset belonging to the subclassification class, and dividing the dataset into a subtraining dataset and a subevaluation dataset based on the distance from the centroid.

[0011] Furthermore, in the dataset generation method according to the present invention, the number of data points included in the sub-training dataset divided by the division step is equal to or greater than the number of data points included in the sub-evaluation dataset.

[0012] Furthermore, in the dataset generation method according to the present invention, in the above invention, if there is one data point belonging to the subclassification class, the division step sets that data point as the training dataset.

[0013] Furthermore, the dataset generation program according to the present invention is a dataset generation program that causes a computer to generate a training dataset used for generating a trained model and an evaluation dataset used for evaluating the trained model from a dataset, and causes the computer to execute the following steps: a feature extraction step of extracting features from datasets classified into classification classes that exhibit different characteristics from each other; a classification step of classifying datasets belonging to each classification class into a plurality of subclassification classes based on the features; a splitting step of splitting datasets belonging to each subclassification class into a subtraining dataset and a subevaluation dataset; and an output step of outputting the subtraining dataset and the subevaluation dataset.

[0014] Furthermore, the dataset generation device according to the present invention is a dataset generation device that generates a training dataset used for generating a trained model and an evaluation dataset used for evaluating the trained model from a dataset, and comprises: a feature extraction unit that extracts features from the datasets classified into classification classes that exhibit different characteristics from each other; a classification unit that classifies the datasets belonging to each classification class into a plurality of subclassification classes based on the features; a splitting unit that divides the datasets belonging to each subclassification class into a subtraining dataset and a subevaluation dataset; and an output unit that outputs the subtraining dataset and the subevaluation dataset. [Effects of the Invention]

[0015] According to the present invention, even when the number of datasets is small, it is possible to set up training and evaluation datasets that can generate a trained model with suppressed deterioration in prediction accuracy. [Brief explanation of the drawing]

[0016] [Figure 1] Figure 1 is a diagram showing a schematic configuration of an estimation system according to an embodiment of the present invention. [Figure 2] Figure 2 is a block diagram showing a configuration of a learning device included in the estimation system according to an embodiment of the present invention. [Figure 3] Figure 3 is a block diagram showing a configuration of an estimation device included in the estimation system according to an embodiment of the present invention. [Figure 4] Figure 4 is a diagram for explaining a flow of estimation processing performed by the estimation system according to an embodiment of the present invention. [Figure 5] Figure 5 is a flowchart showing a flow of dataset setting processing performed by the learning device according to an embodiment of the present invention. [Figure 6] Figure 6 is an image diagram showing a flow of dataset setting processing performed by the learning device according to an embodiment of the present invention. [Figure 7] Figure 7 is a diagram for explaining an example of an image used for learning. [Figure 8] Figure 8 is a diagram for explaining a classification result by a learned model learned by the segmentation method according to the present embodiment. [Figure 9] Figure 9 is a diagram for explaining a classification result by a learned model learned by a random segmentation method. [Figure 10] Figure 10 is a diagram for explaining evaluation results by the segmentation method according to the present embodiment and the random segmentation method. [Figure 11] Figure 11 is a diagram for explaining an example of an image used for evaluation. [Figure 12] Figure 12 is a diagram for explaining an evaluation result when using the image example shown in Figure 11.

Mode for Carrying Out the Invention

[0017] Embodiments of the dataset generation program, dataset generation method, and dataset generation apparatus according to the present invention will be described in detail below with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, the individual embodiments of the present invention are not independent but can be combined and implemented as appropriate.

[0018] (Embodiment) Figure 1 is a diagram showing a schematic configuration of an estimation system according to one embodiment of the present invention. The estimation system 1 shown in these figures comprises a learning device 2 that creates a training dataset and generates a trained model using the created training dataset, an estimation device 3 that estimates the defect type of the target to be estimated using the trained model generated by the learning device 2, a display device 4 that displays information including the estimation results of the estimation device 3, an input device 5, and a data acquisition device 6.

[0019] In this embodiment, the defect type estimated by the estimation device 3 is, for example, the type of defect such as scratches or foreign matter that may occur during the manufacturing of a product. In this embodiment, the estimation of the classification class of defects captured in image data is described, but the method according to the present invention can be applied to the classification of other features such as one-dimensional signal data, tabular data, three-dimensional voxels, and point cloud data.

[0020] In this embodiment, a dataset is used for generating a trained model in which images containing defects are associated with the corresponding defect types.

[0021] The learning device 2 is electrically connected to the estimation device 3, the input device 5, and the data acquisition device 6. The learning device 2 selectively extracts a training dataset and generates and outputs a trained model by training using the extracted training dataset. Figure 2 is a block diagram showing the configuration of the learning device in the estimation system according to an embodiment of the present invention. The learning device 2 has a training dataset generation unit 21, a learning unit 22, a control unit 23, and a storage unit 24. In this embodiment, the dataset generation device is composed of at least the training dataset generation unit 21 and the storage unit 24.

[0022] The training dataset generation unit 21 includes a feature extraction unit 211, a classification unit 212, a splitting unit 213, and an integration unit 214.

[0023] The feature extraction unit 211 extracts features from the image data contained in the data. For example, the feature extraction unit 211 reads a dataset stored in the memory unit 24 and extracts features from the image data contained in this dataset. For example, the feature extraction unit 211 calculates features using an autoencoder. In this case, the feature extraction unit 211 compresses the image data into 16-dimensional features using a reconstruction model that reconstructs the image data. Note that the above classification method is just one example, and is not limited to the above method as long as features can be calculated from the data and classified according to those features, for example, features based on human-defined rules or a pre-trained neural network that has been trained on general-purpose data in advance.

[0024] The classification unit 212 classifies the dataset based on its features. For example, the classification unit 212 may classify the dataset into multiple classification classes according to pre-set conditions or input instructions, or it may classify the datasets belonging to each classification class into multiple subclassification classes based on their features.

[0025] The splitting unit 213 splits the dataset belonging to the subclassification class into a subtraining dataset and a subevaluation dataset.

[0026] The integration unit 214 integrates all the divided sub-training datasets into a training dataset, and also integrates all the sub-evaluation datasets into an evaluation dataset.

[0027] The learning unit 22 generates a trained model through training using the training dataset generated by the training dataset generation unit 21 and validation using the evaluation dataset. The training performed by the learning unit 22 can employ known machine learning methods such as linear discriminant analysis, logistic regression, k-nearest neighbors, support vector machines, decision trees, random forests, gradient boosting trees, and Gaussian processes, as well as deep learning. The trained model is, for example, a neural network consisting of an input layer, hidden layers, and output layers, with each layer having one or more nodes. In the case of image data, a convolutional neural network with weight parameters only for neighboring pixels called convolutional layers, or a Transformer with a self-attention mechanism, can be used. Information such as network parameters in the trained model is stored in the memory unit 24. Network parameters include information about the weights and biases between layers of the neural network.

[0028] Furthermore, when the learning unit 22 generates a trained model using, for example, a neural network, it first makes a prediction using a neural network with randomly initialized weights, and calculates the error (cross-entropy) between the obtained prediction result and the actual training label. Next, it backpropagates the obtained error from the output layer to the input layer and updates the weight parameters of each layer of the neural network using stochastic gradient descent. By repeating this step many times, a trained model with improved classification accuracy can be obtained. Initialization of the neural network is not always necessary; training time can be shortened and classification accuracy can be improved by using a method called fine-tuning, which involves further training this dataset using a trained model that has already been trained on another dataset.

[0029] The control unit 23 comprehensively controls the operation of the learning device 2.

[0030] The memory unit 24 stores data including various programs for operating the learning device 2, and various parameters necessary for the operation of the learning device 2. The various programs include a dataset generation program that generates a dataset for generating a trained model, and a trained model generation program that generates a trained model by training using a training dataset and an evaluation dataset. The memory unit 24 also has a dataset storage unit 241 for storing datasets.

[0031] The memory unit 24 is composed of a ROM (Read Only Memory) on which various programs are pre-installed, and RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., which store calculation parameters and data for each process.

[0032] Various programs can be recorded on computer-readable recording media such as HDDs, flash memory, CD-ROMs, DVD-ROMs, and Blu-ray® discs and widely distributed. Furthermore, the learning device 2 can acquire various programs via a communication network. This communication network can be comprised of existing public telephone networks, LANs (Local Area Networks), WANs (Wide Area Networks), etc., and can be wired or wireless.

[0033] The learning device 2 having the above functional configuration is a computer composed of one or more hardware components such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array).

[0034] The estimation device 3 is electrically connected to the learning device 2, the display device 4, the input device 5, and the data acquisition unit 6. The estimation device 3 uses the trained model acquired from the learning device 2 to output a classification class of defects present in the image data. Figure 3 is a block diagram showing the configuration of the estimation device in the estimation system according to an embodiment of the present invention. The estimation device 3 has a calculation unit 31, a control unit 32, and a storage unit 33.

[0035] The calculation unit 31 calculates a classification class for defects in image data, which is estimated using the data to be estimated (for example, image data showing defects in a product) acquired from the data acquisition unit 6 and a trained model acquired from the learning device 2.

[0036] Figure 4 is a diagram illustrating the flow of the estimation process performed by the estimation system according to an embodiment of the present invention. The trained model 100 is generated by training using the training dataset from the dataset group IP, which is the training dataset, and evaluation using the evaluation dataset. The calculation unit 31 acquires this trained model 100 and uses the trained model 100 to acquire, for example, the estimation classification class OP.

[0037] The control unit 32 comprehensively controls the operation of the estimation device 3. The control unit 32 includes a display control unit 321 that displays the calculation results (estimation results) of the calculation unit 31 on the display device 4. In addition to the estimation results, the display control unit 321 may also display information about the target of estimation (for example, image data of the target) on the display device 4.

[0038] The memory unit 33 stores data including various programs for operating the estimation device 3, and various parameters necessary for the operation of the estimation device 3. The various programs include estimation programs that are executed using trained models. The memory unit 33 is composed of a ROM with various programs pre-installed, and RAM, HDD, SSD, etc., for storing calculation parameters and data for each process.

[0039] Various programs can be recorded on computer-readable recording media such as HDDs, flash memory, CD-ROMs, DVD-ROMs, and Blu-ray® discs and widely distributed. Furthermore, the estimation device 3 can acquire various programs via a communication network. This communication network can be, for example, an existing public network, LAN, or WAN, and can be wired or wireless.

[0040] The estimation device 3, having the above functional configuration, is a computer composed of one or more hardware components such as a CPU, GPU, ASIC, and FPGA.

[0041] The display device 4 is a display made of liquid crystal or organic EL (Electro-Luminescence), and is electrically connected to the estimation device 3. The display device 4 acquires and displays display data output from the estimation device 3 under the control of the display control unit 321. The display device 4 may also have an audio output function such as a speaker.

[0042] Input device 5 accepts various types of information, including settings related to the process of estimating characteristic values, and outputs the received information to learning device 2 and estimation device 3. Input device 5 is configured using a user interface such as a keyboard, mouse, microphone, and touch panel.

[0043] The data acquisition device 6 is the part that acquires image data of the product, and acquires image data of the product from an existing product inspection device. The data acquisition method is not limited to this; image data may be acquired directly from a camera, or image data, training labels, and other information may be acquired from external storage via a communication network. The communication network referred to here is configured using, for example, an existing public telephone network, LAN (Local Area Network), WAN (Wide Area Network), etc., and can be wired or wireless.

[0044] Next, we will explain the processes of the learning device 2 and the estimation device 3. In the estimation system 1, the learning unit 22 generates a trained model by learning using the training dataset generated by the training dataset generation unit 21, and the estimation device 3 uses the trained model to output the estimated defect classification class from the target image.

[0045] First, we will explain the method by which the training dataset generation unit 21 generates training datasets and evaluation datasets. Figure 5 is a flowchart showing the flow of the dataset setting process performed by a learning device according to one embodiment of the present invention. Figure 6 is an illustrative diagram showing the flow of the dataset setting process performed by a learning device according to one embodiment of the present invention.

[0046] The control unit 23 acquires a dataset that has been classified into multiple classification classes (step 101). In this case, the control unit 23 acquires the dataset by referring to the storage unit 24 or reading from an external server. As described above, this dataset is a group of data in which image data containing defects and corresponding training labels, in this case defect types, are associated, and has been pre-classified into multiple classification classes based on user input or other instructions.

[0047] For example, as shown in Figure 6(a), multiple data points are divided into three distinct groups (Defect A, Defect B, and Defect C) in the feature space. Each group corresponds to a different type of defect, such as scratches, internal state changes, or foreign matter adhesion observed in images of a product.

[0048] Then, the feature extraction unit 211 extracts features from each of the multiple data points (step S102). The feature extraction unit 211 extracts features of defects that appear in the image, for example, using an autoencoder. At this time, the feature extraction unit 211 compresses the image to, for example, 16-dimensional features using a reconstruction model that reconstructs the image. This compression extracts features specific to the defects.

[0049] After feature extraction, the classification unit 212 groups the datasets for each classification class into subclassification classes (step S103). The classification unit 212 divides the datasets (feature distributions) in the feature space into multiple groups (subclassification classes), for example, using a Gaussian mixture model (GMM). The classification model here is not limited to a Gaussian mixture model; any unsupervised classification method or clustering method can be used, including k-means, k-means++, x-means, spectral clustering, DBSCAN, HDBSCAN, Ward's method, OPTICS, BIRCH, MeanShift, etc.

[0050] The classification unit 212 classifies the data into N / 10 subclassification classes, for example, when the number of data belonging to a classification class is N. In this way, it is preferable that the number of subclassification classes be set such that multiple data points are assigned to each subclass, relative to the number of data points belonging to a classification class. The above classification method is just one example; it is not limited to this method, as long as the characteristic quantities of defects in the image can be calculated and the images can be classified according to those characteristic quantities.

[0051] For example, as shown in Figure 6(b), for example, for a group belonging to defect A, multiple data points can be divided into four subgroups (first subgroup GrA1, second subgroup GrA2, third subgroup GrA3, and fourth subgroup GrA4) in the feature space. Each group corresponds to a different shape or color in a data set that is grouped together as similar defects. Figure 6(b) shows an example of a defect in each subgroup.

[0052] Returning to Figure 5, after classification into subclasses, the splitting unit 213 splits the multiple data and labels belonging to each subclass into a subtraining dataset and a subevaluation dataset (step S104). The splitting unit 213 calculates the centroid from the distribution of multiple data within the subclass in the feature space, and splits the data into a subtraining dataset and a subevaluation dataset according to the distance from the centroid. The splitting unit 213 distributes the data and labels to the subtraining dataset and the subevaluation dataset based on the distance from the centroid, for example, so that the number of data in the subtraining dataset and the number of data in the subevaluation dataset are in a predetermined ratio. In this case, it is preferable that the number of data in the subtraining dataset is greater than or equal to the number of data in the subevaluation dataset.

[0053] One method of sorting based on distance from the centroid is to include data with a distance close to the mean in the subtraining dataset. For example, in the example shown in Figure 6(b), the centroid C of the distribution in space is used for each of the first subgroup GrA1, the second subgroup GrA2, and the third subgroup GrA3. A1 , C A2 , C A3 The centroid is calculated, and the data is distributed based on the distance from this centroid. The shape of the distribution may be set based on a predefined shape such as a circle, ellipse, or polygon that includes all the data belonging to the subgroup, or the user may set the shape of the subgroups based on the distribution of the data.

[0054] In addition, the splitting unit 213 may, for example, randomly distribute data between the sub-training dataset and the sub-evaluation dataset so that the number of data points in the sub-training dataset and the number of data points in the sub-evaluation dataset are in a predetermined ratio, or it may alternately distribute data between the sub-training dataset and the sub-evaluation dataset in order of proximity to the centroid. In this case, if there is only one data point and label in a subclass, it is preferable to allocate that data point and label to the training dataset in order to prioritize increasing the comprehensiveness of the types of data used for this training.

[0055] For example, as shown in Figure 6(c), for the first group GrA, the dataset is divided in the feature space into a training subgroup (GrA-L) and an evaluation subgroup (GrA-E), respectively, from the first subgroup GrA1, the second subgroup GrA2, and the third subgroup GrA3, to be used as the training dataset. Since the fourth subgroup GrA4 has only one data point, this data is included in the training subgroup (GrA-L).

[0056] Returning to Figure 5, after the dataset is split, the integration unit 214 integrates the sub-training dataset and the sub-evaluation dataset (step S105). The integration unit 214 integrates all the sub-training datasets that were split by the splitting unit 213 into a training dataset, and also integrates all the sub-evaluation data into an evaluation dataset. As a result, the training dataset and evaluation dataset are set from the dataset read by the control unit 23.

[0057] After the datasets are integrated, the control unit 23 outputs the integrated dataset (step S106). The output destinations are the storage unit 24 and the learning unit 22. As a result, the integrated dataset is stored in the storage unit 24, or the learning unit 22 executes the process of generating a trained model.

[0058] The learning unit 22 generates a trained model using the divided training dataset and evaluation dataset. As described above, the learning unit 22 generates a trained model by training with the training dataset and verifying the estimation accuracy with the evaluation dataset. This trained model can, for example, take an image containing defects as input and output the estimation result of the classification class of the defects (e.g., Group 1 GrA, Group 2 GrB, Group 3 GrC).

[0059] The estimation device 3 then inputs the product image into a trained model generated by the learning unit 22 via the calculation unit 31, estimates the group of defects (classification class) visible in the image, and outputs the estimation result.

[0060] Here, we will explain the classification accuracy of the trained model based on the method of splitting the training and evaluation datasets, referring to Figures 7 to 12. First, we will explain the images used in this test. Figure 7 is a diagram illustrating an example of images used for training. 369 images (dataset) showing defects were divided into five main types of defects (Defect A to Defect E), and the images were separated as shown in Figure 7. This grouping was done by the user's visual inspection.

[0061] Using the division method according to this embodiment, when the images (datasets) of each defect were divided into a training dataset and an evaluation dataset, they were separated as shown in Table 1. [Table 1]

[0062] Furthermore, when the image data for each defect was separated into a training dataset and an evaluation dataset using a random splitting method, the results were as shown in Table 2. In the random splitting method, the dataset was split so that the ratio of training dataset to evaluation dataset was 8:2. [Table 2]

[0063] Comparing Table 1 and Table 2, there was no significant difference in the ratio of training data to evaluation data; however, for each defect, the number of training data and evaluation data differed in their respective data splitting patterns.

[0064] Furthermore, a trained model was obtained using the divided training dataset. Next, the obtained trained model was used to classify evaluation data, and the classification accuracy of the trained model was evaluated by comparing the classification results with the results obtained by human visual inspection (correct answers).

[0065] Figure 8 illustrates the classification results of the evaluation dataset in Table 1 by a trained model trained using the training dataset in Table 1 obtained by the splitting method according to this embodiment. Figure 9 illustrates the classification results of the evaluation dataset in Table 2 by a trained model trained using the training dataset in Table 2 obtained by the random splitting method. Figures 8 and 9 show the relationship between visual judgment and judgment by the trained model. Visual judgment was considered correct, and if the judgment result by the trained model matched the shortcomings of the visual judgment, the judgment by the trained model was considered correct. As shown in Figures 8 and 9, the trained model trained by the splitting method according to this embodiment showed a higher accuracy rate (number of correct answers / total number of data points).

[0066] Here, the recall rate between the partitioning method according to this embodiment and the random partitioning method was determined by the following equation (1). Recall = ((Accuracy of answering flaw A) + (Accuracy of answering flaw B) + (Accuracy of answering flaw C)) + (Accuracy rate for defect D) + (Accuracy rate for defect E)) / Number of defect types ...(1) This recall rate is calculated as the average accuracy rate at which the trained model correctly identified each defect type. In this case, there are five defect types: A through E. The reason for using recall rate rather than overall accuracy to evaluate the trained model is to prevent the influence of the accuracy rate of classes with smaller sample sizes from being diminished, as the accuracy rate of classes with larger sample sizes is easily influenced by the larger sample size.

[0067] The recall rate was 0.80 for the trained model trained using the partitioning method according to this embodiment, and 0.72 for the trained model trained using the random partitioning method. This indicates that the trained model trained using the partitioning method according to this embodiment has a higher recall rate. In particular, the trained model trained using the partitioning method according to this embodiment showed a higher accuracy rate in cases where the sample size is small compared to the trained model trained using the random partitioning method.

[0068] Figure 10 illustrates the splitting method according to this embodiment and the evaluation results of the random splitting method. Figure 10 shows the variability of recall when the trained model using the splitting method according to this embodiment and the trained model using the random splitting method are evaluated five times each. As shown in Figure 10, the trained model using the splitting method according to this embodiment has a higher recall and smaller variability than the trained model using the random splitting method. While the trained model using the random splitting method may sometimes yield a model with high recall due to bias in the types of datasets or combinations of evaluation dataset types, overall, the recall of the generated trained models was low.

[0069] In the evaluation results shown in Figure 10, the recall (mean ± σ) was 0.806 ± 0.044 for the trained model trained using the partitioning method according to this embodiment, and 0.726 ± 0.088 for the trained model trained using the random partitioning method. This indicates that the trained model trained using the partitioning method according to this embodiment has a higher recall and a smaller σ. As a result, it can be concluded that with the random partitioning method, defects belonging to the first subgroup GrA1 may be included in the evaluation dataset but not in the training dataset. Therefore, a trained model that has not learned defects belonging to the first subgroup GrA1 may not be able to correctly identify defects belonging to the first subgroup GrA1.

[0070] Next, using the trained models generated from the datasets shown in Tables 1 and 2, we predicted image defects using a different dataset than the one shown in Figure 7. Figure 11 is a diagram illustrating the example images used for evaluation. The images (dataset) shown in Figure 11 are different from the images (dataset) shown in Figure 7 and were not used for training or evaluation.

[0071] Figure 12 is a diagram illustrating the evaluation results when using the image example shown in Figure 11. Figure 12 shows the variability of recall when the trained model using the division method according to this embodiment and the trained model using the random division method are each evaluated 25 times. As shown in Figure 12, the trained model using the division method according to this embodiment has a higher recall than the trained model using the random division method, and the distribution of its variability also stays within a high recall range.

[0072] In the evaluation results shown in Figure 12, the recall (mean ± σ) was 0.664 ± 0.047 for the trained model trained using the splitting method according to this embodiment, and 0.619 ± 0.039 for the trained model trained using the random splitting method. This result shows that the trained model trained using the splitting method according to this embodiment has a higher recall. These evaluation results also indicate that the trained model using the partitioning method according to this embodiment can achieve a higher recall rate than the trained model using the random partitioning method.

[0073] In the embodiment described above, a dataset classified into multiple classification classes that represent different types of defects is divided into a training dataset and an evaluation dataset by classifying the datasets belonging to each classification class into subclassification classes, and then dividing the datasets belonging to the subclassification classes into a subtraining dataset and a subevaluation dataset. According to this embodiment, since datasets that are finely classified within the same type of defect are divided into training datasets and evaluation datasets, it becomes possible to divide each type evenly into training datasets and evaluation datasets. As a result, even when the number of datasets is small, it is possible to generate a trained model that suppresses the decrease in prediction accuracy.

[0074] In the embodiments described above, the classification into classification classes was explained as being done in advance by the user. However, the classification unit 212 may classify the dataset into multiple classification classes. For example, the classification unit 212 classifies the dataset into multiple classification classes based on features, such as classifying into subclasses. In this case, when classifying into subclasses, the classification unit 212 may use the features used when classifying into classification classes to classify into subclasses, or it may newly calculate features calculated according to the conditions for classification into subclasses and classify into subclasses.

[0075] Furthermore, although the above-described embodiment explains an example in which the sub-training dataset and the sub-evaluation dataset are integrated by the integration unit 214, it is also possible to have a configuration in which the sub-training dataset and the sub-evaluation dataset are output and stored in the storage unit 24 without integration by the integration unit 214, each belonging to a subclassification class.

[0076] (Other embodiments) While embodiments for carrying out the present invention have been described so far, the present invention should not be limited to the embodiments described above. For example, the estimation device may also include the function of a learning unit. [Explanation of Symbols]

[0077] 1. Estimation System 2 Learning device 3 Estimation device 4 Display device 5 Input devices 6. Data Acquisition Device 21. Training Dataset Generation Unit 22 Learning Department 23, 32 Control Unit 24, 33 Storage section 31 Calculation Section 211 Feature Extraction Unit 212 Classification Department 213 Split section 214 Integration Department 241 Dataset Storage Unit 321 Display Control Unit

Claims

1. A method for generating datasets in which a computer generates a training dataset for generating a trained model and an evaluation dataset for evaluating the trained model from a dataset, wherein A feature extraction step in which features are extracted from the dataset, which has been classified into classification classes that exhibit different characteristics from each other, A classification step in which datasets belonging to each classification class are each classified into multiple subclasses based on their features, A splitting step in which the datasets belonging to each subclass are divided into a subtraining dataset and a subevaluation dataset, An output step which outputs the aforementioned supplementary training dataset and the aforementioned supplementary evaluation dataset, A method for generating a dataset that includes this.

2. The aforementioned dataset is classified into multiple classification classes based on its features. The method for generating a dataset according to claim 1.

3. Integration step of integrating all of the aforementioned sub-training datasets and setting them as a training dataset, and integrating all of the aforementioned sub-evaluation datasets and setting them as an evaluation dataset, It further includes, The output step outputs the integrated training dataset and the evaluation dataset. The method for generating a dataset according to claim 1.

4. The division step involves calculating the centroid from the feature distribution of the dataset belonging to the subclass in the subclass, and dividing the dataset into a subtraining dataset and a subevaluation dataset based on the distance from the centroid. The method for generating a dataset according to claim 1.

5. The number of data points included in the subtraining dataset divided by the division step is greater than or equal to the number of data points included in the subevaluation dataset. The method for generating a dataset according to claim 1.

6. The aforementioned splitting step, if there is one data point belonging to the subclassification class, sets that dataset as training data. The method for generating a dataset according to claim 1.

7. A dataset generation program that causes a computer to generate a training dataset for generating a trained model and an evaluation dataset for evaluating the trained model from a dataset, A feature extraction step in which features are extracted from the dataset, which has been classified into classification classes that exhibit different characteristics from each other, A classification step in which datasets belonging to each classification class are each classified into multiple subclasses based on their features, A splitting step in which the datasets belonging to each subclass are divided into a subtraining dataset and a subevaluation dataset, An output step which outputs the aforementioned supplementary training dataset and the aforementioned supplementary evaluation dataset, A dataset generation program that causes the aforementioned computer to execute.

8. A dataset generation device that generates a training dataset used to generate a trained model, and an evaluation dataset used to evaluate the trained model, from a dataset, A feature extraction unit extracts features from the dataset, which has been classified into classification classes that exhibit different characteristics from each other. A classification unit that classifies datasets belonging to each classification class into multiple subclasses based on their features, A splitting unit that divides the datasets belonging to each subclass into a subtraining dataset and a subevaluation dataset, An output unit that outputs the aforementioned supplementary training dataset and the aforementioned supplementary evaluation dataset, A dataset generation device equipped with the following features.

Images