Sorting device, sorting method, and sorting program

Abstract

To provide a classification device, method and program for improving classification accuracy of data to be used for supplemental learning.SOLUTION: A method executes: a calculation process S602 for calculating a first training effect evaluation value indicating the extent to which classification target data contributes to supplemental training of a prediction model on the basis of a first degree of uncertainty indicating the level of ambiguity in a first prediction result output as a result of inputting classification target data with no correct-answer label attached to the prediction model which can access a training dataset with correct-answer labels attached and has learned with the training dataset; a classification process S603 in which the classification target data is classified into either one of supplemental training data or non-supplemental training data for the prediction model on the basis of the first training effect evaluation value calculated through the calculation process; a setting process S604 for configuring a setting so that a correct-answer label can be attached to the supplemental training data classified through the classification process; and a supplementation process S605 for supplementing the training dataset with the supplemental training data to which the correct-answer label has been attached in the setting process.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a filtering device, a filtering method, and a filtering program for filtering data. [Background technology]

[0002] The sorting device 100 is a device that learns a model using a known learning means such as a neural network and outputs a prediction based on input data. The sorting device 100 is composed of a model that makes a prediction based on input data and a learning device that generates the model using a known means such as a neural network. For example, a technology is known that takes an image captured by a camera as input and outputs the identity of an object appearing in the image.

[0003] Since a sorting device can improve the predictive performance of a model by learning data of various patterns, a user needs to select data with a high learning effect that contributes to improving the performance of the model from a large amount of data output from sensors, cameras, user interfaces, etc. Then, a model trained using the selected data needs to satisfy the demands of predictive performance.

[0004] However, if the amount of data is very large, the time and labor costs required for the user to select data with a high learning effect become enormous, which in turn increases the time and labor costs required to generate a model.

[0005] Furthermore, in order to determine the effectiveness of the learning process, it is necessary to understand in advance how the model's performance will change when it is trained on various types of data, which poses the challenge of requiring the sorters to have advanced know-how.

[0006] Therefore, a technology is known that automates the aforementioned data selection process by introducing an index that indicates the level of learning effect of each piece of data and selecting data with a high learning effect based on the index (see, for example, Patent Document 1 below).

[0007] The image inspection device in Patent Document 1 calculates a stability evaluation value that indicates the stability of pass / fail judgment for multiple images, and presents candidate images to the user for additional learning by the classifier based on the stability evaluation value. The image inspection device accepts designation by the user of additional learning images from among the presented candidate images. The image inspection device additionally learns the additional learning images designated by the user as pass / fail images and updates the classifier.

[0008] Furthermore, although there are various methods for calculating indices that represent the level of learning effectiveness, they may not correlate with the actual learning effectiveness. For example, there is a method that calculates the uncertainty of the prediction results output by a model and selects data with a high degree of uncertainty as data with a high learning effectiveness (for example, see Patent Document 2 below).

[0009] The information processing device of Patent Document 2 includes a division unit, a calculation unit, and an integration unit. The division unit divides input data, which is the target of processing that inputs data and outputs processing results, into multiple partial data. The calculation unit executes processing for each of the multiple partial data and calculates multiple degrees of uncertainty that indicate the uncertainty of the processing. The integration unit integrates the multiple degrees of uncertainty and outputs the integrated degree of uncertainty as the degree of uncertainty of the input data. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2020-187072 [Patent Document 2] Patent Publication No. 2021-149818 Summary of the Invention [Problem to be solved by the invention]

[0011] However, the data to be selected includes data with low learning effectiveness, such as outlier data that deviates from the population and data that has been incorrectly assigned correct labels that represent ideal prediction results required for model training.

[0012] For some of this data with low learning effectiveness, the aforementioned high uncertainty value is obtained. Therefore, in a method of selecting data with high uncertainty as data with high learning effectiveness, data with low learning effectiveness is mistakenly selected, and learning from such data will degrade model performance.

[0013] An object of the present invention is to improve the accuracy of selecting data to be used for additional learning. [Means for solving the problem]

[0014] A selection device according to one aspect of the invention disclosed in the present application is a selection device having a processor that executes a program and a storage device that stores the program, and is configured to select a group of learning data to which each correct label is assigned. A set of reference data each with a correct answer label attached, and the processor has access to a division learning process of dividing the reference data group into a plurality of subsets based on a second uncertainty indicating the ambiguity of a second prediction result for each of the reference data output as a result of inputting each of the reference data in the reference data group into a prediction model trained with the training data group, additionally training the prediction model for each of the plurality of subsets, and calculating a performance evaluation value of the prediction model after the additional training for each of the subsets; and a generation process of generating a conversion formula of converting a first uncertainty indicating the ambiguity of a first prediction result output as a result of inputting selection target data to which the correct answer label has not been assigned into a prediction model trained with the training data group, into a first learning effect evaluation value indicating the degree to which the selection target data contributes to the additional training of the prediction model, based on a representative value of the second uncertainty for each of the subsets and the performance evaluation value for each of the subsets calculated by the division learning process. The aforementioned The first degree of uncertainty is input into the conversion formula generated by the generation process, thereby obtaining the first learning effect evaluation value. a selection process for selecting the selection target data as additional learning data or non-additional learning data for the prediction model based on the first learning effect evaluation value calculated by the calculation process; a setting process for setting the additional learning data selected by the selection process so that the correct answer label can be assigned; and an addition process for adding the additional learning data assigned the correct answer label by the setting process to the learning data group. [Effects of the Invention]

[0015] According to the exemplary embodiment of the present invention, it is possible to improve the accuracy of selecting data to be used for additional learning. Problems, configurations, and effects other than those described above will become clear from the following description of the examples. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a sorting device. [Figure 2]FIG. 2 is a block diagram illustrating an example of a functional configuration of the sorting device. [Figure 3] FIG. 3 is a block diagram showing a detailed functional configuration of the model generation unit. [Figure 4] FIG. 4 is a block diagram of a detailed functional configuration of the data selection unit according to the first embodiment. [Figure 5] FIG. 5 is an explanatory diagram showing a specific example of the division learning unit and the transformation formula generation unit. [Figure 6] FIG. 6 is a flowchart of an example of a data selection process procedure performed by the selection device according to the first embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of a conversion formula generation process performed by the division learning unit and the conversion formula generation unit. [Figure 8] FIG. 8 is a block diagram of a detailed functional configuration of the data selection unit according to the second embodiment. [Figure 9] FIG. 9 is a flowchart of an example of a data selection process procedure performed by the selection device according to the second embodiment. [Figure 10] FIG. 10 is a flowchart showing a detailed example of the procedure of the determination process (step S900) by the determination unit. DETAILED DESCRIPTION OF THE INVENTION [Example]

[0017] <Example of hardware configuration for sorting equipment> FIG. 1 is a block diagram showing an example of the hardware configuration of a sorting device. The sorting device 100 includes a processor 101, a storage device 102, an input device 103, an output device 104, and a communication interface (communication IF) 105. The processor 101, the storage device 102, the input device 103, the output device 104, and the communication IF 105 are connected via a bus 106. The processor 101 controls the sorting device 100. The storage device 102 serves as a working area for the processor 101. The storage device 102 is a non-transitory or temporary recording medium that stores various programs and data. Examples of the storage device 102 include a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device 103 inputs data. Examples of the input device 103 include a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, and a sensor. The output device 104 outputs data. The output device 104 includes, for example, a display, a printer, and a speaker. The communication IF 105 connects to a network and transmits and receives data.

[0018] <Example of functional configuration of sorting device 100> 2 is a block diagram showing an example of the functional configuration of the selection device 100. The selection device 100 is a computer that supports data selection, and includes a correct answer label setting unit 201, a model generation unit 202, a data selection unit 203, a selection target data pool 210, a learning data storage unit 211, and an evaluation data storage unit 212.

[0019] Specifically, the correct label setting unit 201, the model generation unit 202, and the data selection unit 203 are realized, for example, by having the processor 101 execute a program stored in the storage device 102 shown in Fig. 1. Furthermore, the selection target data pool 210, the learning data storage unit 211, and the evaluation data storage unit 212 are realized by the storage device 102 shown in Fig. 1.

[0020] The selection target data pool 210 is a database that stores selection target data. The selection target data may be image data or text data. No correct answer label is assigned to the selection target data.

[0021] The training data storage unit 211 is a database that stores training data. The training data is selection target data (excluding evaluation data) that is randomly selected from the selection target data pool 210 each time the model generation unit 202 trains a prediction model (hereinafter simply referred to as "model") M. A correct answer label is assigned to the training data. The evaluation data is selection target data that is used in the performance evaluation of the model M by the model generation unit 202, and is selected in advance from the selection target data pool 210. A correct answer label is assigned to the evaluation data.

[0022] The correct label setting unit 201 extracts selection target data from the selection target data pool 210 and sets the extracted selection target data so that a correct label can be assigned. Specifically, for example, the correct label setting unit 201 displays the extracted selection target data. When an input of a correct label is accepted by a user operation, learning data with a correct label and evaluation data with a correct label are set.

[0023] The model generation unit 202 receives training data with correct labels stored in the training data storage unit 211 and generates a model M that performs prediction. The model generation unit 202 also performs prediction using the generated model M on the evaluation data stored in the evaluation data storage unit 212. The model generation unit 202 regards the magnitude of the difference between the correct label and the prediction result as the performance of the model M and calculates a performance evaluation value of the model M. The smaller the difference between the correct label and the prediction result, the higher the performance evaluation value of the model M.

[0024] The data selection unit 203 selects data to be subjected to additional learning (hereinafter referred to as additional learning data) from the selection target data pool 210 based on the prediction result by the model generation unit 202. The data selection unit 203 requests the correct label setting unit 201 to set a correct label for the additional learning data selected from the selection target data pool 210. In this case, the correct label setting unit 201 that has received the request sets a correct label for the additional learning data selected by the data selection unit 203.

[0025] 3 is a block diagram showing a detailed functional configuration of the model generation unit 202. The model generation unit 202 has a model learning unit 301, a model evaluation unit 302, and a prediction processing unit 303. The model learning unit 301 inputs learning data with correct answer labels stored in the learning data storage unit 211, and learns a model M that performs prediction.

[0026] The model evaluation unit 302 inputs the evaluation data with the correct answer labels stored in the evaluation data storage unit 212 and evaluates the model M generated by the model training unit 301. Specifically, for example, the model evaluation unit 302 calculates a performance evaluation value based on a loss function, based on the difference between the output data output as a result of inputting the evaluation data to the model M and the correct answer labels assigned to the evaluation data. For example, a higher performance evaluation value indicates better performance. This performance evaluation value is passed to the model training unit 301, and the model training unit 301 re-trains the model M so as to increase the performance evaluation value.

[0027] The prediction processing unit 303 executes a prediction process on the selection target data stored in the selection target data pool 210 using the model M generated by the model learning unit 301, and outputs the prediction result. The data selection unit 203 selects the selection target data based on the prediction result output by the prediction processing unit 303.

[0028] 4 is a block diagram illustrating a detailed functional configuration of the data selecting unit 203 according to the first embodiment. As illustrated in FIG. 4, the data selecting unit 203 includes an uncertainty calculating unit 401, a learning effect calculating unit 402, a selecting processing unit 403, a conversion formula storage unit 404, a reference data storage unit 405, a division learning unit 406, and a conversion formula generating unit 407. Specifically, the uncertainty calculating unit 401, the learning effect calculating unit 402, the selecting processing unit 403, the division learning unit 406, and the conversion formula generating unit 407 are realized, for example, by causing the processor 101 to execute a program stored in the storage device 102 illustrated in FIG. 1. The conversion formula storage unit 404 and the reference data storage unit 405 are realized by the storage device 102 illustrated in FIG. 1.

[0029] The uncertainty calculation unit 401 calculates the uncertainty indicating the uncertainty of the prediction result based on the prediction result output by the model generation unit 202, as shown in Patent Document 2, for example. Uncertainty is an index indicating the vagueness of the prediction result for the data by the model M. The degree of uncertainty is the uncertainty.

[0030] In other words, the uncertainty is an evaluation value that makes it possible to grasp whether the reliability of the prediction result by the generated model M is high or low, and is correlated with the learning effect. Generally, when the uncertainty is high, the prediction result by the model M for the data to be selected is uncertain, and the data to be selected is determined to be data that highly requires additional learning. On the other hand, when the uncertainty value is low, the prediction result by the model M for the data to be selected is reliable, and the data to be selected is determined to be data that does not highly require additional learning.

[0031] Specifically, the uncertainty can be, for example, the probability of the most likely correct label (= predicted result), the probability difference between the most likely correct label and the second most likely correct label, or the magnitude of the entropy of the predicted distribution of model M.

[0032] The learning effect calculation unit 402 calculates a learning effect evaluation value based on the conversion formula stored in the conversion formula storage unit 404 from the uncertainty calculated by the uncertainty calculation unit 401. The learning effect evaluation value is a value that evaluates the learning effect that contributes to improving the performance of model M. In other words, the learning effect evaluation value is an evaluation value that indicates that performance will improve (i.e., there will be a learning effect) when selection target data to which no correct answer label has been assigned is used for additional learning of model M, and selection target data with a larger learning effect evaluation value indicates that it should be selected as data for additional learning.

[0033] The selection processing unit 403 selects the selection target data stored in the selection target data pool 210 as data for additional learning or data for non-additional learning based on the learning effect evaluation value. Specifically, for example, if the learning effect evaluation value from the learning effect calculation unit 402 is equal to or greater than a threshold, the selection processing unit 403 selects the selection target data corresponding to the prediction result from which the learning effect evaluation value was calculated as data for additional learning, and if the learning effect evaluation value is less than the threshold, the selection processing unit 403 selects the selection target data as data for non-additional learning. The selection processing unit 403 then requests the correct answer label setting unit 201 to set a correct answer label for the additional learning data.

[0034] The selection processing unit 403 may delete selection target data for which the correct label setting unit 201 has not been requested to set a correct label (i.e., non-additional learning data) from the selection target data pool 210. This can reduce the memory usage by the selection target data pool 210.

[0035] The conversion formula storage unit 404 stores conversion formulas. The conversion formula is a conversion formula that converts the uncertainty of an arbitrary prediction result into a learning effect evaluation value that indicates the level of learning effect when the model M is trained using the selection target data. The purpose of the conversion formula is to remove selection target data with low learning effect, such as outlier data that deviates from the selection target data group in the selection target data pool 210 or data with an incorrect correct answer label assigned to the selection target data, even if the selection target data has a high degree of uncertainty.

[0036] Even if such selection target data has a high degree of uncertainty, the conversion formula determines that the learning effect evaluation value is low. Furthermore, for selection target data other than the selection target data with a high degree of uncertainty and low learning effect, a high learning effect evaluation value is calculated. Therefore, even if the selection target data pool 210 includes selection target data that should not be used for additional learning, it is possible to accurately select selection target data that is highly in need of additional learning.

[0037] The reference data storage unit 405 stores reference data. The reference data is data extracted from the selection target data pool 210, and is assigned a correct answer label, similar to the learning data.

[0038] The division learning unit 406 acquires the model M generated by the model generation unit 202. The division learning unit 406 inputs the reference data stored in the reference data storage unit 405 into the acquired model M and outputs a prediction result. The division learning unit 406 outputs the prediction result to the uncertainty calculation unit 401 and causes the uncertainty calculation unit 401 to calculate the uncertainty of the prediction result. The division learning unit 406 acquires the uncertainty for each prediction result and divides the reference data group into multiple batches based on the uncertainty. The division learning unit 406 performs additional learning of the model M using the batches.

[0039] The conversion formula generation unit 407 generates a conversion formula for converting the uncertainty into a learning effect evaluation value from the correlation between the average uncertainty of the reference data included in the batch in the division learning unit 406 and the performance evaluation value of the model for each batch additionally learned in the division learning unit 406, and stores the conversion formula in the conversion formula storage unit 404.

[0040] 5 is an explanatory diagram showing a specific example of the division learning unit 406 and the conversion formula generation unit 407. The division learning unit 406 outputs each reference data R of the reference data group Rs stored in the reference data storage unit 405 to the uncertainty calculation unit 401, and acquires the uncertainty of each reference data R from the uncertainty calculation unit 401.

[0041] The division learning unit 406 arranges the reference data R in ascending order of the uncertainty of each reference data R, and divides the arranged reference data group Rs into k batches B1 to Bk (k is an integer equal to or greater than 2). When the batches B1 to Bk are not distinguished, they are represented as batches Bi (i is an integer satisfying 1≦i≦k).

[0042] For example, if the number of reference data R is N and k batches are generated, the reference data R are sorted in order of decreasing uncertainty, and the 1st to N / kth reference data R are defined as the first batch B1, and then the (N / k+1)th to 2N / kth reference data R are defined as the second batch B2, and so on.The reference data group Rs is then divided into k batches B1 to Bk.After that, additional learning L1 to Lk of model M is performed using the multiple batches B1 to Bk generated, and models M1 to Mk, the same number as the number of batches k, are generated.

[0043] The conversion formula generation unit 407 calculates the average uncertainty of the reference data R included in each of the batches B1 to Bk used in the additional learning L1 to Lk of each of the models M1 to Mk in the division learning unit 406. Note that although the average value of the uncertainty of the reference data R is calculated here, it is not limited to the average value, and a representative value (representative uncertainty) such as the maximum, minimum or median value of the uncertainty within the batch Bi may also be used.

[0044] Furthermore, the conversion formula generation unit 407 calculates a performance evaluation value of each of the models M1 to Mk using the evaluation data, similar to the model evaluation unit 302. Then, the conversion formula generation unit 407 obtains data points P1 to Pk for the batch Bi based on the correlation between the average uncertainty of the reference data R and the performance evaluation value of the model Mi, and generates a conversion formula F that converts the uncertainty into a performance evaluation value by fitting the obtained data points P1 to Pk.

[0045] Specifically, for example, the conversion formula generation unit 407 plots data points P1 to Pk on graph 500, with the horizontal axis representing the average uncertainty of batch Bi and the vertical axis representing the performance evaluation value of model Mi after learning Li, and generates conversion formula F by polynomial approximation. For model Mi that has undergone additional training with batch Bi that contains a large amount of reference data R with a high learning effect evaluation value, the performance evaluation value will be further improved. Therefore, the output when uncertainty is input into conversion formula F, which was generated based on the correlation between the average uncertainty of batch Bi and the performance evaluation value of model Mi, is used as a learning effect evaluation value for evaluating the learning effect, and is output to the data selection unit 203.

[0046] 5, for data points P1 to Pi, the performance evaluation value increases as the average uncertainty increases. Beyond data point Pi, this relationship disappears and the performance evaluation value decreases, but since the performance evaluation value is high in region 501, for data points within region 501, selection target data with high uncertainty and high learning effect evaluation value can be selected as additional training data.

[0047] Specifically, for example, if the uncertainty is equal to or less than the first threshold value T1 and the learning effect evaluation value output from the transformation formula F is equal to or greater than the second threshold value T2, the selection target data is selected as additional learning data. This makes it possible to, for example, prevent selection target data with a high learning effect evaluation value from being overlooked, and to prevent erroneous detection such as selecting data as additional target data despite having a low learning effect evaluation value.

[0048] Furthermore, when the average uncertainty increases from region 501 to region 502, the performance evaluation value further decreases. Therefore, the selection device 100 can avoid using such selection target data in additional learning by excluding such selection target data from the selection target.

[0049] Specifically, for example, if the uncertainty is greater than the first threshold T1 and the learning effect evaluation value output from the transformation formula F is equal to or less than the third threshold T3 (>T2), the selection target data is selected as non-additional learning data. As a result, for example, selection target data in which targets are not clear (for example, out-of-focus images) or in which occlusion has occurred can be excluded from selection as selection target data with a high uncertainty and a low learning effect evaluation value.

[0050] <Data sorting process> 6 is a flowchart illustrating an example of a data selection process procedure by the selection device 100 according to the embodiment 1. First, the prediction processing unit 303 performs prediction on the data in the selection target data pool 210 using the model M generated by the model learning unit 301, and outputs the prediction result (step S601).

[0051] Next, the uncertainty calculation unit 401 receives the prediction result and calculates the uncertainty of the prediction result. The learning effect calculation unit 402 then inputs the uncertainty into the conversion formula stored in the conversion formula storage unit 404 to calculate a performance evaluation value (step S602).

[0052] Thereafter, the selection processing unit 403 selects the selection target data in the selection target data pool 210 as additional learning data based on the converted performance evaluation value, and requests the correct answer label setting unit 201 to set a correct answer label for the selected additional learning data (step S603).

[0053] As described above, the performance of a model Mi obtained by additional training of a batch Bi that includes a large amount of reference data R with a high learning effect improves, and therefore the calculated performance evaluation value becomes the learning effect evaluation value. Therefore, the selection processing unit 403 can select the selection target data as data for additional learning based on the learning effect evaluation value, which is the calculated performance evaluation value. In this way, the higher the learning effect evaluation value, the more likely the selection target data is to be selected as data for additional learning, and the lower the learning effect evaluation value, the more likely the data is to be selected as data not for additional learning.

[0054] The correct label setting unit 201 executes correct label setting (step S604). Specifically, for example, the correct label setting unit 201 displays additional learning data, accepts input of a correct label from the user, and associates the input correct label with the additional learning data (step S604).

[0055] After step S604, the correct label setting unit 201 adds the additional training data to which the correct label has been assigned to the training data storage unit 211 (step S605). The model generation unit 202 updates the model M by performing additional training on the model M using the additional training data with the correct label by the model training unit 301 (step S606).

[0056] The model evaluation unit 302 evaluates the performance of the updated model M using the evaluation data stored in the evaluation data storage unit 212 (step S607). If the evaluation result shows that the performance of the model M has reached the required performance (step S608: Yes), the model evaluation unit 302 ends the data selection process, and if not (step S608: No), executes the data selection process again from step S601.

[0057] 6, the learning effect evaluation value of each data item can be calculated with high accuracy, and data items with high learning effect evaluation values ​​can be selected. This enables efficient learning of the model M.

[0058] 7 is a flowchart showing an example of a conversion formula generation process procedure by the division learning unit 406 and the conversion formula generation unit 407. First, the division learning unit 406 inputs the reference data R stored in the reference data storage unit 405 to the model M and outputs a prediction result for each reference data R (step S701).

[0059] Next, the division learning unit 406 causes the uncertainty calculation unit 401 to calculate the uncertainty based on the prediction result for each reference data R, and acquires the uncertainty for each reference data R from the uncertainty calculation unit 401 (step S702). Then, the division learning unit 406 divides the reference data group Rs into k batches B1 to Bk based on the uncertainty for each reference data R (step S703).

[0060] Next, the division learning unit 406 starts a performance evaluation loop (step S704). Specifically, for example, the division learning unit 406 initializes i to i=1 and executes steps S705 and S706. If it is determined in step S707 that the performance evaluation has not ended, the division learning unit 406 increments i in step S701 and executes steps S705 and S706 for the incremented i.

[0061] In step S707, the division learning unit 406 determines whether i=k, and if not, determines that the performance evaluation has not ended and proceeds to step S704. On the other hand, if i=k, it determines that the performance evaluation of models M1 to Mk has ended and proceeds to step S708.

[0062] In steps S705 and S706 in the i-th loop, the decomposition learning unit 406 executes learning Li of the model Mi using the batch Bi (step S705), evaluates the performance of the learned model Mi (step S706), and proceeds to step S70.

[0063] If it is determined in step S707 that the performance evaluation of models M1 to Mk has ended, the conversion formula generation unit 407 calculates an average uncertainty by averaging the uncertainties for the reference data included in each batch Bi, and generates a conversion formula F for each batch Bi based on the correlation between the average uncertainty of batch Bi and the performance evaluation value of model Mi that executed learning Li in that batch Bi, and stores the generated conversion formula F in the conversion formula storage unit 404 (step S708). This completes the conversion formula generation process.

[0064] The performance evaluation value of a model Mi obtained by additional training of a batch Bi that contains a large amount of reference data R with a high learning effect evaluation value improves more. Therefore, by using the output obtained when uncertainty is input into a conversion formula F generated by the correlation between the average uncertainty of a batch Bi and the performance evaluation value of a model Mi as the learning effect evaluation value for evaluating the learning effect, it becomes possible to accurately select data to be selected using the learning effect evaluation value. [Example]

[0065] A description will be given of Example 2. Example 2 is an example in which unnecessary selection target data is set to be outside the selection target in Example 1. The same components as those in Example 1 are given the same reference numerals, and the description thereof will be omitted.

[0066] Fig. 8 is a block diagram showing a detailed functional configuration of the data selecting unit 203 according to the second embodiment. As shown in Fig. 8, the data selecting unit 203 includes an uncertainty storage unit 800 and a determination unit 801. Specifically, the determination unit 801 is realized by, for example, causing the processor 101 to execute a program stored in the storage device 102 shown in Fig. 1. The uncertainty storage unit 800 is realized by the storage device 102 shown in Fig. 1.

[0067] The uncertainty storage unit 800 stores the uncertainty before and after the additional learning of the model M calculated by the uncertainty calculation unit 401 for each of the selection target data.

[0068] The determining unit 801 calculates the amount of change in the uncertainty for each of the selection target data before and after the additional learning, based on the uncertainties stored in the uncertainty storage unit 800. Specifically, for each of the selection target data, the determining unit 801 calculates the amount of change as the difference obtained by subtracting the uncertainty before learning from the uncertainty after learning. Selection target data with a large amount of change is likely to be outlier data whose prediction accuracy will not improve even if additional learning is performed.

[0069] The determining unit 801 determines the data to be selected as non-selectable data based on the amount of change. Specifically, for example, the selecting processing unit 403 determines that the data to be selected, whose amount of change is equal to or greater than a threshold, is outlier data for which the accuracy of prediction will not improve even if additional learning is performed, and determines the data to be non-selectable.

[0070] Furthermore, the determination unit 801 may determine that the top n selection target data in terms of the magnitude of change among the group of selection target data for which the change amount and learning effect evaluation value (performance evaluation value) have been calculated are outlier data for which accurate predictions will not be possible even with additional learning, and set the selection target data as excluding data from selection.

[0071] The determining unit 801 outputs the selection target data that has not been set as non-selection target data (selection target data that is not outlier data) to the selection processing unit 403. Note that the determining unit 801 may set the selection target data as non-selection target data by deleting the selection target data that has been determined to be outlier data. This can reduce the memory usage by the determining unit 801.

[0072] Furthermore, the determining unit 801 or the sorting processing unit 403 may assign an outlier label indicating that the outlier data is outlier data, and overwrite and save the data in the sorting target data pool 210. In this case, the sorting processing unit 403 does not extract sorting target data that has an outlier label. This makes it possible to prevent the extraction of sorting target data that does not improve prediction accuracy even after additional learning.

[0073] Furthermore, the determining unit 801 or the screening processing unit 403 may delete screening target data determined by the determining unit 801 to be outlier data from the screening target data pool 210. This makes it possible to prevent the extraction of screening target data whose prediction accuracy does not improve even after additional learning. Also, it is possible to reduce the memory usage by the screening target data pool 210.

[0074] <Data sorting process> 9 is a flowchart illustrating an example of a data selection process procedure by the selection device 100 according to Example 2. After step S602, the determination unit 801 executes a determination process (step S900), and the process proceeds to step S603.

[0075] 10 is a flowchart showing a detailed example of the process steps of the determination process (step S900) by the determination unit 801. When the selection target data is input, the determination unit 801 calculates the amount of change based on the degree of uncertainty before and after additional learning (step S1001).

[0076] The determining unit 801 determines whether each piece of selection target data is outlier data based on the amount of change (step S1002). If it is outlier data (step S1002: Yes), the determining unit 801 determines that the selection target data is not to be selected (step S1003) and proceeds to step S603. On the other hand, if it is not outlier data (step S1002: No), the determining unit 801 proceeds to step S603.

[0077] As described above, according to the second embodiment, it is possible to accurately select selection target data having a high learning effect evaluation value.

[0078] In the first and second embodiments described above, the conversion formula generation unit 407 generates the conversion formula F, but the conversion formula F may be stored in the conversion formula storage unit 404 in advance.

[0079] Furthermore, the model generation unit 202, the data pool to be selected 210, the learning data storage unit 211, the evaluation data storage unit 212, the reference data storage unit 405, and the uncertainty calculation unit 401 do not need to be implemented within the sorting device 100, but may be implemented on a computer outside the sorting device 100 that is accessible from the sorting device 100 via a network such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0080] Furthermore, in the above-described first and second embodiments, the prediction model may be a two-class classifier (e.g., classifying whether it is a person or not) or a three-class or more multi-class classifier (e.g., classifying whether it is male, female, or neither).

[0081] As described above, the above-described selection device 100 makes it possible to accurately select selection target data that is effective for learning as additional learning data based on the uncertainty of the prediction result output by the model M for the input selection target data. Therefore, efficient additional learning can be performed using additional learning data that has a high learning effect.

[0082] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added to, deleted from, or replaced with other configurations.

[0083] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0084] Information such as programs, tables, files, etc. that realize each function can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0085] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0086] 100 Sorting Device 101 processors 102 Storage Devices 201 Correct label setting section 202 Model Generation Unit 203 Data Selection Department 210 Data Pool for Selection 211 Learning data storage unit 212 Evaluation data storage unit 301 Model Learning Department 302 Model Evaluation Department 303 Prediction processing unit 401 Uncertainty Calculation Unit 402 Learning Effect Calculation Unit 403 Sorting Processing Unit 404 Conversion formula storage section 405 Reference Data Storage Unit 406 Split Learning Unit 407 Conversion Formula Generation Unit 800 Uncertainty memory unit 801 Decision Section

Claims

1. A selection device having a processor that executes a program and a storage device that stores the program, The method has access to a set of training data, each of which is assigned a correct answer label, and a set of reference data, each of which is assigned a correct answer label; The processor, a division learning process for dividing the reference data group into a plurality of subsets based on a second uncertainty indicating ambiguity of a second prediction result for each of the reference data output as a result of inputting each of the reference data of the reference data group into a prediction model trained with the training data group, additionally learning the prediction model for each of the plurality of subsets, and calculating a performance evaluation value of the prediction model after the additional learning for each of the subsets; a generation process for generating a conversion formula for converting a first uncertainty indicating the ambiguity of a first prediction result output as a result of inputting selection target data to which the correct answer label is not assigned to a prediction model trained with the learning data group, into a first learning effect evaluation value indicating the degree to which the selection target data contributes to additional learning of the prediction model, based on a representative value of the second uncertainty for each of the subsets and the performance evaluation value for each of the subsets calculated by the division learning process; a calculation process of calculating the first learning effect evaluation value by inputting the first uncertainty into a conversion formula generated by the generation process; a selection process of selecting the selection target data as additional learning data or non-additional learning data for the prediction model based on the first learning effect evaluation value calculated by the calculation process; a setting process for setting the additional learning data selected by the selection process so that the correct answer label can be assigned; an addition process of adding the additional learning data to which the correct answer label is assigned by the setting process to the learning data group; A sorting apparatus comprising:

2. The sorting device according to claim 1, In the calculation process, the processor calculates the first uncertainty based on the first prediction result. A sorting device characterized by:

3. The sorting device according to claim 1, The processor, a deletion process of deleting the selection target data selected as the non-additional learning data by the selection process from a selection target data group including the selection target data; A sorting apparatus comprising:

4. A sorting apparatus as claimed in claim 1, In the division learning process, the processor divides the reference data group into the plurality of subsets, with a certain number of the reference data in order of magnitude of the second uncertainty as a unit of the subset. A sorting device characterized by:

5. A sorting apparatus as claimed in claim 1, In the generation process, the processor generates the conversion formula based on a correlation between the representative value of the second uncertainty for each of the subsets and the performance evaluation value for each of the subsets. A sorting device characterized by:

6. A sorting device having a processor that executes a program and a storage device that stores the program, You have access to a set of training data, each of which has a correct answer label attached to it. The processor, a calculation process for calculating a first learning effect evaluation value indicating the degree to which the selection target data contributes to additional learning of the prediction model, based on a first uncertainty indicating the ambiguity of a first prediction result output as a result of inputting the selection target data to which the correct answer label is not assigned to the prediction model trained with the learning data group; a selection process of selecting the selection target data as additional learning data or non-additional learning data for the prediction model based on the first learning effect evaluation value calculated by the calculation process; a setting process for setting the additional learning data selected by the selection process so that the correct answer label can be assigned; an addition process of adding the additional learning data to which the correct answer label is assigned by the setting process to the learning data group; In the selection process, the processor selects, as the additional learning data, the selection target data in which the first uncertainty is equal to or less than a first threshold value and the first learning effect evaluation value is equal to or greater than a second threshold value. A sorting device characterized by:

7. A sorting device having a processor that executes a program and a storage device that stores the program, Access to training data sets with correct labels attached to each data set. The processor, a calculation process for calculating a first learning effect evaluation value indicating the degree to which the selection target data contributes to additional learning of the prediction model, based on a first uncertainty indicating the ambiguity of a first prediction result output as a result of inputting the selection target data to which the correct answer label is not assigned to the prediction model trained with the learning data group; a selection process of selecting the selection target data as additional learning data or non-additional learning data for the prediction model based on the first learning effect evaluation value calculated by the calculation process; a setting process for setting the additional learning data selected by the selection process so that the correct answer label can be assigned; an addition process of adding the additional learning data to which the correct answer label is assigned by the setting process to the learning data group; In the selection process, the processor selects the selection target data in which the first uncertainty is greater than a first threshold value and the first learning effect evaluation value is equal to or less than a third threshold value as the non-additional learning data. A sorting device characterized by:

8. A sorting device having a processor that executes a program and a storage device that stores the program, Access to training data sets with correct labels attached to each data set. The processor, a calculation process for calculating a first learning effect evaluation value indicating the degree to which the selection target data contributes to additional learning of the prediction model, based on a first uncertainty indicating the ambiguity of a first prediction result output as a result of inputting the selection target data to which the correct answer label is not assigned to the prediction model trained with the learning data group; a selection process of selecting the selection target data as additional learning data or non-additional learning data for the prediction model based on the first learning effect evaluation value calculated by the calculation process; a setting process for setting the additional learning data selected by the selection process so that the correct answer label can be assigned; an addition process of adding the additional learning data to which the correct answer label is assigned by the setting process to the learning data group; a determination process for determining that the selection target data is not to be selected based on a difference between the first uncertainty level output as a result of inputting the selection target data into the prediction model before the addition by the addition process and the third uncertainty level output as a result of inputting the selection target data into the prediction model additionally trained with the learning data group after the addition by the addition process; In the calculation process, the processor calculates a second learning effect evaluation value indicating an extent to which the selection target data contributes to additional learning of the prediction model after the additional learning, based on the third uncertainty degree; and In the selection process, the processor selects the selection target data that has not been determined as non-selection target data by the determination process, as additional learning data or non-additional learning data for the prediction model after the additional learning, based on the second learning effect evaluation value. A sorting device characterized by:

9. A sorting method performed by a sorting device having a processor that executes a program and a storage device that stores the program, comprising: the selection device is capable of accessing a set of training data, each of which is assigned a correct answer label, and a set of reference data, each of which is assigned a correct answer label; The selection method includes: The processor, a division learning process for dividing the reference data group into a plurality of subsets based on a second uncertainty indicating ambiguity of a second prediction result for each of the reference data output as a result of inputting each of the reference data of the reference data group into a prediction model trained with the training data group, additionally learning the prediction model for each of the plurality of subsets, and calculating a performance evaluation value of the prediction model after the additional learning for each of the subsets; a generation process for generating a conversion formula for converting a first uncertainty indicating the ambiguity of a first prediction result output as a result of inputting selection target data to which the correct answer label is not assigned to a prediction model trained with the learning data group, into a first learning effect evaluation value indicating the degree to which the selection target data contributes to additional learning of the prediction model, based on a representative value of the second uncertainty for each of the subsets and the performance evaluation value for each of the subsets calculated by the division learning process; a calculation process of calculating the first learning effect evaluation value by inputting the first uncertainty into a conversion formula generated by the generation process; a selection process of selecting the selection target data as additional learning data or non-additional learning data for the prediction model based on the first learning effect evaluation value calculated by the calculation process; a setting process for setting the additional learning data selected by the selection process so that the correct answer label can be assigned; an addition process of adding the additional learning data to which the correct answer label is assigned by the setting process to the learning data group; A selection method comprising the steps of:

10. A processor having access to a set of training data, each of which is assigned a correct answer label, and a set of reference data, each of which is assigned a correct answer label, a division learning process for dividing the reference data group into a plurality of subsets based on a second uncertainty indicating ambiguity of a second prediction result for each of the reference data output as a result of inputting each of the reference data of the reference data group into a prediction model trained with the training data group, additionally learning the prediction model for each of the plurality of subsets, and calculating a performance evaluation value of the prediction model after the additional learning for each of the subsets; a generation process for generating a conversion formula for converting a first uncertainty indicating the ambiguity of a first prediction result output as a result of inputting selection target data to which the correct answer label is not assigned to a prediction model trained with the learning data group, into a first learning effect evaluation value indicating the degree to which the selection target data contributes to additional learning of the prediction model, based on a representative value of the second uncertainty for each of the subsets and the performance evaluation value for each of the subsets calculated by the division learning process; a calculation process of calculating the first learning effect evaluation value by inputting the first uncertainty into a conversion formula generated by the generation process; a selection process of selecting the selection target data as additional learning data or non-additional learning data for the prediction model based on the first learning effect evaluation value calculated by the calculation process; a setting process for setting the additional learning data selected by the selection process so that the correct answer label can be assigned; an addition process of adding the additional learning data to which the correct answer label is assigned by the setting process to the learning data group; A selection program comprising:

Images