Learning support device, learning support method, and learning support program

Abstract

To a learning support device, a method for supporting learning, and a program which suppress the number of steps of correction to operational data while using operational data even if a data drift is happening in the operational data.SOLUTION: A method in an information processor includes the steps of: acquiring learning data and operational data (S101); extracting the respective feature amounts from the learning data and the operational data (S102); selecting operational data to be a target of correcting to be used as learning data from operational data as target data on the basis of the feature amounts of the learning data and the feature amount (S103); and outputting the target data (S104). The method further includes the steps of: determining whether the results of correcting all of the selected target data are a class corrected to each learning data (S105); ending selection of target data and correcting the class corrected to the learning data to all of the operational data of which feature amounts included in a second region are extracted (S106) if the results of correcting all of the selected target data are determined to be a class corrected to each learning data (S105: YES); and adding the learning data to learning data (S108).SELECTED DRAWING: Figure 9A

Description

Technical Field

[0001] The present invention relates to a learning support device, a learning support method, and a learning support program. Background Art

[0002] In recent years, development of systems that use machine learning models for data classification, product abnormality detection, and the like has been progressing. Examples of such systems include those that classify user feedback reports on products into predetermined classes, and those that detect product abnormalities from sensor data such as image data at production sites.

[0003] However, training data used for training a machine learning model is different from operational data input to the machine learning model at the actual operational site. It is also known that, as the characteristics of operational data change over time and thus deviate from the characteristics of training data at the time of training, so-called data drift reduces the prediction accuracy of the machine learning model. Specific causes of data drift include, for example, changes in background in images, changes in materials used for products, changes in product appearance specifications, aging deterioration of imaging cameras, changes in writing style (e.g., terms used) in sentences due to changes in report creators, changes in data formats, and the like.

[0004] In relation to a technique for preventing a decrease in prediction accuracy of a machine learning model due to data drift, the following prior art is disclosed in Patent Document 1 below. Based on data values for each data item of training data, range information indicating possible values of the data value for each data item is calculated. Among operational data to be subjected to abnormality detection, operational data whose data value for each data item is included in the range information is stored as in-range data, and operational data not included in the range information is stored as out-of-range data. Whether retraining of the machine learning model is necessary is determined based on a comparison result between a tendency of the in-range data and a tendency of the out-of-range data. Prior Art Documents Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2021-193497 [Overview of the initiative] [Problems that the invention aims to solve]

[0006] However, while the aforementioned prior art can determine whether data drift has occurred and whether the machine learning model needs to be retrained based on operational data, it has the problem that obtaining the training data to be used for retraining requires marking a relatively large amount of data with correct answers, thus increasing the effort required for marking correct answers.

[0007] This invention was made to solve these problems. Specifically, it aims to provide a learning support device, a learning support method, and a learning support program that can utilize operational data while suppressing the effort required to assign correct answers to operational data, even when data drift occurs in the operational data. [Means for solving the problem]

[0008] The above-mentioned problems of the present invention are solved by the following means.

[0009] (1) A learning support device comprising: an acquisition unit that acquires learning data and operational data; a feature extraction unit that extracts features from the learning data and the operational data, respectively; a target data selection unit that, based on the features of the learning data and the features of the operational data, selects from the operational data the operational data to be used as learning data for correct answer assignment; and an output unit that outputs the selection result by the target data selection unit.

[0010] (2) The learning support device described in (1) above, wherein the target data selection unit selects the target data based on predetermined criteria.

[0011] (3) The learning support device according to (1) above, further comprising a correct answer result receiving unit that receives the correct answer result for the target data.

[0012] (4) The learning support device according to (3) above, wherein the target data selection unit re-selects the operational data to be subject to the correct answer assignment from the operational data based on the correct answer assignment result received.

[0013] (5) The learning support device according to (4) above, wherein the acquisition unit acquires multiple operational data as an operational data group, and the target data selection unit selects the target data from the operational data based on the feature quantities of the learning data and multiple feature quantities extracted from each operational data in the operational data group.

[0014] (6) The learning support device according to (5) above, wherein the acquisition unit acquires a plurality of learning data as a learning data group, and the target data selection unit selects the target data from the operational data based on a plurality of features extracted from each learning data in the learning data group and a plurality of features extracted from each operational data in the operational data group.

[0015] (7) The learning support device according to (6) above, wherein the target data selection unit selects the target data from the operational data based on the feature quantities extracted from the operational data at the boundary of a region that includes all of the multiple feature quantities extracted from each training data in the training data group which includes only the training data that has been correctly assigned to the same class, and all of the multiple feature quantities extracted from each operational data in the operational data group.

[0016] (8) The learning support device according to (7) above, wherein the target data selection unit deselects the operational data selected as target data and whose class is assigned a correct answer by the correct answer result is different from the class assigned to each learning data in the learning data group, and repeats the selection of target data until the correct answer result for all the target data is the class assigned to each learning data in the learning data group.

[0017] (9) The learning support device according to (8) above, wherein the target data selection unit determines that the correct answer result for all the target data is the class to which each learning data in the learning data group has been correctly assigned.

[0018] (10) The learning support device according to (7) above, further comprising a preprocessing unit that performs preprocessing to divide a plurality of feature quantities extracted from each operational data in the operational data group into a plurality of groups by clustering in the feature space, wherein the target data selection unit selects the target data from the operational data based on a representative point of the group to which the feature quantity at the boundary of the region belongs, and for each feature quantity included in the group to which the feature quantity at the boundary of the region belongs, the target data is not selected from the operational data from which the feature quantity other than the representative point has been extracted.

[0019] (11) The learning support device according to (9) above, further comprising a correct answering unit that, when the selection of target data is completed, correctly assigns the classes correctly assigned to the learning data to the operational data from which all the features contained in the region have been extracted.

[0020] (12) A learning support method comprising: a step (a) of acquiring learning data and operation data; a step (b) of extracting a feature amount from each of said learning data and said operation data; and a step (c) of selecting, based on the feature amount of said learning data and the feature amount of said operation data, said operation data to be subjected to correct answer labeling for use as said learning data from among said operation data as target data, and a step (d) of outputting a selection result in said step (c).

[0021] (13) The learning support method according to (12) above, wherein in said step (c), said target data is selected based on a predetermined criterion.

[0022] (14) The learning support method according to (12) above, further comprising a step (e) of receiving a correct answer labeling result for said target data.

[0023] (15) The learning support method according to (14) above, further comprising a step (f) of re-selecting, based on the received correct answer labeling result, said operation data to be subjected to correct answer labeling from among said operation data as said target data.

[0024] (16) In said step (a), a plurality of pieces of operation data are acquired as an operation data group, The learning support method according to (15) above, wherein in said step (c), said target data is selected from among said operation data based on the feature amount of said learning data and a plurality of feature amounts extracted from each piece of operation data of said operation data group.

[0025] (17) The learning support method according to (16) above, wherein in said step (a), a plurality of pieces of learning data are acquired as a learning data group, and in said step (c), said target data is selected from among said operation data based on a plurality of feature amounts extracted from each piece of learning data of said learning data group and a plurality of feature amounts extracted from each piece of operation data of said operation data group.

[0026] (18) The learning support method according to (17) above, wherein in said step (c), in the feature space, based on said feature quantity extracted from said operation data, which is on a boundary of a region that encloses all of a plurality of feature quantities extracted from each piece of learning data in said learning data group containing only said learning data correctly labeled to the same class, and all of a plurality of feature quantities extracted from each piece of operation data in said operation data group, said target data is selected from among said operation data.

[0027] (19) The learning support method according to (18) above, wherein in said step (c), selection of said operation data whose class, which is selected as said target data and correctly labeled based on said correct labeling result, is different from the class correctly labeled to each piece of learning data in said learning data group is canceled, and selection of said target data is repeated until said correct labeling results for all of said target data match the classes correctly labeled to each piece of learning data in said learning data group.

[0028] (20) The learning support method according to (19) above, wherein in said step (c), when it is determined that said correct labeling results for all of said target data are the classes correctly labeled to each piece of learning data in said learning data group, selection of said target data is terminated.

[0029] (21) The learning support method according to (18) above, further comprising step (g) of performing preprocessing for dividing a plurality of feature quantities extracted from each piece of operation data in said operation data group into a plurality of groups by clustering in a feature space, wherein in said step (c), said target data is selected from among said operation data based on a representative point of the group to which said feature quantity on the boundary of said region belongs, and for each feature quantity included in the group to which said feature quantity on the boundary of said region belongs, said target data is not selected from among said operation data from which said feature quantities other than said representative point are extracted.

[0030] (22) The learning support method according to (20) above, further comprising the step (h) of assigning the classes assigned to the learning data to the operational data from which all the features contained in the region have been extracted when the selection of the target data has been completed.

[0031] (23) A learning support program that causes a computer to execute any of the learning support methods described in (12) to (22) above. [Effects of the Invention]

[0032] The system extracts features from both the training data and the operational data. Based on these extracted features, it selects the operational data to be used as training data for ground truth correction and outputs the selection results. This allows for the utilization of operational data while reducing the effort required to correct the operational data, even if data drift occurs. [Brief explanation of the drawing]

[0033] [Figure 1] This is a diagram illustrating the schematic configuration of an information processing system. [Figure 2] This is a block diagram showing the hardware configuration of an information processing device. [Figure 3] This is an explanatory diagram illustrating the function of the control unit during training using training data. [Figure 4] This is an explanatory diagram illustrating the function of the control unit that extracts features from training data. [Figure 5] This is an explanatory diagram illustrating the functions of extracting features from operational data, selecting target data based on the features of the training data and the operational data, and re-selecting target data based on the results of assigning correct answers to the selected target data. [Figure 6] This is an explanatory diagram showing the distribution of the first feature group and the second feature group in the feature space. [Figure 7] This is an explanatory diagram to explain the selection of target data. [Figure 8] This is an explanatory diagram to explain the re-selection of target data. [Figure 9A] This is a flowchart showing the operation of an information processing device. [Figure 9B] This is a flowchart showing the operation of an information processing device. [Modes for carrying out the invention]

[0034] Hereinafter, a learning support device, a learning support method, and a learning support program according to embodiments of the present invention will be described with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and redundant explanations are omitted. Also, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0035] In this specification, the training data used during training in machine learning, and the operational data targeted for prediction during operation (prediction), include image data and / or text data. Text data may, for example, be text data of sentences. The training data and operational data also include data other than image data and text data, such as audio data, temperature data, and odor data.

[0036] Figure 1 is a diagram showing the schematic configuration of the information processing system 10 according to the embodiment.

[0037] The information processing system 10 includes an information processing device 100 and a shooting device 200. The information processing device 100 and the shooting device 200 are connected to each other so as to be able to communicate with each other via a network or the like. The information processing device 100 constitutes a learning support device. The shooting device 200 may be used when the learning data and operational data are image data. When the learning data and operational data are text data, the shooting device 200 may be replaced by a scanner or the like that scans paper and reads the characters on the paper. When the learning data and operational data are audio data, temperature data, or odor data, the shooting device 200 may be replaced by an audio recognition device that converts audio to text data, a temperature sensor, or an odor sensor, respectively.

[0038] The imaging device 200 captures images of products (including parts) and other items to be inspected, and outputs them as image data. Products include, for example, bolts, flat plates, semiconductor chips, semiconductor wafers, and vehicle bodies. If the imaging device 200 is replaced by a scanner, the scanner scans documents (paper), for example, converts them into text data, and outputs them as document data. Documents include, for example, user feedback reports on products, failure reports, and care records. If the documents are data (electronic data), the scanner can be omitted, and the documents, which are data, can be directly input into the information processing device 100 as document data.

[0039] The imaging device 200 is comprised of, for example, a visible light camera. The imaging device 200 is installed, for example, in a production site (factory) so as to have an appropriate imaging range and direction depending on the size and position of the product to be inspected.

[0040] Image data may be a still image, for example, but it may also be a moving image.

[0041] The imaging device 200 may be used to obtain image data to be used as training data and image data to be used as operational data. The image data to be used as training data may be image data captured by an imaging device other than the imaging device 200.

[0042] The imaging device 200 transmits image data to the information processing device 100.

[0043] The information processing device 100 receives image data from the imaging device 200. If the training data and operational data are in the form of text data, the information processing device 100 may receive text data from the scanner.

[0044] The information processing device 100 extracts features from the training data and features from the operational data, respectively, using machine learning. Based on the features of the operational data, the information processing device 100 calculates prediction results from the operational data using machine learning. The prediction results include the likelihood of a predetermined class. Specifically, the prediction result is, for example, the likelihood of the class with the highest likelihood among the predetermined classes. The predetermined classes include, for example, product defect classes and document classification classes. Product defect classes include, for example, scratches, chips, cracks, stains, and good products. Document classification classes include, for example, user-induced failures, equipment-induced failures, and no failures.

[0045] The information processing device 100 selects operational data from the operational data to be used as training data for correct answer assignment (hereinafter also referred to as "target data") based on the features of the training data and the features of the operational data, and outputs the selection result.

[0046] The functions of the information processing device 100 will be explained further. For simplicity, the following explanation will use the example where image data is product image data and text data is failure report data.

[0047] Figure 2 is a block diagram showing the hardware configuration of the information processing device 100.

[0048] The information processing device 100 includes a control unit 110, a storage unit 120, a display unit 130, an input unit 140, and a communication unit 150. These components are interconnected via a bus. The information processing device 100 may be configured by a computer. The control unit 110 constitutes the correct answer assignment unit.

[0049] The control unit 110 is composed of a CPU (Central Processing Unit) and performs control and calculation processing on each part of the information processing device 100 according to the program. Details of the functions of the control unit 110 will be described later.

[0050] The memory unit 120 may consist of, for example, RAM (Random Access Memory), ROM (Read Only Memory), and SSD (Solid State Drive). The RAM temporarily stores programs and data as a working area for the control unit 110. The ROM stores various programs and data in advance. The SSD stores various programs and data, including the operating system.

[0051] The display unit 130 is, for example, a liquid crystal display, and displays various information.

[0052] The input unit 140 is composed of, for example, a touch panel and various keys. The input unit 140 is used for various operations and inputs.

[0053] The communication unit 150 is an interface for communicating with external devices. Network interfaces conforming to standards such as Ethernet®, SATA, PCI Express, USB, and IEEE1394 may be used for communication. In addition, wireless communication interfaces such as Bluetooth®, IEEE802.11, and 4G may be used for communication.

[0054] The functions of the control unit 110 of the information processing device 100 will now be described in detail. The functions of the control unit 110 described below are realized when a program stored in the memory unit 120 is executed using various parameters (for example, weighting parameters of a learning model) stored in the memory unit 120.

[0055] Figure 3 is an explanatory diagram illustrating the function of the control unit 110 during training using training data. Figure 4 is an explanatory diagram illustrating the function of the control unit 110 in extracting features from training data. Figure 5 is an explanatory diagram illustrating the function of extracting features from operational data, selecting target data based on the features of the training data and the features of the operational data, and re-selecting target data based on the results of assigning correct answers to the selected target data.

[0056] (Functions of the control unit 110 during learning) As shown in Figure 3, the control unit 110 functions as a feature extractor 111a, a classifier 112a, and a learning unit 113 during training. The feature extractor 111a and the classifier 112a constitute the learning model.

[0057] The feature extractor 111a acquires training data received from the imaging device 200, etc., and stored in the memory unit 120 by reading it from the memory unit 120. The feature extractor 111a extracts features from each training data in a group of training data (hereinafter also referred to as the "training data group"). If the training data is image data, the feature extractor 111a can use a CNN (Convolutional Neural Network) to extract features from each training data in the training data group. If the training data is text data, the feature extractor 111a can use a natural language processing model such as BERT (Bidirectional Encoder Representations from Transformers) to extract features from each training data in the training data group.

[0058] The classifier 112a classifies the features of each training data into one of the predetermined classes and outputs them as prediction results. Specifically, for example, the classifier 112a calculates and outputs the likelihood for each predetermined class for the features of the training data. As a result, each training data is classified into one of the predetermined classes by classifying the features of each training data into the class with the highest likelihood. For example, a fully connected layer of a neural network is used for the classifier 112a.

[0059] The learning unit 113 can train the feature extractor 111a and the classifier 112a by backpropagation so that the difference between the correct label assigned to each training data in the training data set and the prediction result output from the classifier 112a becomes small. The correct labels can be associated with each training data by the user, input into the input unit 140, and stored in the storage unit 120. The correct labels may be a predetermined class that is the correct prediction result by the classifier 112a. The correct labels may also be predetermined class-specific likelihoods, where the likelihood of the class that is the correct prediction result is 1 and the likelihood of the other classes is 0.

[0060] For simplicity, we will assume that the correct label assigned to each training data point in the training dataset is the same (for example, the "good product" class). However, the correct labels assigned to each training data point in the training dataset do not necessarily have to be the same. That is, for example, the feature extractor 111a and the classifier 112a may be trained using both product images of good products and product images of defective products, each assigned the correct labels "good product" and "defective product," respectively.

[0061] (Functions of the control unit 110 when extracting features from the training data set) As shown in Figure 4, the control unit 110 functions as a feature extractor 111 when extracting features from each training data in the training data set. The feature extractor 111 may use a trained CNN or BERT model that has been trained by the learning process shown in Figure 3. The feature extractor 111 may use only a part, not all, of the trained feature extractor 111a. In this case, the degree of abstraction of the features extracted by the feature extractor 111 may be lower than the degree of abstraction of the features extracted by the feature extractor 111a. For the sake of simplicity, the following explanation assumes that the trained CNN or BERT model that has been trained by the learning process shown in Figure 3 is used as the feature extractor 111. That is, the explanation assumes that the trained feature extractor 111a is used as the feature extractor 111. The feature extractor 111 and the classifier 112 constitute the learning model.

[0062] The feature extractor 111 extracts features from each training data point in the training data set. The feature extractor 111 stores the multiple features extracted from each training data point in the training data set (hereinafter also referred to as the "first feature group") in the storage unit 120. The first feature group may also be stored in the storage unit 120 as data on the distribution of features in the feature space.

[0063] (Functions of the control unit 110 when assigning correct answers to operational data) As shown in Figure 5, the control unit 110 functions as a feature extractor 111 and a target data selection unit 114 when assigning correct answers to operational data. The feature extractor 111 may be a feature extractor 111 (trained model) that has been trained using the training data set. The feature extractor 111 constitutes an acquisition unit and a feature extraction unit. The target data selection unit 114 constitutes a selection unit, an output unit, and a correct answer assignment result receiving unit.

[0064] The feature extractor 111 acquires multiple operational data (hereinafter also referred to as the "operational data group") by receiving data from the imaging device 200, etc. The feature extractor 111 may also acquire the operational data group by reading it from the storage unit 120, which has been received from the imaging device 200, etc. and stored in the storage unit 120.

[0065] The feature extractor 111 extracts features from each operational data in the operational data set. The feature extractor 111 stores the multiple features extracted from each operational data in the operational data set (hereinafter also referred to as the "second feature group") in the storage unit 120. The second feature group may also be stored in the storage unit 120 as data on the distribution of features in the feature space.

[0066] The target data selection unit 114 selects the operational data (operational data group) from the operational data to be used as training data for correct answer assignment, based on the first feature group and the second feature group, and outputs the selection result. The selection result includes information that can identify the target data, such as the data name of the target data.

[0067] Figure 6 is an explanatory diagram showing the distribution of the first and second feature groups in the feature space. The black circles represent each feature in the first feature group, and the white circles represent each feature in the second feature group. The number (types) of features extracted by the feature extractor 111 is relatively large, and the feature space of the features becomes multidimensional. In the example in Figure 6, for the sake of simplicity, the features of the training data group and the features of the operational data are shown in a two-dimensional feature space that shows the relationship between the two features, the first and second features.

[0068] Figure 7 is an explanatory diagram illustrating the selection of target data. Figure 8 is an explanatory diagram illustrating the re-selection of target data.

[0069] As mentioned above, each training data point in the training dataset is a training data point that has been assigned the same correct label. To simplify the explanation, we will now use the example of a case where the classifier 112a classifies two classes, Class A and Class B. Let Class A be the correct label assigned to each training data point.

[0070] In Figure 7, the first region, which encompasses all of the multiple features (first feature group) extracted from each training data point in the training data set, is shown as the region enclosed by the dashed line. Furthermore, the second region (region), which includes all of the multiple features (first feature group) extracted from each training data point in the training data set and all of the multiple features (second feature group) extracted from each operational data point in the operational data set, is shown as the region enclosed by the dashed line.

[0071] The target data selection unit 114 selects target data based on predetermined criteria. Specifically, the target data selection unit 114 selects target data from the operational data group based on the feature quantities extracted from the operational data at the boundary of the second domain. In the example in Figure 7, the feature quantities extracted from operational data 1, operational data 2, and operational data 3, respectively, correspond to the feature quantities extracted from the operational data at the boundary of the second domain.

[0072] The target data selection unit 114 outputs the result of the target data selection. The output includes transmission to other devices by the communication unit 150 and display on the display unit 130. Figure 7 shows the data names of the operational data selected as target data as a result of the target data selection.

[0073] The target data selection unit 114 outputs the selection result for the target data, and then receives the user's visual confirmation result for the target data. The visual confirmation result for the target data constitutes the correct answer assignment result for the target data.

[0074] In Figure 7, the results of visual verification of each target data are shown in relation to the target data. Operational data 1 and operational data 2 are "Class A," which are the correct labels assigned to the training data as a result of visual verification by users. On the other hand, operational data 3 is "Class B," which are not the correct labels assigned to the training data as a result of visual verification by users.

[0075] In Figure 7, the prediction results by classifier 112a for the target data are shown in relation to each target data. The prediction result for operational data 3 is "Class A". Since operational data 3 is "Class B" based on visual inspection, it is thought that the accuracy of the prediction results has decreased due to data drift.

[0076] The target data selection unit 114 re-selects target data from the operational data based on the received visual verification results. Specifically, the target data selection unit 114 re-selects target data from the operational data based on the first feature group, the second feature group, and the visual verification results for the initially selected target data. More precisely, the target data selection unit 114 deselects operational data from the selected target data that belong to a class different from the class correctly assigned to the training data, based on the results of the visual verification. The target data selection unit 114 then re-selects operational data as target data from which the boundary features of the second region have been extracted after removing the features of the deselected operational data.

[0077] As shown by the "X" in Figure 8, operational data 3 is "Class B," which is different from "Class A," which was correctly identified in the training data, and is therefore deselected. The second region, after removing the features of the deselected operational data 3, is shown as the region enclosed by the dashed line. The features extracted from operational data 4 and operational data 5 correspond to the features at the boundary of this second region. Therefore, operational data 4 and operational data 5 are reselected as new target data.

[0078] The target data selection unit 114 outputs the result of re-selecting the target data, and then accepts the user's visual confirmation result for the target data. In Figure 8, operational data 4 and operational data 5 are "Class A," which is the correct label assigned to the training data as a result of the user's visual confirmation. Since the visual confirmation result for all target data is the class assigned to the training data (i.e., "Class A"), the target data selection unit 114 finishes selecting the target data. In other words, ultimately, operational data 1, operational data 2, operational data 4, and operational data 5 are selected as operational data. The selected operational data are added to the training data.

[0079] The target data selection unit 114 repeatedly deselects and selects target data until the visual confirmation results for all target data indicate that each training data is assigned to the correct class. The target data selection unit 114 terminates the selection of target data when it determines that the visual confirmation results for all target data indicate that each training data is assigned to the correct class.

[0080] The target data selection unit 114 may, when the selection of target data is complete, assign the correct class (in the example in Figure 8, "Class A") assigned to the training data to the operational data from which all features contained within the second region have been extracted. This reduces the effort required to assign correct classes to operational data other than the operational data selected as target data.

[0081] (modified version) The control unit 110 can function as a preprocessing unit that performs preprocessing in the feature space by dividing each feature of the second feature group into multiple groups through clustering.

[0082] The target data selection unit 114 selects target data from the operational data set based on the representative point of the group to which the feature quantities at the boundary of the second domain belong. For each feature quantity included in the group to which the feature quantities at the boundary of the second domain belong, it is not necessary to select target data from the operational data from which features other than the representative point have been extracted. This makes it possible to reduce the amount of work required to obtain correct answer results, such as visual inspection, due to the increase in target data, even when feature quantities are concentrated at the boundary of the second domain.

[0083] The operation of the information processing device 100 will be described below.

[0084] Figures 9A and 9B are flowcharts showing the operation of the information processing device 100. This flowchart is executed by the control unit 110 according to the program stored in the storage unit 120.

[0085] The control unit 110 acquires the learning data set and the operation data set (S101). The control unit 110 can acquire the learning data set by reading it from the storage unit 120. The control unit 110 can acquire the operation data set by reading it from the storage unit 120 or by receiving it from the imaging device 200 or the like.

[0086] The control unit 110 extracts the features of each training data from the training data group and the features of each operational data from the operational data group, respectively (S102). The control unit 110 may also perform a process in advance to extract the distribution of multiple features from each training data from the training data group, store it in the storage unit 120, and use this feature distribution. Similarly, the control unit 110 may perform a process to extract the distribution of multiple features from each operational data from the operational data group, store it in the storage unit 120, and use this feature distribution.

[0087] The control unit 110 selects the operational data from which each feature quantity of the boundary of the second region has been extracted as the target data (S103).

[0088] The control unit 110 outputs the result of the target data selection (S104).

[0089] The control unit 110 determines whether the correct assignment result for all selected target data is the same class that was correctly assigned to each training data (S105).

[0090] If the control unit 110 determines that the correct assignment result for all selected target data is the same as the class assigned to each training data (S105: YES), it terminates the selection of target data and assigns the same class assigned to the training data to all operational data from which all features contained in the second region have been extracted (S106).

[0091] The control unit 110 adds the corrected operational data to the training data (S108).

[0092] On the other hand, if the control unit 110 determines that the correct answer result for all selected target data is not the class that was correctly assigned to each training data (S105: NO), it deselects the operational data to which classes not correctly assigned to the training data were correctly assigned, and reselects the operational data to which the boundary features of the second region have been extracted after removing the features of the deselected operational data as target data (S107).

[0093] The control unit 110 repeatedly executes steps S104, S105, and S107 until the correct answer assignment result for all selected target data becomes the class to which each training data has been correctly assigned.

[0094] This embodiment provides the following effects.

[0095] The system extracts features from both the training data and the operational data. Based on these extracted features, it selects the operational data to be used as training data for ground truth correction and outputs the selection results. This allows for the utilization of operational data while reducing the effort required to correct the operational data, even if data drift occurs.

[0096] Furthermore, target data is selected based on predetermined criteria. This enables efficient assignment of correct answers to operational data.

[0097] Furthermore, it accepts the results of assigning correct answers to the target data. This improves the validity of the operational data used as training data.

[0098] Furthermore, based on the received correct answer results, operational data is re-selected as target data from within the operational data. This further improves the validity of the operational data used as training data.

[0099] Furthermore, multiple operational data sets are acquired as an operational data set, and target data is selected from the operational data based on the features of the training data and multiple features extracted from each operational data set in the operational data set. This increases the amount of appropriate operational data used as training data.

[0100] Furthermore, multiple training data sets are acquired as a training data set, and target data is selected from the operational data based on multiple features extracted from each training data set in the training data set, and multiple features extracted from each operational data set in the operational data set. This increases the amount of operational data used as training data and further improves the validity of the operational data.

[0101] Furthermore, in the feature space, target data is selected from the operational data based on features extracted from the operational data at the boundary of the region that encompasses all of the multiple features extracted from each training data point in the training data set (which contains only training data correctly assigned to the same class) and all of the multiple features extracted from each operational data point in the operational data set. This allows for an efficient increase in the amount of operational data used as training data, and further improves the validity of the operational data.

[0102] Furthermore, the system deselects operational data that have been selected as target data and whose class is corrected based on the correct answer result differs from the class corrected for each training data in the training data set. This process of selecting target data is repeated until the correct answer result for all target data matches the class corrected for each training data in the training data set. This allows for a more efficient increase in the amount of operational data used as training data, and further improves the validity of the operational data.

[0103] Furthermore, the selection of target data is terminated when it is determined that the correct answer assignment result for all target data corresponds to the class assigned to each training data point in the training data set. This allows for a more appropriate reduction in the effort required for correct answer assignment.

[0104] Furthermore, preprocessing is performed on multiple features extracted from each operational data point in the operational data set by dividing them into multiple groups using clustering. Based on the representative point of the group to which the feature at the boundary of the above region belongs, target data is selected from the operational data. For each feature included in the group to which the feature at the boundary of the region belongs, target data is not selected from the operational data from which features other than the representative point have been extracted. This reduces the effort required to obtain correct answer results, such as visual inspection, due to the increase in target data, even when features are concentrated at the boundary of the second region.

[0105] Furthermore, once the selection of target data is complete, the classes assigned to the training data are applied to the operational data from which all features contained within the above-mentioned regions have been extracted. This reduces the effort required to assign correct answers to operational data other than the operational data selected as target data.

[0106] The configuration of the information processing system 10 described above is intended to illustrate the main configuration in order to explain the features of the above-described embodiment, and is not limited to the above configuration, and can be modified in various ways within the scope of the claims. Furthermore, it does not preclude the configurations that are generally found in information processing systems.

[0107] For example, the information processing device 100 may be composed of multiple devices.

[0108] Furthermore, the flowchart described above may omit some steps, and other steps may be added. Also, some steps may be executed simultaneously, and one step may be divided into multiple steps for execution.

[0109] Furthermore, the means and methods for performing the various processing tasks in the information processing system 10 described above can be implemented using either dedicated hardware circuits or a programmed computer. The program may be provided, for example, on a computer-readable recording medium such as a USB memory stick or a DVD (Digital Versatile Disc)-ROM, or it may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a storage unit such as a hard disk. The program may also be provided as a standalone application software, or it may be incorporated as a function into the software of a device such as a detection unit. [Explanation of symbols]

[0110] 10 Information Processing Systems, 100 Information Processing Devices, 110 Control unit, 111a, 111 Feature extractor, 112a classifier, 113 Learning Department, 114 Target data selection unit, 120 storage section, 130 Display section, 140 Input section, 150 Communications Department, 200 imaging devices.

Claims

1. An acquisition unit that acquires learning data and operational data, A feature extraction unit that extracts features from the aforementioned training data and the aforementioned operational data, respectively. A target data selection unit selects, from the operational data, the operational data to be used as training data for correct answer assignment, based on the features of the training data and the features of the operational data. An output unit that outputs the selection result by the target data selection unit, It includes a correct answer result receiving unit that receives the correct answer result for the aforementioned target data, The target data selection unit, based on the received correct answer result, re-selects the operational data from the operational data that is subject to correct answer assignment as the target data. The acquisition unit acquires multiple operational data as an operational data group, The target data selection unit selects the target data from the operational data based on the features of the training data and multiple features extracted from each operational data in the operational data group. The acquisition unit acquires multiple training data as a training data set, The target data selection unit selects the target data from the operational data based on a plurality of features extracted from each training data in the training data group and a plurality of features extracted from each operational data in the operational data group. The target data selection unit is a learning support device that, in the feature space, selects the target data from the operational data based on the features extracted from the operational data at the boundary of a region that includes all of the multiple features extracted from each training data in the training data group, which includes only the training data that has been correctly assigned to the same class, and all of the multiple features extracted from each operational data in the operational data group.

2. The learning support device according to claim 1, wherein the target data selection unit deselects the operational data selected as target data and whose class is correctly assigned by the correct answer result is different from the class correctly assigned to each learning data in the learning data group, and repeats the selection of target data until the correct answer result for all the target data is the class correctly assigned to each learning data in the learning data group.

3. The learning support device according to claim 2, wherein the target data selection unit terminates the selection of target data when it determines that the correct answer assignment result for all of the target data is the class to which each learning data in the learning data group has been correctly assigned.

4. The feature space further includes a preprocessing unit that performs preprocessing to divide multiple feature quantities extracted from each operational data in the operational data group into multiple groups by clustering, The learning support device according to claim 1, wherein the target data selection unit selects the target data from the operational data based on a representative point of the group to which the feature quantity at the boundary of the region belongs, and for each feature quantity included in the group to which the feature quantity at the boundary of the region belongs, the target data is not selected from the operational data from which the feature quantity other than the representative point has been extracted.

5. The learning support device according to claim 3, further comprising a correct answering unit that, when the selection of target data is completed, correctly assigns the classes correctly assigned to the training data to the operational data from which all the features contained within the region have been extracted.

6. A step (a) of acquiring training data and operational data, (b) a step of extracting features from the training data and the operational data, respectively, Step (c) of selecting the operational data from the operational data to be used as training data for correct answer assignment, based on the features of the training data and the features of the operational data, Step (d) outputs the selection result in step (c), Step (e) receiving the result of assigning correct answers to the aforementioned target data, The process includes (f) a step of re-selecting the operational data to be subject to the correct answer assignment from the operational data based on the correct answer assignment result received, In step (a) above, multiple operational data are acquired as an operational data set, In step (c), the target data is selected from the operational data based on the features of the training data and multiple features extracted from each operational data in the operational data group. In step (a) above, multiple training data are acquired as a training data set, In step (c), the target data is selected from the operational data based on the multiple features extracted from each training data in the training data set and the multiple features extracted from each operational data in the operational data set. In step (c) above, a learning support method that selects the target data from the operational data based on the features extracted from the operational data at the boundary of a region that includes all of the multiple features extracted from each training data in the training data group, which includes only the training data that has been correctly assigned to the same class, and all of the multiple features extracted from each operational data in the operational data group.

7. The learning support method according to claim 6, in step (c), the selection of the operational data that has been selected as target data and assigned a correct answer by the correct answer result is different from the class assigned to each learning data in the learning data group, and the selection of target data is repeated until the correct answer result for all the target data is the class assigned to each learning data in the learning data group.

8. The learning support method according to claim 7, wherein in step (c), when it is determined that the correct answer result for all the target data is the class to which each of the learning data in the learning data group has been correctly assigned, the selection of the target data is terminated.

9. The process further includes a step (g) of preprocessing in which multiple feature quantities extracted from each operational data in the operational data group are divided into multiple groups by clustering in the feature space, The learning support method according to claim 6, in step (c), the target data is selected from the operational data based on a representative point of the group to which the feature quantity at the boundary of the region belongs, and for each feature quantity included in the group to which the feature quantity at the boundary of the region belongs, the target data is not selected from the operational data from which the feature quantity other than the representative point has been extracted.

10. The learning support method according to claim 8, further comprising the step (h) of assigning the classes assigned to the learning data to the operational data from which all the features contained within the region have been extracted when the selection of the target data has been completed.

11. A learning support program for causing a computer to execute the learning support method described in any one of claims 6 to 10.

Images