Machine learning support method, program, and machine learning support system

Abstract

To efficiently collect training data required for improving the accuracy of predicting an inference model.SOLUTION: A machine learning support method which is executed by a computer includes: a step of receiving data from a data acquisition apparatus; a step of calculating, based on training data, a prediction probability of the received data, as a degree of reliability of the data; and a step of preparing first reliability threshold with which a false detection or noise is identified and second reliability threshold with which reliable data is identified, and determining that the data having a degree of reliability equal to or lower than the first reliability threshold is discarded data which is a false detection or noise, that the data having a degree of reliability higher than the first reliability threshold and less than the second reliability threshold is low-reliability data which could have been trained insufficiently, and that the data having a degree of reliability equal to or higher than the second reliability threshold is high-reliability data which is reliable.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a machine learning support method, program, and machine learning support system that efficiently collect training data necessary to improve the predictive accuracy of an inference model. [Background technology]

[0002] Some inference models use training data, while others do not. In inference models that use training data, if differences in characteristics arise between the training data used when the inference model was first developed and the training data used after commercialization, the predictive accuracy of the inference model will deteriorate. To prevent this deterioration, a large amount of refined training data that considers various cases during training is required. A related technology that solves this problem is known as augmentation, which involves duplicating and expanding training data (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-95212 Summary of the Invention [Problem to be solved by the invention]

[0004] In related technologies, a small amount of training data is processed and used multiple times, which can lead to overfitting, i.e., overlearning, and actually degrading prediction accuracy. Furthermore, if the training data used for padding is far removed from the actual data, padding such training data is unlikely to result in training data that is useful for predicting actual data, making it difficult to improve the prediction accuracy of the inference model.

[0005] In order to solve the above problem, the present disclosure aims to efficiently collect training data necessary to improve the predictive accuracy of an inference model. [Means for solving the problem]

[0006] To achieve the above object, the machine learning assistance method, program, and machine learning assistance system disclosed herein classify data to be added to training data from acquired data and extract trends in the data to be added.

[0007] Specifically, the machine learning assistance method according to the present disclosure includes: receiving data from a data acquisition device; calculating a predicted probability of the received data as a reliability of the data based on training data; a step of setting a first reliability threshold for determining false positives or noise and a second reliability threshold for determining reliable data, and determining that data whose reliability is equal to or less than the first reliability threshold is discarded data that is false positives or noise, that data whose reliability is greater than the first reliability threshold but less than the second reliability threshold is low reliability data that may be due to insufficient learning, and that data whose reliability is equal to or greater than the second reliability threshold is high reliability data that can be trusted; and is executed by a computer.

[0008] The machine learning assistance method according to the present disclosure may further include a step of outputting the data when it is determined that the reliability is equal to or greater than the second reliability threshold.

[0009] The machine learning assistance method according to the present disclosure includes: storing the data when the reliability is determined to be greater than the first reliability threshold and less than the second reliability threshold; creating a histogram relating to each of the settings and times based on the cumulative totals of the settings and times of the data acquisition device when each of the stored data was acquired; may further comprise:

[0010] In the machine learning assistance method according to the present disclosure, after the step of creating a histogram for each of the setting and the time, extracting the most frequent setting and the most frequent time from the histogram; may further comprise:

[0011] In the machine learning assistance method according to the present disclosure, after the steps of setting the most frequent time and extracting the most frequent time, a step of instructing the data acquisition device to set the most frequent value and the time of the most frequent value; may further comprise:

[0012] In the machine learning assistance method according to the present disclosure, after the steps of setting the most frequent time and extracting the most frequent time, a step of instructing another data acquisition device of the most frequent setting and the most frequent time; may further comprise:

[0013] The present disclosure is a program for implementing the machine learning assistance method on a computer.

[0014] The machine learning assistance system according to the present disclosure includes: receiving data from a data acquisition device; Calculating a predicted probability of the received data as the reliability of the data based on training data; A first reliability threshold is set to determine whether the data is a false positive or noise, and a second reliability threshold is set to determine whether the data is reliable. Data whose reliability is equal to or less than the first reliability threshold is determined to be discarded data, which is a false positive or noise; data whose reliability is greater than the first reliability threshold and less than the second reliability threshold is determined to be low-reliability data that may be due to insufficient learning; and data whose reliability is equal to or greater than the second reliability threshold is determined to be high-reliability data that can be trusted.

[0015] The above inventions can be combined as much as possible. [Effects of the Invention]

[0016] According to the present disclosure, it is possible to efficiently collect training data necessary to improve the predictive accuracy of an inference model. [Brief explanation of the drawings]

[0017] [Figure 1] 1 illustrates an example of a schematic configuration of a machine learning support system according to an embodiment. [Figure 2] 1 shows an example of a procedure of a machine learning assistance method according to an embodiment. [Figure 3] 10 shows a histogram of camera settings according to an embodiment. [Figure 4] 10 shows a histogram of shooting times according to an embodiment. [Figure 5] 10 shows a histogram of camera settings according to an embodiment. [Figure 6] 10 shows a histogram of shooting times according to an embodiment. [Figure 7] FIG. 2 is a diagram illustrating modes of the data acquisition device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings indicate the same components.

[0019] (Embodiment) FIG. 1 shows an example of a schematic configuration of a machine learning support system according to this embodiment. The machine learning support system 30 according to this embodiment is composed of a processing device 10. The machine learning support system 30 may further include a data acquisition device 20. The processing device 10 includes a data receiving unit 11, a reliability calculation unit 12, a data determination unit 13, a low-reliability data analysis unit 14, an instruction unit 15, and an additional training data storage unit 16. The processing device 10 can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0020] An example of the procedure executed by the arithmetic processing device 10 according to this embodiment is shown in Fig. 2. Below, a case will be described in which the arithmetic processing device 10 of the machine learning support system 30 shown in Fig. 1 performs steps S101 to S109 of the machine learning support method of Fig. 2.

[0021] In step S101, the data receiving unit 11 receives data from the data acquisition device 20. If the data from the data acquisition device 20 is an analog signal, the data receiving unit 11 has an A / D conversion function. If the data acquisition device 20 is installed separately from the arithmetic processing device 10, the two may be connected via a public communication network, LAN, LTE, or Wi-Fi. The data receiving unit 11 may also receive data-attached information along with the data acquired by the data acquisition device 20. The data-attached information refers to the settings and time of the data acquisition device 20 when the data acquisition device 20 acquired the data.

[0022] For example, if the data acquisition device 20 is a camera, the data receiving unit 11 may receive, along with image data captured by the camera, the camera settings and the time of capture of the image data as data-attached information. Examples of camera settings include shutter speed and aperture. Further, examples of data-attached information include the weather at the time the image data was captured, or the size or color of the subject. The weather at the time the image data was captured, or the size or color of the subject included in the image data, may be acquired by another sensor or may be input into the data receiving unit 11 by a person. The reliability calculation unit 12, which will be described later, may analyze the image data and add the weather at the time the image data was captured, or the size or color of the subject included in the image data, to the data-attached information.

[0023] When the data acquisition device 20 is a camera, it can recognize various objects visually. Therefore, for example, the present invention can be applied to the recognition of meter readings by having the data receiving unit 11 learn images of meter readings as training data and then receiving the meter images captured by the camera as image data. Other examples include the recognition of numeric characters, the detection of object areas, and the determination of flooding and inundation. When the data acquisition device 20 is a camera, it recognizes objects by capturing images with the camera, which is considered to be effective when it is difficult to attach a sensor directly to the object.

[0024] Furthermore, if the data acquisition device 20 is a sensor that acquires waveform data such as acceleration data or voice data, the data receiving unit 11 may receive, along with the value detected by the sensor, the sensor settings at the time of detection and the detection time as data-attached information. Examples of sensor settings include sampling settings. Further, examples of data-attached information include the weather at the time of sensor detection. This weather may be acquired by another sensor or may be manually input into the data receiving unit 11. Furthermore, the reliability calculation unit 12, which will be described later, may analyze the values ​​detected by the sensor over a certain period of time and add information on the waveform shape and frequency components of the sensor-detected value to the data-attached information.

[0025] In the following description, it is assumed that the data acquisition device 20 is a camera, and the data receiving unit 11 receives image data, as well as the camera settings and the time of photographing the image data.

[0026] In step S102, the reliability calculation unit 12 calculates the reliability of the image data received by the data receiving unit 11. The reliability in this embodiment refers to the predicted probability of data based on training data. In this embodiment, the reliability is the reliability used in a general object detection model, that is, the product of the predicted probability of an object region candidate detected from the image data and the predicted probability of class classification of the object detected from the image data. The predicted probability of the detected object region candidate is an index indicating whether the detected object region candidate is an object and not a background. The predicted probability of class classification of the detected object is an index indicating whether the features of the detected object are similar to the features of the classified class. In this embodiment, an AI inference model that has learned a general object detection model by deep learning based on training data serves as the reliability calculation unit 12 to calculate the reliability in step S102.

[0027] In step S103, the data determination unit 13 classifies the image data into discarded data, low-reliability data, or high-reliability data based on the reliability X calculated by the reliability calculation unit 12. In this embodiment, a first reliability threshold L and a second reliability threshold H, which is higher than the first reliability threshold L, are set as reliability thresholds for classifying image data. The data determination unit 13 classifies image data with a reliability X equal to or less than L as discarded data. Discarded data refers to data with extremely low reliability and therefore unusable, such as when the background in an image is mistakenly detected as an object or when an object in an image is classified into the wrong class. The data determination unit 13 classifies image data with a reliability X greater than L and less than H as low-reliability data. Low-reliability data is data with lower reliability compared to current training data, i.e., data that may have insufficient learning with respect to previous training data, making it difficult for the AI ​​inference model to make predictions even when comparing image data with the current training data, and requires analysis to be added to the training data. The data determination unit 13 classifies image data having a reliability X of H or more as high-reliability data. High-reliability data is data that is highly reliable and does not require analysis for addition to training data. Here, the reliability thresholds L and H may be predetermined values. The reliability thresholds L and H may also be changed depending on the classification results of the image data. For example, as described below, the low-reliability data analysis unit 14 may monitor the additional training data storage unit 16, and if a certain number of low-reliability data are not stored in the additional training data storage unit 16, the reliability thresholds L and H may be changed so that low-reliability data is stored.

[0028] Depending on the classification result of step S103, the data determination unit 13 performs one of step S104, step S105, and step S106. If the image data is classified as discardable data, the data determination unit 13 discards the image data in step S104.

[0029] If the image data is classified as highly reliable data, the data determination unit 13 outputs the image data to a user application, for example, a monitor, in step S105. Note that image data classified as highly reliable data is highly reliable, so the data is used as is.

[0030] If the image data is classified as low-reliability data, the data determination unit 13 stores the image data, the camera settings at the time the image data was captured, and the capture time in the additional learning data storage unit 16 at step S106.

[0031] The low-reliability data analysis unit 14 monitors the additional training data storage unit 16. When a certain amount of low-reliability data has been stored in the additional training data storage unit 16, the low-reliability data analysis unit 14 refers to the camera settings and shooting times when the image data stored in the additional training data storage unit 16 were captured, and creates a histogram for each of the camera settings and shooting times when the image data stored in the additional training data storage unit 16 were captured, based on the cumulative total of the camera settings and shooting times when the image data stored in the additional training data storage unit 16 were captured. Step S107 will be specifically described with reference to FIGS. 3 and 4.

[0032] When a predetermined number N or more of image data are stored in the additional training data storage unit 16, the low-reliability data analysis unit 14 determines camera settings for collecting additional training data based on the cumulative total of camera settings at the time of capturing each piece of image data stored in the additional training data storage unit 16. Specifically, for the parameters used in the camera settings, as shown in FIG. 3, a histogram is created by frequency of occurrence based on the cumulative total of camera settings at the time of capturing the image data stored in the additional training data storage unit 16. Note that when multiple parameters are focused on among the parameters used in the camera settings, a histogram is created for each parameter. The range of the histogram classes is calculated using Sturges's formula. The predetermined number N is 1 or more and may be variable.

[0033] When a predetermined number N or more low-reliability data are stored in the additional training data storage unit 16, the low-reliability data analysis unit 14 creates a histogram for the shooting time by occurrence frequency based on the cumulative total of the shooting times of the image data stored in the additional training data storage unit 16, as shown in Fig. 4. The range of the classes in the histogram for the shooting time is, for example, one hour. The range of the classes in the histogram for the shooting time may be changeable in any time unit.

[0034] In step S108, the low-reliability data analysis unit 14 extracts the most frequent camera settings and shooting times from the histogram of camera settings and shooting times created in step S107 shown in Figures 3 and 4. Step S108 will be described in detail with reference to Figures 5 and 6.

[0035] As shown in FIG. 5, the low-reliability data analysis unit 14 extracts the most frequent class of camera settings from the histogram of camera settings. The low-reliability data analysis unit 14 may use the median value of the most frequent class as the camera setting for collecting additional learning data. Alternatively, the camera settings included in the most frequent class may be sorted in order of value, and the median value may be used as the camera setting for collecting additional learning data. Note that when analyzing trends for multiple camera setting parameters, for example, both shutter speed and aperture, a histogram may be created for each parameter, and the most frequent class may be found for each parameter.

[0036] The low-reliability data analysis unit 14 extracts the most frequent class from the histogram of shooting times and sets it as the shooting time for collecting additional learning data, as shown in Fig. 6. In the following, the shooting time for collecting additional learning data will be described as being in the 17:00 range, as shown in Fig. 6.

[0037] In step S109, the instruction unit 15 instructs the camera according to the camera settings and the shooting time for collecting additional learning data determined in step S108. This step will be specifically described with reference to FIG. 7. For example, as shown in FIG. 7, the camera according to this embodiment has two modes: a normal mode in which shooting is performed periodically, and an additional learning data collection mode in which shooting can be performed between regular shootings. In FIG. 7, shooting in the normal mode of the camera is performed at 10-minute intervals, with each shooting session lasting 2 minutes. Shooting in the normal mode of the camera may also be performed at 60-minute intervals or daily intervals. Based on the fact that the shooting time for collecting additional learning data determined in step S107 is in the 5:00 p.m. range, the instruction unit 15 may interrupt the additional learning data collection mode immediately after the end of shooting in the normal mode around 5:00 p.m., as shown in FIG. 7, to cause the camera to collect additional learning data. Although the shooting time for each additional learning data collection mode is set to 2 minutes, the same as in the normal mode, this is not limiting. During the shooting time for collecting additional learning data, shooting may be performed continuously except in the normal mode. In any case, it is desirable to give priority to the normal mode and not overlap the normal mode and the additional learning data collection mode. Note that if the camera settings for collecting additional learning data are the same as the camera settings for the normal mode, additional learning data may be collected in the normal mode without using the additional learning data collection mode.

[0038] The instruction unit 15 instructs the camera so that the normal mode and the additional learning data collection mode do not overlap, so that additional learning data can be collected in the spare time without interfering with the normal mode operation of the camera.

[0039] Furthermore, multiple machine learning support systems 30 may work together to collect additional training data. For example, the instruction unit 15 of one main machine learning support system 30 is connected to the instruction units 15 of the other machine learning support systems 30. In this case, the instruction unit 15 of one machine learning support system 30 may send an instruction to the instruction unit 15 of the other machine learning support systems 30, causing the other machine learning support systems 30 to collect additional training data required by the one machine learning support system 30. Furthermore, one machine learning support system 30 may cause both its own machine learning support system 30 and the other machine learning support systems 30 to collect the additional training data it requires.

[0040] The additional training data is stored in the additional training data storage unit 16 and used for training the AI ​​inference model. Any means may be used to store the additional training data in the additional training data storage unit 16. For example, the data determination unit 13 may determine the reliability of the received additional training data as described above, and store only the additional training data classified as low-reliability data in the additional training data storage unit 16. Alternatively, the data receiving unit 11 may acquire, together with the data, the mode of the data acquisition device 20 when the data acquisition device 20 acquired the data as data-attached information, and the data determination unit 13 may use the mode of the data acquisition device 20 for determination and store all of the additional training data acquired in the aforementioned additional training data collection mode in the additional training data storage unit 16. Alternatively, a cloud may be used as the additional training data storage unit 16.

[0041] As described above, the machine learning support method, program, and machine learning support system can efficiently collect the training data necessary to improve the predictive accuracy of the inference model by classifying the data to be added to training data from the acquired data and extracting trends in the data to be added. [Industrial Applicability]

[0042] The machine learning assistance method, program, and machine learning assistance system according to the present disclosure can be applied to the information and communications industry. [Explanation of symbols]

[0043] 10: Processing unit 11: Data receiving unit 12: Reliability calculation unit 13: Data judgment section 14: Low-reliability data analysis unit 15: Instruction part 16: Additional learning data storage unit 16 20: Data acquisition device 30: Machine learning support system

Claims

1. A computer comprising: receiving image data from a data acquisition device; calculating a predicted probability when the received image data is input to a trained model based on training data as the reliability of the image data; a step of setting a first reliability threshold for determining false detection or noise and a second reliability threshold for determining reliable image data, determining that the image data whose reliability is equal to or less than the first reliability threshold is discarded data that is false detection or noise, that the image data whose reliability is greater than the first reliability threshold but less than the second reliability threshold is low reliability data that may be due to insufficient learning, and that the image data whose reliability is equal to or greater than the second reliability threshold is high reliability data that can be trusted; Equipped with The predicted probability is a product of a predicted probability of an object region candidate detected from the received image data and a predicted probability of a class classification of the object detected from the image data. Machine learning assisted methods.

2. outputting the image data when it is determined that the reliability is equal to or greater than the second reliability threshold; The machine learning assistance method of claim 1 further comprising:

3. storing the image data when the reliability is determined to be greater than the first reliability threshold and less than the second reliability threshold; creating a histogram relating to each of the settings and times of the data acquisition device based on the cumulative total of the settings and times when each of the stored image data was acquired; The machine learning assistance method according to claim 1 or 2, further comprising:

4. After the step of creating a histogram for each of the settings and the time, extracting the most frequent setting and the most frequent time from the histogram; The machine learning assistance method of claim 3 further comprising:

5. After the steps of setting the most frequent time and extracting the most frequent time, a step of instructing the data acquisition device to set the most frequent value and the most frequent time; The machine learning assistance method of claim 4 further comprising:

6. After the steps of setting the most frequent time and extracting the most frequent time, a step of instructing another data acquisition device of the most frequent setting and the most frequent time; The machine learning assistance method according to claim 4 or 5, further comprising:

7. A program for causing a computer to implement the machine learning assistance method according to any one of claims 1 to 6.

8. receiving image data from a data acquisition device; Calculating a predicted probability when the received image data is input to a trained model based on training data as the reliability of the image data; A machine learning support system that sets a first reliability threshold for determining false positives or noise and a second reliability threshold for determining reliable image data, and determines image data whose reliability is equal to or less than the first reliability threshold as discarded data that is false positives and noise, image data whose reliability is greater than the first reliability threshold but less than the second reliability threshold as low reliability data that may be due to insufficient learning, and image data whose reliability is equal to or greater than the second reliability threshold as high reliability data that can be trusted, The predicted probability is a product of a predicted probability of an object region candidate detected from the received image data and a predicted probability of a class classification of the object detected from the image data. Machine learning assisted systems.

Images