Information processing apparatus and information processing method
By extracting and statistically processing the feature amount of input data, the accuracy threshold of the neural network is automatically set, which solves the problem of difficulty in adjusting classification accuracy in the prior art and improves the inference accuracy and flexibility of the neural network.
Patent Information
- Application Number
- CN202280093861.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-25
- Publication Date
- 2024-11-01
AI Technical Summary
When existing neural networks classify input data, it is difficult to determine the appropriate accuracy, and need to be adjusted through empirical rules or trial and error, resulting in redesign when data changes.
Design an information processing device to extract the feature quantity of input data, calculate the accuracy of various types, and rearrange the accuracy, extract the maximum value label and compare the positive solution label, store consistent and inconsistent classes, and perform statistical processing to Set thresholds to determine classification results.
It realizes automatic determination of appropriate accuracy based on machine learning and input data, reduces dependence on empirical rules and trial and error, improves inference accuracy, and reduces adjustment complexity.
Smart Images

Figure CN118891641A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an information processing device and an information processing method. Background Art
[0002] Generally, a neural network used for classification of input data such as image recognition outputs an inference result based on the accuracy of each classification result when classifying the input data (see Patent Document 1).
[0003] Prior art literature
[0004] Patent Literature
[0005] Patent Document 1: Japanese Patent Application Publication No. 2013-117861 Summary of the invention
[0006] Problem that the invention aims to solve
[0007] However, it is usually difficult to determine the accuracy that serves as a benchmark when inferring results based on the accuracy of each classification result, and the appropriate accuracy must be determined through empirical rules or trial and error. Therefore, machine learning such as the neural network used or the input data must be redesigned every time it changes.
[0008] The present disclosure is used to solve the above-mentioned problems, and its purpose is to provide an information processing device and an information processing method that can determine appropriate accuracy based on the machine learning used, the input data used, and the inference results of the machine learning.
[0009] Means used to solve problems
[0010] The information processing device disclosed in the present invention is characterized in that it comprises: a first feature quantity extraction unit, which extracts feature quantities of input data; a first accuracy calculation unit, which performs inference on the input data based on the feature quantities extracted by the first feature quantity extraction unit, and calculates the accuracy of classifying the input data for each class in a first number of classes; and a first classification unit, which classifies the input data into at least one class in the first number of classes based on the accuracy calculated by the first accuracy calculation unit, and the first classification unit performs: a first process, classifying the input data in an ascending or descending order according to the accuracy calculated by the first accuracy calculation unit The first process is to rearrange the input data; the second process is to extract the label with the maximum accuracy in the rearranged input data; the third process is to compare the label with the maximum value with the correct answer label associated with the input data; the first storage process is to store the classes obtained in the first process that are consistent with the comparison results of the third process; the second storage process is to store the classes obtained in the first process that are inconsistent with the comparison results of the third process; the first statistical process is to perform statistical processing on the classes stored by the first storage process; and the second statistical process is to perform statistical processing on the classes stored by the second storage process.
[0011] Effects of the Invention
[0012] According to the present disclosure, since it is configured as described above, it is possible to determine appropriate accuracy based on the inference result of the machine learning used and the input data used. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 This is a configuration diagram showing an example of the hardware configuration of the information processing device according to Embodiment 1.
[0014] Figure 2 This is a block diagram showing the structure of the information processing device according to the first embodiment.
[0015] Figure 3 This is a flowchart showing the processing performed by the information processing device according to Embodiment 1.
[0016] Figure 4 This is a flowchart showing a process of setting a threshold value performed by the information processing device according to the first embodiment.
[0017] Figure 5 This is a flowchart showing a modified example of the processing performed by the information processing device according to the first embodiment.
[0018] Figure 6 This is a diagram showing an example of a data set of images input to the information processing device according to the first embodiment.
[0019] Figure 7This is a diagram showing an example of a data set of a graph input to the information processing device according to the first embodiment.
[0020] Figure 8 This is a diagram showing an example of a natural language data set input to the information processing device according to Embodiment 1.
[0021] Fig. 9 This is a diagram showing an example of a data set of a time waveform of a signal input to the information processing device according to Embodiment 1.
[0022] Fig.10 This is a flowchart showing an example of a neural network for multi-value classification and binary classification in the information processing device of Embodiment 1.
[0023] Fig.11 This is a diagram showing an example of a second data set generated by the information processing device according to the first embodiment.
[0024] Fig.12 This is a diagram showing the number of binary classifications calculated for the threshold value in 10,000 test data of CIFAR10 by the information processing device of Embodiment 1.
[0025] Fig.13 This is a diagram showing experimental data of inference results when the information processing device according to Embodiment 1 uses binary classification for CIFAR10 and when it does not use binary classification.
[0026] Fig.14 This is a diagram showing experimental data of the time taken for the information processing apparatus according to Embodiment 1 to perform inference on 10,000 data with respect to the threshold of CIFAR10.
[0027] Fig.15 This is a diagram showing an example of the second data set generated by the information processing device according to the third embodiment.
[0028] Fig.16 This is a table showing the accuracy of inference performed by the second learning unit of the information processing device according to the third embodiment.
[0029] Fig.17 This is a graph showing the average value of the inference accuracy of the information processing device according to Embodiments 1 and 5.
[0030] Fig.18 This is a graph showing the median value of the inference accuracy of the information processing device according to Embodiments 1 and 5. DETAILED DESCRIPTION
[0031] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.
[0032] Implementation method 1.
[0033] First, refer to Figure 1 The hardware configuration of the information processing device 100 according to the first embodiment will be described. Figure 1 1 is a block diagram showing an example of the hardware structure of the information processing device 100 of Embodiment 1. The information processing device 100 may be an independent computer not connected to an information network, or a server or client of a server-client system connected to a cloud or the like via an information network. In addition, the information processing device 100 may also be a smartphone or a microcomputer. In addition, the information processing device 100 may also be a computer used in a closed network environment in a factory, which is called edge computing.
[0034] For example, the information processing device 100 includes a CPU (Central Processing Unit) 1, a ROM (Read Only Memory) 2a, a RAM (Random Access Memory) 2b, a hard disk (HDD) 2c, and an input / output interface 4, which are connected to each other via a bus wiring 3. In addition, for example, the information processing device 100 includes an output unit 5, an input unit 6, a communication unit 7, and a drive 8 connected to the input / output interface 4.
[0035] The input unit 6 is composed of, for example, a keyboard, a mouse, a microphone, and a camera. The output unit 5 is composed of, for example, an LCD (Liquid Crystal Display) and a speaker. When a command is input to the CPU 1 via the input / output interface 4 by the user operating the input unit 6, the CPU 1 executes the program stored in the ROM 2a. In addition, the CPU 1 loads the program stored in the hard disk 2c or SSD (Solid State Drive, not shown) into the RAM (Random Access Memory), reads and writes and executes it as needed. Thus, the CPU 1 performs various processes to enable the information processing device 100 to function as a device having prescribed functions.
[0036] The CPU 1 outputs the results of various processes via the input / output interface 4. For example, the CPU 1 outputs the results of various processes from the output device as the output unit 5. In addition, for example, the CPU 1 outputs (transmits) the results of various processes to an external device from the communication device as the communication unit 7. In addition, for example, the CPU 1 outputs the results of various processes and records them in the storage unit 20 (see FIG. 2 ) such as the hard disk 2c. Figure 2 For example, various information input from the input unit 6 and the communication unit 7 via the input / output interface 4 is recorded in the hard disk 2c. The CPU 1 calls out various information recorded in the hard disk 2c from the hard disk 2c for use as needed.
[0037] For example, the program executed by the CPU 1 is pre-recorded in the hard disk 2c or ROM 2a as a recording medium built into the information processing device 100. In addition, for example, the program executed by the CPU 1 is stored (recorded) in a removable storage medium 9 connected via a drive 8. Such a removable storage medium 9 can also be provided as so-called packaged software. As the removable storage medium 9, for example, there are floppy disks, CD-ROMs (Compact Disc Read Only Memory), DVDs (Digital Versatile Discs), magnetic disks, semiconductor memories, etc.
[0038] In addition, for example, a system (Comport) such as WWW (World Wide Web) that connects a plurality of hardwares via either or both of wired and wireless connections may transmit and receive a program executed by the CPU 1 via the communication unit 7. In addition, for example, when the information processing device 100 performs learning described later, parameters obtained through learning, especially weight functions in a neural network, may be transmitted and received by the above method.
[0039] For example, CPU 1 functions as a machine learning device that performs machine learning calculations. In addition, such a machine learning device can be composed of general hardware such as a GPU (Graphics Processing Unit) that is good at parallel calculations, in addition to a CPU, and can also be composed of an FPGA (Field-Programmable Gate Array) or dedicated hardware.
[0040] In addition, the information processing device 100 may also be composed of multiple computers connected via a communication port, and may also be implemented by hardware with different structures in which learning and reasoning described later are independent of each other. In addition, the information processing device 100 may also receive a single or multiple sensor signals from an external sensor connected via a communication port. In addition, the information processing device 100 may also prepare multiple virtual hardware environments within one hardware and virtually process each virtual hardware as a separate hardware.
[0041] Next, refer to Figure 2 The functions of the information processing device 100 will be described. Figure 2 1 is a block diagram showing the configuration of the information processing device 100 according to Embodiment 1. The information processing device 100 is configured by the above-mentioned hardware configuration to include a control unit 10 , an input unit 6 , an output unit 5 , a communication unit 7 , and a storage unit 20 .
[0042] Input data from the input unit 6, the communication unit 7, and the storage unit 20 are input to the control unit 10. The storage unit 20 is composed of, for example, a ROM 2a, a RAM 2b, a hard disk 2c, a drive 8, etc., and stores various information used by the information processing device 100 and various data and information such as results of calculations performed by the information processing device 100.
[0043] The control unit 10 includes a first learning unit 11, a second learning unit 12, a first feature extraction unit 13A, a second feature extraction unit 13B, a learning data generation unit 14, a threshold setting unit 15, an accuracy determination unit 16, and a classification result selection unit 17. Based on the data input from the input unit 6 and the communication unit 7 and the data and information obtained from the storage unit 20, the first learning unit 11, the second learning unit 12, the first feature extraction unit 13A, the second feature extraction unit 13B, the learning data generation unit 14, the threshold setting unit 15, the accuracy determination unit 16, and the classification result selection unit 17 perform various processes. For example, the control unit 10 outputs the results obtained by performing various processes to the outside via the output unit 5 and the communication unit 7. In addition, for example, the control unit 10 causes the results obtained by performing various processes to be stored in the storage unit 20. In addition, the input unit 6, the communication unit 7, and the storage unit 20 constitute the input unit in the first embodiment. In addition, the output unit 5, the communication unit 7, and the storage unit 20 constitute the output unit in the first embodiment.
[0044] The first learning unit 11 and the second learning unit 12 learn based on the input data from the input unit 6, the communication unit 7, and the storage unit 20, and infer the input data from the input unit 6, the communication unit 7, and the storage unit 20 in the state of learning, and classify the input data into any class among a plurality of classes. The first feature quantity extraction unit 13A and the second feature quantity extraction unit 13B extract the feature quantity of the input data from the input unit 6, the communication unit 7, and the storage unit 20. In other words, the first feature quantity extraction unit 13A and the second feature quantity extraction unit 13B quantify the feature quantity of the input data from the input unit 6, the communication unit 7, and the storage unit 20. In addition, the first feature quantity extraction unit 13A and the second feature quantity extraction unit 13B extract the feature quantity of input data that is different from each other.
[0045] The learning data generating unit 14 generates learning data for the second learning unit 12 to learn based on the learning data for the first learning unit 11 to learn input from the input unit 6, the communication unit 7 and the storage unit 20. The threshold setting unit 15 sets a threshold value to be referred to when the control unit 10 performs a prescribed process. The accuracy determination unit 16 determines whether the accuracy of the estimation made by the first learning unit 11 is below or exceeds the threshold value set by the threshold setting unit 15. The classification result selection unit 17 selects and outputs either the classification result of the first learning unit 11 or the classification result of the second learning unit 12 based on the determination result of the accuracy determination unit 16. The details of the learning data generating unit 14, the threshold setting unit 15, the accuracy determination unit 16 and the classification result selection unit 17 are described in detail later.
[0046] The first learning unit 11 includes a first model generating unit 11A, a first accuracy calculating unit 11B, and a first classifying unit 11C. The first model generating unit 11A performs learning based on input data from the input unit 6, the communication unit 7, and the storage unit 20, and generates a first learned model.
[0047] The first accuracy calculation unit 11B performs inference (recognition) of the input data from the input unit 6, the communication unit 7 and the storage unit 20 based on the feature extracted by the first feature extraction unit 13A and the first learned model, and calculates the accuracy of the input data being classified into each of the plurality of pre-set classes by the first learned model. In addition, in the first embodiment, the accuracy of the input data being classified into each of the plurality of pre-set classes by the learned model is also referred to as the accuracy of inference. For example, in a classification problem for three classes, three numbers are obtained by inputting the input data into the learned model. These three numbers are, for example, 0.3, 0.6, and 0.1. In the present embodiment, the numbers are referred to as the accuracy of inference. In this example, it is shown that the total accuracy is 1 by normalization, but it does not have to be 1. The first classifying unit 11C classifies input data from the input unit 6 , the communication unit 7 , and the storage unit 20 into at least one of a plurality of preset classes using the first learned model based on the inference accuracy calculated by the first accuracy calculating unit 11B.
[0048] The second learning unit 12 includes a second model generating unit 12A, a second accuracy calculating unit 12B, and a second classifying unit 12C. The second model generating unit 12A performs learning based on input data from the input unit 6, the communication unit 7, and the storage unit 20, and generates a second learned model.
[0049] The second accuracy calculation unit 12B performs inference (recognition) of input data from the input unit 6, the communication unit 7, and the storage unit 20 based on the feature extracted by the second feature extraction unit 13B and the second learned model, and calculates the accuracy (inference accuracy) of the input data being classified into each of the plurality of pre-set classes by the second learned model. The second classification unit 12C classifies the input data from the input unit 6, the communication unit 7, and the storage unit 20 into any class among the plurality of pre-set classes by the second learned model based on the inference accuracy calculated by the second accuracy calculation unit 12B.
[0050] In this way, the first learning unit 11 and the second learning unit 12 function as a learning device, which generates a learned model by learning based on the learning data input from the input unit 6, the communication unit 7 and the storage unit 20, and infers the input data from the input unit 6, the communication unit 7 and the storage unit 20 based on the generated learned model, thereby classifying the input data.
[0051] Next, refer to Figure 2 and Figure 3 An overview of the processing performed by the information processing device 100 will be described. Figure 3 1 is a flowchart showing the processing performed by the information processing device 100 according to Embodiment 1. The processing performed by the information processing device 100 can be divided into a processing for performing learning and a processing for performing inference.
[0052] First, an overview of learning is described. When the information processing device 100 is learning, it obtains the first data set, which includes learning data as a plurality of first input data, and correct labels of N-value classification (first quantity classification) problems respectively attached to the plurality of learning data (step ST1). In other words, when the information processing device 100 is learning, it obtains the first data set, which includes a plurality of correct labels corresponding to a plurality of classes, and a plurality of input data corresponding to the plurality of correct labels, i.e., learning data. In addition, the above-mentioned N as the first quantity is a natural number specified such that 3≤N. In addition, when the information processing device 100 is learning, it can obtain the first data set via the input unit 6 and the communication unit 7 each time, or it can read and use data previously obtained and stored in the storage unit 20.
[0053] When performing the processing of step ST1, the information processing device 100 learns the N-value classification problem through the first model generation unit 11A to generate the first learned model. In addition, when performing the processing of step ST1, the information processing device 100 re-labels the correct labels of the first data set through the learning data generation unit 14 in a manner that the number of classes is M-value classification (second number classification) different from the N-value classification, and produces the second data set (step ST3). In other words, the information processing device 100 re-labels the correct labels of the first data set through the learning data generation unit 14 in a manner that the number of classes is M (second number) M-value classification (second number classification), and produces the second data set. In embodiment 1, the correct labels of the first data set are re-labeled in a manner that the correct labels of the first data set become binary classifications, and the second data set is generated. In addition, the above-mentioned M as the second number can be a natural number specified to satisfy M≤N.
[0054] When performing the processing of step ST3, the information processing device 100 uses the generated second data set to learn the binary classification through the second model generation unit 12A to generate a second learned model (step ST4). In addition, the second learned model can be a single learned model that outputs one result for one input data, or it can be composed of multiple learned models so that multiple results are output for one input data.
[0055] Next, an overview of the inference is described. When performing the processing of step ST2, the information processing device 100 infers unknown input data (for example, test data) that is not included in the first data set through the first learning unit 11 (step ST5). The information processing device 100 performs inference through the first accuracy calculation unit 11B, and calculates the accuracy of the inference of the input test data for each N value (class). In this process, the information processing device 100 classifies the input data into the class with the highest inference accuracy (first class) among the N (first number) classes of inference candidates (classification candidates) of the input data through the first classification unit 11C. In addition, in the following description, the class with the highest inference accuracy is also referred to as the first inference candidate, and the class with the second highest inference accuracy (second class) is also referred to as the second inference candidate. In addition, this embodiment can also be applied to the case where there are more than two correct labels for one input data, such as MultiMNIST, which is one of the data sets. When it is known that there are two correct labels, the first inference candidate and the second inference candidate are set as inference values, and the label corresponding to the inference value is set as the inference label. However, when there are multiple correct labels, the same processing is used as when there is one correct label. Therefore, in this embodiment, the case where there is one correct label is described.
[0056] When performing the process of step ST5 , the information processing device 100 uses the accuracy determination unit 16 to determine whether the accuracy of the first inference candidate is equal to or less than the threshold value set in advance by the threshold setting unit 15 (step ST6 ).
[0057] In the processing of step ST6, when the accuracy of the inference of the first inference candidate exceeds the threshold value (No in step ST6), the information processing device 100 selects and outputs the classification result of the first classification unit 11C from the classification results of the first classification unit 11C and the classification results of the second classification unit 12C through the classification result selection unit 17, that is, the value of the class based on the first classification unit 11C as the first inference candidate.
[0058] In addition, in the process of step ST6, when the accuracy of the inference of the first inference candidate is below the threshold value (yes in step ST6), the information processing device 100 selects and outputs the classification result of the second classifier 12C from the classification result of the first classifier 11C and the classification result of the second classifier 12C, performs inference of binary classification for the input data through the second accuracy calculation unit 12B, and calculates the accuracy of the inference for each of the two classes. In addition, the information processing device 100 classifies the input data into the class with the higher accuracy of the inference of the two classes that are the inference candidates of the input data through the second classifier 12C. The value is output as the classification result and the inference result. When performing any of the processes in step ST6 and step ST7, the information processing device 100 outputs any one of the classification result of the first classifier 11C and the classification result of the second classifier 12C from the control unit 10 to any one of the output unit 5, the communication unit 7, and the storage unit 20 based on the selection result of the classification result selection unit 17.
[0059] In addition, in the process of step ST6, the information processing device 100 determines whether the accuracy of the inference performed by the first learning unit 11 is below the threshold value through the accuracy determination unit 16, but the present invention is not limited thereto. The information processing device can determine whether the accuracy of the inference performed by the first learning unit is greater than the threshold value or less than the threshold value through the accuracy determination unit, and can determine whether the accuracy of the inference performed by the first learning unit is less than the threshold value, or whether the accuracy of the inference performed by the first learning unit is greater than the threshold value, or whether the accuracy of the inference performed by the first learning unit exceeds the threshold value.
[0060] In addition, the information processing device 100 of the first embodiment uses the accuracy of inference and the threshold value which are both positive values for processing, but is not limited to this. In the case where the calculated accuracy of inference and the threshold value are negative values, the information processing device can also be configured so that, in the processing performed by the above-mentioned accuracy determination unit, when the accuracy of inference performed by the first learning unit exceeds the threshold value, the inference result is output based on the inference performed by the first learning unit, and when the accuracy of inference performed by the first learning unit is below the threshold value, the inference result is output based on the inference performed by the second learning unit. The method of setting the threshold value by the threshold setting unit 15 will be described later. For example, the information processing device 100 performs statistical processing on the correctly inferred result and the incorrectly inferred result, and sets the value between them as the threshold value.
[0061] Next, use Figure 4 The threshold value is described. Figure 4 : is a flowchart showing the process of setting a threshold value performed by the information processing device 100 .
[0062] like Figure 4 As shown, for example, the information processing device 100 performs: a first process of rearranging the input data by the first classification unit 11C in such a way that the accuracy calculated by the first accuracy calculation unit 11B is in ascending or descending order; a second process of extracting the label with the maximum accuracy in the rearranged input data; a third process of comparing the label with the maximum value with the correct answer label associated with the input data; a first storage process of storing the classes obtained in the first process that are consistent with the comparison results of the third process; a second storage process of storing the classes obtained in the first process that are inconsistent with the comparison results of the third process; a first statistical process of statistically processing the classes stored in the first storage process; and a second statistical process of statistically processing the classes stored in the second storage process. The threshold setting unit 15 sets a threshold value set between the first statistical value calculated by the first statistical process and the second statistical value calculated by the second statistical process, and the first classification unit 11C classifies the input data based on the comparison result between the accuracy calculated by the first accuracy calculation unit 11B and the threshold value. The first statistical process and the second statistical process are, for example, processes for calculating any one of the average value, the central value, the standard deviation, or the information entropy. In addition, the first statistical process and the second statistical process may also be processes for calculating a combination of two or more of the average value, the central value, the standard deviation, or the information entropy. The second process is, for example, a process for extracting a label that has a minimum value, and the third process is, for example, a process for comparing a label that has a minimum value with a correct answer label associated with the input data.
[0063] Specifically, the information processing device 100 first obtains a first data set, which includes a plurality of first input data and correct answer labels of N-value classification problems respectively attached to the plurality of first input data (step ST1). After performing the processing of step ST1, the information processing device 100 refers to the information stored in the storage unit 20, calls out the first learned model for inference by the first learning unit 11 (step ST8), and infers the N-value classification problem for the input first input data through the first learning unit 11, and calculates the accuracy of the inference for each first input data (step ST5). For example, in the processing of step ST5, the information processing device 100 calculates the accuracy of the inference for a plurality of input data that are not used in the generation of the first learned model.
[0064] When performing the processing of step ST5, the information processing device 100 rearranges the inferred inference data in such a way that the calculated accuracy is in ascending or descending order (first process, step ST19). In other words, the information processing device 100 sorts the inferred inference data in such a way that the calculated accuracy is in ascending or descending order. When performing the processing of step ST19, the information processing device 100 extracts the label (inference label) with the maximum accuracy for each inference data after sorting (second process), and determines whether the extracted inference label is consistent with the correct answer label (third process, step ST20).
[0065] In the process of step ST20, when the inference label and the correct answer label are consistent (yes in step ST20), the first storage unit of the storage unit 20 stores the corresponding sorted inference data (first storage process, step ST21). When performing the process of step ST22, the information processing device 100 performs statistical processing on the sorted inference data stored in the first storage unit through the first statistical unit of the threshold setting unit 15 (first statistical process, step ST22).
[0066] In the process of step ST20, when the inference label and the correct answer label do not match (No in step ST20), the corresponding sorted inference data is stored in the second storage unit of the storage unit 20 (second storage process, step ST23). When performing the process of step ST23, the information processing device 100 performs statistical processing on the sorted inference data stored in the second storage unit through the second statistical unit of the threshold setting unit 15 (second statistical process, step ST24).
[0067] After performing the processes of step ST22 and step ST24 , the information processing device 100 sets a threshold value based on the results of these statistical processes (step ST25 ).
[0068] In addition, for example, the threshold setting unit 15 sets the threshold value to be less than the first statistical value calculated by the first statistical process. Thus, for values above the first statistical value that become the threshold value, it can be determined that the accuracy is high enough and no analysis is required, so the threshold value can be locked. In addition, the threshold setting unit 15 sets the threshold value between the first statistical value calculated by the first statistical process and the second statistical value calculated by the second statistical process. In other words, the threshold setting unit 15 sets the threshold value to a value that is less than the first statistical value calculated by the first statistical process and greater than the second statistical value calculated by the second statistical process. Thus, for values above the first statistical value that become the threshold value, it can be determined that the accuracy is high enough, and for values below the second statistical value, it can be determined that the possibility of being difficult to classify is high regardless of the method used, so the range of the locked threshold value can be narrowed. In addition, for example, the threshold setting unit 15 sets the threshold value to the average value of the first statistical value and the second statistical value. In addition, for example, the threshold setting unit 15 sets the threshold value to a weighted average value with the number of input data assigned to the first statistical value and the second statistical value as the weight. In addition, the threshold setting unit 15 can also use a combination of the average value and the weighted average of the first statistical value, the standard deviation other than the average, and the central value, and determine the condition that does not satisfy all values as the threshold. It can also use a combination of the average value and the weighted average of each of the first statistical value and the second statistical value, the standard deviation other than the average, and the central value, and determine the value between the statistical values of the first statistical value and the second statistical value as the threshold.
[0069] For example, when the highest accuracy among the accuracies for classifying each class in the first number of classes calculated by the first accuracy calculation unit is set as the 5th accuracy, the threshold setting unit 15 may also set the threshold to a value that is between either the average value or the central value of the 5th accuracy when a result consistent with the class corresponding to the correct answer label is obtained, and between either the average value or the central value of the 5th accuracy when a result inconsistent with the class corresponding to the correct answer label is obtained in the result of classifying multiple input data of the first data set by the first classification unit.
[0070] In addition, the threshold setting unit 15 can also set the threshold to a value that is between the average value of the 5th accuracy when the result of classifying the multiple input data of the first data set by the first classification unit is consistent with the class corresponding to the correct answer label, and the average value of the 5th accuracy when the result of classifying the multiple input data of the first data set by the first classification unit is inconsistent with the class corresponding to the correct answer label, and is between the central value of the 5th accuracy when the result of classifying the multiple input data of the first data set by the first classification unit is consistent with the class corresponding to the correct answer label, and the central value of the 5th accuracy when the result of classifying the multiple input data of the first data set by the first classification unit is inconsistent with the class corresponding to the correct answer label.
[0071] In addition, when the second highest accuracy of the highest accuracy among the accuracies for classifying each class in the first number of classes calculated by the first accuracy calculation unit (or the accuracy of any class after the second largest accuracy) is set as the sixth accuracy, the threshold setting unit 15 may also set the threshold to a value that is between either the average value or the central value of the sixth accuracy when a result consistent with the class corresponding to the correct answer label is obtained in the result of classifying the multiple input data of the first data set by the first classification unit, and between either the average value or the central value of the sixth accuracy when a result inconsistent with the class corresponding to the correct answer label is obtained in the result of classifying the multiple input data of the first data set by the first classification unit.
[0072] In addition, the threshold setting unit 15 can also set the threshold to a value that is between either the average value and the central value of the 5th accuracy when the result of classifying the multiple input data of the first data set by the first classification unit is consistent with the class corresponding to the correct answer label, and either the average value and the central value of the 6th accuracy when the result of classifying the multiple input data of the first data set by the first classification unit is consistent with the class corresponding to the correct answer label, and between either the average value and the central value of the 5th accuracy when the result of classifying the multiple input data of the first data set by the first classification unit is inconsistent with the class corresponding to the correct answer label, and either the average value and the central value of the 6th accuracy when the result of classifying the multiple input data of the first data set by the first classification unit is inconsistent with the class corresponding to the correct answer label.
[0073] Furthermore, the threshold setting unit 15 may set the threshold for each subset of input data included in the first data set, or may set the threshold for each of a plurality of classes classified by the first classification unit.
[0074] Furthermore, for example, when the value of the label extracted in the second process that is the object of comparison with the threshold value set by the threshold setting unit 15 is less than the threshold value, the information processing device 100 performs inference using the second feature quantity extraction unit 13B through the second classification unit 12C. Furthermore, for example, when the maximum value of the accuracy in the second process of the input data is less than the threshold value set by the threshold setting unit 15, the information processing device 100 performs inference using the second feature quantity extraction unit through the second classification unit 12C.
[0075] By the above method, the condition of the threshold value can be locked, so it can be obtained by a method that does not rely on empirical rules. In addition, when trial and error (parameter scanning) is performed for the purpose of further optimization, the search range is also narrowed, so the optimal value can be reached with a smaller number of trials. In addition, this method does not depend on the machine learning used and the input data used, so the appropriate accuracy can be determined no matter what is used.
[0076] According to the present invention, it is clear that data with a small maximum value of accuracy that is independent of the size of the data set tends to be prone to errors. Moreover, by setting a threshold value for the accuracy, even if learning is performed with a small data set, data with a small accuracy can be removed, thereby achieving an effect of improving the inference accuracy. By using an information processing device that not only further removes but also achieves a higher accuracy, inference can be performed with a higher accuracy, resulting in an effect of improving the inference accuracy.
[0077] <Data used for the first learning section>
[0078] Next, the first data set and the learning and inference of the first learning unit 11 , and the second data set and the learning and inference of the second learning unit 12 will be described in sequence.
[0079] The data input to the information processing device 100 is, for example, an image, a graph, a text, and a time waveform. The information processing device 100 processes the input data as a multi-value classification problem, i.e., an N-value classification problem, and outputs a classification result. Multi-value classification is, for example, an example of classification using machine learning, which infers (recognizes) which of the 10 values 0 to 9 the input data is through a learned model, and outputs an inference result (classification result, recognition result).
[0080] The learning data used by the information processing device 100 in machine learning is supervised data. The supervised data has one or more classification values for each of the multiple input data. In embodiment 1, the classification value for the above-mentioned supervised data is called a correct answer label. For example, the correct answer label of "handwritten character 5" in MNIST (Modified National Institute of Standards and Technology database) is "5". In addition, the above-mentioned group of learning data and correct answer labels is called a data set.
[0081] Next, the correct answer label is explained. In the case of 10-value classification, the correct answer label usually uses integers from 0 to 9, but is not limited to consecutive integers or numbers starting from 0. Other methods such as One Hot Vector, in which only 1 is put into the corresponding correct answer label in a manner such as setting the 1 to (1, 0, 0), setting the 2 to (0, 1, 0), and setting the 3 to (0, 0, 1) are also effective. For example, in the case of 10-value classification, the correct answer label can also be defined in a 10×10 matrix. In addition, in Implementation 1, 10-value classification is used for easy understanding, but the classification performed by the information processing device only needs to be N-value classification with 3≤N, for example, it can also be the classification of a data set with 20,000 correct answer labels for 14 million input data, such as ImageNet, which is a famous data set in image recognition. In addition, in a regression problem different from a classification problem, when the range of the correct answer label of the regression is a real number such as 0 to 100, the correct answer label is converted into 100 discrete values such as 0 to 1, 1 to 2,..., 99 to 100. Thus, by converting it into a classification problem classified into three or more values, the regression problem can also be applied to the information processing device 100.
[0082] Next, the information processing device 100 is described. The information processing device 100 of Embodiment 1 has a structure for classifying input data into N values. The information processing device 100 may also be a different algorithm such as deep learning, gradient boosting, support vector machine, logistic regression, k-nearest neighbor method, decision tree, naive Bayes, and a combination thereof, which has a structure for classifying input data into N values.
[0083] In the first embodiment, as learning performed by the information processing device, deep learning, which is an example of learning with high inference accuracy (accuracy of inference) and preferred, is used as an example to illustrate. As algorithms for deep learning, various algorithms are known according to the input data. For example, if the input data is image data, algorithms such as CNN (convolutional neural network), MLP (Multi-Layer Perceptron), and Transformer are known. In addition, in CNN, algorithms such as Vgg, ResNet, DenseNet, MobileNet, and EfficientNet are also known, which have the common point of performing convolution. In addition, in MLP, algorithms such as a combination of pure full connection and MLP-Mixer are also known. In Transformer, algorithms such as an algorithm combined with feature extraction of CNN and Vision Transformer are also known. The information processing device can use these single methods or a method of combining multiple of them. In addition, in implementation mode 1, the first learning unit 11 and the second learning unit 12 are described, but the first learning unit and the second learning unit can be different algorithms. In addition, the second learning unit can be composed of more than two devices, and each device uses more than two different algorithms.
[0084] Next, the presence or absence of learning will be described. Information processing device 100 performs learning and inference using a learning data set. In Embodiment 1, learning refers to the process of optimizing parameters within information processing device 100, and inference refers to the operation of input data based on the optimized parameters.
[0085] Figure 5 1 is a flowchart showing a modified example of the processing performed by the information processing device 100 of Embodiment 1. For example, after performing the processing of step ST1, the information processing device 100 may refer to the information stored in the storage unit 20, call out the learned model for inference by the first learning unit 11 (step ST8), and perform inference on the N-value classification problem for the input data by the first learning unit 11 (step ST5).
[0086] Furthermore, in the process of step ST5, when the accuracy calculated by the first learning unit 11 is below the threshold value (yes in step ST6), the information processing device 100 may refer to the information stored in the storage unit 20, call out the learned model for inference by the second learning unit 12 (step ST9), and infer the binary classification problem for the input data by the second learning unit 12 (step ST7). In this way, the information processing device 100 may store the learned model in the storage unit 20 in advance, and call out the learned model for inference as needed.
[0087] Next, refer to Figures 6 to 9 , the data input to the information processing device 100 and the classification problem processed in the information processing device 100 are explained. Figure 6 1 is a diagram showing an example of a data set of images input to the information processing device 100 . Figure 6 The images shown on the left side of include static images and dynamic images, but the dynamic image can be considered to be obtained by continuously combining static images. Therefore, in embodiment 1, the case of inputting static image data to the information processing device 100 is described.
[0088] The static image data input to the information processing device 100 may be a color image composed of a combination of two or more channels such as RGB, or a monochrome image composed of one channel. In addition, regarding the processing when there are multiple channels, various processing is known depending on the algorithm of the information processing device 100, but generally, processing is performed to aggregate the channels into one channel using a weight matrix for coupling the channels.
[0089] In addition, the size of the image data input to the information processing device 100 may be 32 pixels × 32 pixels like MNIST or CIFAR10 (Canadian Institute For Advanced Research10), 96 pixels × 96 pixels like STL10, or other sizes, or other than square. In addition, when the size of the image data input to the information processing device 100 is smaller, the operation time is shorter.
[0090] The input image data can be a sensor signal that converts physical data into numerical data through a CCD (Charge Coupled Device) camera, CMOS (Complementary MOS) camera, infrared camera, ultrasonic measuring device, antenna and other devices that capture electromagnetic waves, or a graphic created on a computer using CAD (Computer Aided Design).
[0091] Figure 7 FIG. 1 is a diagram showing an example of a data set of a graph input to the information processing device 100. Figure 7 In the classification problem in the graph shown on the left side of , multiple problem settings are considered. The graph is composed of nodes, which are points, and edges, which are lines connecting points. The nodes and edges have arbitrary graph information. For example, the main classification problems in such a graph include the problem of classifying nodes based on edges and graph information, the problem of classifying edges based on nodes and graph information, and the problem of learning multiple graphs to classify graphs.
[0092] For example, a circuit can be represented as a graph. For example, as a problem of classifying nodes, consider the following problem: when the data input to the information processing device is set as a circuit diagram and the data output by the information processing device is set as the output voltage between any terminals of the circuit, select the circuit components in such a way that the desired output voltage is obtained. For example, there are a limited number of capacitors, coils, diodes, resistors, etc. as circuit components, so the problem of selecting circuit components in such a way that the desired output voltage is obtained in the above circuit can be handled as a classification problem.
[0093] In addition, for example, as a problem of classifying edges, in a circuit diagram containing all necessary components, when the configuration positions of the components are set as nodes of the diagram and the wiring connecting the components is set as the edges of the diagram, the problem of optimizing the wiring connecting the components can be handled as a classification problem. In order to classify the information processing device 100 of embodiment 1, more than two nodes are required, but if there are more than two components, it can be handled as a multi-value classification problem. In addition, for example, when a diagram that becomes a circuit diagram is provided, the problem of classifying the diagram into a boost power supply circuit, a buck power supply circuit, a buck-boost power supply circuit, an insulating circuit, a non-insulating circuit, etc., and the problem of classifying which one is a power supply circuit, a sensor circuit, a communication circuit, and a control circuit, etc. can be handled as a classification problem for classifying the diagram.
[0094] Figure 8 is a diagram showing an example of a natural language data set input to the information processing device 100. Figure 8In the classification problem of classifying the natural language shown on the left side of , consider the following case: as the input data, data obtained by cutting out a part of the article block such as a sentence, a paragraph, a section, or the whole text is provided. For example, when data of a certain news report is provided, the problem of reasoning to classify it into any of the following: economics, politics, sports, and science is a classification problem.
[0095] Such a classification problem can also be a classification problem that is evaluated through a sentence or a paragraph. For example, it can be a classification problem that provides a novel and infers the author and type of the novel. It can also be a problem that classifies the source code of a programming language, the G code of an NC milling cutter, etc. by function. It can also classify the provided sentences into categories such as joy, anger, sorrow, and happiness to conduct emotional analysis.
[0096] Fig. 9 1 is a diagram showing an example of a data set of a time waveform of a signal input to the information processing device 100. Fig. 9 The problem of classifying the time waveform, which is a set of continuously changing numerical values including time series data shown on the left side of the figure, is to classify the time waveform. For example, when the time waveform of a signal with time as the horizontal axis and voltage, wave height value or any other physical information as the vertical axis is used as input data, the time waveform is classified. For example, the problem of classifying whether the time waveform of a signal in a circuit is a power supply circuit, a sensor circuit, a communication circuit or a control circuit based on the time waveform can be handled as a classification problem. In addition, the horizontal axis of the data input to the information processing device 100 is not limited to time, and can be any feature quantity as long as it is a feature quantity with a physical range such as frequency and coordinates.
[0097] In the above, an example of data input to the information processing device 100 is described. However, the data input to the information processing device 100 may be, for example, an iris dataset classified into three categories based on four types of numerical feature quantities, a numerical dataset, or the like, which is data that can be input to AI (artificial intelligence). As long as it can be converted into a form that is output in the classification result, the data may be any data.
[0098] Next, the processing of the input data by the information processing device 100 just before the output layer of deep learning is described. In deep learning, information processing is performed on the input data such as the above-mentioned images and graphs. At this time, the information processing device 100 performs processing based on full connection or nonlinear functions in the processing just before the output. The full connection processing is performed to unify the results obtained by extracting feature quantities from the input data through convolution operations, etc. into the desired number of categories. Usually, after the full connection processing, the result of the processing using an activation function such as a softmax function (soft maximum function) as a nonlinear function is output.
[0099] In addition, full connection processing is not necessarily required. Although the inference accuracy will mostly decrease, the information processing device can also gather the desired number of classifications in the feature extraction stage described below. For example, the information processing device can also compare the inference value obtained by the output of these fully connected processing results or the feature extraction with the correct answer label. In addition, usually, by implementing the processing using the softmax function, a clear difference is generated between the inference candidates, and the improvement of inference accuracy is expected. Therefore, the information processing device expects to implement the processing using the softmax function on the input data. In addition, the information processing device can also replace the softmax function and implement the processing using a nonlinear function such as log-softmax that is obtained by deforming the softmax function on the input data.
[0100] Next, an example of a process in which the information processing device 100 extracts feature quantities for various input data is shown. When the data input to the information processing device 100 is image data, CNN (convolutional neural network), MLP (Multi-Layer Perceptron), and Transformer are mostly used when extracting feature quantities as described above. In addition, the image can also be processed by GNN (Graph Neural Network) used in graph theory shown below, RNN (Relational Neural Network) used in time series processing, and technologies applying them.
[0101] In addition, although deep learning is described above, the information processing device 100 may also use logistic regression, support vector machine, gradient boosting method, etc., and various algorithms may be considered as these algorithms. In particular, various algorithms are known in deep learning, and the information processing device may also use algorithms such as Vgg, ResNet, AlexNet, MobileNet, and EfficientNet.
[0102] In addition, although the information processing device can process images only by pure full connection in MLP, methods such as MLP-Mixer using MLP are known, and such methods can also be used. In addition, methods of combining feature extraction with Vision Transformer or CNN are also known in Transformer, and the information processing device can also use these single methods or combined methods.
[0103] The information processing device 100 uses GNN (Graph Neural Network), GCN (Graph Convolutional Network) that performs convolution on nearby nodes, etc. as graph data. Since graph data cannot define coordinates like image data, graph data cannot be directly input into deep learning.
[0104] Therefore, when the data input to the information processing device 100 is graph data, the graph data is input by performing a conversion based on an adjacency matrix or a degree matrix as a reversible conversion. Here, the adjacency matrix expresses whether there is a connection between the nodes of the graph in the form of a matrix, and becomes an N×N matrix when there are N nodes. In addition, the adjacency matrix becomes a symmetric matrix when the graph is an undirected graph whose edges do not have directions, and becomes an asymmetric matrix in the case of a directed graph.
[0105] In addition, the degree matrix represents the number of edges contained in each node in the form of a matrix, and becomes an N×N matrix when there are N nodes, which becomes a diagonal matrix. The information processing device converts the input graph data into matrix data, inputs the matrix data into GNN, GCN, etc., learns through multiple hidden layers, and implements processing using full connection, softmax function, etc. before the output layer for output, but this method is the same as the deep learning in the above-mentioned image, so the description is omitted. Usually, in deep learning, when the input data is time waveform data, RNN is mostly used, and the GRU (Gated recurrent unit) and LSTM (Long short-term memory) of the extended RNN become the main technologies.
[0106] In addition, there are also known technologies that use Transformer or a combination of technologies of the Attention mechanism that is the basis of Transformer, TCN (Temporal convolutional network) that uses discrete one-dimensional convolution, etc. By using these technologies on input data, data can be input to deep learning. Regarding output, the information processing device 100 also extracts the feature value of the input data by the above method, and then performs processing using full connection, softmax function, etc. before the output layer to output data. This method is the same as the deep learning in the above image, so the description is omitted.
[0107] When the data input to the information processing device 100 is natural language data, there are known LSTMs for processing the above-mentioned time waveforms, a technology called Seq2Seq (sequence to sequence) which is a development of LSTM, an Attention mechanism which is a development of Seq2Seq, and a Transformer technology which is a further development of Seq2Seq. The information processing device 100 is able to classify natural language data by using these technologies.
[0108] In the past, LSTM was able to predict language based on the context of an article, but it could only process signals of fixed length, so the accuracy of reasoning varied depending on the length of the article. However, by using the concept of Encoder-Decoder in LSTM's Seq2Seq, the above problem was solved.
[0109] However, the reasoning accuracy of this method is not enough. Attention is a method that introduces probabilities between words that make up an article to improve the reasoning accuracy. However, Attention cannot be parallelized and cannot process large-scale data sets. Therefore, a method that can parallelize Attention using dedicated hardware such as GPU is Transformer. Although there are differences in reasoning accuracy or calculation time of Transformer, the underlying technology is common, so the information processing device 100 can also use any of them. Regarding the output, the information processing device 100 also extracts the feature value of the input data through the above-mentioned method, and then implements processing using full connection, softmax function, etc. before the output layer to output the data, but this method is the same as the deep learning in the above-mentioned image, so the description is omitted.
[0110] Next, the amount of data input to the information processing device 100 will be described.
[0111] The number of data such as images, graphs, time waveforms, and texts input to the information processing device 100 is preferably 100 or more, and more preferably 1000 or more, relative to each correct answer label. In addition, regarding the learning data set input to the information processing device 100, it is not preferred to have a small variance of similar data in one correct answer label, but it is preferred to have a distribution that can include the expected result during inference.
[0112] When the data input to the information processing device 100 is image data, "data augmentation" can be performed to increase the data for learning by affine transformation, etc. However, augmentation cannot be used for all data. For example, when the data input to the information processing device 100 is image, text, or time waveform data, it is usually difficult to perform the above-mentioned data padding.
[0113] When the amount of data used for learning is small, the information processing device 100 can improve the accuracy of inference by using a similar data set that obtains more data, or a data set of time waveforms obtained by a similar sensor for learning. In addition, the information processing device 100 can also use the variables and weight matrices obtained through learning as initial values, and perform transfer learning or fine-tuning through a small amount of data that has been obtained to perform learning. When learning is performed in this way, the amount of data input to the information processing device 100 can also be less than 100.
[0114] In addition, transfer learning is the learning of changing the elements of the variables and weight matrices that become the initial values so as to reduce the learning rate, and fine-tuning is a method of fixing the variables and weight matrices and only learning the full connection. Usually, transfer learning and fine-tuning are mostly used in combination, and the information processing device 100 can also be configured to initially try multiple fine-tunings to optimize the parameters during repeated calculations, and then try transfer learning. In addition, in such a case, it is not necessary to set all variables and weight matrices to initial values, and only some variables, weight matrices, and parameters can be shared.
[0115] The above describes the case where the information processing device 100 performs supervised learning, but the information processing device 100 can also perform semi-supervised learning. In the case where the information processing device 100 performs semi-supervised learning, there is the following disadvantage: compared with supervised learning, there are fewer data labeled with correct answer labels, and accordingly learning produces bias and reduces the accuracy of reasoning. Therefore, it is also possible that the information processing device 100 can perform learning even through a method of providing a correct answer after learning by unsupervised learning such as self-supervised learning called control learning. In this case, the learning data without correct answer labels is expected to be more than 1000 relative to each correct answer label, and the data labeled with correct answer labels is expected to be more than 100.
[0116] Next, the first data set including the above-mentioned image, graph, text, time series and other data and the method of using the information processing device 100 are described. In embodiment 1, the information processing device 100 processes the classification problem when N is an integer greater than 3. The upper limit of N is not particularly limited, but the larger the N, the larger the data set required for learning of the information processing device 100, and the greater the amount of calculation required for learning. Therefore, it is desirable that N is as small as possible. The data set is divided into learning data, verification data and test data according to each correct answer label, or is divided only into learning data and test data.
[0117] For example, MNIST (Modified National Institute of Standards and Technology database) contains 60,000 pieces of learning data and 10,000 pieces of testing data, but the information processing device 100 can also use all of them as learning data, for example, 50,000 pieces of data can be used as learning data and 10,000 pieces of data can be used as verification data.
[0118] In addition, the data used for learning is expected to contain the same degree of learning data, verification data, and test data relative to each of the N correct labels, and is expected to be randomly selected to avoid deviations due to the correct labels. In addition, when a part of the data is used as verification data, first, the information processing device 100 can also use the learning data for learning, and use the data not used for learning as verification data to confirm the accuracy of the reasoning based on the verification data. In this way, it is possible to prevent the learning performed by the information processing device 100 from becoming over-learning relative to the test data. However, when a part of the data is used as verification data, the data that can be used as test data is reduced, and therefore, the accuracy of the reasoning for the test data is likely to decrease, and it is expected to be used separately according to the size of the data set that can be prepared.
[0119] <Study in the 1st Study Section>
[0120] Next, a method will be described in which learning data is input to the information processing device 100 and an output classified into a desired number of categories is obtained through deep learning or gradient boosting. Fig.10It is a flowchart showing an example of a neural network in deep learning of multi-value classification and binary classification. In the neural network of implementation mode 1, input data is first input into the input layer (step ST11), and after repeatedly performing the extraction of feature quantities in the hidden layer (step ST12), processing based on the activation function (step ST13), extraction of feature quantities in the hidden layer (step ST14), and processing based on the activation function (step ST15), full connection is performed (step ST16), and processing based on the activation function is performed again (step ST17), and the result is output (step ST18).
[0121] In deep learning, various methods are known depending on the type of input data, but the information processing device 100 that performs deep learning is the same as other learning devices that perform normal learning other than deep learning in that feature quantities are extracted from each hidden layer, and a hidden layer immediately before or before the output is fully connected to output the target N-value classification. In addition, the information processing device 100 that performs deep learning is the same as other learning devices that perform normal learning in terms of using loss functions, optimization functions, and error back propagation.
[0122] In addition, the learning device that performs normal learning defines the learned model as outputting the label with the maximum value (accuracy) obtained by processing the input data using the softmax function as the inference result (classification result), while the first learning unit 11 is different in that the neural network is defined as being able to output the classification result based on inference for all labels. The information processing device 100 learns the data set classified by N values in this way, that is, updates the variables, weight matrix, parameters, etc., and stores the updated learning results in the storage unit 20 of the information processing device 100.
[0123] <Data used for the second learning section>
[0124] The use of the second learning data is a major feature of the information processing device 100 of implementation mode 1. The information processing device 100 uses a part of the input data as the first learning data to change the correct answer label of the first learning data through the learning data generation unit 14, thereby generating the second learning data. The first data set is labeled with N types of correct answer labels as described above. The following is an example of the case where N is 10, but as long as N is greater than 3, it can also be other integers. For example, when the information processing device 100 generates the second learning data, first, it selects 1 correct answer label (the second correct answer label) from the 10 types of correct answer labels.
[0125] Next, the information processing device 100 converts the input data other than the selected correct answer label into data marked with one label (third correct answer label). For example, when the information processing device 100 generates the second learning data, first, it selects 1 among the 10 types of integers from 0 to 9 whose correct answer label is 1, then groups the learning data corresponding to 0 and 2 to 9 other than 1, and assigns one correct answer label to the data corresponding to 0 and 2 to 9. For example, the information processing device 100 newly assigns a correct answer label such as 0 to the input data of 1, and newly assigns a correct answer label such as 1 to the data corresponding to 0 and 2 to 9.
[0126] Next, the details of the second data set generated by the information processing device 100 will be described. Fig.11 : is a diagram showing an example of a second data set generated by the information processing device 100. The second data set (second learning data) is a data set used for learning by the second learning unit 12, and is, for example, data generated as described above and classified into two types of correct answer labels 0 and 1.
[0127] The second data set is data classified as binary correct answer labels. When the number of input data classified as 0 is M0, the number of data classified as 1 is M1, etc., the number of data classified as i0 in the second data set as a whole is M. i0 , the number of data classified as other than this is expressed as formula (1). The second data set generated in this way becomes binary classified data with a deviation in number according to the correct answer label. The information processing device 100 performs the above processing from i0=0 to i0=9 respectively, and generates the second data set as a binary classified data set.
[0128]
[0129] In addition, in the first embodiment, the case where the second data set is a binary classification data set is described, but when the first data set is an N-value classification data set, the second data set can be an M-value classification data set with M≤N-1. However, when M is 3 or more, the number of data combinations increases compared to the case where M is 2, and the amount of calculation when the information processing device 100 performs learning and reasoning increases. Therefore, if there is no special reason, it is desirable to set M to 2. In addition, the second learning unit 12 can also use M-value classification and multi-value classification other than M-value classification in combination.
[0130] <Study in the Second Study Section>
[0131] Next, the learning method of the second learning unit 12 using the above-mentioned second learning data is described. As described above, the second learning unit 12 performs learning of M (≤N-1) value classification. Below, for simplicity, the case where the second learning unit 12 performs learning of binary classification is explained as an example. For example, the loss function (Hinge Loss) of binary classification is represented by formula (2). This loss function is a function that outputs 1-t×y when 1-t×y is less than 0 and is 0 when it is greater than 0. In addition, t is the output result of the second learning unit 12, and y is the correct answer label.
[0132] l(y)=max(0,1-t×y)…(2)
[0133] In the binary classification performed by the second learning unit 12, a sigmoid function or a log sigmoid function may be used as a nonlinear activation function immediately before the output layer. In addition, in the case where the second learning unit 12 performs M-value classification such that 3≤M, it is desirable to use a softmax function, as in the first learning unit 11. Cross entropy (information entropy) can also be used as a loss function in binary classification. When used, a binary value is output from an information processing device for binary classification, and a result is output by applying a softmax function and cross entropy to the binary value. The sum of the binary values before being input to the cross entropy becomes 1 due to the effect of the softmax function. That is, it becomes a value such as [0.63, 0.37]. On the other hand, when the hinge function or sigmoid function described above is used, a value of 1 is output from the information processing device for binary classification. Due to the effect of the hinge function, the result becomes a value of 1 between 0 and 1, and the inference value is changed depending on whether it is close to 0 or close to 1. In addition, regarding the results of using CIFAR10 and only changing the loss function in the same neural network (VGG13), the average binary classification of the test data set when the hinge function is used is 98.375%, while the average when the cross entropy is used is 98.694%, which is not much different. In addition, the second learning unit 12 can also perform deep learning, and can also learn by algorithms other than deep learning.
[0134] In addition, the information processing device 100 is not limited to the first learning unit 11 and the second learning unit 12 both performing deep learning. In the case where both the first learning unit 11 and the second learning unit 12 perform deep learning, the neural network used by the second learning unit 12 may also be a deep learning neural network that is smaller than the first learning unit 11. Here, a smaller neural network refers to a neural network with a relatively small number of hidden layers and adjustable parameters. For example, MobileNet (with approximately 3 million parameters) can be said to be a smaller neural network than ResNet18 (with approximately 12 million parameters).
[0135] For example, the information processing device 100 is configured such that, for the input of CIFAR10, the first learning unit 11 uses ResNet50 as a neural network for deep learning, and the second learning unit 12 uses ResNet18 as a neural network for deep learning. Thus, the information processing device 100 can shorten the computing time consumed for learning, and can reduce the size of the learned model stored in the hardware. In this way, the information processing device 100 utilizes the following characteristics: compared with the 10-value classification, even a smaller network can easily obtain higher inference accuracy for binary classification.
[0136] In addition, the second learning unit 12 may also be composed of multiple binary classification learning devices. In this case, the second learning unit 12 does not need to use the same machine learning algorithm in different binary classification learning devices, and different machine learning algorithms may be used when the inference accuracy is low. For example, in the above, an example is described in which the second learning unit 12 uses ResNet18 for learning, but when sufficient inference accuracy cannot be obtained, the second learning unit 12 may switch the algorithm used to ResNet32. When the inference accuracy of ResNet32 and ResNet18 is 100%, the algorithm used may be switched to ResNet18 as a smaller network. In addition, even if the multiple learning devices in the second learning unit 12 use different networks, it is expected that the second learning unit 12 uses the same softmax function for output immediately before the output layer or uses the same loss function for output, and evaluates different networks with the same indicators.
[0137] In addition, when the outputs of different learning devices cannot be evaluated with the same index, the second learning unit 12 can also use the difference or deviation between the first inference value and the second inference value in the binary classification, and correct with the maximum and minimum values, etc., to define the evaluation index and correction coefficient corresponding to the function used. In this way, the second learning unit 12 learns the binary classification problem and saves the learning results in the storage unit 20 such as the ROM, RAM, hard disk or external storage medium of the information processing device. In addition, the second learning unit 12 performs multiple operations that are lightweight and similar to each other compared to the first learning unit 11. Therefore, it is not necessary to perform learning through a large computer as in the previous machine learning, and learning can also be performed in a distributed manner through multiple small computers.
[0138] <Reasoning in the first learning section>
[0139] For example, when the first learning unit 11 performs inference, it forwardly calculates the variables, weight matrix, and parameters obtained through learning for the matrix as input data. The result of the calculation performed by the first learning unit 11 becomes the output of the softmax function used in the learning of the first learning unit 11, and the output of the softmax function refers to the accuracy of each classification for the N-value classification, that is, the degree of certainty. The information processing device 100 of embodiment 1 sets the candidate with the highest accuracy among the N candidates as the classification result (inference result) of the first learning unit 11.
[0140] In addition, as long as the information processing device 100 can calculate the degree of certainty of each classification for the N-value classification, it can also use algorithms other than deep learning for learning. In the following description, the candidate with the highest accuracy among the inference candidates is set as the first inference candidate, and the candidate with the second highest accuracy is set as the second inference candidate. At this time, the information processing device 100 is characterized in that when the value (accuracy) of the first inference candidate is smaller than a threshold value (first threshold value) defined separately, or when the value of the second inference candidate is larger than the threshold value (second threshold value), the classification result obtained using the second learning unit 12 is output. In addition, the first threshold and the second threshold can be the same value, or different values such that the second threshold value < the first threshold value.
[0141] In any case where the accuracy of the first inference candidate is smaller than the threshold value and the second inference candidate is larger than the threshold value, when the first inference candidate based on the first learning unit 11 is set as the classification result of the information processing device 100, it is easy to become a result different from the classification result requested by the user. In this way, the information processing device 100 pre-sets a threshold for determining the accuracy of inference, and when it is determined that the accuracy of the inference performed by the first learning unit 11 is low, the second learning unit 12 performs inference, thereby improving the accuracy of inference.
[0142] <Reasoning in the second learning section>
[0143] When the accuracy of the first inference result is lower than the threshold, the information processing device 100 performs inference through the second learning unit 12. For example, when the data input to the information processing device 100 is image data, in the following description, the input data that results in the accuracy of the first inference result being lower than the threshold is referred to as the first input image data.
[0144] The second learning unit 12 processes the first input image data. First, when the first input image data is input to the information processing device 100, the second learning unit 12 calls the learned models in sequence. For example, all learned models learned through combinations of binary classification of 0 and (1 to 9), binary classification of 1 and (0, 2 to 9), and binary classification of 2 and (0 to 1, 3 to 9) are called. The information processing device 100 uses all the learned models to infer the first input image data through the second learning unit 12, and outputs the result of the inference when the accuracy is classified as the correct label in each learned model, that is, when it is classified as 0 in the binary classification of 0 and (1 to 9), and the output content is stored in the storage unit 20.
[0145] When the information processing device 100 performs inference based on the second learning unit 12 and has two or more inference results classified as correct labels, the inference result with the highest accuracy, that is, the inference result with the largest value calculated when the softmax function is used, is output as the inference result of the second learning unit 12, and is stored in the storage unit 20. In addition, when the information processing device 100 performs inference based on the second learning unit 12 and has no inference result classified as a correct label, it outputs the label corresponding to the first inference result in the first learning unit 11. In addition, this processing is a processing of calling the binary classification model one by one for the first input image, so it consumes processing time. Therefore, the information processing device 100 can also use a parallel computing device such as a GPU to process the input data that needs to be inferred by the second learning unit 12 because the accuracy is below the threshold according to a partial set or batch of results.
[0146] <Threshold value of the first learning section>
[0147] Next, the above-mentioned threshold value is explained. The values of the first inference candidate and the second inference candidate are calculated for a plurality of inference results, and the results are statistically processed, thereby setting the above-mentioned threshold value based on, for example, a data set, an algorithm used in the first learning unit 11, a loss function, etc. For example, by using the average value of the first inference candidate for the threshold value, a simple and high inference accuracy can be obtained.
[0148] Specifically, when the information processing device 100 performs inference through the first learning unit 11 after the first learning unit 11 performs learning using the learning data, the accuracy of the first inference candidate is stored in the storage unit 20. In addition, the information processing device 100 calculates the average value of the accuracy of the past first inference candidates through the accuracy determination unit 16 based on the accuracy of the past first inference candidates stored in the storage unit 20, and stores the calculation result as a threshold in the storage unit 20. In addition, the information processing device 100 may update the threshold stored in the storage unit 20 to a new threshold each time the first learning unit 11 performs inference, and may calculate the threshold based on the result of inference based on the first learning unit 11 using a plurality of verification data or a plurality of test data.
[0149] In addition, for example, the information processing device 100 first infers a plurality of input data through the first learning unit 11 and outputs an inference result (classification result). Based on the inference result output by the information processing device 100, the user determines whether a plurality of first inference candidates are consistent with the correct answer label, and inputs each determination result to the information processing device 100. Based on the determination result input by the user, the information processing device 100 calculates the average value of the accuracy when the first inference candidate is consistent with the correct answer label through the accuracy determination unit 16, and stores the calculation result as a threshold in the storage unit 20. In this way, the information processing device 100 can obtain a simple and high inference accuracy by using the average value of the accuracy of the first inference candidate.
[0150] In addition, the threshold value may be, for example, a median, 25th percentile, 75th percentile, or other percentile, or a statistical value obtained by performing an exponential or logarithmic operation on them. By using these values other than the average value as the threshold value according to the deviation of the data in the data set, the inference accuracy can be further improved. In addition, for example, the threshold value is set to a value between a statistical value including the average value of the accuracy of the first inference candidate when the inference result of the first learning unit 11 is equal to the correct answer label and a statistical value including the average value of the accuracy of the first inference candidate when the inference result of the first learning unit 11 is different from the correct answer label.
[0151] Specifically, first, the information processing device 100 performs inference on a plurality of input data through the first learning unit 11, and outputs an inference result (classification result). Based on the inference result output by the information processing device 100, the user determines whether a plurality of first inference candidates are consistent with the correct answer label, and inputs each determination result to the information processing device 100. Based on the determination result input by the user, the information processing device 100 calculates the average value of the accuracy when the first inference candidate is consistent with the correct answer label and the average value of the accuracy when the first inference candidate is inconsistent with the correct answer label through the accuracy determination unit 16, sets a predetermined value between the average value of the accuracy when it is inconsistent with the correct answer label and the average value of the accuracy when it is inconsistent with the correct answer label through the accuracy determination unit 16, and stores the value as a threshold value in the storage unit 20.
[0152] More specifically, the information processing device 100 calculates the average value of the accuracy when the label is inconsistent with the correct answer and the median value (average value) of the average value of the accuracy when the label is inconsistent with the correct answer through the accuracy determination unit 16, and stores the calculation result as a threshold in the storage unit 20.
[0153] In addition, for example, the information processing device 100 first performs inference on multiple verification data through the first learning unit 11, and based on the inference result, the accuracy judgment unit 16 respectively determines whether multiple first inference candidates are consistent with the correct answer label, and the accuracy judgment unit 16 calculates the average value of the accuracy when the first inference candidate is consistent with the correct answer label and the average value of the accuracy when the first inference candidate is inconsistent with the correct answer label, and the accuracy judgment unit 16 sets a specified value between the average value of the accuracy when it is inconsistent with the correct answer label and the average value of the accuracy when it is inconsistent with the correct answer label, and the value is stored as a threshold by the storage unit 20.
[0154] More specifically, the information processing device 100 calculates the average value of the accuracy when the label is inconsistent with the correct answer and the median value (average value) of the average value of the accuracy when the label is inconsistent with the correct answer through the accuracy determination unit 16, and stores the calculation result as a threshold in the storage unit 20.
[0155] In addition, for example, the threshold value may be set in such a way that the inference accuracy is maximized by parameter scanning that continuously changes the threshold value. In addition, for example, the threshold value may be calculated using a parallel computing device such as a GPU. When there is a spatial or temporal deviation in the input data, a difference is likely to occur between the threshold value set statistically and the threshold value set by parameter scanning. By calculating the optimal value of the threshold value by parameter scanning for the data set, the inference accuracy can be improved.
[0156] In addition, a method of changing the threshold value for each inference candidate is also effective. In the above example, a fixed threshold value is set regardless of the value of the first inference candidate. In contrast, in the case of 10-value classification, when the first inference candidate is 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9, a threshold value is calculated for each inference candidate based on statistical information. However, in the case where there is little data that is incorrectly classified due to high inference accuracy or a small amount of inference data, specifically, when there are less than 100 data, the value as statistical information becomes small. Therefore, a method of changing the threshold value for each inference candidate is not desirable. In this case, a method of using a fixed threshold value regardless of the value of the first inference candidate is desirable.
[0157] In addition, when the second inference candidate is used as a threshold, a statistical method such as an average value or a median value may also be used. However, if the inference time or the computing resources provided for the inference allow, a method of determining the second inference candidate by parameter scanning is also an effective means. In addition, in an environment where a parallel computing device such as a GPU cannot be used, in order to reduce the computing time, it is not necessary to use the second learning unit 12 to infer all the first input data that is below the threshold, and it is also desirable to use the second learning unit 12 only when the first learning unit 11 pre-classifies it as a correct answer label that is prone to error.
[0158] <Experimental Results>
[0159] Next, refer to Figures 12 to 14 , the experimental results of classification performed in the information processing device 100 are described. Fig.12 10,000 test data of CIFAR10, and the number of binary classification operations performed by the information processing device 100 relative to the threshold. In this experiment, CIFAR10 was used as a data set input to the information processing device 100. CIFAR10 is a data set that includes 50,000 learning image data and 10,000 test image data and is classified into 10 values: airplane, car, bird, cat, deer, dog, frog, horse, ship, and truck. In this experiment, no verification data was prepared, and 50,000 learning data were input to the information processing device 100, and the first learning unit 11 was learned by ResNet50, which is a method of CNN.
[0160] ResNet50 consists of 48 convolutional layers, 1 maximum pooling layer, and 1 average pooling layer. Poisson regression (Poisson negative log likelihood loss) is used as the loss function, but any function such as cross entropy, minimum square error (MSE), mean absolute error (MAE), or defining and using your own error function can also be used. In addition, Adam with a learning rate of 0.01 is used as the optimization function, but momentum, RMSprop, SGD (Stochastic gradient descent), or defining and using your own error function can also be used. In addition, for the scheduler that changes the learning rate, the Step LR function is used, but multiple schedulers such as the Cosine Annealing LR function and the Cyclic LR function are known, and any function can be used as the loss function or optimization function if the inference accuracy for the test data can be ensured. Xavier's initial value is used for the initial value of the convolution weight matrix, that is, the filter.
[0161] When the learning batch size is set to 64, the test batch size is set to 1000, and the round (epoch) is set to 20 times, the inference accuracy of the first learning unit 11 is confirmed to be 86.28% for the test data set. In this definition, the inference value takes a real number between 0 and 1, so Fig.12 Calculation results of the number of first inference candidates taking numbers between 0.30 and 0.99 are shown in . For example, when the number is 0.9, it means that 2617 of the 10000 test data are inferred by binary classification.
[0162] Next, binary classification is described. For binary classification data sets, 10 data sets are created based on the first data set, such that they are airplanes and others, cars and others, birds and others, cats and others, deer and others, dogs and others, frogs and others, horses and others, ships and others, and trucks and others. For example, in the case of airplanes and others, the correct answer label of airplanes is defined as 0, and the correct answer label of others is defined as 1. In this way, the data set of airplanes is 5,000, and the data set of others is 45,000.
[0163] The second learning unit 12 uses ResNet18, which is a method of CNN. Although Hinge Loss is used for the loss function, any function may be used, such as defining and using your own error function. In addition, although Adam with a learning rate of 0.01 is used as the optimization function, any function may be used, such as defining and using your own error function. In addition, although the Cosine AnnealingWarm Restarts function is used for the scheduler that changes the learning rate, any function may be used as long as the inference accuracy for the test data can be ensured, as with the loss function or the optimization function. For the initial value of the convolution weight matrix, i.e., the filter, the initial value of Xavier is used in the same way as in the first learning unit 11. When learning was performed with the batch size for learning set to 250, the batch size for testing set to 1000, and the number of rounds set to 10, in the binary classification of the test data set, the inference results were: airplane: 97.01%, car: 98.90%, bird: 96.02%, cat: 94.85%, deer: 96.96%, dog: 96.31%, frog: 98.36%, horse: 98.35%, ship: 98.71%, and truck: 98.30%.
[0164] Next, the inference results using the first learning unit 11 and the second learning unit 12 will be described. Fig.13 1 is a graph showing experimental data of the inference results of the information processing device when binary classification is used for CIFAR10 and when binary classification is not used. Figure 5 The method of explanation is the same. Here, the results of an experiment performed without notifying the second learning unit 12 of the inference candidates of the first learning unit 11 are shown. The comparison standard is the inference accuracy of 86.28% when only the first learning unit 11 is used. Fig.13 The figure shows the inference results using the first learning unit 11 and the second learning unit 12 when the threshold for the first inference candidate is changed from 0.3 to 0.99. As shown in the figure, as the threshold increases, the amount of data to be binary classified increases, and the inference accuracy improves, reaching a maximum value of 88.70% when the threshold is 0.85.
[0165] On the other hand, it can be seen that when the threshold exceeds 0.86, the inference accuracy decreases. This result means that the inference accuracy is improved by more than 2% compared to the 86.28% inference accuracy that serves as the benchmark, showing the effect of combining multi-value classification with binary classification. It should be noted that by using the second learning unit 12, the following results are obtained, which exceed the results obtained by inference only by the first learning unit 11 at all thresholds of 0.3 to 0.99, and it can be seen that at least under the above conditions, regardless of the threshold, the inference accuracy can be improved by using the second inference candidate.
[0166] Fig.14 The inference time for the threshold is shown. Fig.14 This is a graph of experimental data showing the time taken by the information processing device 100 to infer 10,000 data with a threshold of CIFAR10. In the inference, no parallelization such as GPU was used, but calculations were performed sequentially by the CPU. Observing the results, it can be seen that the inference was completed in 6 seconds when binary classification was not used, but it was 570 seconds when the threshold was 0.86, requiring about 100 times the inference calculation time. Most of this calculation time is the time spent on calling the learned model from ROM. Therefore, if parallelization is not possible, it is desirable to call the learned binary classification model to RAM in advance. In addition, Fig.14 The results obtained by pre-saving data below the threshold and processing it with the GPU are also shown. It can be seen that at the most time-consuming threshold of 0.99, it took 1119 seconds when using the CPU, while it took 16.6 seconds when using the GPU, which is a reduction of 98.5%. In addition, this result is not much different from the 3 seconds when no threshold is used.
[0167] At present, the memory of many AI-specific hardware has become larger, and it is not difficult to place the learned model on the GPU memory. In particular, regarding the size of the learned model this time, the 10-value classification is 103MB, and the binary classification is 47MB×10, which is small enough even considering the memory of GPUs in recent years. In addition, in order to solve the N-value classification problem, N parallel ASICs can also be prepared to perform binary classification inference in parallel through each operation unit. In addition, ResNet50 or ResNet18, for example, compared with EfficientNet or MobileNet, even with the same reasoning accuracy, the file size, that is, the number of parameters of the weight matrix, is larger. Therefore, when the file size becomes a problem, the problem can be solved simply by changing the model.
[0168] In this way, the information processing device 100 of embodiment 1 selects to output the classification result of the first classification unit 11C when the accuracy of the inference performed by the first classification unit 11C exceeds a preset threshold value, and outputs the classification result of the second classification unit 12C classified into a smaller number of classes than the first classification unit 11C when the accuracy of the inference performed by the first classification unit 11C is below the threshold value. Therefore, regardless of the amount of input data when generating the learned model, the accuracy of the inference of the input data can be improved.
[0169] In addition, even without using a large-scale machine learning device, a higher inference accuracy can be obtained. Therefore, the amount of calculation required to obtain the same inference accuracy as before can be reduced, so computing resources can be reduced, learning time can be shortened, and low cost can be achieved. In addition, the amount of data required to obtain the same inference accuracy as before can be reduced. Therefore, not only can the machine learning device be learned with a low-cost and simple device structure, but the difficulty of using machine learning can also be reduced. In particular, significant differences occur in neural networks that require a lot of data. In addition, in the past, a large-scale N-value classification machine learning device needed to be learned by a large-scale computer, but the N-value classification learning device can also be miniaturized. Instead, the learning of multiple M-value classification devices is distributed to different small computers, such as computers that are not equipped with dedicated hardware such as GPUs for learning. Therefore, the use of machine learning devices becomes easy.
[0170] Implementation method 2.
[0171] <Reasoning in the second learning section>
[0172] The feature of the second embodiment is that, when the result of the inference of the first learning unit 11 is that the accuracy becomes below the threshold, the first learning unit 11 performs inference and hands over the first inference candidate with the highest accuracy to the second learning unit 12. In addition, the second learning unit 12 is a device that has been learned using the data set composed of each binary combination described in the first embodiment, and initially uses the first inference candidate and the learned model learned using other data to perform judgment. The judgment result is that, when a result different from the first inference candidate is obtained, the inference result with the highest accuracy is set as the inference result of the second learning unit 12 by performing inference through all the combinations.
[0173] When CIFAR10 shown in the first embodiment is taken as an example, for example, when the first inference candidate is an airplane, in the second learning unit 12, inference is performed by binary classification learned using the airplane and other data sets. When the inference result is an airplane, that is, when the accuracy (first accuracy) of the class of the first inference candidate calculated by the second accuracy calculation unit 12B is higher than the accuracy (second accuracy) of other classes, the second learning unit 12 outputs the airplane, that is, the class of the first inference candidate. When the inference result becomes other than this, inference is performed by all learning devices of airplane and other than this, car and other than this, bird and other than this, cat and other than this, deer and other than this, dog and other than this, frog and other than this, horse and other than this, ship and other than this, truck and other than this, and compared with the inference candidates that do not become other than this, the inference result is determined based on the comparison result. For example, the result with the smallest value is set as the inference result, or the result with the largest value is set as the inference result according to the difference in the output function.
[0174] For example, when the airplane and others are 1.0 and 1.5 respectively, and the ship and others are 0.8 and 2.6 respectively, 1.0 and 0.8, which are smaller values, are used for comparison. Since 0.8 is smaller, the ship is set as the inference result. In addition to the minimum value, the result with the larger difference can also be compared. That is, in the above example, (1.5-1.0=0.5) and (2.6-0.8=1.8) are compared, and the difference 0.5 and 1.8 are compared, and the ship with the larger difference is set as the inference result. Although the binary classification is described, the same is true for the three-value classification and above. In the case of the three-value classification and above, the difference between the upper two inference results can be used. However, when the result of the above calculation is that the inference results of all the binary classifications are classified as others, the first inference candidate is output as the inference result of the second learning unit 12. By using this method, the time consumed for inference can be reduced without reducing the inference accuracy.
[0175] Implementation method 3.
[0176] <Data used for the second learning section>
[0177] In the third embodiment, the data set used for the second learning unit 12 is described. In the first and second embodiments, the data set used for the second learning unit 12 is N in the case of N-value classification. On the other hand, regarding the data set in the present embodiment, in the case of N-value classification, when the natural number less than N is set to L (the third number), any L (the third number) correct answer labels (the first correct answer labels) are selected, and the second data set is constructed by the input data annotated with the L correct answer labels. Fig.15 The following is an example of the structure of a portion of a data set. Fig.15In this way, L correct labels are selected in the N value classification, and a data set for the L value classification is produced. Therefore, the following A data sets are produced. In the following, for the sake of easy understanding, the case where N is 10 and L is 2 is explained, but it can also be an integer other than this.
[0178] A=(N,L)
[0179] The feature is that when N is 10 and L is 2, the 10 values are classified into a combination of binary values. For simplicity, in the case of a three-value classification of 0 to 2, different correct labels are combined such as 0 and 1, 0 and 2, and 1 and 2 to form a second data set. When the combination is performed in this way, A becomes the following A1, that is, a data set of 45 combinations is produced. The data sets thus classified into binary values are respectively input to the second learning unit 12 for learning. The second learning unit 12 is the same as the first embodiment.
[0180] A1=(10,2)
[0181] The second learning unit 12 for learning needs to have 45 of the same data sets as the data set, among which there is a case where the inference accuracy for the test data set that is not used for learning data deteriorates. In this case, it is also possible to change to an algorithm that improves the accuracy. In addition, the accuracy for the test data set sometimes becomes 100%. In this case, the calculation time and amount of calculation can be reduced by changing to a simpler algorithm, which is also the same as Implementation Method 1. Therefore, in addition to the difference between the second learning unit 12 and the first learning unit 11, in the second learning unit 12, different algorithms can be used for each data set, but as shown in Implementation Method 1, the loss function and the activation function immediately before the output layer are expected to use the same function.
[0182] exist Fig.16 , the results of learning binary classification using CIFAR10 by the method based on the present embodiment, and reasoning using the test data set for each binary classification are shown. 0 represents an airplane, 1 represents a car, 2 represents a bird, 3 represents a cat, 4 represents a deer, 5 represents a dog, 6 represents a frog, 7 represents a horse, 8 represents a ship, and 9 represents a truck. It can be seen that although the results of the reasoning accuracy are roughly above 90%, the classification of cats and dogs at 3 and 5 is 84.5%, and the accuracy becomes lower. In such a problem, it is expected to use a larger-scale network to improve the reasoning accuracy, or use data enhancement to improve the reasoning accuracy in the case of images.
[0183] In this way, the learned parameters of the second learning unit 12 that has been learned are saved in advance, and when the degree of certainty of the output result of the first learning unit 11 becomes below the threshold, the second learning unit 12 performs inference. However, in order to measure the reduction in the amount of calculation, as in the first embodiment, it is not necessary to use the second learning unit 12 for all data that are below the threshold, and it is also possible to use binary classification to reduce the calculation time only when the first inference result becomes a combination that is prone to error or when the first inference result is a classification value that is prone to error. For example, in the CIFAR10 data set, there are combinations that are prone to error, such as cats and dogs, ships and airplanes, so the second learning unit 12 can also be used only when cats, dogs, ships, and airplanes become the first inference candidates. For the ease of error, it is expected to perform an inference once and quantify the combination of erroneous data for evaluation.
[0184] In the above description, the second learning unit 12 is described as a binary classification, but it can also be a three-value classification or more. This is because the less the number of classifications, the higher the inference accuracy. However, when the number of classifications such as three-value classification becomes more than 2, the number of combinations increases, and when the 10-value classification is divided into three-value classification, 120 second learning units 12 are required. Therefore, as described above, it is necessary to reduce the amount of calculation consumed by inference by using only when the first learning unit 11 is used for the label that is prone to error.
[0185] Implementation method 4.
[0186] <Reasoning in the second learning section>
[0187] The feature of the fourth embodiment is that, when the inference result of the first learning unit 11 becomes below the threshold, the first inference candidate and the second inference candidate, which are the top two inference candidates with high accuracy obtained by the inference of the first learning unit 11, are handed over to the second learning unit 12. At this time, the second learning unit 12 uses the learned model of the N binary classifications described in the first embodiment or the learned model of the A1 binary classifications described in the second embodiment to perform inference.
[0188] In the case of using a learned model of N binary classifications, for example, when the first inference candidate is 5 and the second inference candidate is 6, inference is performed using a learned model learned using a second data set consisting of 5 and other results. If 5 becomes the inference result, 5 is output. In the case of other results, inference is performed using a learned model learned using a second data set consisting of 6 and other results. When the accuracy of classification as 6 (third accuracy) is higher than the accuracy of classification as other than 6 (fourth accuracy), 6 is output. In addition, in the case of using the above-mentioned N binary classifications, when there is enough computing resources, inference is performed using the learned models of both 5 and 6, the certainty of the two inference results is compared, and a more accurate result, such as 5, is output.
[0189] When using the learned model of the above-mentioned A1 binary classifications, for example, when the first inference candidate is 5 and the second inference candidate is 6, the learned model learned by the second data set consisting of 5 and 6 is used for inference. When this inference is performed, either 5 or 6 becomes a result with high accuracy, so, for example, 5 is output as the inference result. In this embodiment, it is described that the top two inference candidates of the first learning unit 11 are output, but the top P can also be handed over to the second learning unit 12. As described above, when using the learned model of N binary classifications, the more accurate inference result among the top P inference results is output.
[0190] In particular, when N binary classifications are used, if the inference values rearranged according to the degree of certainty in the order of the inference candidates of the first learning unit 11, such as the third inference candidate and the fourth inference candidate, are obtained, the inferences are sequentially performed in such a manner that the third inference candidate is inferred when the second inference candidate is other than the above, and the fourth inference candidate is inferred when the third inference candidate is other than the above, and if the second inference candidate is not other than the above, the inference values can be used as inference results in the second learning unit 12. However, if all the second inference results are other than the above, the first inference candidate is output as the inference value.
[0191] Implementation method 5.
[0192] <Threshold value of the first learning section>
[0193] In implementation mode 5, a method for determining a threshold value is described. It is characterized in that a threshold value is obtained by statistically processing the result of the N value output in the reasoning of the first learning unit 11. For example, when the number of test data sets for reasoning is set to 10,000, and the number of correct answers in the reasoning of the first learning unit 11 is set to 9,000, a matrix of 9,000×N is formed when only correct answers are collected, and it is called a correct answer matrix. In addition, a matrix of 1,000×N is formed when only incorrect answers are collected, and it is set to an error matrix. Then, by rearranging each matrix in such a way that, for example, the smaller the column, the higher the degree of certainty, a correct answer matrix of 9,000×N with 1 column as the maximum value and N columns as the minimum value and an error matrix of 1,000×N can be formed.
[0194] That is, a matrix is created by arranging the output of the softmax function in descending order for each data set. This time, for simplicity, 1 column is used as the first inference candidate for explanation. According to the definition of the loss function, the first inference candidate of the minimum value can be N columns, or it can be arranged so that the minimum value appears in 1 column and the maximum value appears in N columns.
[0195] The correct matrix and the error matrix are statistically processed. The average value and percentile are considered in the statistical processing. In particular, the 50th percentile is the central value. First, the average value is used as an example. When comparing the values of the first column of the correct matrix and the error matrix, the value of the first column of the correct matrix is greater than the value of the first column of the error matrix. Fig.16 2 shows the average value of the inference results in the first learning unit 11 that achieves an inference accuracy of 86.28% in CIFAR10 described in Embodiment 1. The solid line in the figure is the average value of the correct answer matrix, and the dotted line is the average value of the error matrix.
[0196] In this inference value, it is desirable to set the value between the average value of the correct solution matrix in the first column and the error matrix in the first column as the threshold. Fig.16 The value of the first column of the correct solution matrix is 0.93, and the value of the first column of the error matrix is 0.70. Therefore, it is desirable to set the threshold between 0.70 and 0.93. In particular, when the threshold is increased, the number of binary classifications increases, and the amount of calculation consumed by reasoning increases, but setting a larger value can improve the reasoning accuracy. Therefore, the threshold can be determined based on the computing resources consumed by the operation, the computing time, and the required computing accuracy. Fig.16 The calculation accuracy of the threshold is the same as that of Fig.12 The thresholds shown are calculated with the same precision, Fig.12 The maximum value in is when the threshold is set to 0.85, and therefore is included in the above-mentioned range of 0.70 to 0.93.
[0197] The same is true when using the median, 25th percentile, and 75th percentile. Fig.17 The result of calculating the median value for the above-mentioned correct matrix and error matrix is shown. In the median value, it is also expected that the value between the median value of the correct matrix in the first column and the median value of the error matrix in the first column is set as the threshold value, similar to the above-mentioned average value. That is, it is expected to be set between 0.56 and 0.96. In this case, it can also be seen that when the threshold value is 0.85, Fig.12 The maximum value in holds. The case of the central value is the same as the case of the average value. The expected threshold is large, but the threshold can also be determined to match the computing resources, computing time, and required computing accuracy. In addition, this is the result of learning CIFAR10 through ResNet50, so it is the above result, but in the case of data other than images, or even if the image has features extracted by other algorithms, although the value is different according to the definition of the loss function, the method for determining the threshold is expected to follow the above method.
[0198] In addition, statistical values such as their average values or central values can also be used in combination. For example, the following method is also a desired method of use: when the average value of the first column of the correct solution matrix is 0.8, the average value of the first column of the error matrix is 0.6, the central value of the first column of the correct solution matrix is 0.9, and the central value of the first column of the error matrix is 0.5, the upper limit of the threshold is set to the average value of the first column of the correct solution matrix, that is, 0.8, the lower limit of the threshold is set to the central value of the first column of the error matrix, that is, 0.5, and the range of the threshold is set to between 0.5 and 0.8.
[0199] Implementation method 6.
[0200] <Threshold value of the first learning section>
[0201] In Embodiment 5, the correct matrix and the error matrix are described. In Embodiment 6, the following method is described: a threshold is derived from the statistical information of the second column that is the second largest value for the same correct matrix and error matrix. As in Embodiment 5, calculation is performed based on the average value and the median value of the second column. For example, the result obtained by inference using CIFAR10 as a data set in the average is as follows: Fig.16 As shown in , the threshold value of the second column is 0.047 in the correct matrix and 0.207 in the error matrix. Therefore, it is desirable to select a threshold value between 0.047 and 0.21. Similarly, when the median value is used as the reference of the threshold value, Fig.17 As shown, the threshold value of the second column is 0.00025 in the correct matrix and 0.0953 in the error matrix. Therefore, it is desirable to set the threshold value between 0.00025 and 0.0953.
[0202] and Fig.12 Similarly, when the inference accuracy of the test data set is calculated with intervals of 0.01 for thresholds from 0.01 to 0.30, the case of 0.10 is the maximum, with an accuracy of 88.66%. Fig.12 The same level of inference accuracy as the maximum value of 88.70% shown in the figure shows that the same level of inference accuracy can be achieved even without using the first inference candidate as the threshold. In addition, the threshold value based on the above average value is 0.047 to 0.21. Fig.12 It is known that when the value is above 0.15, the inference accuracy decreases, so if the threshold is defined within the range of the average value, the maximum effect can be obtained. In addition, the median value is 0.00025 to 0.0953, which is close to 0.1, which is the maximum value of the inference accuracy.
[0203] Although the case where the first inference candidate is used in the fifth embodiment and the case where the second inference candidate is used in the sixth embodiment is shown, the difference between the first inference candidate and the second inference candidate may also be used. That is, when the average value of the difference between the first inference candidate and the second inference candidate in the correct answer matrix is called the correct answer average value and the average value of the difference between the first inference candidate and the second inference candidate in the error matrix is called the error average value, the correct answer average value is always greater than the error average value. Therefore, the threshold value can also be defined by setting the threshold value to be greater than the error average value and less than the correct answer average value.
[0204] Alternatively, the average value and the central value of the first inference candidate and the average value and the central value of the second inference candidate may be combined, and a value between the average value of the first inference candidate and the average value of the second inference candidate and between the central value of the first inference candidate and the central value of the second inference candidate may be set as a threshold. Here, the average value and the central value are used for explanation, but a value extracted by other statistical methods may also be set as a threshold.
[0205] Implementation method 7.
[0206] <Threshold value of the first learning section>
[0207] The correct answer matrix and the error matrix shown in Embodiment 5 and Embodiment 6 are matrices produced based on the results of the inference by the first learning unit 11 for all the test data. However, when the test data is large or the computing resources are small, the computing time and amount of computing consumed by the inference become large. In addition, when a device capable of parallel processing such as a GPU is used, the test data is usually not put into the first learning unit 11 one by one during the inference, but is input as a batch of aggregated sets. The size of the batch depends on the amount of memory that the GPU has.
[0208] In the seventh embodiment, instead of performing statistical processing after the inference of all the test data is completed, a part of the test data or a matrix after one batch processing is completed is used to calculate the correct answer matrix and the error matrix. For example, when the test data is 10,000, when 1,000 data are collected as part of the data, or when 1,000 data are collected in a batch and put into a device capable of parallel processing, one batch is calculated, and the correct answer matrix and the error matrix are prepared based on the results.
[0209] At this time, by retaining the accuracy data for each classification value used for inference in the memory (RAM), there is no need to use N-value classification for inference multiple times, and the binary classification device shown in embodiments 1 to 4 can also be used to infer the results that do not meet the threshold in the data on the memory.
[0210] In the above-mentioned processing, the correct answer matrix and the error matrix are calculated each time a set or a batch of processing is completed. This method is effective when there is a deviation in the correct answer labels of the test data, for example, in the case of a set or batch with more photos of airplanes in the example of CIFAR10. On the other hand, when the test data is configured randomly enough, the following method can be used. That is, the threshold derived from the correct answer matrix and the error matrix calculated from the processing of one set or one or more batches is also applied to the remaining test data. This is true when the above-mentioned set or one or more batches become a partial set close to the test data as a whole, thereby reducing the amount of calculation consumed by reasoning and shortening the reasoning time.
[0211] In addition, the present disclosure allows free combination of the various embodiments, modification of arbitrary components of the various embodiments, or omission of arbitrary components in the various embodiments.
[0212] Industrial Applicability
[0213] The information processing device of the present disclosure can be used to classify input data.
[0214] Description of Reference Numerals
[0215] 11A first model generating unit, 11B first accuracy calculating unit, 11C first classification unit, 12A second model generating unit, 12B second accuracy calculating unit, 12C second classification unit, 13A first feature value extracting unit, 13B second feature value extracting unit, 14 learning data generating unit, 15 threshold setting unit, 17 classification result selecting unit, 100 information processing device.
Claims
1. An information processing device, characterized in that: The information processing device comprises: a first feature quantity extraction unit for extracting a feature quantity of input data; a first accuracy calculation unit that performs inference on the input data based on the feature quantity extracted by the first feature quantity extraction unit, and calculates the accuracy of classifying the input data for each of the first number of classes; as well as a first classification unit that classifies the input data into at least one of the first number of classes based on the accuracy calculated by the first accuracy calculation unit, The first classification section is to: A first step of rearranging the input data in such a manner that the accuracy calculated by the first accuracy calculation unit is in ascending or descending order; In the second step, the label with the maximum accuracy in the rearranged input data is extracted; The third step is to compare the label that becomes the maximum value with the correct answer label associated with the input data; A first collection process for collecting the classes obtained in the first process that are consistent with the comparison results of the third process; A second collection process for collecting the classes obtained in the first process that are inconsistent with the comparison results of the third process; A first statistical process is to perform statistical processing on the classes collected by the first collection process; as well as The second statistical process performs statistical processing on the classes stored in the second storage process.
2. The information processing device according to claim 1, characterized in that The first statistical process and the second statistical process are processes of calculating any one of the average value, the median value, the standard deviation, and the information entropy, or combining two or more of the average value, the median value, the standard deviation, and the information entropy to perform calculations.
3. The information processing device according to claim 1 or 2, characterized in that: The information processing device includes a threshold setting unit that sets a threshold to be equal to or less than a first statistical value calculated by the first statistical process. The first classifying unit classifies the input data based on a comparison result between the accuracy calculated by the first accuracy calculating unit and the threshold value.
4. The information processing device according to claim 3, characterized in that: The threshold setting unit sets the threshold to be equal to or greater than a second statistical value calculated by the second statistical process.
5. The information processing device according to claim 4, characterized in that The threshold setting unit sets the threshold as an average value of the first statistical value and the second statistical value.
6. The information processing device according to claim 4, characterized in that The threshold setting unit sets the threshold as a weighted average value using the numbers of input data assigned to the first statistical value and the second statistical value as weights.
7. The information processing device according to any one of claims 3 to 6, characterized in that: The information processing device includes a second feature extraction unit that extracts a feature different from that of the first feature extraction unit from the input data. When the value of the label extracted in the second step, which is the object of comparison with the threshold value, is equal to or smaller than the threshold value, inference is performed using the second feature quantity extraction unit.
8. The information processing device according to claim 7, characterized in that: When the maximum value of the accuracy in the second process is equal to or less than the threshold value for the input data, the second feature quantity extraction unit is used to perform inference.
9. The information processing device according to any one of claims 3 to 6, characterized in that: The information processing device includes a second feature extraction unit that extracts a feature different from that of the first feature extraction unit from the input data. The first classification unit performs a process of extracting a value having a precision less than or equal to the second largest value in the input data rearranged in the first process. When the value of the label extracted in the process, which is the comparison target with the threshold value, is equal to or greater than the threshold value, inference is performed using the second feature quantity extraction unit.
10. The information processing device according to claim 9, characterized in that: When the maximum value of the accuracy in the second process is equal to or greater than the threshold value, the second feature quantity extraction unit is used to perform inference on the input data.
11. The information processing device according to any one of claims 3 to 8, characterized in that: The information processing device comprises: a second feature extraction unit configured to extract a feature different from that of the first feature extraction unit from the input data; a second accuracy calculation unit that performs inference on the input data based on the feature quantity extracted by the second feature quantity extraction unit, and calculates the accuracy of classifying the input data for each of a second number of classes less than the first number; a second classification unit that classifies the input data into any class among the second number of classes based on the accuracy calculated by the second accuracy calculation unit; as well as a classification result selection unit for selecting which of the first classification partial classification result and the second classification partial classification result to output; The first accuracy calculation unit performs inference on the input data based on the feature quantity extracted by the first feature quantity extraction unit, and calculates the accuracy of classifying the input data for each of the first number of classes. The first classification unit classifies the input data into a class having the highest accuracy calculated by the first accuracy calculation unit among the first number of classes. When the accuracy calculated by the first accuracy calculation unit for the class into which the input data is classified by the first classification unit exceeds a preset threshold, the classification result selection unit selects to output the partial classification result of the first classification unit; when the accuracy calculated by the first accuracy calculation unit for the class into which the input data is classified by the first classification unit is below the threshold, the classification result selection unit selects to output the partial classification result of the second classification unit.
12. The information processing device according to claim 11, characterized in that: The second classification unit classifies the input data into two classes based on the feature quantity extracted by the first feature quantity extraction unit.
13. The information processing device according to claim 12, characterized in that: The second accuracy calculation unit calculates a first accuracy of the input data being classified into the first class among the first number of classes and a second accuracy of the input data being classified into a class other than the first class among the first number of classes, when the accuracy calculated by the first classification unit for the class into which the input data is classified is below the threshold value, the first class being the class with the highest accuracy calculated by the first accuracy calculation unit, The second classifying unit classifies the input data into the first class when the first accuracy is higher than the second accuracy.
14. The information processing device according to claim 13, characterized in that: The second accuracy calculation unit calculates a third accuracy of the input data being classified as a second class among the first number of classes and a fourth accuracy of the input data being classified as a class other than the second class among the first number of classes when the first accuracy is lower than the second accuracy, the second class being a class whose accuracy calculated by the first accuracy calculation unit is second only to the first class, The second classification unit classifies the input data into the second class when the third accuracy is higher than the fourth accuracy.
15. The information processing device according to claim 11, characterized in that: The second accuracy calculation unit calculates a first accuracy of the input data being classified into the first category among the first number of categories and a third accuracy of the input data being classified into the second category among the first number of categories, the first category being the category with the highest accuracy calculated by the first accuracy calculation unit, and the second category being the category with the second highest accuracy calculated by the first accuracy calculation unit after the first category, The second classifying unit classifies the input data into a class corresponding to a higher accuracy between the first accuracy and the third accuracy, of the first class and the second class.
16. The information processing device according to claim 11, characterized in that: The information processing device comprises: A first model generation unit that generates a first learned model based on a first data set, the first data set including a correct answer label of the first quantity classification and a plurality of input data corresponding to the correct answer labels of the first quantity classification; as well as a second model generating unit for generating a second learned model based on a second data set, the second data set including the correct answer labels of the second quantity classification and a plurality of input data of the first data set corresponding to the correct answer labels of the second quantity classification, The first accuracy calculation unit performs inference of the input data based on the first learned model, The second accuracy calculation unit performs inference of the input data based on the second learned model.
17. The information processing device according to claim 16, characterized in that: The second classifier classifies the input data in a state where the first learned model is generated by the first model generator.
18. The information processing device according to claim 16, characterized in that: The number of adjustable parameters of the second learned model is smaller than that of the first learned model.
19. The information processing device according to claim 16, characterized in that: The second model generation unit generates a plurality of learned models using a plurality of different algorithms. The second accuracy calculation unit calculates the accuracy of classifying the input data into each of the second number of classes using the plurality of learned models.
20. The information processing device according to claim 16, characterized in that: The second model generating unit generates the second learned model using a plurality of computers capable of performing calculations independently of each other.
21. The information processing device according to claim 16, characterized in that: When a third number of correct answer labels that are different from each other among the correct answer labels of the first number category of the first data set are set as first correct answer labels, The second accuracy calculation unit infers the input data based on the feature extracted by the feature extraction unit, and calculates the accuracy of classifying the input data for each of the third number of classes corresponding to the first correct answer label. The second classification unit classifies the input data into the third number of classes corresponding to the first correct answer labels based on the accuracy calculated by the second accuracy calculation unit.
22. The information processing device according to claim 16, characterized in that: When one of the correct answer labels of the first quantity classification of the first data set is set as the second correct answer label, and the correct answer label of the learning data that does not correspond to the second correct answer label among the correct answer labels of the first quantity classification of the first data set is set as the third correct answer label, The second classification unit classifies the input data into two classes corresponding to the second correct answer label and the third correct answer label.
23. The information processing device according to claim 22, characterized in that: The information processing device includes a learning data generating unit, which generates the following second data set based on the first data set, wherein the second data set includes the second correct answer label and the third correct answer label, and a plurality of learning data corresponding to the second correct answer label and the third correct answer label of the first data set.
24. The information processing device according to claim 23, characterized in that: When the highest accuracy among the classification accuracies for each of the first number of classes calculated by the first accuracy calculation unit is assumed to be the fifth accuracy, The threshold setting unit sets the threshold to a value between either the average value or the central value of the fifth accuracy when a result consistent with the class corresponding to the correct answer label is obtained in the result of classification of the multiple input data of the first data set by the first classification unit, and between the average value or the central value of the fifth accuracy when a result inconsistent with the class corresponding to the correct answer label is obtained in the result of classification of the multiple input data of the first data set by the first classification unit.
25. The information processing device according to claim 23, characterized in that: When the accuracy next to the highest accuracy among the accuracies of classification of each of the first number of classes calculated by the first accuracy calculation unit is assumed to be the sixth accuracy, The threshold setting unit sets the threshold to a value between either an average value or a central value of the sixth accuracy when a result consistent with a class corresponding to a correct answer label is obtained in the result of classification of the multiple input data of the first data set by the first classification unit, and between either an average value or a central value of the sixth accuracy when a result inconsistent with a class corresponding to a correct answer label is obtained in the result of classification of the multiple input data of the first data set by the first classification unit.
26. The information processing device according to claim 23, characterized in that When the highest accuracy among the classification accuracies for each of the first number of classes calculated by the first accuracy calculation unit is assumed to be the fifth accuracy, The threshold setting unit sets the threshold to a value that is between an average value of the fifth accuracy when a result consistent with a class corresponding to a correct answer label is obtained from classification of the multiple input data of the first data set by the first classification unit and an average value of the fifth accuracy when a result inconsistent with a class corresponding to a correct answer label is obtained from classification of the multiple input data of the first data set by the first classification unit, and between a central value of the fifth accuracy when a result consistent with a class corresponding to a correct answer label is obtained from classification of the multiple input data of the first data set by the first classification unit and a central value of the fifth accuracy when a result inconsistent with a class corresponding to a correct answer label is obtained from classification of the multiple input data of the first data set by the first classification unit.
27. The information processing device according to claim 23, characterized in that: When the highest accuracy among the accuracies of classification of each class in the first number of classes calculated by the first accuracy calculation unit is assumed to be the fifth accuracy, and the accuracy next to the highest accuracy among the accuracies of classification of each class in the first number of classes calculated by the first accuracy calculation unit is assumed to be the sixth accuracy, The threshold setting unit sets the threshold to a value that is between either an average value or a central value of the fifth accuracy when a result consistent with a class corresponding to a correct answer label is obtained from classification of the multiple input data of the first data set by the first classification unit, and either an average value or a central value of the sixth accuracy when a result consistent with a class corresponding to a correct answer label is obtained from classification of the multiple input data of the first data set by the first classification unit, and between either an average value or a central value of the fifth accuracy when a result inconsistent with a class corresponding to a correct answer label is obtained from classification of the multiple input data of the first data set by the first classification unit, and either an average value or a central value of the sixth accuracy when a result inconsistent with a class corresponding to a correct answer label is obtained from classification of the multiple input data of the first data set by the first classification unit.
28. The information processing device according to any one of claims 24 to 27, characterized in that: The threshold setting unit sets the threshold for each subset of input data included in the first data set.
29. The information processing device according to any one of claims 24 to 27, characterized in that: The threshold setting unit sets the threshold for each of the plurality of classes classified by the first classification unit.
30. The information processing device according to any one of claims 11 to 27, characterized in that: The first classifying unit and the second classifying unit classify the input data using a parallel computing device capable of performing parallel computing.
31. The information processing device according to any one of claims 11 to 27, characterized in that: The input data is image data.
32. The information processing device according to any one of claims 11 to 27, characterized in that: The input data is graph data, which includes at least two nodes and an edge connecting the two nodes.
33. The information processing device according to any one of claims 11 to 27, characterized in that: The input data is natural language data.
34. The information processing device according to any one of claims 11 to 27, characterized in that: The input data is a set of continuously changing numerical values including time series data.
35. An information processing method, which is an information processing method performed by an information processing device, the information processing device comprising a feature extraction unit, a first accuracy calculation unit, a first classification unit, a second accuracy calculation unit, a second classification unit and a classification result selection unit, characterized in that: The information processing method comprises the following steps: The feature extraction unit extracts the feature of the input data; The first accuracy calculation unit performs inference on the input data based on the feature extracted by the feature extraction unit, and calculates the accuracy of classification of the input data for each of the first number of classes; The first classification unit classifies the input data into a class having the highest accuracy calculated by the first accuracy calculation unit among the first number of classes; The second accuracy calculation unit performs inference on the input data based on the feature extracted by the feature extraction unit, and calculates the accuracy of classifying the input data for each of the second number of classes that is smaller than the first number; The second classification unit classifies the input data into any class among the second number of classes based on the accuracy calculated by the second accuracy calculation unit; as well as The classification result selection unit selects which of the first classification partial classification result and the second classification partial classification result to output, When the accuracy calculated by the first accuracy calculation unit for the class into which the input data is classified by the first classification unit exceeds a preset threshold, the classification result selection unit selects to output the partial classification result of the first classification unit; when the accuracy calculated by the first accuracy calculation unit for the class into which the input data is classified by the first classification unit is below the threshold, the classification result selection unit selects to output the partial classification result of the second classification unit.
36. The information processing device according to claim 1, characterized in that The second step is to extract the label with the minimum value. The third step is a process of comparing the label having the minimum value with the correct answer label associated with the input data.
