Classification device, method, and computer program for performing classification processing using machine learning models

Abstract

To provide a technique for improving accuracy of classification processing.SOLUTION: A machine learning model including one or more known classes and one or more dummy classes, is trained with a first training data group for the known classes and a second training data group for the dummy classes. A classification processing unit determines a class of data to be classified as "unknown" when a result of classifying the data to be classified using the machine learning model is a dummy class.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a classification device, a method, and a computer program that perform classification processing using a machine learning model. [Background technology]

[0002] Patent Documents 1 and 2 disclose a vector neural network type machine learning model that uses vector neurons, called a capsule network. A vector neuron is a neuron whose input and output are vectors. A capsule network is a machine learning model that uses vector neurons called capsules as network nodes. A vector neural network type machine learning model such as a capsule network can be used to classify input data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 5,210,798 [Patent Document 2] International Publication No. 2019 / 083553 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, the classification result for data to be classified that should be determined as unknown may be erroneously determined as a known class, and therefore there has been a demand for improving the accuracy of the classification process. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a classification device that performs classification processing of data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers. The classification device includes a classification processing unit that performs the classification processing. The machine learning model is configured to generate a classification data set based on one or more known classes. Judgment value of and one or more dummy classes Judgment value of and, The output layer outputs and the machine learning model is trained using a first group of training data for the known class and a second group of training data for the dummy class. The classification processing unit determines that the class of the classified data is unknown when a class classification result of the classified data using the machine learning model is the dummy class. When N is an integer greater than or equal to 2, the classification device has N machine learning models, and the classification processing unit determines that the class of the classified data is unknown if all of the class classification results from the N machine learning models are the dummy class. (1) The second training data group for the dummy class of each of the machine learning models may be data selected from the N-1 first training data groups for the other N-1 machine learning models, and may be selected to include a portion of each of the N-1 first training data groups. (2) Each of the N machine learning models may have N-1 dummy classes, and the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models may be data selected from the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups may be selected to correspond to the N-1 first teacher data groups. (3) Each of the first teacher data groups included in the first teacher data group is data extracted from sample data larger in size than the first teacher data, and the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models are data extracted from N-1 average sample data obtained by averaging the N-1 sample data groups used to extract the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups may be extracted to correspond to the N-1 average sample data.

[0006] According to a second aspect of the present disclosure, there is provided a method for performing a classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, the method comprising: (a) inputting the data to be classified into the machine learning model and obtaining a classification result; process and (b) determining that the class of the classified data is unknown if the classification result is a dummy class. , which is how a computer executes . The machine learning model has an output layer that outputs judgment values ​​of one or more known classes and judgment values ​​of one or more dummy classes, and the machine learning model is trained using a first group of training data for the known classes and a second group of training data for the dummy classes. When N is an integer equal to or greater than 2, there are N machine learning models, and in step (a), if the classification results from the N machine learning models are all the dummy classes, the class of the data to be classified is determined to be unknown. (1) The second training data group for the dummy class of each of the machine learning models may be data selected from the N-1 first training data groups for the other N-1 machine learning models, and may be selected to include a portion of each of the N-1 first training data groups. (2) Each of the N machine learning models may have N-1 dummy classes, and the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models may be data selected from the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups may be selected to correspond to the N-1 first teacher data groups. (3) Each of the N machine learning models has N-1 dummy classes; Each of the first teacher data included in the first teacher data group may be data extracted from sample data larger in size than the first teacher data, and the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models may be data extracted from N-1 average sample data obtained by averaging the N-1 sample data groups used to extract the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups may be extracted so as to correspond to the N-1 average sample data.

[0007] According to a third aspect of the present disclosure, there is provided a computer program for causing a processor to execute a class classification process for data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, the computer program comprising: (a) inputting the data to be classified into the machine learning model and obtaining a class classification result; process and (b) determining that the class of the classified data is unknown when the classification result is a dummy class. It is a computer program . The machine learning model has an output layer that outputs judgment values ​​of one or more known classes and judgment values ​​of one or more dummy classes, and the machine learning model is trained using a first group of training data for the known classes and a second group of training data for the dummy classes. When N is an integer equal to or greater than 2, there are N machine learning models, and the process (a) determines that the class of the data to be classified is unknown when all of the classification results from the N machine learning models are the dummy classes. (1) The second training data group for the dummy class of each of the machine learning models may be data selected from the N-1 first training data groups for the other N-1 machine learning models, and may be selected to include a portion of each of the N-1 first training data groups. (2) Each of the N machine learning models may have N-1 dummy classes, and the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models may be data selected from the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups may be selected to correspond to the N-1 first teacher data groups. (3) Each of the N machine learning models has N-1 dummy classes; Each of the first teacher data included in the first teacher data group may be data extracted from sample data larger in size than the first teacher data, and the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models may be data extracted from N-1 average sample data obtained by averaging the N-1 sample data groups used to extract the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups may be extracted so as to correspond to the N-1 average sample data. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a classification system according to a first embodiment. [Figure 2] FIG. 1 is an explanatory diagram showing an example of the configuration of a machine learning model. [Figure 3] FIG. 2 is an explanatory diagram showing the class configuration of a machine learning model in the first embodiment. [Figure 4] 5 is a flowchart showing the processing procedure of a preparation step in the first embodiment. [Figure 5] FIG. 10 is an explanatory diagram showing how a patch image is created from a sample image. [Figure 6] FIG. [Figure 7] FIG. 2 is an explanatory diagram showing the configuration of a group of known characteristic spectra. [Figure 8] 10 is a flowchart showing the processing procedure of a classification step in the first embodiment. [Figure 9] FIG. 10 is an explanatory diagram showing the class configuration of a machine learning model in the second embodiment. [Figure 10] 10 is a flowchart showing the processing procedure of a preparation step in the second embodiment. [Figure 11]FIG. 10 is an explanatory diagram showing the clustering process of the sample image. [Figure 12] FIG. 10 is an explanatory diagram showing grouped sample images. [Figure 13] FIG. 10 is an explanatory diagram showing a group of training data made up of patch images extracted from a sample image. [Figure 14] FIG. 10 is an explanatory diagram showing a method for creating a group of training data for a dummy class in the second embodiment. [Figure 15] 10 is a flowchart showing the processing procedure of a classification step in the second embodiment. [Figure 16] FIG. 11 is an explanatory diagram showing the class configuration of a machine learning model in the third embodiment. [Figure 17] FIG. 11 is an explanatory diagram showing a method for creating a group of training data for a dummy class in the third embodiment. [Figure 18] FIG. 13 is an explanatory diagram showing a method for creating a group of training data for a dummy class in the fourth embodiment. [Figure 19] 13 is a flowchart showing the processing procedure of a preparation step in the fifth embodiment. [Figure 20] FIG. 13 is an explanatory diagram showing a performance preference selection screen according to the fifth embodiment. [Figure 21] FIG. 10 is an explanatory diagram showing a first calculation method of class-specific similarity. [Figure 22] FIG. 10 is an explanatory diagram showing a second calculation method of class-specific similarity. [Figure 23] FIG. 10 is an explanatory diagram showing a third method for calculating class similarity. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. First embodiment: FIG. 1 is a block diagram showing a classification system according to a first embodiment. The classification system includes an information processing device 100 and a camera 400. The camera 400 is used to capture images of items to be classified. The camera 400 may be a camera that captures color images, or a camera that captures monochrome or spectral images. In this embodiment, images captured by the camera 400 are used as training data and classified data, but data other than images may also be used as training data and classified data. In this case, a classified data acquisition device appropriate for the type of data is used instead of the camera 400.

[0010] The information processing device 100 has a processor 110, a memory 120, an interface circuit 130, and an input device 140 and a display device 150 connected to the interface circuit 130. A camera 400 is also connected to the interface circuit 130. For example, but not limited to, the processor 110 not only has the function of executing the processes described in detail below, but also has the function of displaying on the display device 150 data obtained by the processes and data generated in the course of the processes.

[0011] The processor 110 functions as a learning execution unit 112 that executes learning of a machine learning model, and a classification processing unit 114 that executes classification processing of data to be classified. The classification processing unit 114 includes a similarity calculation unit 310 and a class determination unit 320. The learning execution unit 112 and the classification processing unit 114 are each implemented by the processor 110 executing a computer program stored in the memory 120. However, the learning execution unit 112 and the classification processing unit 114 may also be implemented as hardware circuits. The term "processor" in the present disclosure also includes such hardware circuits. Furthermore, one or more processors that execute the learning processing and the classification processing may be processors included in one or more remote computers connected via a network.

[0012] The memory 120 stores a machine learning model 200, a first training data group TD1 for known classes, a second training data group TD2 for dummy classes, and a known feature spectrum group GKSp. The machine learning model 200 is used for processing by the class classification processing unit 114. An example configuration and operation of the machine learning model 200 will be described later. The training data groups TD1 and TD2 are sets of labeled data used for training the machine learning model 200. In this embodiment, the training data groups TD1 and TD2 are sets of image data. The known feature spectrum group GKSp is a set of feature spectra obtained when training data is input again into the trained machine learning model 200. The feature spectra will be described later.

[0013] FIG. 2 is an explanatory diagram showing the configuration of a machine learning model 200. This machine learning model 200 has an input layer 210, a hidden layer 280, and an output layer 260. The hidden layer 280 includes a convolutional layer 220, a primary vector neuron layer 230, a first convolutional vector neuron layer 240, and a second convolutional vector neuron layer 250. The output layer 260 is also referred to as a "classification vector neuron layer 260." Of these layers, the input layer 210 is the lowest layer, and the output layer 260 is the highest layer. In the following description, the hidden layer 280 and the output layer 260 are referred to as the "Conv layer 220," the "PrimeVN layer 230," the "ConvVN1 layer 240," the "ConvVN2 layer 250," and the "ClassVN layer 260," respectively.

[0014] 2, two convolution vector neuron layers 240 and 250 are used, but the number of convolution vector neuron layers is arbitrary, and a convolution vector neuron layer may be omitted. However, it is preferable to use one or more convolution vector neuron layers.

[0015] An image of 32×32 pixels is input to the input layer 210. The configuration of each layer other than the input layer 210 can be described as follows. ·Conv layer 220: Conv[32,4,2] ·PrimeVN layer 230: PrimeVN[16,1,1] ·ConvVN1 layer 240:ConvVN1[12,3,2] ·ConvVN2 layer 250: ConvVN2[8,4,1] ·ClassVN layer 260:ClassVN[M,4,1] Vector dimension VD: VD=16 In the description of each layer, the character string before the parentheses is the layer name, and the numbers in the parentheses are, in order, the number of channels, the kernel surface size, and the stride. For example, the layer name of the Conv layer 220 is "Conv," the number of channels is 32, the kernel surface size is 4x4, and the stride is 2. In Figure 2, these descriptions are shown below each layer. The hatched rectangles drawn in each layer represent the kernel surface size used when calculating the output vector of the adjacent higher layer. In this embodiment, since the input data is image data, the kernel surface size is also two-dimensional. Note that the parameter values ​​used in the description of each layer are merely examples and can be changed as desired.

[0016] The input layer 210 and the Conv layer 220 are layers composed of scalar neurons. The other layers 230 to 260 are layers composed of vector neurons. A vector neuron is a neuron that uses vectors as input and output. In the above description, the dimension of the output vector of each vector neuron is constant at 16. In the following, the term "node" is used as a superordinate concept of scalar neurons and vector neurons.

[0017] FIG. 2 shows the first axis x and second axis y that define the planar coordinates of the node array for the Conv layer 220, and the third axis z that represents depth. It also shows that the sizes of the Conv layer 220 in the x, y, and z directions are 15, 15, and 32. The sizes in the x and y directions are called "resolution." The size in the z direction is the number of channels. These three axes x, y, and z are also used in other layers as coordinate axes that indicate the position of each node. However, in FIG. 2, these axes x, y, and z are omitted from the illustration for layers other than the Conv layer 220.

[0018] As is well known, the resolution W1 after convolution is given by the following equation: W1=Ceil{(W0-Wk+1) / S} (A1) Here, W0 is the resolution before convolution, Wk is the surface size of the kernel, S is the stride, and Ceil{X} is a function that rounds up the decimal point of X. The resolution of each layer shown in FIG. 2 is an example in which the resolution of the input data is 32, and the actual resolution of each layer is changed appropriately depending on the size of the input data.

[0019] The ClassVN layer 260 has M channels. M is the number of classes that can be distinguished using the machine learning model 200. In this embodiment, M is an integer equal to or greater than 2. The M channels of the ClassVN layer 260 output judgment values ​​Class_1 to Class_M for the M classes. Typically, the class having the largest value among these judgment values ​​Class_1 to Class_M is judged to be the corresponding class of the classified data. Furthermore, if the largest value among the judgment values ​​Class_1 to Class_M is less than a predetermined threshold, it is also possible to judge that the class of the classified data is unknown.

[0020] FIG. 2 also illustrates subregions Rn in each layer 220, 230, 240, 250, and 260. The subscript "n" in subregion Rn refers to the reference number of the layer. For example, subregion R220 indicates a subregion in the Conv layer 220. A "subregion Rn" is a region in each layer that is identified by a planar position (x, y) defined by the position of the first axis x and the position of the second axis y, and that includes multiple channels along the third axis z. The subregion Rn has dimensions of "Width" × "Height" × "Depth," corresponding to the first axis x, the second axis y, and the third axis z. In this embodiment, the number of nodes included in one "subregion Rn" is "1 × 1 × number of depths," i.e., "1 × 1 × number of channels."

[0021] 2, a feature spectrum Sp, which will be described later, is calculated from the output of the ConvVN2 layer 250 and input to the similarity calculation unit 310. The similarity calculation unit 310 calculates a class-specific similarity, which will be described later, using this feature spectrum Sp and a group of known feature spectra GKSp that has been created in advance. It is also possible to determine the class to which the data to be classified belongs using this class-specific similarity.

[0022] In this disclosure, the vector neuron layer used to calculate the similarity is also referred to as the "specific layer." As the specific layer, a vector neuron layer other than the ConvVN2 layer 250 may be used, and any number of vector neuron layers greater than or equal to one may be used. The configuration of the feature spectrum and the method of calculating the similarity using the feature spectrum will be described later.

[0023] FIG. 3 is an explanatory diagram showing the class configuration of the machine learning model 200 in the first embodiment. Classification processing of data Di to be classified is performed using one machine learning model 200. In the first embodiment, the M classes of the machine learning model 200 are composed of M-1 known classes and one dummy class. That is, the judgment values ​​Class_1 to Class_M of the M classes are composed of judgment values ​​Known_1 to Known_Mk of the M-1 known classes and judgment value Dummy_1 of one dummy class. Here, Mk=M-1. Since M is an integer of 2 or greater, the number M-1 of known classes is 1 or greater. Note that when the number of dummy classes is m, m may be 1 or may be 2 or greater.

[0024] 4 is a flowchart showing the processing steps of the machine learning model preparation process. In step S110, the learning execution unit 112 generates a plurality of sample images by photographing a plurality of samples using the camera 400. In step S120, the learning execution unit 112 applies preprocessing to the sample images to create patch images. For example, the preprocessing can include resolution adjustment, data normalization (min-max normalization), and the like.

[0025] FIG. 5 is an explanatory diagram showing how patch images are created from a sample image. Here, multiple small patch images PD are extracted from the sample image SD. This example shows an example in which the sample image SD is divided into multiple patch images PD along the dividing lines indicated by dashed lines. Each of these patch images PD is used as an image for training data. Note that a larger number of patch images PD may be extracted from a single sample image SD by making the stride of the extraction position of the patch image PD smaller than the size of one side of the patch image PD. In the present disclosure, the sample image SD is also referred to as "sample data SD," and the patch image PD is also referred to as "patch data PD." Note that the sample image SD may be used as an image for training data without extracting the patch image PD.

[0026] In this embodiment, multiple patch images PD extracted from one sample image SD are used as a group of images belonging to one class. Since the number of known classes in the machine learning model 200 shown in Figure 3 is Mk, Mk sample images SD are captured in step S110, and multiple patch images PD are extracted from each sample image SD in step S120.

[0027] In step S130, the learning execution unit 112 creates a first training data group TD1 for known classes by assigning labels to the patch images PD. In this embodiment, Mk training data groups are created by assigning one of Mk labels 1 to Mk to each patch image PD. These labels correspond to the Mk known classes Known_1 to Known_Mk of the machine learning model 200 shown in FIG. 3. In this disclosure, "label" and "class" mean the same thing.

[0028] In step S140, the learning execution unit 112 creates a second training data group TD2 for a dummy class. In the machine learning model 200 shown in FIG. 3, there is one dummy class, and a second training data group TD2 is created for the dummy class Dummy_1. The second training data group TD2 is created to improve the accuracy of classifying known classes. For this reason, it is preferable that the images included in the second training data group TD2 are created so as not to have the characteristics of the images included in the first training data group TD1. Specifically, for example, assume that the numbers 0 to 9 are set as known classes and handwritten numbers are classified. In this case, images of handwritten characters such as alphabets can be used as the images in the second training data group TD2 for the dummy class. This allows handwritten numbers to be correctly classified into one of the classes 0 to 9 and prevents handwritten characters from being mistaken for numbers, thereby improving the accuracy of classifying handwritten numbers. Another expected effect of the dummy class is that by increasing the variation of the learning data, the machine learning model 200 will be able to improve its ability to distinguish between input data. In other words, the performance of identifying input data that is not included in the dummy class, such as Greek letters and symbols other than the alphabet, as unknown will also be improved.

[0029] In step S150, the learning execution unit 112 uses the first teacher data group TD1 for the known classes and the second teacher data group TD2 for the dummy classes to learn the machine learning model 200. When the learning is completed, the learned machine learning model 200 is stored in the memory 120.

[0030] In step S160, the learning execution unit 112 generates a group of known characteristic spectra GKSp by again inputting a plurality of pieces of training data into the trained machine learning model 200. The group of known characteristic spectra GKSp is a collection of characteristic spectra, which will be described below.

[0031] FIG. 6 is an explanatory diagram showing a feature spectrum Sp obtained by inputting arbitrary input data to the trained machine learning model 200. As shown in FIG. 2, in this embodiment, the feature spectrum Sp is created from the output of the ConvVN2 layer 250. The horizontal axis in FIG. 6 represents the position of a vector element in the output vector of multiple nodes included in one subregion R250 of the ConvVN2 layer 250. The position of this vector element is represented by a combination of the element number ND of the output vector at each node and the channel number NC. In this embodiment, since the vector dimension is 16 (the number of elements of the output vector output by each node), the element number ND of the output vector is 16, ranging from 0 to 15. Furthermore, since the ConvVN2 layer 250 has eight channels, the channel number NC is 8, ranging from 0 to 7. In other words, this feature spectrum Sp is obtained by arranging multiple element values ​​of the output vector of each vector neuron included in one subregion R250 across multiple channels along the third axis z.

[0032] The vertical axis of Fig. 6 represents the feature value C at each spectral position. V In this example, the feature value C V is the value of each element of the output vector V ND Feature value C V For the feature value C, statistical processing such as centering to the average value 0 may be performed. V As the value of each element of the output vector V ND Alternatively, the normalization coefficient may be used as is. In the latter case, the feature value C included in the feature spectrum Sp is V The number of is equal to the number of channels, which is 8. The normalization coefficient is a value corresponding to the vector length of the output vector of the node.

[0033] The number of feature spectra Sp obtained from the output of the ConvVN2 layer 250 for one piece of input data is 16, since this is equal to the number of planar positions (x, y) of the ConvVN2 layer 250, that is, the number of subregions R250.

[0034] The learning execution unit 112 inputs the training data again into the trained machine learning model 200, calculates the feature spectrum Sp shown in FIG. 6, and registers it in the memory 120 as a group of known feature spectra GKSp.

[0035] 7 is an explanatory diagram showing the configuration of the known feature spectrum group GKSp. In this example, the known feature spectrum group GKSp is obtained from the output of the ConvVN2 layer 250. Note that it is sufficient that the known feature spectrum group GKSp is obtained from the output of at least one vector neuron layer, and it is also possible to register known feature spectrum groups obtained from the output of the ConvVN1 layer 240 or the ClassVN layer 260.

[0036] Each record in the group of known feature spectra GKSp includes a parameter i indicating the order of the label or class, a parameter k indicating the order of the subregion Rn within the layer, a parameter q indicating the data number, and a known feature spectrum KSp. The known feature spectrum KSp is the same as the feature spectrum Sp in Figure 6.

[0037] The class parameter i takes a value from 1 to M. The parameter k of the subregion Rn takes a value indicating which of the multiple subregions Rn included in a specific layer it is, i.e., which planar position (x, y) it is. For the ConvVN2 layer 250, there are 16 subregions R250, so k = 1 to 16. The data number parameter q indicates the number of training data with the same label, and takes a value from 1 to max1 for class 1 and from 1 to maxM for class M.

[0038] The plurality of training data used in step S160 does not need to be the same as the plurality of training data used in step S150. However, if some or all of the plurality of training data used in step S150 is used in step S160, there is an advantage in that there is no need to prepare new training data.

[0039] FIG. 8 is a flowchart showing the processing steps of the classification step using the trained machine learning model 200. In step S210, the classification processing unit 114 generates classified data by photographing the item to be classified using the camera 400. In step S220, the classification processing unit 114 performs preprocessing on the classified data as necessary. This preprocessing is preferably the same as the preprocessing performed in step S120 of FIG. 4. The classified data after preprocessing is created as an image of the same size as the patch image PD shown in FIG. 5. Note that the preprocessing can be omitted. In step S230, the classification processing unit 114 reads the trained machine learning model 200 and the known feature spectrum group GKSp from the memory 120.

[0040] In step S240, the classification processing unit 114 inputs the data to be classified into the machine learning model 200. In step S250, the classification processing unit 114 obtains the feature spectrum Sp shown in FIG. 6 using the output of the ConvVN2 layer 250, which is a specific layer. In step S260, the similarity calculation unit 310 calculates the similarity using the feature spectrum Sp obtained in step S250 and the known feature spectrum group GKSp shown in FIG. 7. As the similarity, either a class-specific similarity or a maximum similarity that does not take the class into consideration can be used.

[0041] The similarity S(Class) for each class can be calculated using, for example, the following formula. S(Class)=max[G{Sp,KSp(Class,k)}] (A2) Here, "Class" is the ordinal number for the class, G{a,b} is a function for calculating the similarity between a and b, Sp is the feature spectrum obtained from the data to be classified, KSp(Class,k) is all known feature spectra associated with a specific "Class," k is the ordinal number of the known feature spectrum, and max[X] is a logical operation that takes the maximum value of X. The function G{a,b} for calculating the similarity can be, for example, a similarity using a distance such as cosine similarity or Euclidean distance. The similarity S(Class) is the maximum of the similarities calculated between the feature spectrum Sp and all known feature spectra KSp(Class,k) corresponding to a specific class. Such a similarity S(Class) is calculated for each of the M classes. The similarity S(Class) represents the degree to which the data to be classified is similar to the features of each class. This similarity S(Class) can also be used as explanatory information for the class classification results of the data to be classified. Other methods for calculating similarity by class will be described later.

[0042] The maximum similarity S(All) without taking the class into consideration can be calculated using, for example, the following formula. S(All)=max[G{Sp,KSp(k)}] (A3) Here, KSp(k) denotes the k-th spectrum among all known feature spectra. This maximum similarity S(All) is the maximum value among the similarities between the feature spectrum Sp and all known feature spectra KSp. Since it is possible to identify the known feature spectrum KSp(k) that gives the maximum similarity S(All), it is possible to identify the label, i.e., the class, from the group of known feature spectra GKSp shown in FIG. 8. This maximum similarity S(All) can also be used as explanatory information to explain the class classification result, i.e., whether the data to be classified belongs to known data or unknown data.

[0043] In step S270, the class discrimination unit 320 determines the class to which the classified data belongs. This determination can be made using at least one of the judgment values ​​Class_1 to Class_M output from the ClassVN layer 260 and the similarity obtained in step S250. For example, the class to which the classified data belongs may be determined using only the judgment values ​​Class_1 to Class_M output from the ClassVN layer 260. In this case, steps S250 and S260 described above may be omitted. Alternatively, the class to which the classified data belongs may be determined using only the similarity S(Class) for each class. In this case, if the similarity S(Class) for a certain class is equal to or greater than a predetermined threshold, it can be determined that the classified data belongs to that class. On the other hand, if the similarities S(Class) for all classes are less than the threshold, it can be determined that the classified data belongs to an unknown class. When the maximum similarity S(All) is used as the similarity, if the maximum similarity S(All) is equal to or greater than a threshold, the class of the known feature spectrum that gives the maximum similarity S(All) can be determined to be the class of non-classified data; if it is less than the threshold, it can be determined to be an unknown class. The maximum similarity S(All) can also be displayed as explanatory information explaining whether the data to be classified belongs to known data or unknown data. Furthermore, the class of the data to be classified may be determined using both the similarity and the judgment values ​​Class_1 to Class_M of the ClassVN layer 260. For example, if the class determined from the similarity matches the class determined from the judgment values ​​Class_1 to Class_M of the ClassVN layer 260, it can be determined that the data to be classified belongs to that class. Furthermore, if the class determined from the similarity does not match the class determined from the judgment values ​​Class_1 to Class_M of the ClassVN layer 260, it can be determined that the data to be classified belongs to an unknown class.

[0044] If the classification result of the classified data is a dummy class, the class discrimination unit 320 determines that the class of the classified data is unknown. This makes it possible to correctly determine that classified data that does not fall into a known class is unknown.

[0045] As described above, in the first embodiment, the accuracy of the classification process can be improved because the machine learning model 200 having known classes and dummy classes is used to perform the classification process. Furthermore, the use of dummy classes increases the variation of the training data, which is expected to improve the ability of the machine learning model 200 to distinguish between input data.

[0046] B. Second embodiment: FIG. 9 is an explanatory diagram showing the class structure of a machine learning model in the second embodiment. The device configuration of the classification system in the second embodiment is substantially the same as that shown in FIG. 1. As shown in FIG. 9, the classification system in the second embodiment includes N machine learning models 200_1 to 200_N. N is an integer equal to or greater than 2. Each of these machine learning models 200_1 to 200_N has the same configuration as the machine learning model 200 shown in FIG. 2. However, the number of classes of each of the machine learning models 200_1 to 200_N may be different. The additional symbols "_1" to "_N" added to the end of the reference numerals of the machine learning models are added to distinguish between the N machine learning models. When there is no need to distinguish between them, the symbols "_1" to "_N" are omitted and the model will simply be referred to as "machine learning model 200." In the following, the j-th machine learning model among the N models will be referred to as "machine learning model 200_j."

[0047] The j-th machine learning model 200_j has j_M classes. These j_M classes are composed of j_M-1 known classes and one dummy class. That is, the judgment values ​​Class_j_1 to Class_j_M of the j_M classes are composed of judgment values ​​Known_j_1 to Known_j_Mk of the j_Mk known classes and judgment value Dummy_j_1 of one dummy class. Here, j_Mk=j_M-1. j_M is an integer of 2 or more, and the number j_Mk of known classes is 1 or more. When the number of dummy classes is m, m may be 1 or may be 2 or more.

[0048] Fig. 10 is a flowchart showing the processing procedure of the preparation step of the machine learning model in the second embodiment. The only difference from the preparation step in the first embodiment shown in Fig. 4 is that step S115 is added between step S110 and step S120, and step S140 is replaced with step S145; the other steps are almost the same as those in Fig. 4.

[0049] In step S115, the learning execution unit 112 executes a process of dividing the plurality of sample images into N groups. The number N of groups corresponds to the number N of machine learning models 200 shown in FIG.

[0050] FIG. 11 shows an example in which a plurality of sample images are divided into two groups GP_1 and GP_2. The grouping can be performed, for example, by a clustering process using the k-means method. In the example of FIG. 11, a plurality of sample images are classified into two groups GP_1 and GP_2, each centered on its respective center of gravity G1 and G2. Note that the grouping may also be performed using a method other than the k-means method. For example, a plurality of sample images may be arranged and classified into a plurality of groups according to their ordinal numbers. The number of groups N can be set to any value equal to or greater than 2.

[0051] 12 is an explanatory diagram showing sample images grouped into N groups. The j-th group GP_j includes multiple sample images SD_j_1 to SD_j_Mk, where j_Mk is the number of known classes in the j-th machine learning model 200_j shown in FIG. 9.

[0052] The N groups of sample images thus grouped are used to create training data for N machine learning models 200_1 to 200_N in steps S120 and S130 of Fig. 10. That is, patch images are created from the sample images in step S120, and labels are assigned to the patch images in step S130, thereby creating a first training data group for known classes. The processing details of steps S120 and S130 are the same as those in the first embodiment.

[0053] Fig. 13 is an explanatory diagram showing a first training data group made up of patch images extracted from the sample image shown in Fig. 12 in the second embodiment. The j-th first training data group TD_j includes a plurality of patch image groups PD_j_1 to PD_j_Mk. Each of the plurality of patch image groups PD_j_1 to PD_j_Mk includes a plurality of patch images. The first patch image group PD_1_1 is extracted from the first sample image SD_1_1 shown in Fig. 12. The same applies to the other patch image groups.

[0054] In step S145, the learning execution unit 112 creates a second set of teacher data for a dummy class using the teacher data set of the known classes. In the classification system shown in Figure 9, the number of dummy classes in each machine learning model 200_j is one, and a second set of teacher data is created for the dummy class Dummy_j_1. Note that if the number m of dummy classes is 2 or more, a second set of teacher data is created for each dummy class.

[0055] FIG. 14 is an explanatory diagram showing a method for creating a second teacher data group for a dummy class in the second embodiment. The second teacher data group Dummy_PD_1_1 for the dummy class in the first machine learning model 200_1 is data selected from the N-1 first teacher data groups TD_2 to TD_N for the other N-1 machine learning models 200_2 to 200_N. The second teacher data group Dummy_PD_1_1 is selected so as to include a portion of each of the N-1 first teacher data groups TD_2 to TD_N. This selection is preferably performed randomly. To ensure that the second teacher data group Dummy_PD_1_1 always includes a portion of each of the N-1 first teacher data groups TD_2 to TD_N, it is preferable to set the number of patch images constituting the second teacher data group Dummy_PD_1_1 to a number sufficiently larger than N. The second teacher data group Dummy_PD_j_1 for the dummy class in the jth machine learning model 200_j other than the first one is similarly data selected from the N-1 first teacher data groups for the other N-1 machine learning models, and is selected to include a portion of each of those N-1 first teacher data groups.

[0056] When the number m of dummy classes is 2 or more, the teacher data for those dummy classes can be created in a similar manner. That is, the second teacher data group for the m dummy classes of each machine learning model 200 is data selected from the N-1 first teacher data groups for the other N-1 machine learning models, and is selected so as to include a portion of each of the N-1 first teacher data groups.

[0057] By creating the second teacher data group Dummy_PD_j_1 for the dummy class as described above, each machine learning model 200_j can determine that classified data that should be determined to be a known class in other machine learning models is a dummy class. As a result, it is possible to prevent erroneous determination of a data as a known class in two or more machine learning models, thereby improving the accuracy of the classification process in the classification system as a whole. The processing from step S150 onwards in Figure 10 is almost the same as in the first embodiment, so its description will be omitted.

[0058] Fig. 15 is a flowchart showing the processing procedure of the classification step in the second embodiment. The only difference from the classification step in the first embodiment shown in Fig. 8 is that step S235 is added between step S230 and step S240, and that processing from step S280 onwards is added; the other steps are almost the same as those in Fig. 8.

[0059] In step S235, the class discrimination unit 320 selects one of the N machine learning models 200_1 to 200_N. In steps S240 to S270, the class classification process is executed using the selected one machine learning model.

[0060] In step S280, class discrimination unit 320 determines whether or not the processing of steps S235 to S270 has been completed for all N machine learning models 200. If not completed, the processing returns to step S235, and if completed, the processing proceeds to step S300.

[0061] In step S235, the classified data may be classified into one of the groups using the same criteria as in step S115 of Fig. 10, and a machine learning model using the training data of that group may be selected. In this case, step S280 is omitted.

[0062] In step S300, the class discrimination unit 320 determines whether the classification result of any of the N machine learning models 200 is a known class. If a classification result of a known class exists, in step S310, it is determined that the classified data corresponds to that known class. On the other hand, if the classification results of the N machine learning models are all dummy classes, or if the similarity is less than the threshold, in step S320, the class of the classified data is determined to be unknown. In step S330, the classification result of the classified data is output.

[0063] As described above, in the second embodiment, similar to the first embodiment, the class classification process is performed using a machine learning model 200 having known classes and dummy classes, thereby improving the accuracy of the class classification process. Furthermore, similar to the first embodiment, increasing the variation in training data can be expected to improve the ability of the machine learning model 200 to distinguish between input data. In particular, in the second embodiment, as shown in FIG. 9 , when classifying data into a large number of known classes using N machine learning models 200, the accuracy of class classification can be improved. Furthermore, in the second embodiment, as shown in FIG. 14 , the second teacher data group for the dummy class of each machine learning model 200 is selected from N−1 first teacher data groups for the other N−1 machine learning models 200, thereby easily creating the second teacher data group for the dummy class.

[0064] C. Third embodiment: Fig. 16 is an explanatory diagram showing the class configuration of a machine learning model in the third embodiment. The device configuration of the classification system in the third embodiment is almost the same as that shown in Fig. 1. As shown in Fig. 16, the classification system of the third embodiment also includes N machine learning models 200_1 to 200_N, similar to the second embodiment, where N is an integer of 2 or greater.

[0065] The j-th machine learning model 200_j has j_M classes. These j_M classes are composed of j_Mk known classes and N-1 dummy classes. That is, the judgment values ​​Class_j_1 to Class_j_M of the j_M classes are composed of the judgment values ​​Known_j_1 to Known_j_Mk of the j_Mk known classes and the judgment values ​​Dummy_j_1 to Dummy_j_N-1 of the N-1 dummy classes. Here, the total number of classes j_M is 2 or more, and the number of known classes j_Mk is 1 or more. Since N is an integer of 2 or more, the number N-1 of dummy classes is 1 or more. In a typical example, N is 3 or more, and the number N-1 of dummy classes is 2 or more.

[0066] FIG. 17 is an explanatory diagram showing a method for creating a second teacher data group for a dummy class in the third embodiment. Here, only a method for creating a second teacher data group Dummy_PD_1 for a dummy class in the first machine learning model 200_1 is shown. The second teacher data group Dummy_PD_1 includes N-1 data groups Dummy_PD_1_2 to Dummy_PD_1_N selected from the N-1 first teacher data groups TD_2 to TD_N for the other N-1 machine learning models 200_2 to 200_N. These N-1 data groups Dummy_PD_1_2 to Dummy_PD_1_N are selected to correspond to the N-1 first teacher data groups TD_2 to TD_N. Selection from each of the first teacher data groups is preferably performed randomly. Similarly, the second teacher data group Dummy_PD_j_1 for the dummy class in the j-th machine learning model 200_j other than the first one is selected from the N-1 first teacher data groups for the other N-1 machine learning models 200.

[0067] The preparation process and classification process in the third embodiment are almost the same as those in the second embodiment described with reference to FIGS. 10 and 15, and therefore will not be described here.

[0068] The above-described third embodiment also has substantially the same effects as the above-described second embodiment. Furthermore, in the third embodiment, as shown in Fig. 17 , the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models 200 are selected to correspond to the N-1 first teacher data groups for the other N-1 machine learning models 200, so that the second teacher data groups for the dummy classes of the N machine learning models 200 can be easily created.

[0069] D. Fourth embodiment: Fig. 18 is an explanatory diagram showing a method for creating a second group of training data for dummy classes in the fourth embodiment. The configuration of the classification system in the fourth embodiment is the same as that of the third embodiment shown in Fig. 16, and includes N machine learning models 200_1 to 200_N. Also, as shown in Fig. 16, the j_M classes of the j-th machine learning model 200_j are composed of j_Mk known classes and N-1 dummy classes, which is the same as in the third embodiment.

[0070] FIG. 18 shows only a method for creating the second training data group Dummy_PD_1 for the dummy class in the first machine learning model 200_1. In the fourth embodiment, before creating the second training data group, an average sample image SD_j_ave is created by averaging j_Mk sample images SD_j_1 to SD_j_Mk that make up each sample image group GP_j. As a result, N average sample images SD_1_ave to SD_N_ave are obtained. Note that the sample image group GP_j is the same as that shown in FIG. 12 and is the image group used to extract the first training data group for the known class of the machine learning model 200_j. In other words, each piece of first training data included in the first training data group is data extracted from sample data that is larger in size than the first training data.

[0071] The second training data set for the dummy class, Dummy_PD_1, is the N-1 average sample images SD_2_ave to SD_N_ave corresponding to the other N-1 machine learning models 200_2 to 200_N. These N-1 data groups Dummy_PD_1_2 to Dummy_PD_1_N are extracted so as to correspond to the N-1 average sample images SD_1_ave to SD_N_ave. A second teacher data group Dummy_PD_j for the dummy class in the j-th machine learning model 200_j other than the first one is also created in the same way.

[0072] The preparation process and classification process in the fourth embodiment are almost the same as those in the second embodiment described with reference to FIGS. 10 and 15, and therefore will not be described here.

[0073] The above-described fourth embodiment also has substantially the same effects as the above-described second or third embodiment. Furthermore, in the fourth embodiment, as shown in Fig. 18 , the N-1 second teacher data groups for the N-1 dummy classes of each machine learning model 200 are extracted from an average sample image obtained by averaging the sample image groups used to create the first teacher data groups for the other N-1 machine learning models 200. Therefore, the second teacher data groups for the dummy classes of the N machine learning models 200 can be easily created.

[0074] The second to fourth embodiments described above have in common that the second group of training data for the dummy class in the j-th machine learning model 200_j is created using the first group of training data or sample images for the known classes in the other N-1 machine learning models. In this way, classified data determined to belong to a known class by any one machine learning model is determined to belong to a dummy class by the other N-1 machine learning models. Therefore, it is possible to reduce the possibility of the data being erroneously determined to belong to a known class in two or more machine learning models, thereby improving the accuracy of the class classification process.

[0075] In addition, in the experiments conducted by the inventors, of the second to fourth embodiments described above, the second embodiment is the most superior in terms of processing speed. In terms of classification accuracy, the third embodiment is the most superior, followed by the second embodiment. The user can select any of the embodiments taking these performances into consideration.

[0076] E. Fifth embodiment: Fig. 19 is a flowchart showing the processing procedure of the preparation step of the machine learning model in the fifth embodiment. The only difference from the preparation step in the second embodiment shown in Fig. 10 is that step S142 is added between step S130 and step S145, and the other steps are almost the same as in Fig. 10.

[0077] In the fifth embodiment, the number of dummy classes in the N machine learning models 200 and the method for creating the second teacher data group for the dummy classes are determined according to a specification by the user in step S142. In step S142, the learning execution unit 112 receives a performance preference regarding the performance of the machine learning model 200 from the user.

[0078] FIG. 20 is an explanatory diagram showing a screen WD for selecting a performance preference. This screen WD has two options CH1 for "emphasis on speed" and CH2 for "emphasis on accuracy" as performance options for the machine learning model 200. The user can select either of these two options CH1 or CH2. For example, if the user selects the "emphasis on speed" option CH1, the learning execution unit 112 selects the configuration and processing of the second embodiment described in FIGS. 9 to 15 above to realize a classification system with excellent processing speed. On the other hand, if the user selects the "emphasis on accuracy" option CH2, the learning execution unit 112 selects the configuration and processing of the third embodiment described in FIGS. 16 to 17 above to realize a classification system with excellent processing accuracy. In this way, an appropriate number of dummy classes and a method for creating a second training data group can be selected according to a performance preference indicating which of performances, such as processing speed or processing accuracy, should be prioritized.

[0079] The number of dummy classes corresponding to the two options CH1 and CH2 and the method of creating the second teacher data group for the dummy classes are not limited to those in the second and third embodiments described above, and other methods may be adopted. Furthermore, options other than the two options CH1 and CH2 described above may be used as options selectable by performance preference. For example, performance with a good balance between processing speed and processing accuracy may be provided as another option.

[0080] F. Class Similarity Calculation Method: As a method for calculating the above-mentioned class-specific similarity, for example, any of the following methods can be adopted. (1) A first calculation method M1 for calculating class-specific similarity without considering the correspondence between the feature spectrum Sp and the subregion Rn in the known feature spectrum group GKSp. (2) A second calculation method M2 for calculating the class-specific similarity between the feature spectrum Sp and the corresponding subregion Rn of the known feature spectrum group GKSp. (3) A third calculation method M3 for calculating class-specific similarity without considering the subregion Rn at all Below, methods for calculating class-specific similarities from the output of the ConvVN2 layer 250 according to these calculation methods M1, M2, and M3 will be described in order.

[0081] Fig. 21 is an explanatory diagram showing a first calculation method M1 of class-specific similarity. In the first calculation method M1, first, a local similarity S(i, k) indicating the similarity to each class i for each subregion k is calculated from the output of the ConvVN2 layer 250, which is a specific layer, according to a formula described below. In the machine learning model 200 of Fig. 2, the number of subregions R250 in the ConvVN2 layer 250 is 16, so the subregion parameter k takes a value from 1 to 16. From these local similarities S(i, k), one of three types of class-specific similarities Sclass(i) shown on the right side of Fig. 21 is calculated.

[0082] In the first calculation method M1, the local similarity S(i,k) is calculated using the following formula: S(i,k)=max[G{Sp(k), KSp(i,k=all,q=all)}] (C1) where: i is a parameter indicating the class, k is a parameter indicating the subregion Rn, q is a parameter indicating the data number, G{a,b} is a function that calculates the similarity between a and b. Sp(k) is the feature spectrum obtained from the output of a specific subregion k of a specific layer according to the data to be classified. KSp(i, k=all, q=all) is the known feature spectrum of all data numbers q in all subregions k of a specific layer associated with class i in the known feature spectrum group GKSp shown in FIG. 7; max[X] is a logical operation that takes the maximum value of X. As the function G{a, b} for calculating the similarity, for example, an equation for calculating cosine similarity or an equation for calculating similarity according to distance can be used.

[0083] The three types of class-specific similarities Sclass(i) shown on the right side of Fig. 21 are obtained by taking the maximum, average, or minimum value of the local similarities S(i,k) for multiple partial regions k for each class i. Which of the maximum, average, or minimum values ​​to use depends on the purpose of the classification process. Which of these three types of calculations to use is set in advance by the user experimentally or empirically.

[0084] As described above, in the first calculation method M1 of class-specific similarity, (1) Calculating a local similarity S(i, k) between a feature spectrum Sp obtained from the output of a specific subregion k in a specific layer and all known feature spectra Ksp associated with that specific layer and each class i according to the data to be classified; (2) For each class i, the class-specific similarity Sclass(i) is calculated by taking the maximum, average, or minimum value of the local similarities S(i,k) for multiple subregions k. According to this first calculation method M1, the class-specific similarity Sclass(i) can be calculated using relatively simple calculations and procedures.

[0085] 22 is an explanatory diagram showing a second calculation method M2 of class similarity. In the second calculation method M2, the local similarity S(i, k) is calculated using the following equation instead of the above-mentioned equation (C1). S(i,k)=max[G{Sp(k), KSp(i,k,q=all)}] (C2) where: KSp(i, k, q=all) is the known feature spectrum of all data numbers q in a specific partial region k of a specific layer associated with class i, among the known feature spectrum group GKSp shown in FIG.

[0086] While the first calculation method M1 described above uses known feature spectra KSp(i, k=all, q=all) in all partial regions k of a specific layer, the second calculation method M2 uses only known feature spectra KSp(i, k, q=all) for the partial region k that is the same as the partial region k of the feature spectrum Sp(k). The other aspects of the second calculation method M2 are the same as those of the first calculation method M1.

[0087] In the second calculation method M2 of the class-specific similarity, (1) According to the data to be classified, a local similarity S(i, k) is calculated, which is the similarity between the feature spectrum Sp obtained from the output of a specific subregion k of a specific layer and all known feature spectra Ksp associated with the specific subregion k of the specific layer and each class i; (2) For each class i, the class-specific similarity Sclass(i) is calculated by taking the maximum, average, or minimum value of the local similarities S(i,k) for multiple subregions k. This second calculation method M2 also makes it possible to find the class-specific similarity Sclass(i) through relatively simple calculations and procedures.

[0088] 23 is an explanatory diagram showing a third calculation method M3 of class-specific similarity. In the third calculation method M3, class-specific similarity Sclass(i) is calculated from the output of ConvVN2 layer 250, which is a specific layer, without calculating local similarity S(i, k).

[0089] The class-specific similarity Sclass(i) obtained by the third calculation method M3 is calculated using the following formula. Sclass(i)=max[G{Sp(k=all), KSp(i,k=all,q=all)}] (C3) where: Sp(k=all) is a feature spectrum obtained from the output of all partial regions k of a specific layer according to the data to be classified. The class-specific similarity Sclass(i) given by equation (C3) is substantially the same as the class-specific similarity S(Class) given by equation (A2) described in the first embodiment.

[0090] As described above, in the third calculation method M3 of class-specific similarity, (1) The class-specific similarity Sclass(i), which is the similarity between all feature spectra Sp obtained from the output of a specific layer according to the data to be classified and all known feature spectra KSp associated with that specific layer and each class i, is calculated for each class. According to the third calculation method M3, the class-specific similarity Sclass(i) can be calculated using even simpler calculations and procedures.

[0091] The three calculation methods M1 to M3 described above are all methods for calculating class-specific similarities using the output of one specific layer. However, class-specific similarities can be calculated using one or more of the vector neuron layers 240, 250, and 260 shown in Fig. 2 as specific layers. When multiple specific layers are used, it is preferable to use, for example, the minimum value of the multiple class-specific similarities obtained from the multiple specific layers as the final similarity.

[0092] G. How to calculate the output vector of each layer of the machine learning model: The calculation method for the output of each layer shown in Figure 2 is as follows.

[0093] Each node in the PrimeVN layer 230 regards the scalar output of the 1x1x32 nodes in the Conv layer 220 as a 32-dimensional vector and obtains the vector output of that node by multiplying this vector by a transformation matrix. This transformation matrix is ​​an element of a kernel with a surface size of 1x1, and is updated by learning the machine learning model 200. Note that the processing of the Conv layer 220 and the PrimeVN layer 230 can also be integrated into one primary vector neuron layer.

[0094] When the PrimeVN layer 230 is referred to as the "lower layer L" and the ConvVN1 layer 240 adjacent to it on the upper side is referred to as the "upper layer L+1", the output of each node in the upper layer L+1 is determined using the following equation.

number

[0095] As the normalization function F(X), for example, the following formula (E3a) or (E3b) can be used.

number

[0096] In the above equation (E3a), the sum vector u j Norm of |u j The activation value a is normalized by the softmax function | j On the other hand, in equation (E3b), the sum vector u j Norm of |u j | is the norm |u j Activation value a by dividing by the sum of | j It should be noted that a function other than equation (E3a) or (E3b) may be used as the normalization function F(X).

[0097] The ordinal number i in the above equation (E2) is the output vector M of the jth node in the upper layer L+1. L+1 j The integer n is assigned for convenience to the nodes in the lower layer L used to determine the output vector M L+1 j is the number of nodes in the lower layer L used to determine . Thus, the integer n is given by n = Nk × Nc (E5) Here, Nk is the surface size of the kernel, and Nc is the number of channels in the lower layer, the PrimeVN layer 230. In the example of FIG. 2, Nk=9 and Nc=16, so n=144.

[0098] One kernel used to calculate the output vector of the ConvVN1 layer 240 has a surface size of 3 × 3 and a depth of 16 channels in the lower layer, with 3 × 3 × 16 = 144 elements. Each of these elements is a prediction matrix W L ij In addition, 12 sets of this kernel are required to generate output vectors for 12 channels of the ConvVN1 layer 240. Therefore, the prediction matrix W of the kernel used to obtain the output vector of the ConvVN1 layer 240 is L ij The number of prediction matrices W is 144 × 12 = 1728. L ij is updated by learning of the machine learning model 200.

[0099] As can be seen from the above equations (E1) to (E4), the output vector M of each node in the upper layer L+1 L+1 j is calculated by the following calculation: (a) Output vector M of each node in the lower layer L L i The prediction matrix W L ij Multiplying by the predicted vector v ij Seeking (b) Prediction vector v obtained from each node in the lower layer L ij The sum vector u is a linear combination of j Seeking (c) Sum vector u j Norm of |u j The activation value a is normalized by normalizing | j Seeking (d) Sum vector u j norm |u j Divide by | and then use the activation value a j Multiply by.

[0100] In addition, the activation value a j is the norm |u jis the normalization factor obtained by normalizing |. Therefore, the activation value a j can be considered as an index showing the relative output strength of each node among all nodes in the upper layer L+1. The norm used in equations (E3), (E3a), (E3b), and (4) is typically the L2 norm, which represents the vector length. In this case, the activation value a j is the output vector M L+1 j The activation value a corresponds to the vector length of j is only used in the above equations (E3) and (E4), and does not need to be output from the node. However, the activation value a j It is also possible to configure the upper layer L+1 so that it outputs

[0101] The configuration of a vector neural network is almost the same as that of a capsule network, and the vector neurons of a vector neural network correspond to the capsules of a capsule network. However, the calculations according to the above formulas (E1) to (E4) used in a vector neural network are different from the calculations used in a capsule network. The biggest difference between the two is that in a capsule network, the predicted vector v on the right side of the above formula (E2) ij are multiplied by weights, and the weights are searched by repeating dynamic routing multiple times. On the other hand, in the vector neural network of this embodiment, the output vector M is calculated by calculating the above-mentioned equations (E1) to (E4) once in order. L+1 j Therefore, there is no need to repeat dynamic routing, which has the advantage of allowing faster calculations. In addition, the vector neural network of this embodiment has the advantage that it requires less memory for calculations than a capsule network, and according to experiments by the inventors of this disclosure, it only requires about 1 / 2 to 1 / 3 of the memory required.

[0102] Vector neural networks are similar to capsule networks in that they use nodes that use vectors as input and output. Therefore, they share the advantages of using vector neurons with capsule networks. Furthermore, the multiple layers 220-260 are similar to conventional convolutional neural networks in that the higher layers represent features of larger areas and the lower layers represent features of smaller areas. Here, "feature" refers to a characteristic part contained in the input data to the neural network. Vector neural networks and capsule networks are superior to conventional convolutional neural networks in that the output vector of a node contains spatial information representing the spatial information of the feature represented by that node. That is, the vector length of a node's output vector represents the probability of the feature represented by that node, and the vector direction represents spatial information such as the direction and scale of the feature. Therefore, the vector direction of the output vectors of two nodes belonging to the same layer represents the relative positions of the respective features. Alternatively, the vector direction of the output vectors of the two nodes can be said to represent the variation of the feature. For example, for a node corresponding to the "eye" feature, the direction of the output vector can represent variations such as the narrowness of the eyes or the way they are lifted. In conventional convolutional neural networks, it is said that spatial information of features is lost due to the pooling process. As a result, vector neural networks and capsule networks have the advantage of being superior to conventional convolutional neural networks in terms of the performance of identifying input data.

[0103] The advantages of vector neural networks can also be considered as follows. In other words, the advantage of vector neural networks is that the output vectors of nodes represent the features of input data as coordinates in continuous space. Therefore, output vectors can be evaluated such that the closer the vector directions, the more similar the features. Another advantage is that even if the features contained in the input data are not covered by the training data, they can be determined by interpolation. On the other hand, conventional convolutional neural networks have the disadvantage that the features of input data cannot be represented as coordinates in continuous space due to the chaotic compression caused by the pooling process.

[0104] The outputs of each node in the ConvVN2 layer 250 and the ClassVN layer 260 are similarly determined using the above-mentioned equations (E1) to (E4), and therefore detailed explanations are omitted. The ClassVN layer 260, which is the top layer, has a resolution of 1×1 and M channels.

[0105] The output of the ClassVN layer 260 is converted into a plurality of decision values ​​Class_1 to Class_M for a plurality of classes. These decision values ​​are usually normalized by a softmax function. Specifically, for example, the decision value for each class can be obtained by performing the following operation: calculating the vector length of the output vector from the output vector of each node of the ClassVN layer 260, and then normalizing the vector length of each node by a softmax function. As described above, the activation value a obtained by the above formula (E3) is j is the output vector M L+1 j The activation value a at each node in the ClassVN layer 260 is a value corresponding to the vector length of j may be output and used as the judgment value for each class.

[0106] In the above-described embodiment, a vector neural network that determines an output vector by calculating the above equations (E1) to (E4) was used as the machine learning model 200, but instead, a capsule network disclosed in U.S. Pat. No. 5,210,798 or WO 2009 / 083553 may be used.

[0107] Other forms: The present disclosure is not limited to the above-described embodiments and can be realized in various forms without departing from the spirit thereof. For example, the present disclosure can also be realized in the following aspects. The technical features in the above embodiments corresponding to the technical features in each aspect described below can be appropriately replaced or combined to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted.

[0108] (1) According to a first aspect of the present disclosure, there is provided a classification device that performs a classification process on data to be classified using a machine learning model including a vector neural network having multiple vector neuron layers. The classification device includes a classification processing unit that performs the classification process. The machine learning model has one or more known classes and one or more dummy classes, and the machine learning model is trained using a first set of training data for the known classes and a second set of training data for the dummy classes. When a classification result of the data to be classified using the machine learning model is the dummy class, the classification processing unit determines that the class of the data to be classified is unknown. According to this classification device, the accuracy of the classification process can be improved because the classification process is performed using a machine learning model having one or more known classes and one or more dummy classes.

[0109] (2) In the above-mentioned classification device, when N is an integer equal to or greater than 2, the classification device may have N machine learning models, and the classification processing unit may determine that the class of the classified data is unknown if all of the classification results by the N machine learning models are the dummy class. According to this classification device, it is possible to improve the accuracy of classification even when classifying data into a large number of known classes using N machine learning models.

[0110] (3) In the above-mentioned class classification device, the second teacher data group for the dummy class of each of the machine learning models may be data selected from among the N-1 first teacher data groups for the other N-1 machine learning models, and may be selected to include a portion of each of the N-1 first teacher data groups. This classification device can shorten the processing time required for classification, and can easily create a second training data group for the dummy classes of the N machine learning models.

[0111] (4) In the classification device, each of the N machine learning models may have N-1 dummy classes. The N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models may be data selected from the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups may be selected to correspond to the N-1 first teacher data groups. This classification device can improve the accuracy of classification, and can easily create a second group of training data for the dummy classes of the N machine learning models.

[0112] (5) In the classification device, each of the N machine learning models may have N-1 dummy classes, and each of the first teacher data sets may be data extracted from sample data larger in size than the first teacher data. The N-1 second teacher data sets for the N-1 dummy classes of each of the machine learning models may be data extracted from N-1 average sample data sets obtained by averaging N-1 sample data sets used to extract the N-1 first teacher data sets for the other N-1 machine learning models, and the N-1 second teacher data sets may be extracted to correspond to the N-1 average sample data sets. According to this classification device, it is possible to easily create a second group of training data for the dummy classes of the N machine learning models.

[0113] (6) The above-mentioned classification device may further include a learning execution unit that executes learning of the N machine learning models, and the learning execution unit may receive a performance preference selected by a user regarding the performance of the N machine learning models, and select one of a plurality of options regarding the number of dummy classes in the N machine learning models and the method of creating the second training data group in accordance with the performance preference. This classification device can create appropriate dummy classes according to performance preferences that indicate which of the performances, such as processing speed or processing accuracy, should be prioritized.

[0114] (7) The classification device may further include a memory that stores a group of known feature spectra obtained from the output of a specific layer of the machine learning model when a plurality of pieces of training data are input to the machine learning model. The classification processing unit may be configured to (a) read the group of known feature spectra from the memory, (b) calculate a similarity between the group of known feature spectra and a feature spectrum obtained from the output of the specific layer when the data to be classified is input to the machine learning model, and (c) determine a class of the data to be classified using the similarity. According to this classification device, the class of data to be classified can be determined with high accuracy using the similarity between the feature spectrum and a group of known feature spectra.

[0115] (8) In the above-described classification device, the classification processing unit may determine that the class of the classified data is unknown when the class classification result of the classified data using the machine learning model is the dummy class, or when the similarity is less than a threshold value. This classification device can determine with high accuracy whether the class of data to be classified is unknown or not.

[0116] (9) In the classification device, the specific layer may have a configuration in which vector neurons arranged on a plane defined by two axes, a first axis and a second axis, are arranged as multiple channels along a third axis that is oriented differently from the first and second axes. The feature spectrum may be any one of (i) a first type of feature spectrum in which multiple element values ​​of an output vector of a vector neuron at one planar position in the specific layer are arranged across the multiple channels along the third axis, (ii) a second type of feature spectrum obtained by multiplying each element value of the first type of feature spectrum by an activation value corresponding to the vector length of the output vector, and (iii) a third type of feature spectrum in which the activation values ​​at one planar position in the specific layer are arranged across the multiple channels along the third axis. This classification device makes it possible to easily obtain a characteristic spectrum.

[0117] (10) According to a second aspect of the present disclosure, there is provided a method for performing a classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, the method including: (a) inputting the data to be classified into the machine learning model to obtain a classification result, the machine learning model having one or more known classes and one or more dummy classes, and the machine learning model being trained using a first set of training data for the known classes and a second set of training data for the dummy classes; and (b) determining that the class of the data to be classified is unknown when the classification result is the dummy class.

[0118] (11) According to a third aspect of the present disclosure, there is provided a computer program that causes a processor to execute a class classification process for data to be classified using a machine learning model including a vector neural network having multiple vector neuron layers. The computer program causes the processor to execute: (a) a process of inputting the data to be classified into the machine learning model to obtain a class classification result, the machine learning model having one or more known classes and one or more dummy classes, and the machine learning model being trained using a first set of training data for the known classes and a second set of training data for the dummy classes; and (b) a process of determining that the class of the data to be classified is unknown when the class classification result is the dummy class.

[0119] The present disclosure may be realized in various forms other than those described above, such as a computer program for implementing the functions of a classification device, a non-transitory storage medium on which the computer program is recorded, or the like. [Explanation of symbols]

[0120] 100...information processing device, 110...processor, 112...learning execution unit, 114...classification processing unit, 120...memory, 130...interface circuit, 140...input device, 150...display device, 200...machine learning model, 210...input layer, 220...convolutional layer, 230...primary vector neuron layer, 240...first convolutional vector neuron layer, 250...second convolutional vector neuron layer, 260...classification vector neuron layer, 280...hidden layer, 310...similarity calculation unit, 320...classification discrimination unit, 400...camera

Claims

1. A classification device that performs a classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, a classification processing unit that executes the classification process, the machine learning model has an output layer that outputs one or more judgment values ​​of known classes and one or more judgment values ​​of dummy classes; the machine learning model is trained using a first group of training data for the known class and a second group of training data for the dummy class; the class classification processing unit determines that the class of the classified data is unknown when a class classification result of the classified data using the machine learning model is the dummy class; wherein N is an integer equal to or greater than 2, and the classification device has N machine learning models; the class classification processing unit determines that the class of the data to be classified is unknown when all of the class classification results by the N machine learning models are the dummy class; A class classification device, wherein the second teacher data group for the dummy class of each of the machine learning models is data selected from the N-1 first teacher data groups for the other N-1 machine learning models, and is selected to include a portion of each of the N-1 first teacher data groups.

2. A classification device that performs a classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, a classification processing unit that executes the classification process, the machine learning model has an output layer that outputs one or more judgment values ​​of known classes and one or more judgment values ​​of dummy classes; the machine learning model is trained using a first group of training data for the known class and a second group of training data for the dummy class; the class classification processing unit determines that the class of the classified data is unknown when a class classification result of the classified data using the machine learning model is the dummy class; wherein N is an integer equal to or greater than 2, and the classification device has N machine learning models; the class classification processing unit determines that the class of the data to be classified is unknown when all of the class classification results by the N machine learning models are the dummy class; Each of the N machine learning models has N-1 dummy classes, A class classification device, wherein the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models are data selected from the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups are selected to correspond to the N-1 first teacher data groups.

3. A classification device that performs a classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, a classification processing unit that executes the classification process, the machine learning model has an output layer that outputs one or more judgment values ​​of known classes and one or more judgment values ​​of dummy classes; the machine learning model is trained using a first group of training data for the known class and a second group of training data for the dummy class; the class classification processing unit determines that the class of the classified data is unknown when a class classification result of the classified data using the machine learning model is the dummy class; wherein N is an integer equal to or greater than 2, and the classification device has N machine learning models; the class classification processing unit determines that the class of the data to be classified is unknown when all of the class classification results by the N machine learning models are the dummy class; Each of the N machine learning models has N-1 dummy classes, Each of the first teacher data included in the first teacher data group is data extracted from sample data having a size larger than that of the first teacher data, A class classification device, wherein the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models are data extracted from N-1 average sample data obtained by averaging the N-1 sample data groups used to extract the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups are extracted to correspond to the N-1 average sample data.

4. The classification device according to any one of claims 1 to 3, a memory that stores a group of known characteristic spectra obtained from an output of a specific layer of the machine learning model when a plurality of pieces of training data are input to the machine learning model; The class classification processing unit (a) reading the group of known characteristic spectra from the memory; (b) calculating a similarity between a feature spectrum obtained from an output of the specific layer when the data to be classified is input to the machine learning model and the group of known feature spectra; (c) determining a class of the data to be classified using the similarity; A classifier configured to perform the following:

5. The classification device according to claim 4, the class classification processing unit determines that the class of the classified data is unknown when a class classification result of the classified data using the machine learning model is the dummy class or when the similarity is less than a threshold. Classification device.

6. The classification device according to claim 4 or 5, The specific layer has a configuration in which vector neurons arranged on a plane defined by two axes, a first axis and a second axis, are arranged as a plurality of channels along a third axis in a direction different from the two axes, The characteristic spectrum is (i) a first type of feature spectrum in which a plurality of element values ​​of an output vector of a vector neuron at one plane position in the specific layer are arranged across the plurality of channels along the third axis; (ii) a second type feature spectrum obtained by multiplying each element value of the first type feature spectrum by an activation value corresponding to the vector length of the output vector; (iii) a third type of feature spectrum in which the activation values ​​at one planar position of the specific layer are arranged across the plurality of channels along the third axis; A classifier that is one of the following:

7. 1. A method for performing a classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, comprising: (a) inputting the data to be classified into the machine learning model to obtain a class classification result; (b) determining that the class of the classified data is unknown when the classification result is a dummy class; a computer-implemented method comprising: the machine learning model has an output layer that outputs judgment values ​​of one or more known classes and judgment values ​​of one or more dummy classes, and the machine learning model is trained using a first group of training data for the known classes and a second group of training data for the dummy class; When N is an integer equal to or greater than 2, there are N machine learning models, The step (a) determines that the class of the data to be classified is unknown when all of the class classification results by the N machine learning models are the dummy class; A method in which the second training data group for the dummy class of each of the machine learning models is data selected from among the N-1 first training data groups for the other N-1 machine learning models, and is selected to include a portion of each of the N-1 first training data groups.

8. 1. A method for performing a classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, comprising: (a) inputting the data to be classified into the machine learning model to obtain a class classification result; (b) determining that the class of the classified data is unknown when the classification result is a dummy class; a computer-implemented method comprising: the machine learning model has an output layer that outputs judgment values ​​of one or more known classes and judgment values ​​of one or more dummy classes, and the machine learning model is trained using a first group of training data for the known classes and a second group of training data for the dummy class; When N is an integer equal to or greater than 2, there are N machine learning models, The step (a) determines that the class of the data to be classified is unknown when all of the class classification results by the N machine learning models are the dummy class; Each of the N machine learning models has N-1 dummy classes, A method in which the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models are data selected from the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups are selected to correspond to the N-1 first teacher data groups.

9. 1. A method for performing a classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, comprising: (a) inputting the data to be classified into the machine learning model to obtain a class classification result; (b) determining that the class of the classified data is unknown when the classification result is a dummy class; a computer-implemented method comprising: the machine learning model has an output layer that outputs judgment values ​​of one or more known classes and judgment values ​​of one or more dummy classes, and the machine learning model is trained using a first group of training data for the known classes and a second group of training data for the dummy class; When N is an integer equal to or greater than 2, there are N machine learning models, The step (a) determines that the class of the data to be classified is unknown when all of the class classification results by the N machine learning models are the dummy class; Each of the N machine learning models has N-1 dummy classes, Each of the first teacher data included in the first teacher data group is data extracted from sample data having a size larger than that of the first teacher data, A method in which the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models are data extracted from N-1 average sample data obtained by averaging the N-1 sample data groups used to extract the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups are extracted to correspond to the N-1 average sample data.

10. A computer program that causes a processor to perform a class classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, The computer program comprises: (a) inputting the data to be classified into the machine learning model to obtain a class classification result; (b) determining that the class of the classified data is unknown when the classification result is a dummy class; a computer program that causes the processor to execute the the machine learning model has an output layer that outputs judgment values ​​of one or more known classes and judgment values ​​of one or more dummy classes, and the machine learning model is trained using a first group of training data for the known classes and a second group of training data for the dummy class; When N is an integer equal to or greater than 2, there are N machine learning models, The process (a) determines that the class of the data to be classified is unknown when all of the class classification results by the N machine learning models are the dummy class; A computer program, wherein the second teacher data group for the dummy class of each of the machine learning models is data selected from among the N-1 first teacher data groups for the other N-1 machine learning models, and is selected to include a portion of each of the N-1 first teacher data groups.

11. A computer program that causes a processor to perform a class classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, The computer program comprises: (a) inputting the data to be classified into the machine learning model to obtain a class classification result; (b) determining that the class of the classified data is unknown when the classification result is a dummy class; a computer program that causes the processor to execute the the machine learning model has an output layer that outputs judgment values ​​of one or more known classes and judgment values ​​of one or more dummy classes, and the machine learning model is trained using a first group of training data for the known classes and a second group of training data for the dummy class; When N is an integer equal to or greater than 2, there are N machine learning models, The process (a) determines that the class of the data to be classified is unknown when all of the class classification results by the N machine learning models are the dummy class; Each of the N machine learning models has N-1 dummy classes, A computer program, wherein the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models are data selected from the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups are selected to correspond to the N-1 first teacher data groups.

12. A computer program that causes a processor to perform a class classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, The computer program comprises: (a) inputting the data to be classified into the machine learning model to obtain a class classification result; (b) determining that the class of the classified data is unknown when the classification result is a dummy class; a computer program that causes the processor to execute the the machine learning model has an output layer that outputs judgment values ​​of one or more known classes and judgment values ​​of one or more dummy classes, and the machine learning model is trained using a first group of training data for the known classes and a second group of training data for the dummy class; When N is an integer equal to or greater than 2, there are N machine learning models, The process (a) determines that the class of the data to be classified is unknown when all of the class classification results by the N machine learning models are the dummy class; Each of the N machine learning models has N-1 dummy classes, Each of the first teacher data included in the first teacher data group is data extracted from sample data having a size larger than that of the first teacher data, A computer program, wherein the N-1 second teacher data groups for the N-1 dummy classes of each of the machine learning models are data extracted from N-1 average sample data obtained by averaging the N-1 sample data groups used to extract the N-1 first teacher data groups for the other N-1 machine learning models, and the N-1 second teacher data groups are extracted to correspond to the N-1 average sample data.

Images