Classification device, method, and computer program for performing classification processing using machine learning models

Abstract

To provide a technique for obtaining highly reliable explanatory information.SOLUTION: A classification device executes classification processing for data to be classified using a machine learning model including a vector neural network including a plurality of vector neuron layers. The machine learning model includes an input layer, an intermediate layer, and a first output layer and a second output layer that are branched from the intermediate layer. The first output layer is configured to use a first activation function, and the second output layer is configured to use a second activation function that is different from the first activation function.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a classification device, a method, and a computer program that perform classification processing using a machine learning model. [Background technology]

[0002] Patent Documents 1 and 2 disclose a vector neural network type machine learning model that uses vector neurons, called a capsule network. A vector neuron is a neuron whose input and output are vectors. A capsule network is a machine learning model that uses vector neurons called capsules as network nodes. A vector neural network type machine learning model such as a capsule network can be used to classify input data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 5,210,798 [Patent Document 2] International Publication No. 2019 / 083553 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional techniques, although the results of class classification are output from the machine learning model, the classification basis of the output classes is unclear, and it is particularly difficult to know a reliable classification basis. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a classification device that performs classification processing on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, wherein the machine learning model has an input layer, a hidden layer, and first and second output layers branching from the hidden layer, the first output layer being configured to use a first activation function, and the second output layer being configured to use a second activation function different from the first activation function. The classification device includes a classification processing unit that performs the classification process using the machine learning model, and a memory that stores the machine learning model and a group of known feature spectra obtained from the output of the second output layer when a plurality of training data are input to the machine learning model. The classification processing unit is configured to execute the following steps: (a) reading the machine learning model from the memory; (b) reading the group of known feature spectra from the memory; and (c) using the machine learning model to determine a class of the data to be classified. The step (c) includes: (c1) calculating a similarity between the group of known feature spectra and a feature spectrum obtained from the output of the second output layer when the data to be classified is input to the machine learning model, and creating the similarity as explanatory information regarding the classification result of the data to be classified; (c2) determining the class of the data to be classified based on any of the output of the first output layer, the output of the second output layer, and the similarity; and (c3) displaying the class of the data to be classified and the explanatory information.

[0006] According to a second aspect of the present disclosure, there is provided a method for performing a class classification process on data to be classified using a machine learning model including a vector neural network having multiple vector neuron layers, the method including: (a) reading the machine learning model from a memory, the machine learning model having an input layer, a hidden layer, and first and second output layers branching from the hidden layer, the first output layer configured to use a first activation function, and the second output layer configured to use a second activation function different from the first activation function; (b) reading from the memory a set of known feature spectra obtained from the output of the second output layer when multiple pieces of training data are input to the machine learning model; and (c) determining a class of the data to be classified using the machine learning model. The step (c) includes: (c1) calculating a similarity between a feature spectrum obtained from the output of the second output layer when the classified data is input to the machine learning model and the group of known feature spectra, and creating the similarity as explanatory information for class classification of the classified data; (c2) determining the corresponding class of the classified data according to the output of the first output layer, the output of the second output layer, or the similarity; and (c3) displaying the corresponding class of the classified data and the explanatory information. , which is how a computer executes .

[0007] According to a third aspect of the present disclosure, there is provided a computer program that causes a processor to perform a class classification process for data to be classified using a machine learning model including a vector neural network having multiple vector neuron layers. The computer program causes the processor to (a) read the machine learning model from a memory, the machine learning model having an input layer, a hidden layer, and first and second output layers branching from the hidden layer, the first output layer being configured to use a first activation function, and the second output layer being configured to use a second activation function different from the first activation function; (b) read from the memory a set of known feature spectra obtained from the output of the second output layer when multiple pieces of training data are input to the machine learning model; and (c) determine a class corresponding to the data to be classified using the machine learning model. The process (c) includes: (c1) calculating the similarity between the feature spectrum obtained from the output of the second output layer when the classified data is input to the machine learning model and the group of known feature spectra, and creating the similarity as explanatory information for class classification of the classified data; (c2) determining the corresponding class of the classified data according to the output of the first output layer, the output of the second output layer, or the similarity; and (c3) displaying the corresponding class of the classified data and the explanatory information. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a classification system according to an embodiment. [Figure 2] FIG. 1 is an explanatory diagram showing the configuration of a machine learning model. [Figure 3] FIG. 1 is an explanatory diagram showing the configuration of layers other than the branch output layer of a machine learning model. [Figure 4] 10 is a flowchart showing the processing procedure of a preparation step. [Figure 5] FIG. 10 is an explanatory diagram showing a layer whose parameters are adjusted in step S120. [Figure 6]FIG. 10 is an explanatory diagram showing a layer whose parameters are adjusted in step S130. [Figure 7] FIG. [Figure 8] FIG. 2 is an explanatory diagram showing the configuration of a group of known characteristic spectra. [Figure 9] 10 is a flowchart showing the processing procedure of a classification step. [Figure 10] FIG. 10 is an explanatory diagram showing an example of a display of the results of class classification. [Figure 11] FIG. 10 is an explanatory diagram showing another example of displaying the results of class classification. [Figure 12] FIG. 10 is an explanatory diagram showing a comparison of unknown detection rates with and without a branch output layer. [Figure 13] FIG. 10 is an explanatory diagram showing a method for calculating an unknown detection rate. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. Implementation: FIG. 1 is a block diagram showing a classification system according to an embodiment. The classification system includes an information processing device 100 and a camera 400. The camera 400 is used to capture images of an item to be inspected. The camera 400 may be a camera that captures color images, or a camera that captures monochrome or spectral images. In this embodiment, images captured by the camera 400 are used as training data and classified data, but data other than images may also be used as training data and classified data. In this case, a classified data acquisition device appropriate for the type of data is used instead of the camera 400.

[0010] The information processing device 100 has a processor 110, a memory 120, an interface circuit 130, and an input device 140 and a display device 150 connected to the interface circuit 130. A camera 400 is also connected to the interface circuit 130. For example, but not limited to, the processor 110 not only has the function of executing the processes described in detail below, but also has the function of displaying on the display device 150 data obtained by the processes and data generated in the course of the processes.

[0011] The processor 110 functions as a learning execution unit 112 that executes learning of a machine learning model, and a classification processing unit 114 that executes classification processing of data to be classified. The classification processing unit 114 includes a similarity calculation unit 310 and a class determination unit 320. The learning execution unit 112 and the classification processing unit 114 are each implemented by the processor 110 executing a computer program stored in the memory 120. However, the learning execution unit 112 and the classification processing unit 114 may also be implemented as hardware circuits. The term "processor" in the present disclosure also includes such hardware circuits. Furthermore, one or more processors that execute classification processing may be processors included in one or more remote computers connected via a network.

[0012] The memory 120 stores a machine learning model 200, a training data group TD, and a known feature spectrum group GKSp. The machine learning model 200 is used for processing by the class classification processing unit 114. An example configuration and operation of the machine learning model 200 will be described later. The training data group TD is a collection of labeled data used for training the machine learning model 200. In this embodiment, the training data group TD is a collection of image data. The known feature spectrum group GKSp is a collection of feature spectra obtained when the training data group TD is input to the trained machine learning model 200. The feature spectrum will be described later.

[0013] FIG. 2 is an explanatory diagram showing the configuration of a machine learning model 200. This machine learning model 200 has an input layer 210, a hidden layer 280, and an output layer 290. The hidden layer 280 includes a convolutional layer 220, a primary vector neuron layer 230, a first convolutional vector neuron layer 240, and a second convolutional vector neuron layer 250. The output layer 290 includes a classification vector neuron layer 260 and a branching output layer 270. These two output layers 260 and 270 are configured as layers branched from the hidden layer 280. The branching output layer 270 includes a pre-branching classification vector neuron layer 271 and a post-branching classification vector neuron layer 272. Of these layers, the input layer 210 is the lowest layer, and the output layer 290 is the highest layer. The input layer 210 and the convolutional layer 220 are layers composed of scalar neurons, while the other layers 230, 240, 250, 260, 271, and 272 are layers composed of vector neurons. In the following explanation, the layers constituting the intermediate layer 280 will be referred to as the "Conv layer 220," the "PrimeVN layer 230," the "ConvVN1 layer 240," and the "ConvVN2 layer 250," respectively. The layers 260, 271, and 272 constituting the output layer 290 will be referred to as the "ClassVN layer 260," the "PreBranchedClassVN layer 271," and the "PostBranchedClassVN layer 272," respectively.

[0014] 2, two convolution vector neuron layers 240 and 250 are used, but the number of convolution vector neuron layers is arbitrary, and a convolution vector neuron layer may be omitted. However, it is preferable to use one or more convolution vector neuron layers.

[0015] The ClassVN layer 260 corresponds to the "first output layer" in the present disclosure, and the branch output layer 270 corresponds to the "second output layer." The PreBranchedClassVN layer 271 corresponds to the "pre-layer," and the PostBranchedClassVN layer 272 corresponds to the "post-layer." In this embodiment, the branch output layer 270 is composed of two layers, the pre-layer 271 and the post-layer 272. However, one or more vector neuron layers may be added between these layers 271 and 272. The post-layer 272 may be omitted, and the branch output layer 270 may be composed of only the pre-layer 271. However, configuring the branch output layer 270 to include the post-layer 272 is preferable, since it increases the reliability of the explanatory information obtained from the output of the pre-layer 271.

[0016] The ClassVN layer 260 outputs judgment values ​​Class_0 to Class_Nm-1 for Nm classes for the input classified data. Here, Nm is an integer equal to or greater than 2, typically an integer equal to or greater than 3. Similarly, the PostBranchedClassVN layer 272 outputs judgment values ​​#Class_0 to #Class_Nm-1 for Nm classes. How these two types of judgment values ​​Class_0 to Class_Nm-1 and #Class_0 to #Class_Nm-1 are used will be described later.

[0017] In Figure 2, for vector neuron layers from the ConvVN1 layer 240 onwards, the type of activation function is indicated by hatching. That is, the activation functions of layers 240, 250, and 271 are linear functions shown in the following formula (A1), and the activation functions of layers 260 and 272 are softmax functions shown in the following formula (A2). The activation functions that can be used in each layer will be further described later. Note that activation functions are also called "normalization functions."

number

[0018] Figure 3 is an explanatory diagram showing the configuration of each layer of the machine learning model 200 shown in Figure 2. An image with a size of 32 x 32 pixels is input to the input layer 210. The configuration of each layer other than the input layer 210 can be described as follows. ·Conv layer 220: Conv[32,5,2] ·PrimeVN layer 230: PrimeVN[16,1,1] ·ConvVN1 layer 240:ConvVN1[12,3,1] ·ConvVN2 layer 250: ConvVN2[6,7,2] ·ClassVN layer 260:ClassVN[Nm,3,1] ·PreBranchedClassVN layer 271:PreBranchedClassVN[Nm,3,1] ·PostBranchedClassVN layer 272:PostBranchedClassVN[Nm,1,1] Vector dimension VD: VD=16 In the description of each layer, the character string before the parentheses is the layer name, and the numbers in the parentheses are, in order, the number of channels, the kernel surface size, and the stride. For example, the layer name of the Conv layer 220 is "Conv," the number of channels is 32, the kernel surface size is 5x5, and the stride is 2. In Figure 3, these descriptions are shown below each layer. The hatched rectangles drawn in each layer represent the kernel surface size used when calculating the output vector of the adjacent higher layer. In this embodiment, since the input data is image data, the kernel surface size is also two-dimensional. Note that the parameter values ​​used in the description of each layer are merely examples and can be changed as desired.

[0019] The input layer 210 and the Conv layer 220 are layers composed of scalar neurons. The other layers 230-260, 271, and 272 are layers composed of vector neurons. A vector neuron is a neuron that uses vectors as input and output. In the above description, the dimension of the output vector of each vector neuron is constant at 16. In the following, the term "node" is used as a superordinate concept of scalar neurons and vector neurons.

[0020] FIG. 3 shows the first axis x and second axis y that define the planar coordinates of the node array for the Conv layer 220, and the third axis z that represents depth. It also shows that the sizes of the Conv layer 220 in the x, y, and z directions are 14, 14, and 32. The sizes in the x and y directions are called "resolution." The size in the z direction is the number of channels. These three axes x, y, and z are also used in other layers as coordinate axes that indicate the position of each node. However, in FIG. 3, these axes x, y, and z are omitted from the illustration for layers other than the Conv layer 220.

[0021] As is well known, the resolution W1 after convolution is given by the following equation: W1=Ceil{(W0-Wk+1) / S} (A3) Here, W0 is the resolution before convolution, Wk is the surface size of the kernel, S is the stride, and Ceil{X} is a function that rounds up the decimal point of X. The resolution of each layer shown in FIG. 3 is an example in which the resolution of the input data is 32, and the actual resolution of each layer is changed appropriately depending on the size of the input data.

[0022] The ClassVN layer 260 has Nm channels. Generally, Nm is the number of classes that can be distinguished using the machine learning model 200. Nm is an integer equal to or greater than 2, typically equal to or greater than 3. The Nm channels of the ClassVN layer 260 output judgment values ​​Class_0 to Class_Nm-1 for the Nm classes. Similarly, the Nm channels of the PostBranchedClassVN layer 272 output judgment values ​​#Class_0 to #Class_Nm-1 for the Nm classes. The corresponding class of the classified data can be determined using either the judgment values ​​Class_0 to Class_Nm-1 output from the ClassVN layer 260 or the judgment values ​​#Class_0 to #Class_Nm-1 output from the PostBranchedClassVN layer 272. For example, when the judgment values ​​#Class_0 to #Class_Nm-1 of the PostBranchedClassVN layer 272 are used, the class having the largest value among them is determined to be the corresponding class of the classified data. Furthermore, if the largest value among the judgment values ​​#Class_0 to #Class_Nm-1 is less than a predetermined threshold, it is also possible to determine that the class of the classified data is unknown.

[0023] Instead of using the judgment value of the ClassVN layer 260 or the judgment value of the PostBranchedClassVN layer 272, the class to which the classified data belongs may be determined using the similarity by class calculated from the output of the PreBranchedClassVN layer 271. The similarity by class will be described later.

[0024] FIG. 3 also illustrates subregions Rn in each layer 220, 230, 240, 250, 260, 271, and 272. The subscript "n" in subregion Rn refers to the reference number of the layer. For example, subregion R220 indicates a subregion in the Conv layer 220. A "subregion Rn" is a region in each layer that is identified by a planar position (x, y) defined by the position of the first axis x and the position of the second axis y, and that includes multiple channels along the third axis z. The subregion Rn has dimensions of "Width" × "Height" × "Depth," corresponding to the first axis x, the second axis y, and the third axis z. In this embodiment, the number of nodes included in one "subregion Rn" is "1 × 1 × depth count," i.e., "1 × 1 × number of channels."

[0025] 3, a feature spectrum Sp, which will be described later, is calculated from the output of the PreBrachedClassVN layer 271 and input to the similarity calculation unit 310. The similarity calculation unit 310 uses this feature spectrum Sp and a group of known feature spectra GKSp that has been created in advance to calculate similarities for each class, which will be described later.

[0026] In this disclosure, the vector neuron layer used to calculate the similarity is also referred to as the "specific layer." As the specific layer, a vector neuron layer other than the PreBrachedClassVN layer 271 may be used, and any number of vector neuron layers greater than or equal to one may be used. The configuration of the feature spectrum and the method of calculating the similarity using the feature spectrum will be described later.

[0027] The output of the branching output layer 270 can be used to create explanatory information about the classification results. Information other than the above-described class-specific similarity may be used as explanatory information. For example, the output vector of the PreBrachedClassVN layer 271 itself can be used as explanatory information. However, explanatory information using the above-described similarity has the advantage of being easy for users to understand.

[0028] Fig. 4 is a flowchart showing the processing procedure for preparing a machine learning model. Fig. 5 shows the layer whose internal parameters are adjusted in step S120 of Fig. 4, and Fig. 6 shows the layer whose internal parameters are adjusted in step S130 of Fig. 4.

[0029] In step S110, the user creates a machine learning model to be used in the classification process and sets its parameters. In this embodiment, the machine learning model 200 shown in FIGS. 2 and 3 is created and its parameters are set. Steps S120 to S140 are processes for executing learning of the machine learning model 200 using a training data group TD. Each piece of training data is assigned a label in advance. For example, since the machine learning model 200 has Nm known classes, each piece of training data is assigned one of Nm labels corresponding to the Nm classes.

[0030] In this embodiment, it is assumed that images representing the numbers 0 to 9 are used as training data. Therefore, Nm is 10, and one of the labels 0 to 9 is assigned to each training data.

[0031] In step S120, the learning execution unit 112 executes a predetermined number of epochs using training data to adjust the internal parameters of layers other than the branch output layer 270. The number of epochs in the "predetermined number of epochs" may be, for example, 1 or a multiple value such as 100. In step S120, as shown in FIG. 5, the internal parameters of layers 220, 230, 240, 250, and 260 are adjusted. The "internal parameters" include kernel values ​​for convolution operations. Note that the learning in step S120 may be executed using a division method other than the "predetermined number of epochs." For example, learning may be executed until the value of the loss function decreases by a predetermined percentage or a predetermined width from the value before execution of step S120. Alternatively, learning may be executed until the value of the accuracy increases by a predetermined percentage or a predetermined width from the value before execution of step S120.

[0032] In step S130, the learning execution unit 112 executes a predetermined number of epochs using training data to adjust the internal parameters of the branch output layer 270. The number of epochs executed in step S130 is preferably equal to the number of epochs executed in step S120. In step S130, as shown in FIG. 6, the internal parameters of layers 271 and 272 are adjusted, but the internal parameters of layers 220, 230, 240, 250, and 260 are not changed and remain unchanged.

[0033] In step S140, the learning execution unit 112 determines whether learning has finished. This determination is made, for example, based on whether learning has finished for a predetermined number of epochs. If learning has not finished, the process returns to step S120, and the above-described steps S120 and S130 are executed again. If learning has finished, the process proceeds to the next step, S150. Note that if the number of epochs executed in steps S120 and S130 is sufficiently large, step S140 may be omitted and the process may proceed directly to step S150.

[0034] In step S150, the learning execution unit 112 generates a group of known characteristic spectra GKSp by again inputting a plurality of pieces of training data into the trained machine learning model 200. The group of known characteristic spectra GKSp is a collection of characteristic spectra, which will be described below.

[0035] FIG. 7 is an explanatory diagram showing a feature spectrum Sp obtained by inputting arbitrary input data to the trained machine learning model 200. Here, the feature spectrum Sp obtained from the output of the PreBranchedClassVN layer 271 will be described. The horizontal axis in FIG. 7 is the spectral position represented by a combination of the element number ND of the output vector of a node at one planar position (x, y) of the PreBranchedClassVN layer 271 and the channel number NC. In this embodiment, since the vector dimension of the node is 16, the element numbers ND of the output vector are 16, ranging from 0 to 15. Furthermore, since the number of channels in the PreBranchedClassVN layer 271 is Nm, the channel numbers NC are Nm, ranging from 0 to Nm-1.

[0036] The vertical axis of Fig. 7 shows the feature value C at each spectral position. V In this example, the feature value C V is the value of each element of the output vector V ND Note that the feature value C V As the value of each element of the output vector V ND and the activation value a j Alternatively, the activation value a j In the latter case, the feature value C included in the feature spectrum Sp may be used as it is. V The number of activation values ​​is equal to the number of channels, Nm. j is a value corresponding to the vector length of the output vector of the node.

[0037] The feature spectrum Sp is calculated for each plane position (x, y). The number of feature spectra Sp obtained from the output of the PreBranchedClassVN layer 271 for one input data is equal to the number of plane positions (x, y) of the PreBranchedClassVN layer 271, so it is one.

[0038] The learning execution unit 112 inputs the training data again into the trained machine learning model 200 to calculate the feature spectrum Sp shown in FIG.

[0039] FIG. 8 is an explanatory diagram showing the configuration of the known feature spectrum group GKSp. Each record in the known feature spectrum group GKSp includes a record number, a layer name, a label Lb, and a known feature spectrum KSp. The known feature spectrum KSp is the same as the feature spectrum Sp in FIG. 7 obtained in response to input training data. In the example of FIG. 8, known feature spectra KSp associated with each label Lb value are generated and registered from the output of the PreBranchedClassVN layer 271 in response to multiple training data. For example, #0_max known feature spectra KSp are registered in association with label Lb=0, #1_max known feature spectra KSp are registered in association with label Lb=1, and #Nm-1_max known feature spectra KSp are registered in association with label Lb=Nm-1. #0_max, #1_max, and #Nm-1_max are each an integer greater than or equal to 2. As described above, each label Lb corresponds to a different known class. Therefore, it can be understood that each known characteristic spectrum KSp in the group of known characteristic spectra GKSp is registered in association with one of a plurality of known classes.

[0040] The training data used in step S150 does not have to be the same as the training data used in steps S120 and S130. However, if some or all of the training data used in steps S120 and S130 is used in step S150, there is an advantage in that there is no need to prepare new training data.

[0041] FIG. 9 is a flowchart showing the processing steps of a classification step using a trained machine learning model. In step S210, the classification processing unit 114 generates classified data by photographing the inspection target product using the camera 400. In step S220, the classification processing unit 114 performs preprocessing on the classified data as needed. Preprocessing may include clipping and resolution adjustment. Note that preprocessing is optional. In step S230, the classification processing unit 114 reads the trained machine learning model 200 and the known feature spectrum group GKSp from the memory 120.

[0042] In step S240, the class discrimination unit 320 inputs the classified data into the machine learning model 200 and determines the class to which the classified data belongs. This determination can be made using, for example, either the judgment values ​​Class_0 to Class_Nm-1 output from the ClassVN layer 260 or the judgment values ​​#Class_0 to #Class_Nm-1 output from the PostBranchedClassVN layer 272. As will be described later, it is also possible to determine the class to which the classified data belongs using the similarity for each class.

[0043] In step S250, the classification processing unit 114 uses the output of the PreBranchedClassVN layer 271 to obtain the feature spectrum Sp shown in FIG.

[0044] In step S260, the similarity calculation unit 310 calculates the similarity using the feature spectrum Sp obtained in step S250 and the known feature spectrum group GKSp shown in Fig. 8. As will be described below, the similarity can be either a class-specific similarity or a maximum similarity that does not take class into consideration.

[0045] The similarity S(class) for each class can be calculated using, for example, the following formula. S(Class)=max[G{Sp,KSp(Class,k)}] (A4) Here, "Class" is the ordinal number for the class, G{a,b} is a function for calculating the similarity between a and b, Sp is the feature spectrum obtained from the data to be classified, KSp(Class,k) is all known feature spectra associated with a specific "Class," k is the ordinal number of the known feature spectrum, and max[X] is a logical operation for obtaining the maximum value of X. The function G{a,b} for calculating the similarity can be, for example, a similarity using a distance such as cosine similarity or Euclidean distance. The similarity S(Class) is the maximum similarity calculated between the feature spectrum Sp and all known feature spectra KSp(Class,k) corresponding to a specific class. Such a similarity S(Class) is calculated for each of the Nm classes. The similarity S(Class) represents the degree to which the data to be classified is similar to the characteristics of each class. This similarity S(Class) can be used as explanatory information for the class classification results of the data to be classified.

[0046] The maximum similarity S(All) without taking the class into consideration can be calculated using, for example, the following formula. S(All)=max[G{Sp,KSp(k)}] (A5) Here, KSp(k) denotes the k-th spectrum among all known feature spectra. This maximum similarity S(All) is the maximum value among the similarities between the feature spectrum Sp and all known feature spectra KSp. Since it is possible to identify the known feature spectrum KSp(k) that gives the maximum similarity S(All), it is possible to identify the label, i.e., the class, from the group of known feature spectra GKSp shown in FIG. 8. This maximum similarity S(All) can be used as explanatory information to explain the class classification result, i.e., whether the data to be classified belongs to known data or unknown data.

[0047] Note that the similarity S(Class) for each class represents the degree to which the classified data is similar to the characteristics of each class, so the class to which the classified data belongs may be determined using the similarity S(Class) for each class. For example, if the similarity S(Class) for a certain class is equal to or greater than a predetermined threshold, it can be determined that the classified data belongs to that class. On the other hand, if the similarity S(Class) for all classes is less than the threshold, it can be determined that the classified data is unknown. Alternatively, the maximum similarity S(All) may be used to determine the class to which the classified data belongs.

[0048] Furthermore, instead of determining the class of the classified data using only the similarity, the class of the classified data may be determined using the similarity and one of the judgment values ​​Class_0 to Class_Nm-1 of the ClassVN layer 260 or the judgment values ​​#Class_0 to #Class_Nm-1 of the PostBranchedClassVN layer 272. For example, if the class determined from the similarity matches the class determined from the judgment values ​​#Class_0 to #Class_Nm-1 of the PostBranchedClassVN layer 272, it can be determined that the classified data belongs to that class. Furthermore, if the class determined from the similarity does not match the class determined from the judgment values ​​#Class_0 to #Class_Nm-1 of the PostBranchedClassVN layer 272, it can be determined that the classified data belongs to an unknown class.

[0049] In step S270, the classification processing unit 114 displays the similarity as explanatory information on the display device 150 together with the class to which the classified data corresponds. As the similarity, either the above-mentioned class-specific similarity S(Class) or the maximum similarity S(All) can be used. Below, an example will be described in which the class-specific similarity S(Class) is used as explanatory information.

[0050] FIG. 10 is an explanatory diagram showing an example of a display of the results of class classification. The result display window WD displays an image of the classified data GF, the classification result RF, and explanatory information XF. In this example, the classification result RF is the number "6." The explanatory information XF displays, in a bar graph, the similarity S (Class) values ​​for the labels 0 to 9 corresponding to the numbers 0 to 9, i.e., for each of the classes 0 to 9. The similarity for label 6 is sufficiently large compared to the similarities for the other labels, so the user can understand from the explanatory information XF that the classification result RF is reliable. The example in FIG. 10 also displays the threshold Th used when determining the corresponding class using the similarity.

[0051] 11 is an explanatory diagram showing another example of a display of class classification results. In this example, the classification result RF of the classified data is "unknown." The similarities shown in the explanatory information XF are sufficiently small for all labels, so the user can understand from this explanatory information XF that the classification result RF of "unknown" is reliable.

[0052] 12 is an explanatory diagram comparing the unknown detection rates with and without the branch output layer 270. Here, for a hypothetical model in which the branch output layer 270 is omitted from the machine learning model 200 shown in FIG. 3, the unknown detection rate for correctly determining an unknown is shown when classifying unknown data using similarity using the feature spectrum obtained from the output of the ClassVN layer 260. For the machine learning model 200 with the branch output layer 270, the unknown detection rate for correctly determining an unknown is shown when classifying unknown data using similarity using the feature spectrum obtained from the output of the ClassVN layer 260 and the PreBranchedClassVN layer 271.

[0053] FIG. 13 is an explanatory diagram showing a method for calculating the unknown detection rate. The horizontal axis of FIG. 13 represents similarity, and the vertical axis represents frequency. In this process, the mean μ and variance σ of similarity are calculated for test data belonging to a known class, and μ-2σ is used as the threshold Th. Test data with a similarity below the threshold Th is then determined to be unknown, and test data with a similarity equal to or greater than the threshold Th is determined to be known. In this case, the proportion of test data belonging to an unknown class that was correctly determined to be unknown is calculated as the unknown detection rate.

[0054] 12, in the machine learning model 200 provided with the branch output layer 270, the similarity calculated from the output of the PreBranchedClassVN layer 271 is more reliable than the similarity calculated from the output of the ClassVN layer 260 of a machine learning model that does not have the branch output layer 270. Therefore, by providing the branch output layer 270, more reliable explanation information can be generated.

[0055] In general, the softmax function is suitable as an activation function for the output layer of a neural network that performs class classification. However, because the softmax function emphasizes intensity differences and compresses information, it also distorts and compresses the feature spectra in the output layer, tending to reduce the reliability of the explanatory information. Therefore, when the softmax function is used as the activation function for the ClassVN layer 260, which is the first output layer of the machine learning model 200, it is preferable to use an activation function other than the softmax function as the activation function for the PreBranchedClassVN layer 271. This makes it possible to generate highly reliable explanatory information using the output of the PreBranchedClassVN layer 271. Furthermore, because the softmax function emphasizes differences and compresses information, the layer before the layer using the softmax function tends to generate rich information that can withstand compression, which in turn tends to improve the reliability of the explanatory information. Therefore, by branching and creating a second output layer, the reliability of the explanatory information in the layer before the original first output layer can be maintained.

[0056] In the above-described embodiment, a softmax function is used as the activation function of the ClassVN layer 260, and a linear function is used as the activation function of the PreBranchedClassVN layer 271. However, the PreBranchedClassVN layer 271 may be configured to use an activation function different from that used in the ClassVN layer 260, and other activation functions may be used as the activation functions of the two layers 260 and 271. Even in this case, explanation information regarding the classification results can be generated using one of the two layers 260 and 271. Examples of other activation functions that can be used include the identity function, step function, sigmoid function, tanh function, softplus function, ReLU, Leaky ReLU, Parametric ReLU, ELU, SELU, Swish function, and Mish function.

[0057] As described above, in this embodiment, in addition to the ClassVN layer 260, which is the first output layer, the branch output layer 270, which is the second output layer, is provided, and the second output layer uses an activation function different from that of the first output layer, so that highly reliable explanation information for class classification can be created using one of the first output layer or the second output layer. Also, in this embodiment, the similarity by class between the feature spectrum obtained from the output of the branch output layer 270, which is the second output layer, and the group of known feature spectra can be used as highly reliable explanation information.

[0058] B. How to calculate the output vector of each layer of the machine learning model: The calculation method for the output of each layer shown in Figure 3 is as follows.

[0059] Each node in the PrimeVN layer 230 regards the scalar output of the 1x1x32 nodes in the Conv layer 220 as a 32-dimensional vector and obtains the vector output of that node by multiplying this vector by a transformation matrix. This transformation matrix is ​​an element of a kernel with a surface size of 1x1, and is updated by learning the machine learning model 200. Note that the processing of the Conv layer 220 and the PrimeVN layer 230 can also be integrated into one primary vector neuron layer.

[0060] When the PrimeVN layer 230 is referred to as the "lower layer L" and the ConvVN1 layer 240 adjacent to it on the upper side is referred to as the "upper layer L+1", the output of each node in the upper layer L+1 is determined using the following equation.

number

[0061] As the normalization function F(X), for example, the following formula (E3a) or (E3b) can be used.

number

[0062] In the above equation (E3a), the sum vector u j Norm of |u j The activation value a is normalized by the softmax function | j On the other hand, in equation (E3b), the sum vector u j Norm of |u j | is the norm |u j Activation value a by dividing by the sum of | j The formulas (E3a) and (E3b) are the same as the formulas (A2) and (A1) described above. Note that a function other than the formulas (E3a) and (E3b) may be used as the normalization function F(X).

[0063] The ordinal number i in the above equation (E2) is the output vector M of the jth node in the upper layer L+1. L+1 j The integer n is assigned for convenience to the nodes in the lower layer L used to determine the output vector M L+1 j is the number of nodes in the lower layer L used to determine . Thus, the integer n is given by n = Nk × Nc (E5) Here, Nk is the surface size of the kernel, and Nc is the number of channels in the lower layer, the PrimeVN layer 230. In the example of FIG.

[0064] One kernel used to calculate the output vector of the ConvVN1 layer 240 has a surface size of 3 × 3 and a depth of 16 channels in the lower layer, with 3 × 3 × 16 = 144 elements. Each of these elements is a prediction matrix W L ij In addition, 12 sets of this kernel are required to generate output vectors for 12 channels of the ConvVN1 layer 240. Therefore, the prediction matrix W of the kernel used to obtain the output vector of the ConvVN1 layer 240 isL ij The number of prediction matrices W is 144 × 12 = 1728. L ij is updated by learning of the machine learning model 200.

[0065] As can be seen from the above equations (E1) to (E4), the output vector M of each node in the upper layer L+1 L+1 j is calculated by the following calculation: (a) Output vector M of each node in the lower layer L L i The prediction matrix W L ij Multiplying by the predicted vector v ij Seeking (b) Prediction vector v obtained from each node in the lower layer L ij The sum vector u is a linear combination of j Seeking (c) Sum vector u j Norm of |u j The activation value a is normalized by normalizing | j Seeking (d) Sum vector u j norm |u j Divide by | and then use the activation value a j Multiply by.

[0066] In addition, the activation value a j is the norm |u j is the normalization factor obtained by normalizing |. Therefore, the activation value a j can be considered as an index showing the relative output strength of each node among all nodes in the upper layer L+1. The norm used in equations (E3), (E3a), (E3b), and (4) is typically the L2 norm, which represents the vector length. In this case, the activation value a j is the output vector M L+1 j The activation value a corresponds to the vector length of jis only used in the above equations (E3) and (E4), and does not need to be output from the node. However, the activation value a j It is also possible to configure the upper layer L+1 so that it outputs

[0067] The configuration of a vector neural network is almost the same as that of a capsule network, and the vector neurons of a vector neural network correspond to the capsules of a capsule network. However, the calculations according to the above formulas (E1) to (E4) used in a vector neural network are different from the calculations used in a capsule network. The biggest difference between the two is that in a capsule network, the predicted vector v on the right side of the above formula (E2) ij are multiplied by weights, and the weights are searched by repeating dynamic routing multiple times. On the other hand, in the vector neural network of this embodiment, the output vector M is calculated by calculating the above-mentioned equations (E1) to (E4) once in order. L+1 j Therefore, there is no need to repeat dynamic routing, which has the advantage of allowing faster calculations. In addition, the vector neural network of this embodiment has the advantage that it requires less memory for calculations than a capsule network, and according to experiments by the inventors of this disclosure, it only requires about 1 / 2 to 1 / 3 of the memory required.

[0068] Vector neural networks are similar to capsule networks in that they use nodes that use vectors as input and output. Therefore, they share the advantages of using vector neurons with capsule networks. Furthermore, the multiple layers 220-260, 270 are similar to conventional convolutional neural networks in that the higher layers represent features of larger areas and the lower layers represent features of smaller areas. Here, "feature" refers to a characteristic part contained in the input data to the neural network. Vector neural networks and capsule networks are superior to conventional convolutional neural networks in that the output vector of a node contains spatial information representing the spatial information of the feature represented by that node. That is, the vector length of a node's output vector represents the probability of the feature represented by that node, and the vector direction represents spatial information such as the direction and scale of the feature. Therefore, the vector direction of output vectors of two nodes belonging to the same layer represents the relative positions of their respective features. Alternatively, the vector direction of the output vectors of the two nodes can be said to represent the variation of the feature. For example, for a node corresponding to the feature of "eyes," the direction of the output vector can represent variations such as the narrowness of the eyes or how they are lifted. In conventional convolutional neural networks, it is said that spatial information of features is lost during the pooling process. As a result, vector neural networks and capsule networks have the advantage of being superior to conventional convolutional neural networks in their ability to distinguish input data.

[0069] The advantages of vector neural networks can also be considered as follows. In other words, the advantage of vector neural networks is that the output vectors of nodes represent the features of input data as coordinates in continuous space. Therefore, output vectors can be evaluated such that the closer the vector directions, the more similar the features. Another advantage is that even if the features contained in the input data are not covered by the training data, they can be determined by interpolation. On the other hand, conventional convolutional neural networks have the disadvantage that the features of input data cannot be represented as coordinates in continuous space due to the chaotic compression caused by the pooling process.

[0070] The outputs of each node in the ConvVN2 layer 250 and the ClassVN layer 260 are similarly determined using the above-mentioned formulas (E1) to (E4), and therefore detailed explanations are omitted. The ClassVN layer 260, which is the top layer, has a resolution of 1x1 and Nm channels. The outputs of each node in the PreBranchedClassVN layer 271 and PostBranchedClassVN layer 272 that constitute the branch output layer 270 are similarly determined using the above-mentioned formulas (E1) to (E4).

[0071] The output of the ClassVN layer 260 is converted into a plurality of decision values ​​Class_0 to Class_Nm-1 for known classes. These decision values ​​are usually normalized by a softmax function. Specifically, for example, the decision value for each class can be obtained by calculating the vector length of the output vector from the output vector of each node of the ClassVN layer 260, and then normalizing the vector length of each node by a softmax function. As described above, the activation value a obtained by the above formula (E3) is j is the output vector M L+1 j The activation value a at each node in the ClassVN layer 260 is a value corresponding to the vector length of jmay be output and used as the judgment value for each class as is. The same applies to the judgment values ​​#Class_0 to #Class_Nm-1 of the PostBranchedClassVN layer 272.

[0072] In the above-described embodiment, a vector neural network that determines an output vector by calculating the above equations (E1) to (E4) was used as the machine learning model 200, but instead, a capsule network disclosed in U.S. Pat. No. 5,210,798 or WO 2009 / 083553 may be used.

[0073] Other forms: The present disclosure is not limited to the above-described embodiments and can be realized in various forms without departing from the spirit thereof. For example, the present disclosure can also be realized in the following aspects. The technical features in the above embodiments corresponding to the technical features in each aspect described below can be appropriately replaced or combined to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted.

[0074] (1) According to a first aspect of the present disclosure, there is provided a classification device that performs classification processing on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, wherein the machine learning model has an input layer, a hidden layer, and first and second output layers branching from the hidden layer, the first output layer being configured to use a first activation function, and the second output layer being configured to use a second activation function different from the first activation function. According to this classification device, the second output layer uses an activation function different from that of the first output layer, so that highly reliable explanatory information for class classification can be created using either the first output layer or the second output layer.

[0075] (2) In the above classification device, the first activation function may be a softmax function. According to this classification device, highly reliable explanation information can be generated by using a second output layer that uses a second activation function different from the softmax function.

[0076] (3) In the above classification device, the pre-layer may be configured to use the second activation function, and the post-layer may be configured to use the softmax function. This classification device can generate reliable explanation information using the pre-layer, and the post-layer uses a softmax function, which allows for efficient training of the second output layer.

[0077] (4) The above-mentioned classification device may include a classification processing unit that performs the classification process using the machine learning model, and a memory that stores the machine learning model and a group of known feature spectra obtained from the output of the second output layer when a plurality of teacher data are input to the machine learning model, and the class classification processing unit may be configured to perform the following processes: (a) a process of reading the machine learning model from the memory; (b) a process of reading the group of known feature spectra from the memory; and (c) a process of determining the class of the data to be classified using the machine learning model. The process (c) may include: (c1) calculating a similarity between a feature spectrum obtained from the output of the second output layer when the classified data is input to the machine learning model and the group of known feature spectra, and creating the similarity as explanatory information regarding the class classification result of the classified data; (c2) determining the corresponding class of the classified data according to the output of the first output layer, the output of the second output layer, or the similarity; and (c3) displaying the corresponding class of the classified data and the explanatory information. According to this classification device, the similarity for each class between the feature spectrum obtained from the output of the second output layer and the group of known feature spectra can be used as highly reliable explanatory information.

[0078] (5) In the above classification device, a specific layer included in the second output layer may have a configuration in which vector neurons arranged on a plane defined by two axes, a first axis and a second axis, are arranged as multiple channels along a third axis that is oriented in a direction different from the first and second axes. The feature spectrum may be any one of (i) a first type feature spectrum in which multiple element values ​​of an output vector of a vector neuron at one planar position in the specific layer are arranged across the multiple channels along the third axis, (ii) a second type feature spectrum obtained by multiplying each element value of the first type feature spectrum by an activation value corresponding to the vector length of the output vector, and (iii) a third type feature spectrum in which the activation values ​​at one planar position in the specific layer are arranged across the multiple channels along the third axis. This classification device makes it possible to easily obtain a characteristic spectrum.

[0079] (6) According to a second aspect of the present disclosure, there is provided a method for performing a class classification process on data to be classified using a machine learning model including a vector neural network having multiple vector neuron layers, the method including: (a) reading the machine learning model from a memory, the machine learning model having an input layer, a hidden layer, and first and second output layers branching from the hidden layer, the first output layer configured to use a first activation function, and the second output layer configured to use a second activation function different from the first activation function; (b) reading from the memory a set of known feature spectra obtained from the output of the second output layer when multiple pieces of training data are input to the machine learning model; and (c) determining a class of the data to be classified using the machine learning model. The step (c) includes: (c1) calculating a similarity between the feature spectrum obtained from the output of the second output layer when the classified data is input to the machine learning model and the group of known feature spectra, and creating the similarity as explanatory information for class classification of the classified data; (c2) determining the corresponding class of the classified data according to either the output of the first output layer, the output of the second output layer, or the similarity; and (c3) displaying the corresponding class of the classified data and the explanatory information. According to this method, the similarity between the feature spectrum obtained from the output of the second output layer and the group of known feature spectra by class can be used as highly reliable explanatory information.

[0080] (7) According to a third aspect of the present disclosure, there is provided a computer program causing a processor to perform a class classification process for data to be classified using a machine learning model including a vector neural network having multiple vector neuron layers. The computer program causes the processor to (a) read the machine learning model from a memory, the machine learning model having an input layer, a hidden layer, and first and second output layers branching from the hidden layer, the first output layer being configured to use a first activation function, and the second output layer being configured to use a second activation function different from the first activation function; (b) read from the memory a set of known feature spectra obtained from the output of the second output layer when multiple pieces of training data are input to the machine learning model; and (c) determine a class corresponding to the data to be classified using the machine learning model. The process (c) includes: (c1) calculating the similarity between the feature spectrum obtained from the output of the second output layer when the classified data is input to the machine learning model and the group of known feature spectra, and creating the similarity as explanatory information for class classification of the classified data; (c2) determining the corresponding class of the classified data according to the output of the first output layer, the output of the second output layer, or the similarity; and (c3) displaying the corresponding class of the classified data and the explanatory information.

[0081] The present disclosure may be realized in various forms other than those described above, such as a computer program for implementing the functions of a classification device, a non-transitory storage medium on which the computer program is recorded, or the like. [Explanation of symbols]

[0082] 100...information processing device, 110...processor, 112...learning execution unit, 114...classification processing unit, 120...memory, 130...interface circuit, 140...input device, 150...display device, 200...machine learning model, 210...input layer, 220...convolutional layer, 230...primary vector neuron layer, 240...first convolutional vector neuron layer, 250...second convolutional vector neuron layer, 260...classification vector neuron layer, 270...branching output layer, 271...pre-branching classification vector neuron layer, 272...post-branching classification vector neuron layer, 280...hidden layer, 290...output layer, 310...similarity calculation unit, 320...classification discrimination unit, 400...camera

Claims

1. A classification device that performs a classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, the machine learning model has an input layer, an intermediate layer, and a first output layer and a second output layer branching from the intermediate layer; the first output layer is configured to use a first activation function; the second output layer is configured to use a second activation function different from the first activation function; The classification device a classification processing unit that executes the classification process using the machine learning model; and a memory that stores a group of known feature spectra obtained from the output of the second output layer when a plurality of pieces of training data are input to the machine learning model, and the machine learning model; Equipped with The class classification processing unit (a) reading the machine learning model from the memory; (b) reading the group of known characteristic spectra from the memory; (c) determining a class of the classified data using the machine learning model; is configured to run The process (c) (c1) calculating a similarity between a feature spectrum obtained from the output of the second output layer when the data to be classified is input to the machine learning model and the group of known feature spectra, and creating the similarity as explanatory information regarding the class classification result of the data to be classified; (c2) determining the corresponding class of the data to be classified according to any one of the output of the first output layer, the output of the second output layer, and the similarity; (c3) a process of displaying the corresponding class of the classified data and the explanatory information; A classifier including:

2. The classification device according to claim 1 , The first activation function is a softmax function.

3. The classification device according to claim 2, the second output layer includes a bottom pre-layer and a top post-layer; The pre-layer is configured to use the second activation function, and the post-layer is configured to use the softmax function.

4. The classification device according to claim 1 , The specific layer included in the second output layer has a configuration in which vector neurons arranged on a plane defined by two axes, a first axis and a second axis, are arranged as a plurality of channels along a third axis in a direction different from the two axes, The characteristic spectrum is (i) a first type of feature spectrum in which a plurality of element values ​​of an output vector of a vector neuron at one plane position in the specific layer are arranged across the plurality of channels along the third axis; (ii) a second type feature spectrum obtained by multiplying each element value of the first type feature spectrum by an activation value corresponding to the vector length of the output vector; (iii) a third type of feature spectrum in which the activation values ​​at one planar position of the specific layer are arranged across the plurality of channels along the third axis; A classifier that is one of the following:

5. 1. A method for performing a classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, comprising: (a) reading the machine learning model from a memory, the machine learning model having an input layer, an intermediate layer, and a first output layer and a second output layer branching from the intermediate layer, the first output layer being configured to use a first activation function, and the second output layer being configured to use a second activation function different from the first activation function; (b) reading from the memory a group of known characteristic spectra obtained from the output of the second output layer when a plurality of pieces of training data are input to the machine learning model; (c) using the machine learning model to determine the class of the classified data; Including, The step (c) (c1) calculating a similarity between a feature spectrum obtained from the output of the second output layer when the data to be classified is input to the machine learning model and the group of known feature spectra, and creating the similarity as explanatory information regarding the class classification result of the data to be classified; (c2) determining the corresponding class of the data to be classified according to any one of the output of the first output layer, the output of the second output layer, and the similarity; (c3) displaying the corresponding class of the classified data and the explanatory information; A computer-implemented method, including:

6. A computer program that causes a processor to perform a class classification process on data to be classified using a machine learning model including a vector neural network having a plurality of vector neuron layers, The computer program comprises: (a) a process for reading the machine learning model from a memory, the machine learning model having an input layer, an intermediate layer, and a first output layer and a second output layer branching from the intermediate layer, the first output layer being configured to use a first activation function, and the second output layer being configured to use a second activation function different from the first activation function; (b) reading from the memory a group of known characteristic spectra obtained from the output of the second output layer when a plurality of pieces of training data are input to the machine learning model; (c) determining a class of the classified data using the machine learning model; causing the processor to execute The process (c) (c1) calculating a similarity between a feature spectrum obtained from the output of the second output layer when the data to be classified is input to the machine learning model and the group of known feature spectra, and creating the similarity as explanatory information regarding the class classification result of the data to be classified; (c2) determining the corresponding class of the data to be classified according to any one of the output of the first output layer, the output of the second output layer, and the similarity; (c3) a process of displaying the corresponding class of the classified data and the explanatory information; A computer program comprising:

Images