Method for performing classification processing using machine learning model, information processing device, and computer program

Abstract

To provide a technique for executing highly accurate class classification.SOLUTION: A method includes the steps of: (a) generating N pieces of input data from one target object; (b) inputting the input data to a machine learning model and obtaining M classification output values, one determination class, and a feature spectrum; (c) obtaining a similarity degree between a known feature spectrum group and the feature spectrum for the input data, and obtaining a reliability degree with respect to the determination class as a function of the similarity degree; and (d) executing a vote for the determination class, based on the reliability degree with respect to the determination class, and determining a class determination result of the target object, based on a result of the vote.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to a method, an information processing device, and a computer program for performing a class classification process using a machine learning model. [Background technology]

[0002] Patent Documents 1 and 2 disclose a vector neural network type machine learning model that uses vector neurons, called a capsule network. A vector neuron is a neuron whose input and output are vectors. A capsule network is a machine learning model that uses vector neurons called capsules as network nodes. A vector neural network type machine learning model such as a capsule network can be used to classify input data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 5,210,798 [Patent Document 2] International Publication No. 2019 / 083553 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, the classification result for input data that should be determined as unknown may be erroneously determined as a known class, and therefore there has been a demand for improving the accuracy of the classification process. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, there is provided a method for performing a classification process for M classes using a machine learning model including a vector neural network having a plurality of vector neuron layers, where M is an integer equal to or greater than 2. The method includes the steps of: (a) generating N pieces of input data from one object, where N is an integer equal to or greater than 2; and (b) inputting each of the N pieces of input data into the machine learning model to generate a classification process for M classes. 、 (c) determining, for each of the N pieces of input data, one discriminant class and a feature spectrum obtained from the output of a specific layer of the machine learning model; (d) determining a similarity between a group of known feature spectra obtained from the output of the specific layer when a plurality of training data are input to the machine learning model and each of the feature spectra for the N pieces of input data, and determining a confidence level for the discriminant class for each of the N pieces of input data as a function of the similarity; and (d) voting for the discriminant class for each of the N pieces of input data in accordance with the confidence level for the discriminant class, and determining a class discrimination result for the object in accordance with the result of the voting. The step (c) includes any one of the steps of: (1) setting the similarity as the confidence level; (2) calculating the confidence level by multiplying the similarity level, the classification output value for the discrimination class, and a non-zero positive coefficient; and (3) calculating the confidence level by performing a weighted addition of the similarity level and the classification output value for the discrimination class. The specific layer has a configuration in which vector neurons arranged on a virtual plane defined by two axes, a first axis and a second axis, are arranged as multiple channels along a third axis that is oriented differently from the first two axes. The feature spectrum is one of (i) a first type of feature spectrum in which multiple element values ​​of an output vector of a vector neuron at one planar position in the specific layer are arranged across the multiple channels along the third axis, (ii) a second type of feature spectrum obtained by multiplying each element value of the first type of feature spectrum by an activation value corresponding to the vector length of the output vector, and (iii) a third type of feature spectrum in which the activation values ​​at one planar position in the specific layer are arranged across the multiple channels along the third axis.

[0006] According to a second aspect of the present disclosure, there is provided an information processing device that performs a classification process for M classes using a machine learning model including a vector neural network having a plurality of vector neuron layers, where M is an integer equal to or greater than 2. The information processing device includes a memory that stores the machine learning model, and a processor that executes calculations using the machine learning model. The processor (a) reads out N pieces of input data created from one object from the memory, where N is an integer equal to or greater than 2, and (b) inputs each of the N pieces of input data into the machine learning model to generate a classification result. 、(c) determining, for each of the N pieces of input data, one discriminant class and a feature spectrum obtained from the output of a specific layer of the machine learning model; (d) determining, for each of the N pieces of input data, a similarity between a group of known feature spectra obtained from the output of the specific layer when a plurality of pieces of training data are input to the machine learning model and each of the feature spectra for the N pieces of input data, and determining, for each of the N pieces of input data, a confidence level for the discriminant class as a function of the similarity; and (d) performing voting for the discriminant class for each of the N pieces of input data in accordance with the confidence level for the discriminant class, and determining a class discrimination result for the object in accordance with the result of the voting. The process (c) includes any one of the following: (1) setting the similarity as the confidence; (2) calculating the confidence by multiplying the similarity, the classification output value for the discrimination class, and a non-zero positive coefficient; and (3) calculating the confidence by performing a weighted addition of the similarity and the classification output value for the discrimination class. The specific layer has a configuration in which vector neurons arranged on a virtual plane defined by two axes, a first axis and a second axis, are arranged as multiple channels along a third axis that is oriented differently from the first two axes. The feature spectrum is one of (i) a first type of feature spectrum in which multiple element values ​​of an output vector of a vector neuron at one planar position in the specific layer are arranged across the multiple channels along the third axis, (ii) a second type of feature spectrum obtained by multiplying each element value of the first type of feature spectrum by an activation value corresponding to the vector length of the output vector, and (iii) a third type of feature spectrum in which the activation values ​​at one planar position in the specific layer are arranged across the multiple channels along the third axis.

[0007] According to a third aspect of the present disclosure, there is provided a computer program that causes a processor to execute a classification process for M classes using a machine learning model including a vector neural network having a plurality of vector neuron layers, where M is an integer equal to or greater than 2. The computer program includes: (a) a process of reading, from a memory, N pieces of input data created from one object, where N is an integer equal to or greater than 2; and (b) a process of inputting each of the N pieces of input data into the machine learning model to generate a classification process for M classes. 、 (c) determining, for each of the N pieces of input data, one discriminant class and a feature spectrum obtained from the output of a specific layer of the machine learning model; (d) determining, for each of the N pieces of input data, a similarity between a group of known feature spectra obtained from the output of the specific layer when a plurality of pieces of training data are input to the machine learning model and each of the feature spectra for the N pieces of input data, and determining, for each of the N pieces of input data, a confidence level for the discriminant class as a function of the similarity; and (d) voting for the discriminant class for each of the N pieces of input data in accordance with the confidence level for the discriminant class, and determining a class discrimination result for the object in accordance with the result of the voting. The process (c) includes any one of the following: (1) setting the similarity as the confidence; (2) calculating the confidence by multiplying the similarity, the classification output value for the discrimination class, and a non-zero positive coefficient; and (3) calculating the confidence by performing a weighted addition of the similarity and the classification output value for the discrimination class. The specific layer has a configuration in which vector neurons arranged on a virtual plane defined by two axes, a first axis and a second axis, are arranged as multiple channels along a third axis that is oriented differently from the first two axes. The feature spectrum is one of (i) a first type of feature spectrum in which multiple element values ​​of an output vector of a vector neuron at one planar position in the specific layer are arranged across the multiple channels along the third axis, (ii) a second type of feature spectrum obtained by multiplying each element value of the first type of feature spectrum by an activation value corresponding to the vector length of the output vector, and (iii) a third type of feature spectrum in which the activation values ​​at one planar position in the specific layer are arranged across the multiple channels along the third axis. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a classification system according to a first embodiment. [Figure 2] FIG. 1 is an explanatory diagram showing an example of the configuration of a machine learning model. [Figure 3] 10 is a flowchart showing a processing procedure of a preparation step in the embodiment. [Figure 4] FIG. 10 is an explanatory diagram showing how a patch image is created from a sample image. [Figure 5] FIG. [Figure 6] FIG. 2 is an explanatory diagram showing the configuration of a group of known characteristic spectra. [Figure 7] 10 is a flowchart showing the processing procedure of a classification step in the first embodiment. [Figure 8] FIG. 1 is a functional block diagram of a classification processing device according to a first embodiment. [Figure 9] FIG. 10 is an explanatory diagram showing the number of votes in the first embodiment and the comparative example. [Figure 10] FIG. 10 is a functional block diagram of a classification processing device according to a second embodiment. [Figure 11] 11 is a flowchart showing the processing procedure of a classification step in the third embodiment. [Figure 12] FIG. 11 is a functional block diagram of a classification processing device according to a third embodiment. [Figure 13] FIG. 4 is an explanatory diagram showing a first calculation method of similarity. [Figure 14] FIG. 10 is an explanatory diagram showing a second similarity calculation method. [Figure 15] FIG. 10 is an explanatory diagram showing third and fourth similarity calculation methods. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. First embodiment: FIG. 1 is a block diagram showing a classification system according to a first embodiment. The classification system includes an information processing device 100 and a camera 400. The camera 400 is used to capture an image of an object OB. The camera 400 may be a camera that captures color images, or a camera that captures monochrome images or spectral images. In this embodiment, images captured by the camera 400 are used as training data or input data, but data other than images may also be used as training data or input data. In this case, an input data acquisition device appropriate for the type of data is used instead of the camera 400.

[0010] The information processing device 100 has a processor 110, a memory 120, an interface circuit 130, and an input device 140 and a display device 150 connected to the interface circuit 130. A camera 400 is also connected to the interface circuit 130. For example, but not limited to, the processor 110 not only has the function of executing the processes described in detail below, but also has the function of displaying on the display device 150 data obtained by the processes and data generated in the course of the processes.

[0011] The processor 110 functions as a learning execution unit 112 that executes learning of a machine learning model, and a classification processing unit 114 that executes classification processing of input data. The classification processing unit 114 includes a similarity calculation unit 310, a confidence calculation unit 320, and a voting execution unit 330. The learning execution unit 112 and the classification processing unit 114 are each implemented by the processor 110 executing a computer program stored in the memory 120. However, the learning execution unit 112 and the classification processing unit 114 may also be implemented as hardware circuits. The term "processor" in the present disclosure also includes such hardware circuits. Furthermore, one or more processors that execute the learning processing and the classification processing may be processors included in one or more remote computers connected via a network.

[0012] The memory 120 stores a machine learning model 200, a training data group TD, and a known feature spectrum group GKSp. The machine learning model 200 is used for processing by the class classification processing unit 114. An example configuration and operation of the machine learning model 200 will be described later. The training data group TD is a collection of labeled data used for training the machine learning model 200. In this embodiment, the training data group TD is a collection of image data. The known feature spectrum group GKSp is a collection of feature spectra obtained when training data is input again into the trained machine learning model 200. The feature spectrum will be described later.

[0013] FIG. 2 is an explanatory diagram showing the configuration of a machine learning model 200. This machine learning model 200 has an input layer 210, a hidden layer 280, and an output layer 260. The hidden layer 280 includes a convolutional layer 220, a primary vector neuron layer 230, a first convolutional vector neuron layer 240, and a second convolutional vector neuron layer 250. The output layer 260 is also referred to as a "classification vector neuron layer 260." Of these layers, the input layer 210 is the lowest layer, and the output layer 260 is the highest layer. In the following description, the hidden layer 280 and the output layer 260 are referred to as the "Conv layer 220," the "PrimeVN layer 230," the "ConvVN1 layer 240," the "ConvVN2 layer 250," and the "ClassVN layer 260," respectively.

[0014] 2, two convolution vector neuron layers 240 and 250 are used, but the number of convolution vector neuron layers is arbitrary, and a convolution vector neuron layer may be omitted. However, it is preferable to use one or more convolution vector neuron layers.

[0015] An image of 32×32 pixels is input to the input layer 210. The configuration of each layer other than the input layer 210 can be described as follows. ·Conv layer 220: Conv[32,4,2] ·PrimeVN layer 230: PrimeVN[16,1,1] ·ConvVN1 layer 240:ConvVN1[12,3,2] ·ConvVN2 layer 250: ConvVN2[8,4,1] ·ClassVN layer 260:ClassVN[M,4,1] Vector dimension VD: VD=16 In the description of each layer, the character string before the parentheses is the layer name, and the numbers in the parentheses are, in order, the number of channels, the kernel surface size, and the stride. For example, the layer name of the Conv layer 220 is "Conv," the number of channels is 32, the kernel surface size is 4x4, and the stride is 2. In Figure 2, these descriptions are shown below each layer. The hatched rectangles drawn in each layer represent the kernel surface size used when calculating the output vector of the adjacent higher layer. In this embodiment, since the input data is image data, the kernel surface size is also two-dimensional. Note that the parameter values ​​used in the description of each layer are merely examples and can be changed as desired.

[0016] The input layer 210 and the Conv layer 220 are layers composed of scalar neurons. The other layers 230 to 260 are layers composed of vector neurons. A vector neuron is a neuron that uses vectors as input and output. In the above description, the dimension of the output vector of each vector neuron is constant at 16. In the following, the term "node" is used as a superordinate concept of scalar neurons and vector neurons.

[0017] FIG. 2 shows the first axis x and second axis y that define the planar coordinates of the node array for the Conv layer 220, and the third axis z that represents depth. It also shows that the sizes of the Conv layer 220 in the x, y, and z directions are 15, 15, and 32. The sizes in the x and y directions are called "resolution." The size in the z direction is the number of channels. These three axes x, y, and z are also used in other layers as coordinate axes that indicate the position of each node. However, in FIG. 2, these axes x, y, and z are omitted from the illustration for layers other than the Conv layer 220.

[0018] As is well known, the resolution W1 after convolution is given by the following equation: W1=Ceil{(W0-Wk+1) / S} (A1) Here, W0 is the resolution before convolution, Wk is the surface size of the kernel, S is the stride, and Ceil{X} is a function that rounds up the decimal point of X. The resolution of each layer shown in FIG. 2 is an example in which the resolution of the input data is 32, and the actual resolution of each layer is changed appropriately depending on the size of the input data.

[0019] The ClassVN layer 260 has M channels. M is the number of classes that can be discriminated using the machine learning model 200. In this embodiment, M is an integer equal to or greater than 2. M classification output values ​​Class(1) to Class(M) are output from the M channels of the ClassVN layer 260. The class having the largest value among these classification output values ​​Class(1) to Class(M) is determined to be the discrimination class of the input data. In this embodiment, processing is not performed to determine that the class of the input data is unknown by comparing the largest value of the classification output values ​​Class(1) to Class(M) with a threshold. Therefore, one discrimination class is always determined for one piece of input data.

[0020] FIG. 2 also illustrates subregions Rn in each layer 220, 230, 240, 250, and 260. The subscript "n" in subregion Rn refers to the reference number of the layer. For example, subregion R220 indicates a subregion in the Conv layer 220. A "subregion Rn" is a region in each layer that is identified by a planar position (x, y) defined by the position of the first axis x and the position of the second axis y, and that includes multiple channels along the third axis z. The subregion Rn has dimensions of "Width" × "Height" × "Depth," corresponding to the first axis x, the second axis y, and the third axis z. In this embodiment, the number of nodes included in one "subregion Rn" is "1 × 1 × number of depths," i.e., "1 × 1 × number of channels."

[0021] 2, a feature spectrum Sp, which will be described later, is calculated from the output of the ConvVN2 layer 250 and input to the similarity calculation unit 310. The similarity calculation unit 310 calculates a similarity S(i), which will be described later, using this feature spectrum Sp and a known feature spectrum group GKSp that has been created in advance.

[0022] In this disclosure, the vector neuron layer used to calculate the similarity S(i) is also referred to as the "specific layer." As the specific layer, a vector neuron layer other than the ConvVN2 layer 250 may be used, and any number of vector neuron layers greater than or equal to one may be used. The configuration of the feature spectrum and the method of calculating the similarity using the feature spectrum will be described later.

[0023] 3 is a flowchart showing the processing steps of the machine learning model preparation process. In step S110, the learning execution unit 112 generates a plurality of sample images by photographing a plurality of samples using the camera 400. In step S120, the learning execution unit 112 applies preprocessing to the sample images to create patch images. For example, the preprocessing can include resolution adjustment, data normalization (min-max normalization), and the like.

[0024] FIG. 4 is an explanatory diagram showing how patch images are created from a sample image. Here, multiple small patch images PD are extracted from the sample image SD. This example shows an example in which the sample image SD is divided into multiple patch images PD along the dividing lines indicated by dashed lines. Each of these patch images PD is used as an image for training data. Note that a larger number of patch images PD may be extracted from a single sample image SD by making the stride of the extraction position of the patch image PD smaller than the size of one side of the patch image PD. In the present disclosure, the sample image SD is also referred to as "sample data SD," and the patch image PD is also referred to as "patch data PD." Note that the sample image SD may be used as an image for training data without extracting the patch image PD.

[0025] In this embodiment, multiple patch images PD extracted from one sample image SD are used as a group of images belonging to one class. Since the number of known classes in the machine learning model 200 shown in Fig. 3 is M, M sample images SD are captured in step S110, and multiple patch images PD are extracted from each sample image SD in step S120.

[0026] In step S130, the learning execution unit 112 creates a training data group TD by assigning labels to the patch images PD. In this embodiment, M training data groups are created by assigning one of M labels 1 to M to each patch image PD. These labels correspond to the M classes of the machine learning model 200 shown in FIG. 2. In this disclosure, "label" and "class" mean the same thing.

[0027] In step S140, the learning execution unit 112 uses the training data group TD to execute learning of the machine learning model 200. When the learning is completed, the trained machine learning model 200 is stored in the memory 120.

[0028] In step S150, the learning execution unit 112 generates a group of known characteristic spectra GKSp by again inputting a plurality of pieces of training data into the trained machine learning model 200. The group of known characteristic spectra GKSp is a collection of characteristic spectra, which will be described below.

[0029] FIG. 5 is an explanatory diagram showing a feature spectrum Sp obtained by inputting arbitrary input data to the trained machine learning model 200. As shown in FIG. 2, in this embodiment, the feature spectrum Sp is created from the output of the ConvVN2 layer 250. The horizontal axis in FIG. 5 represents the position of a vector element in the output vector of multiple nodes included in one subregion R250 of the ConvVN2 layer 250. The position of this vector element is represented by a combination of the element number ND of the output vector at each node and the channel number NC. In this embodiment, since the vector dimension is 16 (the number of elements of the output vector output by each node), the element number ND of the output vector is 16, ranging from 0 to 15. Furthermore, since the ConvVN2 layer 250 has eight channels, the channel number NC is 8, ranging from 0 to 7. In other words, this feature spectrum Sp is obtained by arranging multiple element values ​​of the output vector of each vector neuron included in one subregion R250 across multiple channels along the third axis z.

[0030] The vertical axis of Fig. 5 represents the feature value C at each spectral position. V In this example, the feature value C V is the value of each element of the output vector V ND Feature value C V For the feature value C, statistical processing such as centering to the average value 0 may be performed. V As the value of each element of the output vector V ND Alternatively, the normalization coefficient may be used as is. In the latter case, the feature value C included in the feature spectrum Sp is V The number of is equal to the number of channels, which is 8. The normalization coefficient is a value corresponding to the vector length of the output vector of the node.

[0031] The number of feature spectra Sp obtained from the output of the ConvVN2 layer 250 for one piece of input data is 16, since this is equal to the number of planar positions (x, y) of the ConvVN2 layer 250, that is, the number of subregions R250.

[0032] The learning execution unit 112 inputs the training data again into the trained machine learning model 200, calculates the feature spectrum Sp shown in FIG. 5, and registers it in the memory 120 as a group of known feature spectra GKSp.

[0033] 6 is an explanatory diagram showing the configuration of the known feature spectrum group GKSp. In this example, the known feature spectrum group GKSp is shown as being obtained from the output of the ConvVN2 layer 250. Note that it is sufficient that the known feature spectrum group GKSp is obtained from the output of at least one vector neuron layer, and it is also possible to register known feature spectrum groups obtained from the output of the ConvVN1 layer 240 or the ClassVN layer 260.

[0034] Each record in the group of known feature spectra GKSp includes a parameter c indicating a label or class, a parameter k indicating the order of the subregion Rn within the layer, a parameter q indicating a data number, and a known feature spectrum KSp. The known feature spectrum KSp is the same as the feature spectrum Sp in Figure 5.

[0035] The class parameter c takes a value from 1 to M. The subregion Rn parameter k takes a value indicating which of the multiple subregions Rn included in a specific layer it is, i.e., which planar position (x, y) it is. For the ConvVN2 layer 250, there are 16 subregions R250, so k = 1 to 16. The data number parameter q indicates the number of training data with the same label, and takes a value from 1 to max1 for class 1 and from 1 to maxM for class M.

[0036] The plurality of training data used in step S150 does not need to be the same as the plurality of training data used in step S140. However, if some or all of the plurality of training data used in step S140 is used in step S150 as well, there is an advantage in that there is no need to prepare new training data.

[0037] Fig. 7 is a flowchart showing the processing procedure of the class classification process using the trained machine learning model 200, and Fig. 8 is a functional block diagram of the class classification processing device in the first embodiment. Fig. 8 shows the case where the number of classes M is 5, and an example of the class classification result is also shown on the right side of Fig. 8.

[0038] In step S210, the classification processing unit 114 generates N pieces of input data D(i) for one object OB by photographing the object OB using the camera 400. Here, N is an integer equal to or greater than 2, and i is an integer between 1 and N. In the example of FIG. 8, N=275. As described in FIG. 4, the N pieces of input data D(i) can be obtained by photographing a sample image SD of the object OB and extracting N pieces of patch images PD from the sample image SD. Alternatively, instead of extracting patch images PD, the N pieces of input data D(i) may be obtained by photographing the same object OB N times. The classification processing unit 114 may perform the same pre-processing as the pre-processing performed in step S120 of FIG. 3 on the input data D(i).

[0039] In step S220, the class classification processing unit 114 inputs one piece of input data D(i) to the machine learning model 200, obtains M classification output values ​​from the ClassVN layer 260, and determines one discrimination class. The discrimination class is the class that indicates the largest value among the M classification output values. Note that instead of determining the discrimination class from the classification output value, the class with the highest class-specific similarity calculated using the feature spectrum Sp obtained according to the input data D(i) may be determined as the discrimination class. A method for calculating the class-specific similarity will be described later.

[0040] In step S230, the class classification processing unit 114 obtains the feature spectrum Sp shown in Fig. 5 using the output of the ConvVN2 layer 250, which is a specific layer. Fig. 8 shows a case where the number of classes M of the machine learning model 200 is 5, and depicts five classification output values ​​Class(1,i) to Class(5,i) obtained from the machine learning model 200 for each piece of input data D(i), one discriminant class c(i), and a feature spectrum Sp(i). In the following description, the first values ​​"1" to "5" in parentheses of the symbols of the classification output values ​​Class(1,i) to Class(5,i) indicate the class, and the symbol "i" indicates the order i of the input data D(i).

[0041] In step S240, as shown in Fig. 8, the similarity calculation unit 310 calculates the similarity S(i) between the feature spectrum Sp(i) for the input data D(i) and the known feature spectrum group GKSp shown in Fig. 6. The similarity S(i) is an index representing the degree to which the input data D(i) is similar to the features of the discrimination class c(i). The method for calculating the similarity S(i) will be described later.

[0042] In step S250, the reliability calculation unit 320 calculates the reliability R(i) for the discrimination class c(i) from the similarity S(i). The reliability R(i) is calculated, for example, by one of the following formulas. R(i)=S(i) (A2) R(i)=α×S(i)×Class(c,i) (A3) R(i)=β×S(i)+(1-β)×Class(c,i) (A4) where: i is a parameter indicating the order of the input data D(i), c is a parameter indicating the discriminant class c(i) of the input data D(i), S(i) is the similarity between the feature spectrum Sp(i) for the input data D(i) and the known feature spectrum group GKSp. Class(c,i) is the classification output value of the discriminant class c(i) according to the input data D(i), α is a non-zero positive coefficient, β is a coefficient where 0 < β < 1, and is as follows.

[0043] The above equation (A2) shows a function that uses the similarity S(i) itself as the reliability R(i) for the discrimination class c(i). The above equation (A3) shows a function that obtains the reliability R(i) by multiplying the similarity S(i), the classification output value Class(c,i) for the discrimination class c(i), and a non-zero positive coefficient α. The above equation (A4) shows a function that obtains the reliability R(i) by weighted addition of the similarity S(i) and the classification output value Class(c,i) for the discrimination class c(i). By using any of these equations, the reliability R(i) for the discrimination class c(i) can be obtained as a function of the similarity S(i). It is also possible to calculate the reliability R(i) from the similarity S(i) using a function other than the above equations (A2) to (A4). Also in this case, it is preferable to use a function such that the reliability R(i) has a positive correlation with the similarity S(i).

[0044] In steps S260 to S280, the voting execution unit 330 compares the reliability R(i) of the discrimination class c(i) with the reliability threshold Rth and executes voting according to the comparison result. That is, when Rth ≤ R(c,i), in step S270, 1 is added to the vote count of that discrimination class c(i). On the other hand, when R(i) < Rth, in step S280, the vote by the input data D(i) is invalidated. The reliability threshold Rth can be set to, for example, 0.995. In FIG. 8, the result of the voting execution unit 330 counting the vote counts Vn(1) to Vn(5) for the five classes according to the input data D(i) is depicted. The vote count Vn_null is the number of input data D(i) invalidated in step S280.

[0045] In step S290, the classification processing unit 114 determines whether the processing of steps S220 to S280 has ended for all the input data D(i). If the processing for all the input data D(i) has not ended, the process returns to step S220, and the processing of steps S220 to S280 described above is executed for the next input data D(i). If the processing for all the input data D(i) has ended, the process proceeds to step S300.

[0046] In steps S300 to S320, the classification processing unit 114 compares the maximum vote count Vn_max among the vote counts Vn for the M classes with a predetermined vote count threshold Vnth, and determines the final classification result according to the comparison result. That is, when Vnth ≤ Vn_max, in step S310, the class with the maximum vote count Vn_max is determined as the class discrimination result of the object OB. On the other hand, when Vn_max < Vnth, in step S320, the class of the object OB is determined to be unknown. The vote count threshold Vnth can be set, for example, to 15% to 30% of the total vote count N. In the example of FIG. 8, the vote count threshold Vnth is set to 50. Since the maximum vote count Vn_max is the vote count Vn(1) = 100 for class 1, the class discrimination result of the object OB is class 1. The result shown in FIG. 8 is obtained by creating 275 input data D(i) using the same object as the object OB when creating the teacher data for class 1, and using these input data D(i). Therefore, the result shown in FIG. 8 is the correct class discrimination result.

[0047] FIG. 9 is an explanatory diagram showing the number of votes Vn shown in FIG. 8 obtained in the first embodiment and the number of votes Vn' in a comparative example obtained without using the reliability R(i). The number of votes Vn' in the comparative example is the result of voting according to the discrimination class c(i) obtained in step S220 using the same input data D(i) as the number of votes Vn in the first embodiment. In the number of votes Vn' in the comparative example, since class 2 has the most votes, class 2 is the classification result of the object OB, resulting in a misclassification. On the other hand, in the number of votes Vn in the embodiment, class 1 is the classification result of the object OB, resulting in a correct classification. As can be seen from the results of the comparative example, the input data D(i) obtained from the object OB is data that may result in a misclassification if classification is normally performed using the machine learning model 200. Even in such cases, if the discrimination class is determined by voting as in the first embodiment, the class can be correctly classified.

[0048] In step S330, the class classification processing unit 114 outputs the classification result to the display device 150. As the classification result, only the class to which the object OB belongs may be displayed, or the number of votes Vn for each class may be displayed as shown in FIG.

[0049] As described above, in the first embodiment, the reliability R(i) for the discrimination class c(i) is calculated according to the similarity S(i) of the feature spectrum Sp, and the class discrimination result of the object is determined according to the voting result using the reliability R(i), thereby enabling highly accurate class classification. Furthermore, in the first embodiment, when the reliability R(i) is equal to or greater than the reliability threshold Rth, 1 is added to the number of votes Vn for the discrimination class c(i), and when the reliability R(i) is less than the reliability threshold Rth, the vote is invalidated, so that the class discrimination result can be determined according to the number of votes Vn corresponding to the reliability R(i).

[0050] B. Second embodiment: Fig. 10 is a functional block diagram of a classification processing device in the second embodiment. The device configuration of the classification system in the second embodiment is almost the same as that shown in Fig. 1. The processing procedure in the second embodiment is also almost the same as that shown in Figs. 3 and 7.

[0051] As shown in FIG. 10 , the classification system of the second embodiment includes multiple machine learning models 200_1 to 200_3. Each of these machine learning models 200_1 to 200_3 has the same configuration as the machine learning model 200 shown in FIG. 2 . The additional symbols "_1" to "_3" suffixed to the reference symbols of the machine learning models are added to distinguish the three machine learning models. The three machine learning models 200_1 to 200_3 have 5, 3, and 4 classes, respectively, for a total of 12 classes. The three machine learning models 200_1 to 200_3 have substantially the same functions as the single machine learning model 200 of the first embodiment shown in FIG. 2 , where the number of classes M is 12. However, since a large number of classes in a single machine learning model 200 requires a long time for learning, using multiple machine learning models enables faster classification processing. Furthermore, a decrease in classification accuracy can be prevented when the number of distinguishable classes is large. Furthermore, when replacing the training data and retraining the machine learning model, there is no need to retrain the entire model, which has the advantage of speeding up the learning process.

[0052] Input data D(i) is input to three machine learning models 200_1 to 200_3, respectively. The first machine learning model 200_1 outputs classification output values ​​Class(1,i) to Class(5,i) for five classes according to the input data D(i), and calculates a feature spectrum Sp_1(i). Furthermore, one discriminant class c_1(i) is determined from the maximum value of the classification output values ​​Class(1,i) to Class(5,i). For ease of illustration, in FIG. 10, the similarity calculation unit 310 and the reliability calculation unit 320 are depicted as a single block. The dashed-line blocks depicted within the blocks of the similarity calculation unit 310 and the reliability calculation unit 320 indicate that the outputs of the three machine learning models 200_1 to 200_3 are processed separately.

[0053] The second machine learning model 200_2 outputs classification output values ​​Class(6,i) to Class(8,i) for three classes and a discriminant class c_2(i) according to the input data D(i), and also calculates a feature spectrum Sp_2(i). The third machine learning model 200_3 outputs classification output values ​​Class(9,i) to Class(12,i) for four classes and a discriminant class c_3(i) according to the input data D(i), and also calculates a feature spectrum Sp_3(i).

[0054] The similarity calculation unit 310 and the reliability calculation unit 320 calculate the reliability R(c_1,i) for the discrimination class c_1(i) using the classification output values ​​Class(1,i) to Class(5,i) and the feature spectrum Sp_1(i) obtained from the first machine learning model 200_1. The method for calculating the reliability R(c_1,i) is the same as in the first embodiment. Similarly, for the other machine learning models 200_2 and 200_3, the reliability R(c_2,i) for the discrimination class c_2(i) and the reliability R(c_3,i) for the discrimination class c_3(i) are calculated, respectively.

[0055] The voting unit 330 performs voting according to the reliability R(1,i), R(2,i), and R(3,i) obtained for each of the three machine learning models 200_1 to 200_3. As a result, as shown on the right side of FIG. 10, the numbers of votes Vn(1) to Vn(12) for the 12 classes are obtained. The number of votes Vn_1null is the number of input data D(i) for which votes were invalidated in the first machine learning model 200_1. Similarly, the numbers of votes Vn_2null and Vn_2null are the numbers of input data D(i) for which votes were invalidated from the outputs of the machine learning models 200_1 and 200_2. From the numbers of votes Vn(1) to Vn(12) in FIG. 10, the class discrimination result for the object OB is correctly determined to be class 1.

[0056] Alternatively, voting may be performed only on the machine learning model that yields the highest reliability among the reliability ratings R(1,i), R(2,i), and R(3,i). In this case, the total number of votes shown in the rightmost column of FIG. 10 for all machine learning models is 275. Even in this way, the class of the object OB can be correctly determined from the number of votes Vn(1) to Vn(12).

[0057] The second embodiment described above also has the same effects as the first embodiment, and can perform highly accurate class classification. Furthermore, since the second embodiment uses multiple machine learning models, it is possible to perform classification processing at high speed even when the number M of discriminable classes is large, and it is possible to prevent a decrease in classification accuracy.

[0058] C. Third embodiment: Fig. 11 is a flowchart showing the processing procedure of the classification step in the third embodiment, and Fig. 12 is a functional block diagram of a classification processing device in the third embodiment. The device configuration of the classification system in the third embodiment is almost the same as that shown in Fig. 1. In the third embodiment, unlike the first and second embodiments described above, vote values ​​are added up for each class instead of the number of votes, and the classification result is determined according to the addition result.

[0059] The processing procedure of the third embodiment shown in FIG. 11 is the same as that of the first embodiment shown in FIG. 7, except that steps S270, S300, and S310 are replaced with steps S275, S305, and S315, and the other steps are the same as those in FIG.

[0060] If the reliability R(i) of the discrimination class c(i) is equal to or greater than the reliability threshold Rth in step S260, the reliability R(i) is added to the voting value Vv of that discrimination class c(i) in step S275. In the first embodiment shown in FIG. 8, 1 is added to the number of votes Vn, whereas in the third embodiment shown in FIG. 12, the reliability R(i) is added to the voting value Vv. FIG. 12 illustrates how the voting unit 330 calculates the voting values ​​Vv(1) to Vv(5) for five classes according to the input data D(i).

[0061] When adding the voting value Vv, the reliability threshold Rth may be set to the minimum value that the reliability R(i) can take. For example, if the range of possible values ​​for the reliability (c,i) is −1.0 to +1.0, the reliability threshold Rth may be set to −1.0, which is the minimum value. When the reliability threshold Rth is set to the minimum value, all the determinations in step S260 are YES, so the process always proceeds to step S275, where the reliability R(i) is added to the voting value Vv for the discrimination class c(i). This process is essentially equivalent to a process that does not include steps S260 and S280. In other words, the phrase "adding the reliability R(i) as the voting value Vv for the discrimination class c(i) when the reliability R(i) is equal to or greater than the reliability threshold Rth" has a broader meaning that includes "adding the reliability R(i) as the voting value Vv for the discrimination class c(i) without using the reliability threshold Rth." However, if the reliability threshold Rth is set to a value greater than the minimum possible value of the reliability R(i), votes from the discrimination class c(i) with low reliability R(i) can be invalidated, thereby obtaining a more desirable value for the voting value Vv.

[0062] In step S305, the class classification processing unit 114 compares the maximum voting value Vv_max among the voting values Vv for M classes with a predetermined voting value threshold Vvth, and determines the final classification result according to the comparison result. That is, when Vvth ≤ Vv_max, in step S315, the class with the maximum voting value Vv_max is determined as the class discrimination result of the object OB. On the other hand, when Vv_max < Vvth, in step S320, the class of the object OB is determined to be unknown. The voting value threshold Vvth can be set, for example, to 15% - 30% of the total number of votes N. In the example of FIG. 12, the voting value threshold Vvth is set to 50, and the class discrimination result of the object OB is class 1.

[0063] Similar to the first embodiment, in the third embodiment described above, the reliability R(i) for the discrimination class c(i) is obtained according to the similarity S(i) of the feature spectrum Sp, and the class discrimination result of the object is determined according to the voting result using the reliability R(i), so that high-precision class classification can be performed. Also, in the third embodiment, when the reliability R(i) is greater than or equal to the reliability threshold Rth, the reliability R(i) is added to the voting value Vv for the discrimination class c(i), and when the reliability R(i) is less than the reliability threshold Rth, the vote is invalidated, so that the class discrimination result can be determined according to the voting value Vv corresponding to the reliability R(i).

[0064] D. Method for calculating similarity: As the calculation method of the similarity S(i) described above, for example, any of the following methods can be adopted. (1) The first calculation method M1 for obtaining the similarity S(i) for each class without considering the correspondence between the feature spectrum Sp and the partial region Rn in the known feature spectrum group GKSp (2) The second calculation method M2 for obtaining the similarity S(i) for each class between the corresponding partial regions Rn of the feature spectrum Sp and the known feature spectrum group GKSp (3) The third calculation method M3 for obtaining the similarity S(i) for each class without considering the partial region Rn at all (4) A fourth calculation method M4 for calculating the similarity S(i) between the feature spectrum Sp and the known feature spectrum group GKSp without distinguishing classes. Below, methods for calculating the similarity S(i) from the output of the ConvVN2 layer 250 according to these calculation methods M1 to M4 will be described in order.

[0065] FIG. 13 is an explanatory diagram showing a first calculation method M1 of similarity S(i). In the first calculation method M1, first, a local similarity SL(c, k, i), which indicates the similarity to the discriminant class c(i), for each partial region k is calculated from the output of the ConvVN2 layer 250, which is a specific layer, according to a formula described below. In the machine learning model 200 of FIG. 2, the number of partial regions R250 in the ConvVN2 layer 250 is 16, so the partial region parameter k takes a value from 1 to 16. From these local similarities SL(c, k, i), one of the three types of class-specific similarities S(i) shown on the right side of FIG. 13 is calculated.

[0066] In the first calculation method M1, the local similarity SL(c, k, i) is calculated using the following equation: SL(c,k,i)=max[G{Sp(k,i), KSp(c,k=all,q=all)}] (D1) where: c is a parameter indicating the discriminant class c(i), k is a parameter indicating the order of the subregion Rn, i is a parameter indicating the order of the input data D(i), q is a parameter indicating the data number, G{a,b} is a function that calculates the similarity between a and b. Sp(k,i) is the feature spectrum obtained from the output of a specific subregion k of a specific layer according to the input data D(i). KSp(c, k=all, q=all) is the known feature spectrum of all data numbers q in all subregions k of a specific layer associated with class c in the known feature spectrum group GKSp shown in Figure 6. max[X] is a logical operation that takes the maximum value of X. As the function G{a, b} for calculating the similarity, for example, an equation for calculating cosine similarity or an equation for calculating similarity according to distance can be used.

[0067] The three types of similarity S(i) shown on the right side of Fig. 13 are obtained by taking the maximum, average, or minimum value of the local similarity SL(c, k) for multiple partial regions k. Which of the maximum, average, or minimum values ​​to use depends on the purpose of the classification process. Which of these three types of calculations to use is preset by the user experimentally or empirically.

[0068] As described above, in the first calculation method M1 of the similarity S(i), (1) Calculating the local similarity SL(c,k,i) between the feature spectrum Sp(k,i) obtained from the output of a specific subregion k in a specific layer and all known feature spectra KSp associated with that specific layer and class c according to the input data D(i); (2) The similarity S(i) is calculated by taking the maximum, average, or minimum value of the local similarities SL(c,k,i) for a plurality of partial regions k. According to this first calculation method M1, the similarity S(i) for each class can be calculated using relatively simple calculations and procedures.

[0069] 14 is an explanatory diagram showing a second calculation method M2 of the similarity S(i). In the second calculation method M2, the local similarity SL(c, k, i) is calculated using the following equation instead of the above-mentioned equation (D1). SL(c,k,i)=max[G{Sp(k,i), KSp(c,k,q=all)}] (D2) where: KSp(c, k, q=all) is a known feature spectrum of all data numbers q in a specific partial region k of a specific layer associated with class c, among the known feature spectrum group GKSp shown in FIG.

[0070] While the first calculation method M1 described above uses known feature spectra KSp(c,k=all,q=all) in all partial regions k of a specific layer, the second calculation method M2 uses only known feature spectra KSp(c,k,q=all) for the partial region k that is the same as the partial region k of the feature spectrum Sp(k,i). The other aspects of the second calculation method M2 are the same as those of the first calculation method M1.

[0071] In the second calculation method M2 of the similarity S(i), (1) Calculating a local similarity SL(c,k,i) between a feature spectrum Sp(k,i) obtained from the output of a specific subregion k in a specific layer and all known feature spectra KSp associated with the specific subregion k in the specific layer and class c according to input data D(i); (2) The similarity S(i) is calculated by taking the maximum, average, or minimum value of the local similarities SL(c,k,i) for a plurality of partial regions k. This second calculation method M2 also makes it possible to find the similarity S(i) for each class using relatively simple calculations and procedures.

[0072] 15 is an explanatory diagram showing a third calculation method M3 and a fourth calculation method M4 for the similarity S(i). In the third calculation method M3, the similarity S(i) is calculated from the output of the specific layer, ConvVN2 layer 250, without calculating the local similarity SL(c,k,i).

[0073] The similarity S(i) obtained by the third calculation method M3 is calculated using the following formula: S(i)=max[G{Sp(k=all,i), KSp(c,k=all,q=all)}] (D3) where: Sp(k=all, i) is the feature spectrum obtained from the output of all subregions k of a specific layer according to the input data D(i).

[0074] As described above, in the third calculation method M3 of the similarity S(i), (1) The similarity S(i) between all feature spectra Sp obtained from the output of a specific layer according to input data D(i) and all known feature spectra KSp associated with that specific layer and class c is calculated. According to the third calculation method M3, the similarity S(i) for each class can be calculated using even simpler calculations and procedures.

[0075] The three calculation methods M1 to M3 described above are all methods for calculating similarities S(i) for each class. Of the class-specific similarities S(i), the similarity to one discrimination class c(i) is used as the similarity S(i) described in the various embodiments described above. Alternatively, similarities S(i) may be calculated for each of the M classes, and the class with the maximum value may be determined as the discrimination class c(i). In this case, instead of determining the discrimination class c(i) from the classification output value in step S220 of FIGS. 7 and 11, a process of determining one discrimination class c(i) from the class-specific similarities S(i) for the M classes is performed after step S240.

[0076] In the fourth calculation method M4, the similarity S(i) between the feature spectrum Sp and the known feature spectrum KSp is calculated without distinguishing between classes. The similarity S(i) obtained by the fourth calculation method M4 is calculated using the following equation, which is similar to the above equation (D3). S(i)=max[G{Sp(k=all,i), KSp(c=all,k=all,q=all)}] (D4) where: Sp(k=all,i) is the feature spectrum obtained from the output of all subregions k of a specific layer according to the input data D(i), KSp(c=all, k=all, q=all) is the known feature spectrum of all data numbers q in all subregions k of a specific layer for all classes c, among the known feature spectra KSp shown in FIG. Although this fourth calculation method M4 does not take into account the class of the known feature spectrum KSp, the similarity S(i) obtained by the above formula (D4) usually coincides with the similarity S(i) obtained by the above formula (D3) for the discrimination class c(i) determined from the classification output values ​​Class(1,i) to Class(M,i) of the machine learning model 200. Therefore, the fourth calculation method M4 also makes it possible to substantially determine the similarity S(i) for the discrimination class c(i).

[0077] The above-mentioned four calculation methods M1 to M4 are all methods for calculating the similarity S(i) using the output of one specific layer. However, the similarity S(i) can be calculated using one or more of the vector neuron layers 240, 250, and 260 shown in Fig. 2 as specific layers. When multiple specific layers are used, it is preferable to use, for example, the minimum value of the multiple similarities S(i) obtained from the multiple specific layers as the final similarity S(i).

[0078] E. How to calculate the output vector of each layer of the machine learning model: The calculation method for the output of each layer shown in Figure 2 is as follows.

[0079] Each node in the PrimeVN layer 230 regards the scalar output of the 1x1x32 nodes in the Conv layer 220 as a 32-dimensional vector and obtains the vector output of that node by multiplying this vector by a transformation matrix. This transformation matrix is ​​an element of a kernel with a surface size of 1x1, and is updated by learning the machine learning model 200. Note that the processing of the Conv layer 220 and the PrimeVN layer 230 can also be integrated into one primary vector neuron layer.

[0080] When the PrimeVN layer 230 is referred to as the "lower layer L" and the ConvVN1 layer 240 adjacent to it on the upper side is referred to as the "upper layer L+1", the output of each node in the upper layer L+1 is determined using the following equation.

number

[0081] As the normalization function F(X), for example, the following formula (E3a) or (E3b) can be used.

number

[0082] In the above equation (E3a), the sum vector u j Norm of |u j The activation value a is normalized by the softmax function | j On the other hand, in equation (E3b), the sum vector u j Norm of |u j | is the norm |uj Activation value a by dividing by the sum of | j It should be noted that a function other than equation (E3a) or (E3b) may be used as the normalization function F(X).

[0083] The ordinal number i in the above equation (E2) is the output vector M of the jth node in the upper layer L+1. L+1 j The integer n is assigned for convenience to the nodes in the lower layer L used to determine the output vector M L+1 j is the number of nodes in the lower layer L used to determine . Thus, the integer n is given by n = Nk × Nc (E5) Here, Nk is the surface size of the kernel, and Nc is the number of channels in the lower layer, the PrimeVN layer 230. In the example of FIG. 2, Nk=9 and Nc=16, so n=144.

[0084] One kernel used to calculate the output vector of the ConvVN1 layer 240 has a surface size of 3 × 3 and a depth of 16 channels in the lower layer, with 3 × 3 × 16 = 144 elements. Each of these elements is a prediction matrix W L ij In addition, 12 sets of this kernel are required to generate output vectors for 12 channels of the ConvVN1 layer 240. Therefore, the prediction matrix W of the kernel used to obtain the output vector of the ConvVN1 layer 240 is L ij The number of prediction matrices W is 144 × 12 = 1728. L ij is updated by learning of the machine learning model 200.

[0085] As can be seen from the above equations (E1) to (E4), the output vector M of each node in the upper layer L+1 L+1 j is calculated by the following calculation: (a) Output vector M of each node in the lower layer L L i The prediction matrix W L ij Multiplying by the predicted vector v ij Seeking (b) Prediction vector v obtained from each node in the lower layer L ij The sum vector u is a linear combination of j Seeking (c) Sum vector u j Norm of |u j The activation value a is normalized by normalizing | j Seeking (d) Sum vector u j norm |u j Divide by | and then use the activation value a j Multiply by.

[0086] In addition, the activation value a j is the norm |u j is the normalization factor obtained by normalizing |. Therefore, the activation value a j can be considered as an index showing the relative output strength of each node among all nodes in the upper layer L+1. The norm used in equations (E3), (E3a), (E3b), and (4) is typically the L2 norm, which represents the vector length. In this case, the activation value a j is the output vector M L+1 j The activation value a corresponds to the vector length of j is only used in the above equations (E3) and (E4), and does not need to be output from the node. However, the activation value a j It is also possible to configure the upper layer L+1 so that it outputs

[0087] The configuration of a vector neural network is almost the same as that of a capsule network, and the vector neurons of a vector neural network correspond to the capsules of a capsule network. However, the calculations according to the above formulas (E1) to (E4) used in a vector neural network are different from the calculations used in a capsule network. The biggest difference between the two is that in a capsule network, the predicted vector v on the right side of the above formula (E2) ij are multiplied by weights, and the weights are searched by repeating dynamic routing multiple times. On the other hand, in the vector neural network of this embodiment, the output vector M is calculated by calculating the above-mentioned equations (E1) to (E4) once in order. L+1 j Therefore, there is no need to repeat dynamic routing, which has the advantage of allowing faster calculations. In addition, the vector neural network of this embodiment has the advantage that it requires less memory for calculations than a capsule network, and according to experiments by the inventors of this disclosure, it only requires about 1 / 2 to 1 / 3 of the memory required.

[0088] Vector neural networks are similar to capsule networks in that they use nodes that use vectors as input and output. Therefore, they share the advantages of using vector neurons with capsule networks. Furthermore, the multiple layers 220-260 are similar to conventional convolutional neural networks in that the higher layers represent features of larger areas and the lower layers represent features of smaller areas. Here, "feature" refers to a characteristic part contained in the input data to the neural network. Vector neural networks and capsule networks are superior to conventional convolutional neural networks in that the output vector of a node contains spatial information representing the spatial information of the feature represented by that node. That is, the vector length of a node's output vector represents the probability of the feature represented by that node, and the vector direction represents spatial information such as the direction and scale of the feature. Therefore, the vector direction of the output vectors of two nodes belonging to the same layer represents the relative positions of the respective features. Alternatively, the vector direction of the output vectors of the two nodes can be said to represent the variation of the feature. For example, for a node corresponding to the "eye" feature, the direction of the output vector can represent variations such as the narrowness of the eyes or the way they are lifted. In conventional convolutional neural networks, it is said that spatial information of features is lost due to the pooling process. As a result, vector neural networks and capsule networks have the advantage of being superior to conventional convolutional neural networks in terms of the performance of identifying input data.

[0089] The advantages of vector neural networks can also be considered as follows. In other words, the advantage of vector neural networks is that the output vectors of nodes represent the features of input data as coordinates in continuous space. Therefore, output vectors can be evaluated such that the closer the vector directions, the more similar the features. Another advantage is that even if the features contained in the input data are not covered by the training data, they can be determined by interpolation. On the other hand, conventional convolutional neural networks have the disadvantage that the features of input data cannot be represented as coordinates in continuous space due to the chaotic compression caused by the pooling process.

[0090] The outputs of each node in the ConvVN2 layer 250 and the ClassVN layer 260 are similarly determined using the above-mentioned equations (E1) to (E4), and therefore detailed explanations are omitted. The ClassVN layer 260, which is the top layer, has a resolution of 1×1 and M channels.

[0091] The output of the ClassVN layer 260 is converted into a plurality of classification output values ​​Class(1) to Class(M) for a plurality of classes. These classification output values ​​are usually normalized by a softmax function. Specifically, for example, the classification output value for each class can be obtained by performing the following operation: calculating the vector length of the output vector from the output vector of each node of the ClassVN layer 260, and then normalizing the vector length of each node by a softmax function. As described above, the activation value a obtained by the above formula (E3) is j is the output vector M L+1 j The activation value a at each node in the ClassVN layer 260 is a value corresponding to the vector length of j may be output and used as the classification output value for each class.

[0092] In the above-described embodiment, a vector neural network that determines an output vector by calculating the above equations (E1) to (E4) was used as the machine learning model 200, but instead, a capsule network disclosed in U.S. Pat. No. 5,210,798 or WO 2009 / 083553 may be used.

[0093] Other forms: The present disclosure is not limited to the above-described embodiments and can be realized in various forms without departing from the spirit thereof. For example, the present disclosure can also be realized in the following aspects. The technical features in the above embodiments corresponding to the technical features in each aspect described below can be appropriately replaced or combined to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted.

[0094] <1> According to a first aspect of the present disclosure, there is provided a method for performing a classification process for M classes using a machine learning model including a vector neural network having multiple vector neuron layers, where M is an integer equal to or greater than 2. The method includes: (a) generating N pieces of input data from one object, where N is an integer equal to or greater than 2; (b) inputting each of the N pieces of input data into the machine learning model and determining, for each of the N pieces of input data, M classification output values ​​output from an output layer of the machine learning model, one discriminant class, and a feature spectrum obtained from the output of a specific layer of the machine learning model; (c) calculating a similarity between a group of known feature spectra obtained from the output of the specific layer when a plurality of training data are input to the machine learning model and each of the feature spectra for the N pieces of input data, and calculating a confidence level for the discriminant class for each of the N pieces of input data as a function of the similarity; and (d) voting for the discriminant class for each of the N pieces of input data according to the confidence level for the discriminant class, and determining a class classification result for the object according to a result of the voting. According to this method, the reliability of the classification class is calculated according to the similarity of the feature spectra, and the classification result of the object is determined according to the voting result using the reliability, thereby enabling highly accurate classification.

[0095] <2> In the above method, the step (c) may include any one of the steps of: (1) setting the similarity as the reliability; (2) calculating the reliability by multiplying the similarity, the classification output value for the discrimination class, and a non-zero positive coefficient; and (3) calculating the reliability by performing a weighted addition of the similarity and the classification output value for the discrimination class. According to this method, the reliability of the classification class can be calculated as a function of the similarity.

[0096] <3> In the above method, the step (d) may include the steps of: (d1) adding 1 to the number of votes for the discrimination class when the reliability is equal to or greater than a reliability threshold for each of the N pieces of input data, and invalidating the vote when the reliability is less than the reliability threshold; and (d2) determining, as the class discrimination result, the class among the M classes that has the largest number of votes according to the N pieces of input data. According to this method, the result of class discrimination can be determined according to the number of votes corresponding to the reliability.

[0097] <4> In the above method, step (d2) may include the step of determining the class of the object as unknown if the highest number of votes is less than a vote number threshold. This method allows accurate determination when the object class is unknown.

[0098] <5> In the above method, the step (d) may include: (d1) adding, for each of the N pieces of input data, the reliability as a vote value for the discrimination class when the reliability is equal to or greater than a reliability threshold; and (d2) invalidating the vote when the reliability is less than the reliability threshold, and determining, as the class discrimination result, the class among the M classes having the highest vote value corresponding to the N pieces of input data. According to this method, the result of class discrimination can be determined in accordance with the vote value corresponding to the reliability.

[0099] <6> In the above method, step (d2) may include determining that the class of the object is unknown if the maximum vote value is less than a vote value threshold. This method allows accurate determination when the object class is unknown.

[0100] <7> In the above method, the specific layer may have a configuration in which vector neurons arranged on a plane defined by two axes, a first axis and a second axis, are arranged as multiple channels along a third axis that is oriented differently from the first and second axes. The feature spectrum may be any of (i) a first type of feature spectrum in which multiple element values ​​of an output vector of a vector neuron at one planar position in the specific layer are arranged across the multiple channels along the third axis, (ii) a second type of feature spectrum obtained by multiplying each element value of the first type of feature spectrum by an activation value corresponding to the vector length of the output vector, and (iii) a third type of feature spectrum in which the activation values ​​at one planar position in the specific layer are arranged across the multiple channels along the third axis. According to this method, the characteristic spectrum can be easily obtained.

[0101] <8> According to a second aspect of the present disclosure, there is provided an information processing device that performs a classification process for M classes using a machine learning model including a vector neural network having a plurality of vector neuron layers, where M is an integer equal to or greater than 2. The information processing device includes: a memory that stores the machine learning model; and a processor that performs calculations using the machine learning model. The processor performs the following steps: (a) reading from the memory N pieces of input data created from one object, where N is an integer equal to or greater than 2; (b) inputting each of the N pieces of input data into the machine learning model and determining, for each of the N pieces of input data, M classification output values ​​output from an output layer of the machine learning model, one discrimination class, and a feature spectrum obtained from the output of a specific layer of the machine learning model; (c) determining a similarity between a group of known feature spectra obtained from the output of the specific layer when a plurality of training data are input to the machine learning model and each of the feature spectra for the N pieces of input data, and determining a confidence level for the discrimination class for each of the N pieces of input data as a function of the similarity; and (d) voting for the discrimination class for each of the N pieces of input data in accordance with the confidence level for the discrimination class, and determining a class discrimination result for the object in accordance with the result of the voting.

[0102] <9> According to a third aspect of the present disclosure, there is provided a computer program that causes a processor to perform a class classification process for M classes using a machine learning model including a vector neural network having multiple vector neuron layers, where M is an integer greater than or equal to 2. This computer program causes the processor to perform the following steps: (a) reading from memory N pieces of input data created from one object, where N is an integer greater than or equal to 2; (b) inputting each of the N pieces of input data into the machine learning model and determining, for each of the N pieces of input data, M classification output values ​​output from an output layer of the machine learning model, one discriminant class, and a feature spectrum obtained from the output of a specific layer of the machine learning model; (c) determining the similarity between a group of known feature spectra obtained from the output of the specific layer when a plurality of training data are input to the machine learning model and each of the feature spectra for the N pieces of input data, and determining a confidence level for the discriminant class for each of the N pieces of input data as a function of the similarity; and (d) voting for the discriminant class for each of the N pieces of input data in accordance with the confidence level for the discriminant class, and determining a class discrimination result for the object in accordance with the result of the voting.

[0103] The present disclosure may be realized in various forms other than those described above, such as a computer program for implementing the functions of a classification device, a non-transitory storage medium on which the computer program is recorded, or the like. [Explanation of symbols]

[0104] 100...information processing device, 110...processor, 112...learning execution unit, 114...classification processing unit, 120...memory, 130...interface circuit, 140...input device, 150...display device, 200...machine learning model, 210...input layer, 220...convolutional layer, 230...primary vector neuron layer, 240...first convolutional vector neuron layer, 250...second convolutional vector neuron layer, 260...classification vector neuron layer (output layer), 280...hidden layer, 310...similarity calculation unit, 320...confidence calculation unit, 330...voting execution unit, 400...camera

Claims

1. A method for performing a classification process for M classes using a machine learning model including a vector neural network having a plurality of vector neuron layers, where M is an integer equal to or greater than 2, comprising: (a) generating N pieces of input data from one object, where N is an integer equal to or greater than 2; (b) inputting each of the N pieces of input data into the machine learning model to obtain one discriminant class and a feature spectrum obtained from an output of a specific layer of the machine learning model for each of the N pieces of input data; (c) calculating a similarity between a group of known feature spectra obtained from the output of the specific layer when a plurality of training data are input to the machine learning model and each of the feature spectra for the N pieces of input data, and calculating a reliability for the discrimination class for each of the N pieces of input data as a function of the similarity; (d) for each of the N pieces of input data, voting for the discrimination class according to the reliability for the discrimination class, and determining a class discrimination result for the object according to a result of the voting; Including, The step (c) (1) determining the similarity as the reliability; (2) calculating the confidence by multiplying the similarity, the classification output value for the discrimination class, and a non-zero positive coefficient; (3) calculating the reliability by weighting the similarity and the classification output value for the discrimination class; including any of the following: The specific layer has a configuration in which vector neurons arranged on a virtual plane defined by two axes, a first axis and a second axis, are arranged as a plurality of channels along a third axis in a direction different from the two axes, The characteristic spectrum is (i) a first type of feature spectrum in which a plurality of element values ​​of an output vector of a vector neuron at one plane position in the specific layer are arranged across the plurality of channels along the third axis; (ii) a second type feature spectrum obtained by multiplying each element value of the first type feature spectrum by an activation value corresponding to the vector length of the output vector; (iii) a third type of feature spectrum in which the activation values ​​at one planar position of the specific layer are arranged across the plurality of channels along the third axis; A method, which is one of the following:

2. 10. The method of claim 1, The step (d) (d1) for each of the N pieces of input data, if the reliability is equal to or greater than a reliability threshold, adding 1 to the number of votes for the discrimination class, and if the reliability is less than the reliability threshold, invalidating the vote; (d2) determining the class having the largest number of votes according to the N pieces of input data among the M classes as the class discrimination result; A method comprising:

3. 3. The method of claim 2, The method, wherein step (d2) comprises the step of determining the class of the object as unknown if the highest number of votes is less than a vote threshold.

4. 10. The method of claim 1, The step (d) (d1) adding the reliability of each of the N pieces of input data as a vote value for the discrimination class when the reliability is equal to or greater than a reliability threshold; (d2) invalidating the vote when the reliability is less than the reliability threshold, and determining the class with the highest vote value according to the N pieces of input data among the M classes as the class discrimination result; A method comprising:

5. 5. The method of claim 4, The method, wherein step (d2) comprises determining the class of the object as unknown if the maximum vote value is less than a vote value threshold.

6. 1. An information processing device that performs a classification process for M classes using a machine learning model including a vector neural network having a plurality of vector neuron layers, where M is an integer equal to or greater than 2, a memory that stores the machine learning model; a processor that executes calculations using the machine learning model; Equipped with The processor: (a) reading out from the memory N pieces of input data created from one object, where N is an integer equal to or greater than 2; (b) inputting each of the N pieces of input data into the machine learning model, and determining, for each of the N pieces of input data, one discriminant class and a feature spectrum obtained from an output of a specific layer of the machine learning model; (c) calculating a similarity between a group of known feature spectra obtained from the output of the specific layer when a plurality of training data are input to the machine learning model and each of the feature spectra for the N pieces of input data, and calculating a reliability of the discrimination class for each of the N pieces of input data as a function of the similarity; (d) a process of voting for the discrimination class according to the reliability of each of the N pieces of input data, and determining a class discrimination result of the object according to a result of the voting; is configured to run The process (c) (1) a process of determining the similarity as the reliability; (2) calculating the confidence by multiplying the similarity, the classification output value for the discrimination class, and a non-zero positive coefficient; (3) calculating the reliability by weighting the similarity and the classification output value for the discrimination class; including any of the following: The specific layer has a configuration in which vector neurons arranged on a virtual plane defined by two axes, a first axis and a second axis, are arranged as a plurality of channels along a third axis in a direction different from the two axes, The characteristic spectrum is (i) a first type of feature spectrum in which a plurality of element values ​​of an output vector of a vector neuron at one plane position in the specific layer are arranged across the plurality of channels along the third axis; (ii) a second type feature spectrum obtained by multiplying each element value of the first type feature spectrum by an activation value corresponding to the vector length of the output vector; (iii) a third type of feature spectrum in which the activation values ​​at one planar position of the specific layer are arranged across the plurality of channels along the third axis; An information processing device that is one of the above.

7. A computer program that causes a processor to execute a classification process for M classes using a machine learning model including a vector neural network having a plurality of vector neuron layers, where M is an integer equal to or greater than 2, comprising: The computer program comprises: (a) reading N pieces of input data generated from one object from a memory, where N is an integer equal to or greater than 2; (b) inputting each of the N pieces of input data into the machine learning model, and determining, for each of the N pieces of input data, one discriminant class and a feature spectrum obtained from an output of a specific layer of the machine learning model; (c) calculating a similarity between a group of known feature spectra obtained from the output of the specific layer when a plurality of training data are input to the machine learning model and each of the feature spectra for the N pieces of input data, and calculating a reliability of the discrimination class for each of the N pieces of input data as a function of the similarity; (d) a process of voting for the discrimination class according to the reliability of each of the N pieces of input data, and determining a class discrimination result of the object according to a result of the voting; configured to cause the processor to execute The process (c) (1) a process of determining the similarity as the reliability; (2) calculating the confidence by multiplying the similarity, the classification output value for the discrimination class, and a non-zero positive coefficient; (3) calculating the reliability by weighting the similarity and the classification output value for the discrimination class; including any of the following: The specific layer has a configuration in which vector neurons arranged on a virtual plane defined by two axes, a first axis and a second axis, are arranged as a plurality of channels along a third axis in a direction different from the two axes, The characteristic spectrum is (i) a first type of feature spectrum in which a plurality of element values ​​of an output vector of a vector neuron at one plane position in the specific layer are arranged across the plurality of channels along the third axis; (ii) a second type feature spectrum obtained by multiplying each element value of the first type feature spectrum by an activation value corresponding to the vector length of the output vector; (iii) a third type of feature spectrum in which the activation values ​​at one planar position of the specific layer are arranged across the plurality of channels along the third axis; A computer program that is one of the following:

Images