Method for discriminating class of data to be discriminated using machine learning model, information processing apparatus, and computer program

Abstract

To provide a technique that discriminates a class of discriminated data by a method other than a normal method using a discrimination result at an output layer of a neural network.SOLUTION: A class discrimination method includes: a step (a) of preparing M pairs of known characteristic spectrum groups corresponding to M machine learning models; and a step (b) of performing class discrimination processing of discriminated data using the M machine learning models and M known characteristic spectrum groups. The step (b) includes: a step (b1) of calculating M characteristic spectra from an output of a specific layer of the M machine learning models; a step (b2) of calculating a similarity for each class relating to each of the machine learning models as a similarity between the characteristic spectrum and the known characteristic spectrum group, and obtaining a reliability for each class depending on the similarity for each class; and a step (b3) of outputting a discrimination result list in which a plurality of classes are arranged in an order of the reliability of each of the classes.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a method for discriminating the class of data to be discriminated using a machine learning model, an information processing apparatus, and a computer program.

Background Art

[0002] Patent Documents 1 and 2 disclose a vector neural network type machine learning model using vector neurons, called a capsule network. A vector neuron means a neuron whose input and output are vectors. A capsule network is a machine learning model having vector neurons called capsules as nodes of a network. Machine learning models of the vector neural network type such as capsule networks can be used for class discrimination of input data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] The inventor of the present disclosure has found that, in the prior art, there is a problem that the features extracted in the intermediate layer of the neural network are not correctly reflected in the class discrimination result in the output layer, and the class discrimination result may be incorrect. The present disclosure provides a technique for class discrimination of data to be discriminated by a method different from the normal method of directly using the discrimination result in the output layer of the neural network. Further, the present disclosure provides a teacher data processing technique that contributes to improving the discrimination accuracy, whether combined with or separated from the different method.

Means for Solving the Problems

[0005] According to a first aspect of the present disclosure, when M is an integer of 1 or more, there is provided a method for discriminating the class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers. Let Nm, which is an integer of 2 or more, be the number of classes discriminable by the m-th machine learning model among the M machine learning models, and let ΣNm be the total number of classes discriminable by the M machine learning models. The method includes: (a) a step of preparing M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of specific layers among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of the Nm classes are input to each machine learning model; and (b) a step of performing a class discrimination process on the data to be discriminated using the M machine learning models and the M Group sets of known feature spectrum groups. The step (b) includes: (b1) a step of calculating M feature spectra from the outputs of the specific layers in response to the input of the data to be discriminated to the M machine learning models; (b2) a step of obtaining M sets of class-specific reliabilities associated with the M machine learning models by calculating class-specific similarities for each of the Nm classes in each machine learning model as similarities between the M feature spectra and the M sets of known feature spectrum groups, and obtaining class-specific reliabilities depending on the class-specific similarities; and (b3) a step of outputting a discrimination result list in which a plurality of classes, which are at least a part of the ΣNm classes, are arranged in the order of the reliabilities of the classes represented by the M sets of class-specific reliabilities.

[0006] According to a second aspect of the present disclosure, when M is an integer of 2 or more, there is provided a method for discriminating the class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers. This method includes: (a) a step of preparing M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of specific layers among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of Nm classes are input to each machine learning model; and (b) using the M machine learning models and the M Group sets of known feature spectrum groups to perform a class discrimination process on the data to be discriminated. The step (b) includes: (b1) a step of calculating M feature spectra from the outputs of the specific layers and determining M discrimination classes in response to the input of the data to be discriminated to the M machine learning models; (b2) a step of obtaining model reliabilities for each of the M machine learning models by calculating discrimination class similarities for each of the M discrimination classes as similarities between the M feature spectra and the M sets of known feature spectrum groups, and obtaining a model reliability that depends on the discrimination class similarities; and (b3) a step of outputting, as a class discrimination result for the data to be discriminated, the discrimination class obtained from the machine learning model having the highest model reliability among the M discrimination classes.

[0007] According to a third aspect of the present disclosure, when M is an integer of 1 or more, there is provided an information processing apparatus that discriminates a class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers. This information processing apparatus includes a memory that stores the M machine learning models, and a processor that executes operations using the M machine learning models. Let Nm be an integer of 2 or more, which is the number of classes discriminable by the m-th machine learning model among the M machine learning models, and let ΣNm be the total number of classes discriminable by the M machine learning models. The processor executes: (a) a process of reading from the memory M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of a specific layer among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of the Nm classes are input to each machine learning model; and (b) a process of executing a class discrimination process for the data to be discriminated using the M machine learning models and the M Group sets of known feature spectrum groups. The process (b) includes: (b1) a process of calculating M feature spectra from the output of the specific layer in response to input of the data to be discriminated to the M machine learning models; (b2) a process of obtaining M sets of class-by-class reliability associated with the M machine learning models by calculating class-by-class similarity for each of the Nm classes in each machine learning model as the similarity between the M feature spectra and the M sets of known feature spectrum groups, and obtaining class-by-class reliability depending on the class-by-class similarity; and (b3) a process of outputting a discrimination result list in which a plurality of classes, which are at least a part of the ΣNm classes, are arranged in the order of the reliability of each class represented by the M sets of class-by-class reliability.

[0008] According to a fourth aspect of the present disclosure, when M is an integer of 2 or more, there is provided an information processing apparatus that discriminates the class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers. This information processing apparatus includes a memory that stores the M machine learning models, and a processor that executes operations using the M machine learning models. The processor performs (a) a process of reading out M sets of known feature spectrum groups associated with the M machine learning models from the memory, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of specific layers among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of Nm classes are input to each machine learning model, and (b) a process of executing a class discrimination process for the data to be discriminated using the M machine learning models and the M Group sets of known feature spectrum groups. The process (b) includes (b1) a process of calculating M feature spectra from the outputs of the specific layers and determining M discrimination classes in response to the input of the data to be discriminated to the M machine learning models, (b2) a process of obtaining model reliabilities for each of the M machine learning models by calculating discrimination class similarities for each of the M discrimination classes as similarities between the M feature spectra and the M sets of known feature spectrum groups and obtaining a model reliability that depends on the discrimination class similarities, and (b3) a process of outputting, as a class discrimination result for the data to be discriminated, the discrimination class obtained from the machine learning model with the highest model reliability among the M discrimination classes.

[0009] According to a fifth aspect of the present disclosure, when M is an integer of 1 or more, there is provided a computer program that causes a processor to execute a class discrimination process for discriminating a class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers. Let Nm be an integer of 2 or more, which is the number of classes discriminable by the m-th machine learning model among the M machine learning models, and let ΣNm be the total number of classes discriminable by the M machine learning models. The computer program includes: (a) a process of reading from a memory M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of a specific layer among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of the Nm classes are input to each machine learning model; and (b) a process of causing the processor to execute a class discrimination process for the data to be discriminated using the M machine learning models and the M Group sets of known feature spectrum groups. The process (b) includes: (b1) a process of calculating M feature spectra from the output of the specific layer in response to input of the data to be discriminated to the M machine learning models; (b2) a process of obtaining M sets of class-specific reliabilities associated with the M machine learning models by calculating class-specific similarities for each of the Nm classes in each machine learning model as similarities between the M feature spectra and the M sets of known feature spectrum groups, and obtaining class-specific reliabilities depending on the class-specific similarities; and (b3) a process of outputting a discrimination result list in which a plurality of classes, which are at least a part of the ΣNm classes, are arranged in the order of the reliabilities of the classes represented by the M sets of class-specific reliabilities.

[0010] According to a sixth aspect of the present disclosure, when M is an integer of 2 or more, there is provided a computer program that causes a processor to execute a class discrimination process for discriminating a class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers. This computer program includes: (a) a process of reading from a memory M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of specific layers among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of Nm classes are input to each machine learning model; and (b) a process of causing the processor to execute a class discrimination process for the data to be discriminated using the M machine learning models and the M Group sets of known feature spectrum groups. The process (b) includes: (b1) a process of calculating M feature spectra from the outputs of the specific layers and determining M discrimination classes in response to input of the data to be discriminated to the M machine learning models; (b2) a process of obtaining model reliabilities for each of the M machine learning models by calculating discrimination class similarities for each of the M discrimination classes as similarities between the M feature spectra and the M sets of known feature spectrum groups and obtaining a model reliability depending on the discrimination class similarities; and (b3) a process of outputting, as a class discrimination result for the data to be discriminated, the discrimination class obtained from the machine learning model having the highest model reliability among the M discrimination classes.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Mode for Carrying Out the Invention

[0012] A. First Embodiment: FIG. 1 is a block diagram showing a class discrimination system according to the first embodiment. This class discrimination system is a printing system including a printer 10, an information processing apparatus 20, and a spectroscopic measuring device 30. The spectroscopic measuring device 30 can perform spectroscopic measurement on a printing medium PM used in the printer 10 in an unprinted state to obtain a spectral reflectance. In the present disclosure, the spectral reflectance is also referred to as "spectral data". The spectroscopic measuring device 30 includes, for example, a wavelength-variable interference spectroscopic filter and a monochrome image sensor. The spectral data obtained by the spectroscopic measuring device 30 is used as discrimination data to be input to a machine learning model described later. The information processing apparatus 20 executes class discrimination processing of the spectral data using the machine learning model, and discriminates which of a plurality of classes the printing medium PM belongs to. The "class of the printing medium PM" means the type of the printing medium PM. The information processing apparatus 20 controls the printer 10 to execute printing under appropriate printing conditions according to the type of the printing medium PM. Further, the information processing apparatus 20 may display the discriminated type of the printing medium PM on a printer display. In this way, it is possible to prevent the printer from being driven while a medium different from the medium intended by the user is installed. Note that the class discrimination system according to the present disclosure can also be realized as a system other than a printing system. For example, it may be realized as a system that performs class discrimination using, as discrimination data, a discriminated image, one-dimensional data other than spectral data, a spectroscopic image, time-series data, or the like.

[0013] FIG. 2 is a block diagram showing the functions of the information processing apparatus 20. The information processing apparatus 20 includes a processor 110, a memory 120, an interface circuit 130, an input device 140 and a display unit 150 connected to the interface circuit 130. The interface circuit 130 is also connected to a spectroscopic measuring device 30 and a printer 10. Although not limited, for example, the processor 110 not only has a function of executing the processes described in detail below, but also has a function of displaying the data obtained by the processes and the data generated during the processes on the display unit 150.

[0014] The processor 110 functions as a print processing unit 112 that executes a print process using the printer 10, and also functions as a class discrimination processing unit 114 that executes a class discrimination process for spectroscopic data of the print medium PM. The class discrimination processing unit 114 includes a similarity calculation unit 310 and a comprehensive determination unit 320. The print processing unit 112 and the class discrimination processing unit 114 are realized by the processor 110 executing a computer program stored in the memory 120. However, these units 112 and 114 may be realized by a hardware circuit. The processor in this specification is a term that also includes such a hardware circuit. Further, the processor that executes the class discrimination process may be a processor included in a remote computer connected to the information processing apparatus 20 via a network.

[0015] The memory 120 stores a plurality of machine learning models 200, a plurality of teacher data groups TD, a plurality of known feature spectrum groups KSp, and a print setting table PST. The machine learning model 200 is used for the processing by the class discrimination processing unit 114. The configuration example and operation of the machine learning model 200 will be described later. The teacher data group TD is a set of labeled data used for the learning of the machine learning model 200. In the present embodiment, the teacher data group TD is a set of spectral data. The known feature spectrum group KSp is a set of feature spectra obtained when the teacher data group TD is input to the learned machine learning model 200. The feature spectrum will be described later. The print setting table PST is a table in which print settings suitable for each printing medium are registered. When the number of machine learning models 200 is represented by an integer M, M can be set to any number of 1 or more. In the present embodiment, the case of using two machine learning models 200 will be described. For the teacher data group TD and the known feature spectrum group KSp, those corresponding to the machine learning model 200 are used respectively.

[0016] FIG. 3 is an explanatory diagram showing the configuration of the machine learning model 200. This machine learning model 200 includes, in order from the input data IM side, a convolutional layer 210, a primary vector neuron layer 220, a first convolutional vector neuron layer 230, a second convolutional vector neuron layer 240, and a classification vector neuron layer 250. Among these five layers 210 to 250, the convolutional layer 210 is the lowest layer, and the classification vector neuron layer 250 is the highest layer. In the following description, the layers 210 to 250 are also referred to as "Conv layer 210", "PrimeVN layer 220", "ConvVN1 layer 230", "ConvVN2 layer 240", and "ClassVN layer 250", respectively.

[0017] In the present embodiment, since the input data IM is spectral data, it is data in a one-dimensional array. For example, the input data IM is data obtained by extracting 36 representative values every 10 nm from the spectral data in the range of 380 nm to 730 nm.

[0018] In the example of FIG. 3, two convolutional vector neuron layers 230 and 240 are used, but the number of convolutional vector neuron layers is arbitrary, and the convolutional vector neuron layers may be omitted. However, it is preferable to use one or more convolutional vector neuron layers.

[0019] The configurations of the layers 210 to 250 in FIG. 3 can be described as follows. <Description of the configuration of the machine learning model 200> ·Conv layer 210: Conv[32, 6, 2] ·PrimeVN layer 220: PrimeVN[26, 1, 1] ·ConvVN1 layer 230: ConvVN1[20, 5, 2] ·ConvVN2 layer 240: ConvVN2[16, 4, 1] ·ClassVN layer 250: ClassVN[Nm, 3, 1] ·Vector dimension VD: VD = 16 In the description of each of these layers 210 to 250, the string before the parentheses is the layer name, and the numbers inside the parentheses are, in order, the number of channels, the surface size of the kernel, and the stride. For example, the layer name of the Conv layer 210 is "Conv", the number of channels is 32, the surface size of the kernel is 1×6, and the stride is 2. In FIG. 3, these descriptions are shown below each layer. The hatched rectangles drawn inside each layer represent the surface size of the kernel used when calculating the output vector of the adjacent upper layer. In this embodiment, since the input data IM is one-dimensional array data, the surface size of the kernel is also one-dimensional. Note that the values of the parameters used in the description of each of the layers 210 to 250 are examples and can be arbitrarily changed.

[0020] The Conv layer 210 is a layer composed of scalar neurons. The other four layers 220 - 250 are layers composed of vector neurons. A vector neuron is a neuron that takes a vector as input and output. In the above description, the dimension of the output vector of each individual vector neuron is fixed at 16. Hereinafter, the term "node" is used as the superordinate concept of scalar neurons and vector neurons.

[0021] In FIG. 3, for the Conv layer 210, the first axis x and the second axis y that define the planar coordinates of the node array, and the third axis z that represents the depth are shown. Also, it is shown that the sizes of the Conv layer 210 in the x, y, and z directions are 1, 16, and 32. The sizes in the x - direction and y - direction are called "resolution". In this embodiment, the resolution in the x - direction is always 1. The size in the z - direction is the number of channels. These three axes x, y, z are also used as the coordinate axes indicating the positions of each node in other layers. However, in FIG. 3, for layers other than the Conv layer 210, the illustration of these axes x, y, z is omitted.

[0022] As is well - known, the resolution W1 in the y - direction after convolution is given by the following formula. W1 = Ceil{(W0 - Wk + 1) / S} (1) Here, W0 is the resolution before convolution, Wk is the surface size of the kernel, S is the stride, and Ceil{X} is a function that performs the operation of rounding up the decimal part of X. The resolution of each layer shown in FIG. 3 is an example when the resolution in the y - direction of the input data IM is 36. The actual resolution of each layer is appropriately changed according to the size of the input data IM.

[0023] The ClassVN layer 250 has Nm channels. In the example of FIG. 3, Nm = 3. Generally, Nm is an integer of 2 or more, and is the number of known classes that can be discriminated using the machine learning model 200. The number of discriminable classes Nm can be set to different values for each machine learning model 200. The total number of classes discriminable by the M machine learning models 200 is represented by ΣNm. From the three channels of the ClassVN layer 250, determination values Class1 to Class3 for three known classes are output. Usually, the class having the largest value among these determination values Class1 to Class3 is used as the class discrimination result of the input data IM. Also, when the largest value among the determination values Class1 to Class3 is less than a predetermined threshold value, it may be determined that the class of the input data IM is unknown.

[0024] In the present disclosure, as will be described later, instead of using the determination values Class1 to Class3 of the ClassVN layer 250 which is the output layer, a method of determining a discriminant class using the similarity calculated from the output of a specific vector neuron layer may be adopted. Also, instead of determining one discriminant class, it is also possible to output a discriminant result list in which a plurality of classes are arranged in the order of similarity or in the order of reliability depending on the similarity. In the first embodiment, a method of outputting a discriminant result list and presenting it to the user is adopted.

[0025] In FIG. 3, further, sub-regions Rn in each of the layers 210, 220, 230, 240, 250 are drawn. The subscript "n" of the sub-region Rn is the symbol of each layer. For example, the sub-region R210 indicates the sub-region in the Conv layer 210. The "sub-region Rn" is a region that is specified by the planar position (x, y) defined by the position of the first axis x and the second axis y in each layer, and includes a plurality of channels along the third axis z. The sub-region Rn has dimensions of "Width" × "Height" × "Depth" corresponding to the first axis x, the second axis y, and the third axis z. In the present embodiment, the number of nodes included in one "sub-region Rn" is "1×1×number of depths", that is, "1×1×number of channels".

[0026] As shown in FIG. 3, a feature spectrum Sp_ConvVN1 described later is calculated from the output of the ConvVN1 layer 230 and input to the similarity calculation unit 310. Similarly, feature spectra Sp_ConvVN2 and Sp_ClassVN are respectively calculated from the outputs of the ConvVN2 layer 240 and the ClassVN layer 250 and input to the similarity calculation unit 310. The similarity calculation unit 310 calculates class-specific similarities described later using these feature spectra Sp_ConvVN1, Sp_ConvVN, and Sp_ClassVN and a group of known feature spectra KSp created in advance, and calculates class-specific reliabilities depending on the class-specific similarities.

[0027] In the present disclosure, the vector neuron layer used for calculating the similarity is also referred to as a "specific layer". As the specific layer, one or more arbitrary numbers of vector neuron layers can be used. Note that the configuration of the feature spectrum and the calculation methods of the similarity and reliability using the feature spectrum will be described later.

[0028] FIG. 4 is an explanatory diagram showing another configuration of the machine learning model 200. This machine learning model 200 is different from the machine learning model 200 in FIG. 3 that uses one-dimensional array input data in that the input data IM is two-dimensional array data. The configurations of the respective layers 210 to 250 in FIG. 4 can be described as follows. <Description of the configuration of each layer> ·Conv layer 210: Conv[32, 5, 2] ·PrimeVN layer 220: PrimeVN[16, 1, 1] ·ConvVN1 layer 230: ConvVN1[12, 3, 2] ·ConvVN2 layer 240: ConvVN2[6, 3, 1] ·ClassVN layer 250: ClassVN[Nm, 4, 1] ·Vector dimension VD: VD = 16

[0029] The machine learning model 200 shown in FIG. 4 can be used, for example, in a class discrimination system that performs class discrimination on a to-be-discriminated image. However, in the following description, the machine learning model 200 shown in FIG. 3 is used.

[0030] FIG. 5 is a block diagram showing the functions of the class discrimination processing unit 114a in the first embodiment. The class discrimination processing unit 114a includes a similarity calculation unit 310a and a comprehensive determination unit 320a. The "a" appended to the end of these reference numerals indicates that they are for the first embodiment. The comprehensive determination unit 320a includes a list creation unit 322. In this example, two machine learning models 200_1 and 200_2 are used. The "_1" and "_2" at the end of these reference numerals are additional reference numerals for distinguishing the two machine learning models. However, when there is no need to distinguish multiple machine learning models, the reference numeral "200" without additional reference numerals is used.

[0031] When the to-be-discriminated data IM is input into the two machine learning models 200, feature spectra Sp are respectively calculated from specific layers of the two machine learning models 200 and input into the similarity calculation unit 310a. The similarity calculation unit 310a calculates a class-by-class similarity Sclass, which is the similarity between the feature spectrum Sp and a known feature spectrum group KSp of the corresponding specific layer, and obtains a class-by-class reliability Rclass that depends on the class-by-class similarity Sclass. The box drawn with a dashed line inside the similarity calculation unit 310a in FIG. 5 is drawn for convenience to show the input-output relationship. The class-by-class similarity Sclass includes a parameter m indicating the machine learning model 200, a parameter i indicating the class, and a similarity value S_value of class i. The method for calculating the class-by-class similarity Sclass will be described later.

[0032] The class-by-class reliability Rclass includes a parameter m indicating the machine learning model 200, a parameter i indicating the class, and a reliability value R_value of class i. As shown in the following formula, the reliability value R_value is calculated according to a reliability function H that depends on the similarity value S_value of the class-by-class similarity Sclass. R_value(i)=H[S_value(i)] (2) In this equation, the parameter m of the machine learning model is omitted. It is preferable that the reliability function H determines the class-by-class reliability Rclass such that the class-by-class reliability Rclass has a positive correlation with the class-by-class similarity Sclass. Specific examples of the reliability function H will be described later.

[0033] In the example of FIG. 5, it is assumed that the number of distinguishable classes in each of the two machine learning models 200 is three. The list creation unit 322 outputs a discrimination result list RL in which a plurality of classes are arranged in the order of reliability according to the class-by-class reliability Rclass. This discrimination result list RL is displayed on the display unit 150. Specific examples of the class-by-class reliability Rclass and the discrimination result list RL will be described later.

[0034] FIG. 6 is a flowchart showing the processing procedure of the preparation step of the machine learning model. This preparation step is a step executed, for example, by the manufacturer of the printer 10.

[0035] In step S100 of FIG. 6, the class discrimination processing unit 114a performs grouping by clustering a plurality of teacher data. FIG. 7 shows the teacher data grouped by the clustering process. In this example, the plurality of teacher data are grouped into a first teacher data group TD1 and a second teacher data group TD2. As the clustering process, for example, the k-means method can be used. It is also possible to perform the clustering process using distances such as the Euclidean distance and the Mahalanobis distance regarding the optical spectrum of the printed medium. For example, in the clustering process using the Euclidean distance, an average spectrum obtained by averaging the results of a plurality of spectral measurements for each printed medium is obtained, and the printed media are collected in ascending order of the Euclidean distance from the average spectrum of the reference printed medium to form a first group. When the number of collected printed media reaches the reference value, the collection of the first group is terminated. Then, a new reference is set again, and the process of collecting the printed media in the order of the Euclidean distance to form a second group is repeated. Considering the case of using the k-means method, the teacher data groups TD1 and TD2 have representative points G1 and G2 representing the respective teacher data groups TD1 and TD2. These representative points G1 and G2 are, for example, the centroids. When the teacher data is composed of reflectivities at n wavelengths, by regarding one teacher data as data representing a point in an n-dimensional space, it is possible to calculate the distance between teacher data and the centroid of a plurality of teacher data. In FIG. 7, for the sake of illustration, points of a plurality of teacher data are drawn in a two-dimensional space, but actually, the teacher data can be represented as points in an n-dimensional space. These representative points G1 and G2 can be used when determining which of the plurality of teacher data groups TD1 and TD2 the teacher data for an additional class is closest to when adding a new class as the target of the class discrimination process. By using this determination to add the teacher data of the additional class to one of the plurality of teacher data groups TD1 and TD2, it is possible to maintain the teacher data groups TD1 and TD2 after the addition in a state equivalent to the state grouped by the clustering process. Note that, as the representative points G1 and G2, other than the centroids may be used.

[0036] In this embodiment, a plurality of teacher data are grouped into two teacher data groups TD1 and TD2. However, there may be only one teacher data group, or three or more teacher data groups may be created. Also, a plurality of teacher data groups may be created by a method other than the clustering process. However, if a plurality of teacher data are grouped by the clustering process, teacher data that are similar to each other can be grouped into the same group. By using such a plurality of teacher data groups to train a plurality of machine learning models 200, the accuracy of the class discrimination process by each machine learning model 200 can be improved compared to the case where no clustering process is performed. This is due to the following reasons. That is, since the plurality of machine learning models are independent of each other, for example, assume that there are two machine learning models A and B, and there are spectral data Am and Bm that are very similar. If the spectral data Am and Bm are trained by different machine learning models A and B, it means that the learning to separate the characteristics of the spectral data Am and Bm is not performed in the individual machine learning models A and B. As a result, the difference between the spectral data Am and Bm cannot be distinguished, and there is a possibility that both machine learning models A and B will determine the spectral data Am as Bm and the spectral data Bm as Am. On the other hand, if the spectral data Am and Bm are trained by the same machine learning model, since the difference between the spectral data Am and Bm is learned, the spectral data Am and Bm can be discriminated. Thus, by clustering the spectral data, printed media with similar characteristics are trained by the same machine learning model, so that those printed media can be correctly discriminated. Also, in other machine learning models, it becomes easier to be discriminated as an unknown printed media.

[0037] In step S110 of FIG. 6, the class discrimination processing unit 114a executes learning of the machine learning model 200 using a plurality of teacher data groups TD. Labels are pre-assigned to individual teacher data. In the present embodiment, it is assumed that labels of any one of 1 to 3 are assigned to the teacher data of the first machine learning model 200_1. These labels correspond to the three classes Class1 to Class3 of the machine learning model 200 shown in FIG. 3. Also, it is assumed that labels of any one of 11 to 13 are assigned to the teacher data of the second machine learning model 200_2. In the present disclosure, "label" and "class" mean the same thing.

[0038] When learning using a plurality of teacher data groups TD is completed, the learned machine learning model 200 is stored in the memory 120. In step S120 of FIG. 6, a plurality of teacher data are input again to the learned machine learning model 200 to generate a known feature spectrum group KSp. The known feature spectrum group KSp is a set of feature spectra described below.

[0039] FIG. 8 is an explanatory diagram showing a feature spectrum Sp obtained by inputting arbitrary input data to the learned machine learning model 200. Here, the feature spectrum Sp obtained from the output of the ConvVN1 layer 230 will be described. The horizontal axis in FIG. 8 is the position of vector elements regarding the output vectors of a plurality of nodes included in one partial region R230 of the ConvVN1 layer 230. This position of the vector element is represented by a combination of the element number ND of the output vector at each node and the channel number NC. In the present embodiment, since the vector dimension is 16 (the number of elements of the output vector output by each node), the element number ND of the output vector is 16 from 0 to 15. Also, since the number of channels of the ConvVN1 layer 230 is 20, the channel number NC is 20 from 0 to 19. In other words, this feature spectrum Sp is an arrangement of a plurality of element values of the output vectors of each vector neuron included in one partial region R230 over a plurality of channels along the third axis z.

[0040] The vertical axis of FIG. 8 represents the characteristic value C at each spectral position V as shown. In this example, the characteristic value C V is the value V of each element of the output vector ND . Note that as the characteristic value C V , a value obtained by multiplying the value V of each element of the output vector ND by a normalization coefficient described later may be used, or the normalization coefficient may be used as it is. In the latter case, the number of characteristic values C V included in the feature spectrum Sp is equal to the number of channels, which is 20. Note that the normalization coefficient is a value corresponding to the vector length of the output vector of that node.

[0041] For one input data, the number of feature spectra Sp obtained from the output of the ConvVN1 layer 230 is equal to the number of planar positions (x, y) of the ConvVN1 layer 230, that is, the number of sub-regions R230, so it is 6. Similarly, for one input data, 3 feature spectra Sp are obtained from the output of the ConvVN2 layer 240, and 1 feature spectrum Sp is obtained from the output of the ClassVN layer 250.

[0042] When the teacher data is input again to the learned machine learning model 200, the similarity calculation unit 310a calculates the feature spectrum Sp shown in FIG. 8 and registers it in the memory 120 as the known feature spectrum group KSp.

[0043] FIG. 9 is an explanatory diagram showing how the known feature spectrum group KSp is created using the teacher data TD. In this example, by inputting the teacher data TD with labels 1 to 3 into the learned machine learning model 200, feature spectra KSp_ConvVN1, KSp_ConvVN2, and KSp_ClassVN associated with their respective labels or classes are obtained from the outputs of the three vector neuron layers, namely, the ConvVN1 layer 230, the ConvVN2 layer 240, and the ClassVN layer 250. These feature spectra KSp_ConvVN1, KSp_ConvVN2, and KSp_ClassVN are stored in the memory 120 as the known feature spectrum group KSp.

[0044] FIG. 10 is an explanatory diagram showing the configuration of the known feature spectrum group KSp. In this example, the known feature spectrum group KSp_ConvVN1 obtained from the output of the ConvVN1 layer 230 of the first machine learning model 200_1 is shown. The known feature spectrum group KSp_ConvVN2 obtained from the output of the ConvVN2 layer 240 and the known feature spectrum group KSp_ConvVN1 obtained from the output of the ClassVN layer 250 also have a similar configuration, but are not shown in FIG. 10. Note that as the known feature spectrum group KSp, it is sufficient that it is obtained from the output of at least one vector neuron layer.

[0045] Each record of the known feature spectrum group KSp_ConvVN1 includes a parameter m indicating the order of the machine learning model, a parameter i indicating the order of the label or class, a parameter j indicating the order of the specific layer, a parameter k indicating the order of the sub-region Rn, a parameter q indicating the data number, and the known feature spectrum KSp. The known feature spectrum KSp is the same as the feature spectrum Sp in FIG. 8.

[0046] The class parameter i takes values from 1 to 3, the same as the label. The specific layer parameter j takes values from 1 to 3 indicating which of the three specific layers 230, 240, 250 it is. The sub-region Rn parameter k takes a value indicating which of the multiple sub-regions Rn included in the individual specific layer, that is, which of the planar positions (x, y). For the ConvVN1 layer 230, since the number of sub-regions R230 is 6, k = 1 to 6. The data number parameter q indicates the number of the teacher data with the same label, and takes values from 1 to max1 for class 1, from 1 to max2 for class 2, and from 1 to max3 for class 3.

[0047] Note that the plurality of teacher data TD used in step S120 does not have to be the same as the plurality of teacher data TD used in step S110. However, even in step S120, if part or all of the plurality of teacher data TD used in step S110 is utilized, there is an advantage that it is not necessary to prepare new teacher data.

[0048] FIG. 11 is a flowchart showing a processing procedure of a medium discrimination / printing process using a learned machine learning model. This medium discrimination / printing process is executed, for example, by a user who uses the printer 10.

[0049] In step S210, the user instructs the class discrimination processing unit 114a whether class discrimination processing is required for the target printing medium, which is the printing medium to be processed. Even when the user knows the type of the target printing medium, the user may instruct that class discrimination processing is required for confirmation. If class discrimination processing is not necessary, the process proceeds to step S260 where the user selects a print setting suitable for the target printing medium, and in step S270, the print processing unit 112 causes the printer 10 to perform printing using the target printing medium. On the other hand, if the type of the target printing medium is unknown and class discrimination processing is required, the process proceeds to step S220.

[0050] In step S220, the class discrimination processing unit 114a acquires spectral data by causing the spectroscopic measuring device 30 to perform spectroscopic measurement of the target printing medium. This spectral data is used as the data to be discriminated input to the machine learning model 200.

[0051] In step S230, the class discrimination processing unit 114a inputs the discrimination data into the learned machine learning models 200_1 and 200_2 respectively, and calculates the feature spectrum Sp from the outputs of the machine learning models 200_1 and 200_2 respectively. In step S240, the similarity calculation unit 310a calculates the class-by-class similarity from the feature spectrum Sp obtained in response to the input of the discrimination data and the registered known feature spectrum group KSp, and obtains the class-by-class reliability depending on the class-by-class similarity.

[0052] FIG. 12 is an explanatory diagram showing how to obtain the class-by-class similarity Sclass and the class-by-class reliability Rclass for the discrimination data. When the discrimination data is input into the m-th machine learning model 200_m, the class discrimination processing unit 114a calculates the feature spectra Sp_ConvVN1, Sp_ConvVN2, and Sp_ClassVN from the outputs of the ConvVN1 layer 230, the ConvVN2 layer 240, and the ClassVN layer 250 of the machine learning model 200_m respectively. The similarity calculation unit 310a calculates the class-by-class similarity Sclass_ConvVN1 using the feature spectrum Sp_ConvVN1 obtained from the output of the ConvVN1 layer 230 and the known feature spectrum group KSp_ConvVN1. The specific method for calculating the class-by-class similarity will be described later. For the ConvVN2 layer 240 and the ClassVN layer 250, the class-by-class similarities Sclass_ConvVN2 and Sclass_ClassVN are calculated in the same way.

[0053] It is not necessary to generate all the class-by-class similarities Sclass_ConvVN1, Sclass_ConvVN2, and Sclass_ClassVN using the three vector neuron layers 230, 240, and 250 respectively, but it is preferable to calculate the class-by-class similarity using one or more of these vector neuron layers. As described above, in the present disclosure, the vector neuron layer used for calculating the similarity is called the "specific layer".

[0054] The similarity calculation unit 310a calculates the class-based similarity Sclass using at least one of these class-based similarities Sclass_ConvVN1, Sclass_ConvVN2, Sclass_ClassVN, and obtains the class-based reliability Rclass from this class-based similarity Sclass. The class-based similarity Sclass represents the similarity value S_value for the individual class i of the m-th machine learning model. The class-based reliability Rclass represents the reliability value R_value for the individual class i of the m-th machine learning model. As a reliability function for obtaining the reliability value R_value from the similarity value S_value, for example, any of the following can be used. R_value(i)=H 1 [S_value(i)]=S_value(i) (3a) R_value(i)=H 2 [S_value(i)]=Ac(i)×Wt+S_value(i)×(1-Wt) (3b) R_value(i)=H 3 [S_value(i)]=Ac(i)×S_value(i) (3c) Here, Ac(i) is the activation value corresponding to the determination value of class i in the output layer of the machine learning model 200, and Wt is a weight coefficient in the range of 0 < Wt < 1. Note that the activation value Ac(i) is the same as the determination values Class1~Class3 of each class shown in FIG. 3.

[0055] The reliability function H of the above equation (3a) 1 is an identity function that sets the similarity value S_value itself as the reliability value R_value. The reliability function H of the above equation (3b) 2 is a function for obtaining the reliability value R_value by taking a weighted average of the similarity value S_value and the activation value Ac. The reliability function H of the above equation (3c) 3is a function that obtains a reliability value R_value by multiplying a similarity value S_value by an activation value Ac. Also, other reliability functions may be used. For example, a function that uses the power of the similarity value S_value as the reliability value R_value may be used. In this way, the class-specific reliability Rclass can be obtained as depending on the class-specific similarity Sclass. Also, it is preferable that the class-specific reliability Rclass has a positive correlation with the class-specific similarity Sclass.

[0056] The class-specific similarity Sclass is an index indicating the degree to which the data to be discriminated is similar to the teacher data regarding each class. Also, the class-specific reliability Rclass can be used as an index indicating the degree to which the data to be discriminated belongs to each class.

[0057] In step S250, the list creation unit 322 outputs a discrimination result list RL in which a plurality of classes are arranged in the order of the reliability value R_value according to the class-specific reliability Rclass. This discrimination result list RL is displayed on the display unit 150.

[0058] FIG. 13 is an explanatory diagram showing an example of the discrimination result list RL. In this example, the discrimination result list RL includes a priority, a parameter m indicating a machine learning model, a parameter i indicating a class, a class name, and a reliability value R_value, and a plurality of classes are arranged in the order of the reliability value R_value. In other embodiments, the discrimination result list RL may not include the parameter m indicating the machine learning model. The priority indicates the order of the reliability value R_value. As the class name, the name of the printed medium is exemplified. Note that the priority may be omitted. Also, among the class information, either one of the class parameter i and the class name may be omitted. Further, the discrimination result list RL does not necessarily include all the classes that can be discriminated by the M machine learning models, and may include only a part of them. Generally, when the number of classes that can be discriminated by the m-th machine learning model among the M machine learning models is Nm, the total number of classes that can be discriminated by the M machine learning models is represented by ΣNm. The discrimination result list RL may include only a part of the plurality of classes with the highest reliability values among the ΣNm classes.

[0059] When the discrimination result list RL is displayed on the display unit 150, the user can observe this discrimination result list RL and determine the type of the target printed medium to be discriminated. For example, if the class that the user has assumed in advance as the type of the target printed medium exists at the top of the discrimination result list RL, the assumed class can be determined as the type of the target printed medium. Also, when the user has not assumed the type of the target printed medium in advance, by comparing with the reliability value, the class at the top of the discrimination result list RL can be determined as the type of the target printed medium. If the reliability of the class at the top of the discrimination result list RL is low, it may be handled as an unknown printed medium without determining the type of the target printed medium, and corresponding operations may be performed. For example, the user may be warned that an unknown printed medium is set.

[0060] In step S260, the user selects the class of the target printing medium, that is, the type of the target printing medium, from the discrimination result list RL and instructs the printing processing unit 112. Step S260 may include any one of the following: (i) the processor 110 selects one type of printing medium from the discrimination result list RL according to the instruction from the user; (ii) when the user gives an instruction meaning "Accept", the processor 110 selects the type of printing medium at the top of the discrimination result list RL; and (iii) when there is no instruction from the user within a predetermined period, the processor 110 selects the type of printing medium at the top of the discrimination result list RL or stops the process. In step S270, the printing processing unit 112 refers to the printing setting table PST according to the type of the target printing medium and selects a printing setting. In step S280, the printing processing unit 112 executes printing according to the printing setting. According to the procedure in FIG. 11, even when the type of the target printing medium is unclear, the type of the target printing medium can be discriminated using the machine learning model 200, so it is possible to execute printing using a printing setting suitable for the type.

[0061] As described above, in the first embodiment, since the discrimination result list RL in which a plurality of classes are arranged in the order of the reliability of each class is output, a highly reliable class discrimination result can be obtained.

[0062] B. Second Embodiment: FIG. 14 is a block diagram showing the function of the class discrimination processing unit 114b in the second embodiment. The class discrimination processing unit 114b includes a similarity calculation unit 310b and an overall determination unit 320b. The overall determination unit 320b includes a within-class selection unit 321 and a list creation unit 322. Note that the configuration of the apparatus shown in FIGS. 1 and 2 and the procedure of the processing shown in FIGS. 6 and 11 are substantially the same as those in the first embodiment.

[0063] In the second embodiment, a plurality of data to be discriminated IM_1, IM_2, IM_3 obtained from the same object to be discriminated are input into the machine learning models 200_1 and 200_2 respectively. When the number of data to be discriminated IM is represented by an integer P, P can be set to any number of 2 or more. When distinguishing individual data to be discriminated IM, the parameter p is used. In the second embodiment, an example using three data to be discriminated IM_1, IM_2, IM_3 will be described. These data to be discriminated IM_1, IM_2, IM_3 are spectral data measured by the spectroscope 30 from the same printing medium.

[0064] The similarity calculation unit 310b calculates, for each individual data to be discriminated IM_p, the class-specific similarity Sclass, which is the similarity between the characteristic spectrum Sp and the known characteristic spectrum group KSp, and obtains the class-specific reliability Rclass_p that depends on the class-specific similarity. The class-specific reliability Rclass_p includes the parameter m indicating the machine learning model, the parameter i indicating the class, the parameter p indicating the data to be discriminated, and the reliability value R_value of the class. In other words, in the second embodiment, P class-specific reliabilities Rclass_p are obtained for each class according to the P data to be discriminated. These class-specific reliabilities Rclass_p are the same as the class-specific reliabilities Rclass described with reference to FIG. 5 in the first embodiment, except that they include the parameter p indicating the data to be discriminated.

[0065] The in-class selection unit 321 determines, for each class, the reliability representative value, which is the statistical representative value among the P reliability values R_value represented by the P class-specific reliabilities Rclass_p, as the reliability value of that class. The "statistical representative value" means the median, the average value, or the mode. As a result, the in-class selection unit 321 outputs the same class-specific reliability Rclass as that described with reference to FIG. 5.

[0066] The list creation unit 322 outputs a discrimination result list RL in which a plurality of classes are arranged in the order of the reliability value R_value according to the class-by-class reliability Rclass given from the in-class selection unit 321. This process is the same as that of the first embodiment.

[0067] As described above, in the second embodiment, a representative reliability value for each class is obtained from among the P class-by-class reliabilities Rclass_p calculated using the P discriminated data obtained from the same object to be discriminated, and the discrimination result list RL is output using these representative reliability values. Therefore, a more reliable class discrimination result can be obtained.

[0068] C. Third Embodiment: FIG. 15 is a block diagram showing the functions of the class discrimination processing unit 114c in the third embodiment. The class discrimination processing unit 114c includes a similarity calculation unit 310c and a comprehensive determination unit 320c. The comprehensive determination unit 320c includes a result selection unit 325. Note that the configuration of the apparatus shown in FIGS. 1 and 2 and the procedure of the process shown in FIG. 6 are substantially the same as those of the first embodiment.

[0069] In the similarity calculation unit 310c of the third embodiment, in addition to the feature spectrum Sp, the class discrimination result Dc obtained at the output layer of the machine learning model 200 is given. This class discrimination result Dc is a result that specifies one of a plurality of classes that can be discriminated by the machine learning model 200 as a discrimination class. The similarity calculation unit 310c calculates a discrimination class similarity Sdc regarding the discrimination class indicated by the class discrimination result Dc, and obtains a model reliability Rmodel that depends on the discrimination class similarity Sdc. The discrimination class similarity Sdc includes only the similarity value regarding the discrimination class and does not include the similarity values of other classes other than the discrimination class, and is the same as the class-by-class similarity Sclass described in the first embodiment above, and is calculated by the same formula as the class-by-class similarity Sclass. Also, as a method for obtaining the model reliability Rmodel from the discrimination class similarity Sdc, the same method as the method for obtaining the class-by-class reliability Rclass from the class-by-class similarity Sclass described in the first embodiment can be used. The model reliability Rmodel includes a parameter m indicating the machine learning model 200, a parameter Dc indicating the discrimination class, and a reliability value Rb.

[0070] The result selection unit 325 outputs, as a class discrimination result FRD for the data to be discriminated, the discrimination class obtained from the machine learning model with the highest model reliability Rmodel among the M discrimination classes discriminated by the M machine learning models 200. Alternatively, as the class discrimination result FRD, a list in which a plurality of classes are arranged in descending order of the model reliability Rmodel may be output. In the example of FIG. 15, the number M of the machine learning models 200 is 2, but in the third embodiment, any value of 2 or more can be adopted as the integer M.

[0071] FIG. 16 is a flowchart showing the processing procedure of the medium discrimination / printing process in the third embodiment. The procedure of FIG. 16 is obtained by replacing steps S230 to S260 in FIG. 11 described in the first embodiment with steps S310 to S340, and the other steps are the same as those in FIG. 11.

[0072] In step S310, the class discrimination processing unit 114c inputs the discrimination data into each of the plurality of machine learning models 200, and calculates a discrimination class Dc and a feature spectrum Sp from the outputs of the respective machine learning models 200. In step S320, the similarity calculation unit 310c calculates a discrimination class similarity Sdc of the discrimination class Dc from the feature spectrum Sp obtained in response to the input of the discrimination data and the registered known feature spectrum group KSp, and obtains a model reliability Rmodel that depends on the discrimination class similarity Sdc. In step S330, the result selection unit 325 outputs a class discrimination result FRD obtained from the machine learning model with a high model reliability Rmodel. This class discrimination result FRD is given to the printing processing unit 112. In step S340, the printing processing unit 112 selects the type of the target printing medium according to the class discrimination result FRD. The subsequent steps S270 and later are the same as those in the first embodiment.

[0073] As described above, in the third embodiment, among the M discrimination classes obtained from the M machine learning models, the discrimination class obtained from the machine learning model with the highest model reliability Rmodel is output as the class discrimination result FRD for the discrimination data, so that a highly reliable class discrimination result can be obtained.

[0074] D. Fourth Embodiment: FIG. 17 is a block diagram showing the functions of the class discrimination processing unit 114d in the fourth embodiment. The class discrimination processing unit 114d includes a similarity calculation unit 310d and an integrated determination unit 320d. The integrated determination unit 320d includes an in-model selection unit 324 and a result selection unit 325. Note that the configuration of the apparatus shown in FIGS. 1 and 2 and the procedure of the process shown in FIG. 6 are substantially the same as those in the first embodiment, and the procedure of the process shown in FIG. 16 is substantially the same as that in the third embodiment.

[0075] In the fourth embodiment, as in the second embodiment, a plurality of data to be discriminated IM_1, IM_2, IM_3 obtained from the same object to be discriminated are input into the machine learning models 200_1 and 200_2, respectively. Similarity calculation unit 310d is given a feature spectrum Sp and a class discrimination result Dc from each machine learning model 200, as in the third embodiment.

[0076] For each individual data to be discriminated IM_p, the similarity calculation unit 310d calculates a discrimination class similarity Sdc with respect to the discrimination class Dc, and obtains a model reliability Rmodel_p that depends on the discrimination class similarity Sdc. The model reliability Rmodel_p includes a parameter m indicating the machine learning model, a parameter Dc indicating the discrimination class, a parameter p indicating the data to be discriminated, and a model reliability value Rb. In other words, in the fourth embodiment, P model reliabilities Rmodel_p are obtained for each machine learning model according to the P data to be discriminated. These model reliabilities Rmodel_p are the same as the model reliability Rmodel described with reference to FIG. 15 in the third embodiment, except that they include a parameter p indicating the data to be discriminated.

[0077] For each machine learning model 200, the in-model selection unit 324 selects the most frequent class among the P discrimination classes Dc included in the P model reliabilities Rmodel_p as the discrimination class Dc in that machine learning model 200, and adopts the model reliability including that discrimination class Dc. As a result, the in-model selection unit 324 outputs a model reliability Rmodel similar to that described with reference to FIG. 15. The result selection unit 325 outputs a class discrimination result FRD obtained from the machine learning model with the highest model reliability. This process is the same as that in the third embodiment.

[0078] As described above, in the fourth embodiment, by using the P data to be discriminated obtained from the same object to be discriminated, the most frequent class among the P discrimination classes obtained from each machine learning model is adopted as the discrimination class of that machine learning model, so that a more reliable class discrimination result can be obtained.

[0079] E. Fifth Embodiment: FIG. 18 is a block diagram showing the functions of the class discrimination processing unit 114e in the fifth embodiment. The class discrimination processing unit 114e includes a similarity calculation unit 310e and an integrated determination unit 320e. The integrated determination unit 320e includes a within-class selection unit 326, a within-model selection unit 327, and a result selection unit 328. Note that the configuration of the apparatus shown in FIGS. 1 and 2 and the procedure of the process shown in FIG. 6 are substantially the same as those in the first embodiment, and the procedure of the process shown in FIG. 16 is substantially the same as those in the third embodiment.

[0080] In the fifth embodiment, as in the second embodiment, a plurality of discriminated data IM_1, IM_2, IM_3 obtained from the same object to be discriminated are input to the machine learning models 200_1, 200_2, respectively. Similar to the second embodiment, the feature spectrum Sp is given to the similarity calculation unit 310e from each machine learning model 200.

[0081] The similarity calculation unit 310e calculates a class-specific similarity Sclass_p, which is the similarity between the feature spectrum Sp and the known feature spectrum group KSp, for each discriminated data IM_p. The class-specific similarity Sclass_p includes a parameter m indicating the machine learning model, a parameter i indicating the class, a parameter p indicating the discriminated data, and a similarity value S_value of the class. In other words, in the fifth embodiment, P class-specific similarities Sclass_p are obtained according to the P discriminated data.

[0082] The within-class selection unit 326 determines, for each class, a representative similarity value, which is a statistical representative value among the P similarity values S_value represented by the P class-specific similarities Sclass_p, as the similarity value of that class. The "statistical representative value" means the median, the average value, or the mode. As a result, the within-class selection unit 326 outputs a class-specific similarity Sclass including a parameter m indicating the machine learning model, a parameter i indicating the class, and a similarity value S_value of the class.

[0083] For each machine learning model 200, the in-model selection unit 327 determines, as the discrimination class Dc corresponding to the machine learning model 200, the class corresponding to the highest similarity value among the similarity values S_value of each class represented by the class-by-class similarity Sclass. Also, the in-model selection unit 327 adopts the similarity value S_value for this discrimination class Dc as the discrimination class similarity of the discrimination class Dc, and obtains a reliability value Rb of the model reliability Rmodel that depends on the discrimination class similarity S_value. As the method for obtaining the model reliability Rmodel from the discrimination class similarity S_value, the same method as the method for obtaining the class-by-class reliability Rclass from the class-by-class similarity Sclass described in the first embodiment can be used. As a result, from the in-model selection unit 327, a model reliability Rmodel including a parameter m indicating the model, a parameter Dc indicating the discrimination class, and a reliability value Rb is output, similar to the case of FIG. 17. The result selection unit 325 outputs the class discrimination result FRD obtained from the machine learning model with the highest model reliability Rmodel. This process is the same as that in the fourth embodiment.

[0084] Note that in the fifth embodiment, in step S310 of the procedure shown in FIG. 16, instead of using the discrimination result obtained from the output layer of the machine learning model 200, the class corresponding to the highest similarity value among the similarity values S_value of each class represented by the class-by-class similarity Sclass is determined as the discrimination class Dc corresponding to the machine learning model 200, which is different from the fourth embodiment. In this fifth embodiment, it is also possible to consider that the process of determining the discrimination class is performed in step S320 of FIG. 16.

[0085] As described above, in the fifth embodiment, P class-by-class similarities are obtained for each machine learning model using the P discriminated data obtained from the same object to be discriminated, and the discrimination class is determined using the P class-by-class similarities, so that a more reliable class discrimination result can be obtained.

[0086] F. Method for calculating similarity: As the above-described method for calculating the similarity by class, for example, any of the following three methods can be adopted. (1) The first calculation method M1 for obtaining the similarity by class without considering the correspondence of the partial region Rn in the feature spectrum Sp and the known feature spectrum group KSp (2) The second calculation method M2 for obtaining the similarity by class between the corresponding partial regions Rn of the feature spectrum Sp and the known feature spectrum group KSp (3) The third calculation method M3 for obtaining the similarity by class without considering the partial region Rn at all Hereinafter, according to these three calculation methods M1, M2, and M3, the method for calculating the similarity by class Sclass_ConvVN1 from the output of the ConvVN1 layer 230 will be sequentially described. In the following description, the parameters m of the machine learning model 200 and the parameter q of the data to be discriminated are omitted.

[0087] FIG. 19 is an explanatory diagram showing the first calculation method M1 of the similarity by class. In the first calculation method M1, first, from the output of the ConvVN1 layer 230 which is a specific layer, the local similarity S(i, j, k) indicating the similarity for each class i for each partial region k is calculated. Then, from these local similarities S(i, j, k), any of the three types of similarities by class Sclass(i, j) shown on the right side of FIG. 19 is calculated. The similarity by class Sclass(i, j) is the same as the similarity by class Sclass_ConvVN1 shown in FIGS. 3 and 12.

[0088] In the first calculation method M1, the local similarity S(i, j, k) is calculated using the following formula. S(i, j, k)=max[G{Sp(j, k), KSp(i, j, k = all, q = all)}] (c1) Here, i is a parameter indicating the class, j is a parameter indicating the specific layer, k is a parameter indicating the partial region Rn, q is a parameter indicating the data number, G{a, b} is a function for obtaining the similarity between a and b, Sp(j, k) is a feature spectrum obtained from the output of a specific partial region k of a specific layer j according to the data to be discriminated. KSp(i, j, k = all, q = all) is the known feature spectrum of all data numbers q in all partial regions k of a specific layer j associated with class i among the known feature spectrum group KSp shown in FIG. 10. max[X] is a logical operation that takes the maximum value among the values of X. Note that as the function G{a, b} for obtaining the similarity, for example, an expression for obtaining the cosine similarity or an expression for obtaining the similarity according to the distance can be used.

[0089] The three types of class-specific similarities Sclass(i, j) shown on the right side of FIG. 19 are obtained by taking the maximum value, average value, or minimum value of the local similarities S(i, j, k) for a plurality of partial regions k for each class i. Which operation of the maximum value, average value, or minimum value to use depends on the purpose of use of the class discrimination process. For example, when the purpose is to discriminate an object using a natural image, it is preferable to obtain the class-specific similarity Sclass(i, j) by taking the maximum value of the local similarity S(i, j, k) for each class i. Also, when the purpose is to discriminate the type of a printed medium or when the purpose is to perform a pass / fail determination using an image of an industrial product, it is preferable to obtain the class-specific similarity Sclass(i, j) by taking the minimum value of the local similarity S(i, j, k) for each class i. Also, there may be a case where it is preferable to obtain the class-specific similarity Sclass(i, j) by taking the average value of the local similarity S(i, j, k) for each class i. Which of these three types of operations to use is preset by the user experimentally or empirically.

[0090] As described above, in the first calculation method M1 of the class-specific similarity, (1) According to the data to be discriminated, obtain the local similarity S(i, j, k), which is the similarity between the feature spectrum Sp obtained from the output of a specific partial region k of a specific layer j and all the known feature spectra KSp associated with that specific layer j and each class i. (2) For each class i, the class - specific similarity Sclass(i,j) is obtained by taking the maximum value, average value, or minimum value of the local similarities S(i,j,k) for a plurality of sub - regions k. According to this first calculation method M1, the class - specific similarity Sclass(i,j) can be obtained by relatively simple calculations and procedures.

[0091] FIG. 20 is an explanatory diagram showing a second calculation method M2 of the class - specific similarity. In the second calculation method M2, the local similarity S(i,j,k) is calculated using the following formula instead of the above - mentioned formula (c1). S(i,j,k)=max[G{Sp(j,k), KSp(i,j,k,q = all)}] (c2) Here, KSp(i,j,k,q = all) is the known feature spectrum of all data numbers q in a specific sub - region k of a specific layer j associated with class i among the known feature spectrum groups KSp shown in FIG. 10.

[0092] In the above - mentioned first calculation method M1, the known feature spectrum KSp(i,j,k = all,q = all) in all sub - regions k of a specific layer j is used, while in the second calculation method M2, only the known feature spectrum KSp(i,j,k,q = all) for the same sub - region k as the sub - region k of the feature spectrum Sp(j,k) is used. Other methods in the second calculation method M2 are the same as those in the first calculation method M1.

[0093] In the second calculation method M2 of the class - specific similarity, (1) The local similarity S(i,j,k), which is the similarity between the feature spectrum Sp obtained from the output of a specific sub - region k of a specific layer j according to the data to be discriminated and all the known feature spectra KSp associated with that specific sub - region k of that specific layer j and each class i, is obtained. (2) For each class i, the class - specific similarity Sclass(i,j) is obtained by taking the maximum value, average value, or minimum value of the local similarities S(i,j,k) for a plurality of sub - regions k. Even with this second calculation method M2, the class similarity Sclass(i,j) can be obtained by relatively simple calculations and procedures.

[0094] FIG. 21 is an explanatory diagram showing a third calculation method M3 for class similarity. In the third calculation method M3, the class similarity Sclass(i,j) is calculated from the output of the ConvVN1 layer 230, which is a specific layer, without obtaining the local similarity S(i,j,k).

[0095] The class similarity Sclass(i,j) obtained by the third calculation method M3 is calculated using the following formula. Sclass(i,j)=max[G{Sp(j,k=all), KSp(i,j,k=all,q=all)}] (c3) Here, Sp(j,k=all) is a feature spectrum obtained from the outputs of all sub-regions k of the specific layer j according to the data to be discriminated.

[0096] As described above, in the third calculation method M3 for class similarity, (1) For each class, the class similarity Sclass(i,j), which is the similarity between all feature spectra Sp obtained from the output of the specific layer j according to the data to be discriminated and all known feature spectra KSp associated with the specific layer j and each class i, is obtained respectively. According to this third calculation method M3, the class similarity Sclass(i,j) can be obtained by even simpler calculations and procedures.

[0097] The three calculation methods M1 to M3 described above are all methods for calculating the class similarity for each individual specific layer i. As described above, in this embodiment, the class similarity can be calculated using one or more of the plurality of vector neuron layers 230, 240, 250 shown in FIG. 3 as the specific layer. When using a plurality of specific layers, for example, the class similarity can be determined as follows.

[0098] FIG. 22 is an explanatory diagram showing a method for selecting class-wise similarity when using a plurality of specific layers. In this method, among the plurality of specific layers, the class-wise similarity obtained from the specific layer showing the most statistically significant determination result is selected. In the example of FIG. 22, the ConvVN1 layer 230 and the ConvVN2 layer 240 are used as specific layers. First, for each partial region k of the ConvVN1 layer 230, a process is executed to determine the class for which the local similarity S(i, j, k) becomes the maximum value, and assign the class parameter value i of that class to each partial region k. Note that the class parameter value i is a value indicating the order among a plurality of classes. Similarly, for the ConvVN2 layer 240, a process is executed to determine the class for which the local similarity S(i, j, k) becomes the maximum value, and assign the class parameter value i of that class to each partial region k.

[0099] In the method of FIG. 22, further, for each specific layer, the variance is calculated regarding the distribution of the class parameter values i in the plurality of partial regions k. This variance is a statistical variance value for the class parameter value i. In the example of FIG. 22, the variance of the ConvVN1 layer 230 is 0.14, and the variance of the ConvVN2 layer 240 is 0.22. Among these specific layers 230, 240, it is expected that the more biased the distribution of the class parameter value i is, the clearer the determination result is. Therefore, the class-wise similarity for the specific layer with a lower variance is adopted. In other words, among the plurality of specific layers, the class-wise similarity obtained for the specific layer with the smallest variance is adopted. According to this method, even when a plurality of class-wise similarities are obtained using a plurality of specific layers, an appropriate class-wise similarity can be selected from among them.

[0100] G. Calculation method for output vectors of each layer of the machine learning model: The calculation method for the output of each layer in the machine learning model 200 shown in FIG. 3 is as follows. The machine learning model 200 shown in FIG. 4 is also the same except for the values of individual parameters.

[0101] Each node of the PrimeVN layer 220 regards the scalar output of 1×1×32 nodes of the Conv layer 210 as a 32-dimensional vector, and obtains the vector output of that node by multiplying this vector by a transformation matrix. This transformation matrix is an element of a kernel with a surface size of 1×1 and is updated by the learning of the machine learning model 200. It should be noted that it is also possible to integrate the processing of the Conv layer 210 and the PrimeVN layer 220 and configure them as one primary vector neuron layer.

[0102] When the PrimeVN layer 220 is called the "lower layer L" and the ConvVN1 layer 230 adjacent to its upper side is called the "upper layer L+1", the output of each node of the upper layer L+1 is determined using the following formula.

Equation

[0103] As the normalization function F(X), for example, the following formula (E3a) or (E3b) can be used.

number

[0104] In the above formula (E3a), the sum vector u j Norm of |u j The activation value a is normalized by the softmax function j On the other hand, in equation (E3b), the sum vector u j Norm of |u j | is the norm |u j Activation value a by dividing by the sum of | j It should be noted that the normalization function F(X) may be a function other than equation (E3a) or (E3b).

[0105] The ordinal number i in the above equation (E2) is the output vector M L+1 j The integer n is a number that is assigned for convenience to the nodes in the lower layer L used to determine the output vector M L+1 j is the number of nodes in the lower layer L used to determine . Thus, the integer n is given by n = Nk × Nc (E5) Here, Nk is the surface size of the kernel, and Nc is the number of channels in the lower layer, the PrimeVN layer 220. In the example of FIG.

[0106] One of the kernels used to obtain the output vector of the ConvVN1 layer 230 has 1×5×26 = 130 elements with a kernel size of 1×5 as the surface size and 26 channels of the lower layer as the depth. Each of these elements is the prediction matrix W L ij is. Also, to generate the output vectors of 20 channels of the ConvVN1 layer 230, 20 sets of this kernel are required. Therefore, the number of prediction matrices W L ij used to obtain the output vector of the ConvVN1 layer 230 is 130×20 = 2600. These prediction matrices W L ij are updated by the learning of the machine learning model 200.

[0107] As can be seen from the above equations (E1) to (E4), the output vector M L+1 j of each individual node in the upper layer L+1 is obtained by the following operations. (a) Multiply the output vector M L i of each node in the lower layer L by the prediction matrix W L ij to obtain the prediction vector v ij , (b) Obtain the sum vector u ij which is the sum, i.e., the linear combination, of the prediction vectors v j obtained from each node in the lower layer L, (c) Obtain the activation value a j which is the normalization coefficient by normalizing the norm |u j | of the sum vector u j , (d) Divide the sum vector u j by the norm |u j | and further multiply by the activation value a j .

[0108] Note that the activation value a j is the norm |u jis a normalization coefficient obtained by normalizing |. Therefore, the activation value a j can be considered as an index indicating the relative output intensity of each node among all the nodes in the upper layer L+1. The norms used in equations (E3), (E3a), (E3b), and (4) are, in typical examples, the L2 norm representing the vector length. At this time, the activation value a j corresponds to the vector length of the output vector M L+1 j . The activation value a j is only used in the above-mentioned equations (E3) and (E4), so it is not necessary to be output from the node. However, it is also possible to configure the upper layer L+1 to output the activation value a j externally.

[0109] The configuration of the vector neural network is almost the same as that of the capsule network, and the vector neurons of the vector neural network correspond to the capsules of the capsule network. However, the operations according to the above-mentioned equations (E1) to (E4) used in the vector neural network are different from the operations used in the capsule network. The biggest difference between the two is that in the capsule network, weights are multiplied by the prediction vectors v ij on the right side of the above equation (E2), respectively, and the weights are searched by repeating the dynamic routing multiple times. On the other hand, in the vector neural network of the present embodiment, the output vector M L+1 j is obtained by calculating the above-mentioned equations (E1) to (E4) in order once, so there is no need to repeat the dynamic routing, and there is an advantage that the operation is faster. In addition, the vector neural network of the present embodiment has an advantage that the amount of memory required for the operation is less than that of the capsule network, and according to the experiments of the inventors of the present disclosure, it only requires about 1 / 2 to 1 / 3 of the memory amount.

[0110] In terms of using nodes that take vectors as inputs and outputs, the vector neural network is the same as the capsule network. Therefore, the advantages of using vector neurons are also common to the capsule network. Also, the fact that the multiple layers 210 to 250 represent the features of larger regions as you go up and the features of smaller regions as you go down is the same as in a normal convolutional neural network. Here, "features" means the characteristic parts contained in the input data to the neural network. In a vector neural network or a capsule network, the output vector of a certain node contains spatial information representing the spatial information of the features represented by that node, which is superior to a normal convolutional neural network. That is, the vector length of the output vector of a certain node represents the probability of existence of the features represented by that node, and the vector direction represents spatial information such as the direction and scale of the features. Therefore, the vector directions of the output vectors of two nodes belonging to the same layer represent the positional relationship of the respective features. Or, it can be said that the vector directions of the output vectors of the two nodes represent the variations of the features. For example, for a node corresponding to the feature of "eyes", the direction of the output vector can represent variations such as the thinness of the eyes and the way they are lifted. In a normal convolutional neural network, it is said that the spatial information of the features is lost due to the pooling process. As a result, the vector neural network and the capsule network have the advantage of being superior in performance in identifying input data compared to a normal convolutional neural network.

[0111] The advantages of the vector neural network can also be considered as follows. That is, in the vector neural network, the output vector of a node has the advantage of expressing the features of the input data as coordinates in a continuous space. Therefore, the output vector can be evaluated such that if the vector directions are close, the features are similar. Also, there are advantages such as being able to interpolate and discriminate features even if the features contained in the input data cannot be covered by the teacher data. On the other hand, a normal convolutional neural network has the drawback that since unordered compression is applied by the pooling process, it cannot express the features of the input data as coordinates in a continuous space.

[0112] The outputs of the nodes of the ConvVN2 layer 240 and the ClassVN layer 250 are also determined in the same way using the above-mentioned equations (E1) to (E4), so detailed explanations are omitted. The resolution of the topmost ClassVN layer 250 is 1×1, and the number of channels is n1.

[0113] The output of the ClassVN layer 250 is converted into a plurality of determination values Class0 to Class2 for known classes. These determination values are usually values normalized by a softmax function. Specifically, for example, by calculating the vector length of the output vector from the output vector of each node of the ClassVN layer 250 and further normalizing the vector lengths of each node by the softmax function, determination values for individual classes can be obtained. As described above, the activation value a j is a value corresponding to the vector length of the output vector M L+1 j and is normalized. Therefore, the activation value a j output from each node of the ClassVN layer 250 may be directly used as the determination value for each class.

[0114] In the above-described embodiment, the vector neural network that obtains the output vector by the operations of the above equations (E1) to (E4) is used as the machine learning model 200. Instead of this, a capsule network disclosed in U.S. Patent No. 5,210,798 or International Publication No. 2009 / 083553 may be used.

[0115] · Other embodiments: The present disclosure is not limited to the above-described embodiments, and can be realized in various forms without departing from the gist thereof. For example, the present disclosure can also be realized by the following aspects. The technical features in the above embodiments corresponding to the technical features in each of the following aspects can be appropriately replaced or combined in order to solve part or all of the problems of the present disclosure, or to achieve part or all of the effects of the present disclosure. Further, if the technical feature is not described as essential in this specification, it can be appropriately deleted.

[0116] <1>According to a first aspect of the present disclosure, when M is an integer of 1 or more, a method for discriminating the class of data to be discriminated is provided using M machine learning models of a vector neural network type having a plurality of vector neuron layers. Let Nm be an integer of 2 or more, which is the number of classes discriminable by the m-th machine learning model among the M machine learning models, and let ΣNm be the total number of classes discriminable by the M machine learning models. The method includes: (a) a step of preparing M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of specific layers among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of the Nm classes are input to each machine learning model; and (b) a step of performing a class discrimination process on the data to be discriminated using the M machine learning models and the M sets of known feature spectrum groups. The step (b) includes: (b1) a step of calculating M feature spectra from the outputs of the specific layers in response to the input of the data to be discriminated to the M machine learning models; (b2) a step of obtaining M sets of class-wise reliabilities associated with the M machine learning models by calculating class-wise similarities for each of the Nm classes in each machine learning model as similarities between the M feature spectra and the M sets of known feature spectrum groups, and obtaining class-wise reliabilities depending on the class-wise similarities; and (b3) a step of outputting a discrimination result list in which a plurality of classes, which are at least a part of the ΣNm classes, are arranged in the order of the reliabilities of the classes represented by the M sets of class-wise reliabilities. According to this method, since a discrimination result list in which a plurality of classes are arranged in the order of the reliabilities of the classes is output, a highly reliable class discrimination result can be obtained.

[0117] <2>When P is an integer of 2 or more in the above method, the discrimination target data includes P pieces of discrimination target data obtained from the same object to be discriminated. The step (b1) includes a step of calculating P feature spectra from the outputs of the machine learning models according to the input of the P pieces of discrimination target data to each machine learning model. The step (b2) includes, for each class reliability of each of the M sets of class reliabilities, using the P feature spectra obtained from each machine learning model, obtaining P class reliabilities as the class reliability of each set, and determining a reliability representative value, which is a statistical representative value among the P reliability values for each class represented by the P class reliabilities, as the reliability value of each class represented by each set of class reliabilities. According to this method, a reliability representative value for each class is obtained from among the P class reliabilities calculated using the P pieces of discrimination target data obtained from the same object to be discriminated, and a discrimination result list is output using these reliability representative values, so that a more reliable class discrimination result can be obtained.

[0118] <3>In the above method, the class reliability may be any one of (1) equal to the class similarity, (2) the result of taking a weighted average of the similarity values of each class in the class similarity and the activation value corresponding to the determination value of each class in the output layer of the machine learning model, or (3) the result of multiplying the similarity value of each class in the class similarity by the activation value for each class. According to this method, the class reliability can be obtained by simple calculations.

[0119] <4>According to a second aspect of the present disclosure, when M is an integer of 2 or more, there is provided a method for discriminating the class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers. This method includes: (a) a step of preparing M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from the outputs of specific layers among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of Nm classes are input to each machine learning model; and (b) a step of performing a class discrimination process for the data to be discriminated using the M machine learning models and the M sets of known feature spectrum groups. The step (b) includes: (b1) a step of calculating M feature spectra from the outputs of the specific layers and determining M discrimination classes in response to the input of the data to be discriminated to the M machine learning models; (b2) a step of calculating discrimination class similarities for each of the M discrimination classes as similarities between the M feature spectra and the M sets of known feature spectrum groups, and obtaining model reliabilities for each of the M machine learning models by obtaining model reliabilities that depend on the discrimination class similarities; and (b3) a step of outputting, as a class discrimination result for the data to be discriminated, the discrimination class obtained from the machine learning model having the highest model reliability among the M discrimination classes. According to this method, among the M discrimination classes obtained from the M machine learning models, the discrimination class obtained from the machine learning model having the highest model reliability is output as the class discrimination result for the data to be discriminated, so that a highly reliable class discrimination result can be obtained.

[0120] <5>In the above method, when P is an integer of 2 or more, the data to be discriminated includes P pieces of data to be discriminated obtained from the same object to be discriminated. The step (b1) may include: a step of determining P discrimination classes according to the input of the P pieces of data to be discriminated to each machine learning model; and a step of selecting, as the discrimination class in the machine learning model, the most frequent class among the P discrimination classes obtained from each machine learning model. According to this method, by using the P pieces of data to be discriminated obtained from the same object to be discriminated and adopting, as the discrimination class of the machine learning model, the most frequent class among the P discrimination classes obtained from each machine learning model, a more reliable class discrimination result can be obtained.

[0121] <6>In the above method, when P is an integer of 2 or more, the data to be discriminated includes P pieces of data to be discriminated obtained from the same object to be discriminated. The step (b1) may include: for each of the M sets of machine learning models, a step of obtaining P class-based similarities as the class-based similarities associated with the machine learning model by using the P feature spectra obtained from each machine learning model; a step of determining, as the similarity value of each class represented by the P class-based similarities, a representative similarity value which is a statistical representative value among the P similarity values regarding each class represented by the P class-based similarities; and a step of determining, as the discrimination class corresponding to the machine learning model, the class corresponding to the highest similarity value among the similarity values of each class represented by the class-based similarities. The step (b2) may include a step of adopting, as the discrimination class similarity, the highest similarity value used when determining the discrimination class corresponding to each machine learning model in the step (b1). According to this method, by using the P pieces of data to be discriminated obtained from the same object to be discriminated to obtain P class-based similarities for each machine learning model and determining the discrimination class by using the P class-based similarities, a more reliable class discrimination result can be obtained.

[0122] <7>In the above method, the model reliability may be any of the following: (1) equal to the discrimination class similarity; (2) the weighted average of the discrimination class similarity and the activation value corresponding to the determination value of the discrimination class in the output layer of the machine learning model; or (3) the result of multiplying the discrimination class similarity by the activation value for the discrimination class. According to this method, the model reliability can be obtained by simple calculations.

[0123] <8>In the above method, the integer M is 2 or more, the M machine learning models are trained using corresponding M teacher data groups, and the M teacher data groups may be in a state equivalent to being grouped into N sets by clustering processing. According to this method, since the teacher data groups used for training each machine learning model are grouped by clustering processing, the accuracy of the class discrimination processing by the machine learning model can be improved.

[0124] <9>In the above method, the specific layer has a configuration in which vector neurons arranged in a plane defined by two axes, a first axis and a second axis, are arranged as a plurality of channels along a third axis in a direction different from the two axes. In the specific layer, a region defined by a plane position defined by the position of the first axis and the position of the second axis and including the plurality of channels along the third axis is called a partial region. The feature spectrum may be obtained as any of the following for each partial region among the plurality of partial regions included in the specific layer: (i) a first type of feature spectrum in which a plurality of element values of the output vectors of the respective vector neurons included in the partial region are arranged over the plurality of channels along the third axis; (ii) a second type of feature spectrum obtained by multiplying each element value of the first type of feature spectrum by a normalization coefficient corresponding to the vector length of the output vector; and (iii) a third type of feature spectrum in which the normalization coefficient is arranged over the plurality of channels along the third axis. According to this method, the similarity can be obtained by using any one of the three characteristic spectra obtained from the output vector of the specific layer.

[0125] <10>In the above method, the step (b2) may include: according to the data to be discriminated, obtaining a local similarity, which is the similarity between the characteristic spectrum obtained from the output of a specific partial region of the specific layer and all the known characteristic spectra associated with the specific layer and each class, to obtain a plurality of local similarities indicating the similarity for each class with respect to the plurality of partial regions of the specific layer; and for each class, obtaining the class-specific similarity or the discriminant class similarity by taking the maximum value, average value, or minimum value of the plurality of local similarities with respect to the plurality of partial regions. According to this method, the class-specific similarity or the discriminant class similarity can be calculated by relatively simple operations.

[0126] <11>In the above method, the step (b2) may include: according to the data to be discriminated, obtaining a local similarity, which is the similarity between the characteristic spectrum obtained from the output of a specific partial region of the specific layer and all the known characteristic spectra associated with the specific partial region of the specific layer and each class, to obtain a plurality of local similarities indicating the similarity for each class with respect to the plurality of partial regions of the specific layer; and for each class, obtaining the class-specific similarity or the discriminant class similarity by taking the maximum value, average value, or minimum value of the plurality of local similarities with respect to the plurality of partial regions. According to this method, the class-specific similarity or the discriminant class similarity can be calculated by relatively simple operations.

[0127] <12>In the above method, the step (b2) may include a step of obtaining the class-specific similarity or the discrimination-class similarity by respectively obtaining, for each class, the similarity between all the feature spectra obtained from the output of the specific layer and all the known feature spectra associated with the specific layer and each class, according to the data to be discriminated. According to this method, the class-specific similarity or the discrimination-class similarity can be calculated by a simpler operation.

[0128] <13>According to a third aspect of the present disclosure, when M is an integer of 1 or more, there is provided an information processing apparatus that discriminates a class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers. This information processing apparatus includes a memory that stores the M machine learning models, and a processor that executes operations using the M machine learning models. Let Nm, which is an integer of 2 or more, be the number of classes discriminable by the m-th machine learning model among the M machine learning models, and let ΣNm be the total number of classes discriminable by the M machine learning models. The processor executes: (a) a process of reading from the memory M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of specific layers among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of the Nm classes are input to each machine learning model; and (b) a process of executing a class discrimination process for the data to be discriminated using the M machine learning models and the M sets of known feature spectrum groups. The process (b) includes: (b1) a process of calculating M feature spectra from the outputs of the specific layers in response to input of the data to be discriminated to the M machine learning models; (b2) a process of obtaining M sets of class-wise reliability associated with the M machine learning models by calculating class-wise similarity for each of the Nm classes in each machine learning model as the similarity between the M feature spectra and the M sets of known feature spectrum groups, and obtaining class-wise reliability depending on the class-wise similarity; and (b3) a process of outputting a discrimination result list in which a plurality of classes, which are at least a part of the ΣNm classes, are arranged in the order of the reliability of each class represented by the M sets of class-wise reliability. According to this information processing apparatus, since a discrimination result list in which a plurality of classes are arranged in the order of the reliability of each class is output, a highly reliable class discrimination result can be obtained.

[0129] <14>According to a fourth aspect of the present disclosure, when M is an integer of 2 or more, there is provided an information processing apparatus that discriminates a class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers. This information processing apparatus includes a memory that stores the M machine learning models, and a processor that executes operations using the M machine learning models. The processor executes: (a) a process of reading from the memory M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of a specific layer among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of Nm classes are input to each machine learning model; and (b) a process of executing a class discrimination process for the data to be discriminated using the M machine learning models and the M sets of known feature spectrum groups. The process (b) includes: (b1) a process of calculating M feature spectra from the output of the specific layer and determining M discrimination classes in response to input of the data to be discriminated to the M machine learning models; (b2) a process of obtaining model reliabilities for each of the M machine learning models by calculating discrimination class similarities for each of the M discrimination classes as similarities between the M feature spectra and the M sets of known feature spectrum groups, and obtaining a model reliability that depends on the discrimination class similarities; and (b3) a process of outputting, as a class discrimination result for the data to be discriminated, the discrimination class obtained from the machine learning model having the highest model reliability among the M discrimination classes. According to this information processing apparatus, among the M discrimination classes obtained from the M machine learning models, the discrimination class obtained from the machine learning model having the highest model reliability is output as the class discrimination result for the data to be discriminated, so that a highly reliable class discrimination result can be obtained.

[0130] <15>According to a fifth aspect of the present disclosure, when M is an integer of 1 or more, there is provided a computer program that causes a processor to execute a class discrimination process for discriminating a class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers. Let Nm be an integer of 2 or more, which is the number of classes discriminable by the m-th machine learning model among the M machine learning models, and let ΣNm be the total number of classes discriminable by the M machine learning models. The computer program causes the processor to execute: (a) a process of reading from a memory M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of a specific layer among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of the Nm classes are input to each machine learning model; and (b) a process of executing a class discrimination process for the data to be discriminated using the M machine learning models and the M sets of known feature spectrum groups. The process (b) includes: (b1) a process of calculating M feature spectra from the output of the specific layer in response to input of the data to be discriminated to the M machine learning models; (b2) a process of obtaining M sets of class-specific reliabilities associated with the M machine learning models by calculating class-specific similarities for each of the Nm classes in each machine learning model as similarities between the M feature spectra and the M sets of known feature spectrum groups, and obtaining class-specific reliabilities depending on the class-specific similarities; and (b3) a process of outputting a discrimination result list in which a plurality of classes, which are at least a part of the ΣNm classes, are arranged in the order of the reliabilities of the classes represented by the M sets of class-specific reliabilities. According to this computer program, since a discrimination result list in which a plurality of classes are arranged in the order of the reliabilities of the classes is output, a highly reliable class discrimination result can be obtained.

[0131] <16>According to a sixth aspect of the present disclosure, when M is an integer of 2 or more, there is provided a computer program that causes a processor to execute a class discrimination process for discriminating a class of data to be discriminated, using M machine learning models of a vector neural network type having a plurality of vector neuron layers. This computer program includes: (a) a process of reading from a memory M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of specific layers among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of Nm classes are input to each machine learning model; and (b) a process of executing a class discrimination process for the data to be discriminated, using the M machine learning models and the M sets of known feature spectrum groups. The process (b) includes: (b1) a process of calculating M feature spectra from the outputs of the specific layers and determining M discrimination classes in response to input of the data to be discriminated to the M machine learning models; (b2) a process of obtaining model reliability levels for each of the M machine learning models by calculating discrimination class similarity levels for each of the M discrimination classes as similarity levels between the M feature spectra and the M sets of known feature spectrum groups, and obtaining a model reliability level that depends on the discrimination class similarity levels; and (b3) a process of outputting, as a class discrimination result for the data to be discriminated, the discrimination class obtained from the machine learning model having the highest model reliability level among the M discrimination classes. According to this computer program, among the M discrimination classes obtained from the M machine learning models, the discrimination class obtained from the machine learning model having the highest model reliability level is output as the class discrimination result for the data to be discriminated, so that a highly reliable class discrimination result can be obtained.

[0132] The present disclosure can also be realized in various forms other than those described above. For example, it can be realized in the form of a computer program for realizing the function of the class discrimination device, a non-transitory storage medium recording the computer program, and the like.

Explanation of Reference Numerals

[0133] 10…Printer, 20…Information processing device, 30…Spectrophotometer, 110…Processor, 112…Printing processing unit, 114…Class discrimination processing unit, 120…Memory, 130…Interface circuit, 150…Display unit, 200…Machine learning model, 210…Convolutional layer, 220…Primary vector neuron layer, 230…First convolutional vector neuron layer, 240…Second convolutional vector neuron layer, 250…Classification vector neuron layer, 310…Similarity calculation unit, 320…Comprehensive determination unit, 321…Intra-class selection unit, 322…List creation unit, 324…In-model selection unit, 325…Result selection unit, 326…Intra-class selection unit, 327…In-model selection unit, 328…Result selection unit

Claims

1. A method for discriminating the class of data to be discriminated, using M machine learning models of a vector neural network type having a plurality of vector neuron layers when M is an integer of 1 or more, wherein, when the number of classes discriminable by the m-th machine learning model among the M machine learning models is Nm which is an integer of 2 or more, and the total number of classes discriminable by the M machine learning models is ΣNm, the method includes: (a) A step of preparing M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from the output of a specific layer among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of the Nm classes are input to each machine learning model; (b) A step of performing a class discrimination process on the data to be discriminated using the M machine learning models and the M sets of known feature spectrum groups; and includes: The step (b) includes: (b1) A step of calculating M feature spectra from the output of the specific layer in response to the input of the data to be discriminated to the M machine learning models; (b2) As the similarity between the M feature spectra and the M sets of known feature spectrum groups, calculating a class-specific similarity for each of the Nm classes in each machine learning model and obtaining M sets of class-specific reliabilities associated with the M machine learning models by obtaining a class-specific reliability that depends on the class-specific similarity; (b3) A step of outputting a discrimination result list in which a plurality of classes, which are at least a part of the ΣNm classes, are arranged in the order of the reliabilities of the classes represented by the M sets of class-specific reliabilities. A method including the above steps.

2. The method according to Claim 1, wherein when P is an integer of 2 or more, the data to be discriminated includes P pieces of data to be discriminated obtained from the same object to be discriminated, the step (b1) includes a step of calculating P feature spectra from the output of each machine learning model in response to the input of the P pieces of data to be discriminated to each machine learning model, and the step (b2) includes, regarding the class-specific reliabilities of each set among the M sets of class-specific reliabilities, a step of obtaining P class-specific reliabilities as the class-specific reliabilities of each set using the P feature spectra obtained from each machine learning model. A step of determining a reliability representative value, which is a statistical representative value among the P reliability values for each class represented by the P class-specific reliabilities, as the reliability value for each class represented by the class-specific reliabilities of each group; A method including the above.

3. The method according to claim 1 or 2, wherein the class-specific reliability is (1) equal to the class-specific similarity, or (2) the result of taking a weighted average of the similarity value of each class in the class-specific similarity and the activation value corresponding to the determination value of each class in the output layer of the machine learning model, or (3) the result of multiplying the similarity value of each class in the class-specific similarity by the activation value for each class, and is any one of the above.

4. When M is an integer of 2 or more, a method for discriminating the class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers, comprising: (a) A step of preparing M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from the output of a specific layer among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of Nm classes are input to each machine learning model; (b) A step of performing a class discrimination process on the data to be discriminated using the M machine learning models and the M sets of known feature spectrum groups; including The step (b) includes: (b1) A step of calculating M feature spectra from the output of the specific layer and determining M discrimination classes in response to the input of the data to be discriminated to the M machine learning models; (b2) A step of calculating a discrimination class similarity for each of the M discrimination classes as the similarity between the M feature spectra and the M sets of known feature spectrum groups, and obtaining a model reliability for each of the M machine learning models by obtaining a model reliability depending on the discrimination class similarity; (b3) A step of outputting, as a class discrimination result for the data to be discriminated, the discrimination class obtained from the machine learning model with the highest model reliability among the M discrimination classes; and is a method including the above.

5. The method according to claim 4, wherein When P is an integer of 2 or more, the data to be discriminated includes P pieces of data to be discriminated obtained from the same object to be discriminated. The step (b1) is a step of determining P discrimination classes in response to the input of the P pieces of data to be discriminated to each machine learning model; a step of selecting, as the discrimination class in the machine learning model, the most frequent class among the P discrimination classes obtained from each machine learning model; a method including the above.

6. The method according to claim 4, When P is an integer of 2 or more, the data to be discriminated includes P pieces of data to be discriminated obtained from the same object to be discriminated. The step (b1) is, for each of the M machine learning models, a step of obtaining P class-by-class similarity degrees as the class-by-class similarity degrees associated with the machine learning model by using the P feature spectra obtained from each machine learning model; a step of determining, as the similarity degree value of each class represented by the class-by-class similarity degrees, a representative similarity degree value that is a statistical representative value among the P similarity degree values regarding each class represented by the P class-by-class similarity degrees; a step of determining, as the discrimination class corresponding to the machine learning model, the class corresponding to the highest similarity degree value among the similarity degree values of each class represented by the class-by-class similarity degrees; including The step (b2) includes a step of adopting, as the discrimination class similarity degree, the highest similarity degree value used when determining the discrimination class corresponding to each machine learning model in the step (b1). a method.

7. The method according to any one of claims 4 to 6, The model reliability is (1) equal to the discrimination class similarity degree, or (2) the result of taking a weighted average of the discrimination class similarity degree and the activation value corresponding to the determination value of the discrimination class in the output layer of the machine learning model, or (3) the result of multiplying the discrimination class similarity degree by the activation value for the discrimination class, and is any one of the above.

8. The method according to any one of claims 1 to 7, The integer M is 2 or more, The M machine learning models are learned using corresponding M teacher data groups, The M teacher data groups are in a state equivalent to being grouped into N sets by clustering processing.

9. The method according to any one of claims 1 to 8, wherein the specific layer has a configuration in which vector neurons arranged in a plane defined by two axes, a first axis and a second axis, are arranged as a plurality of channels along a third axis in a direction different from the two axes, in the specific layer, when an area including the plurality of channels along the third axis, which is specified at a plane position defined by the position of the first axis and the position of the second axis, is called a partial area, the feature spectrum is for each partial area among the plurality of partial areas included in the specific layer, (i) a first type of feature spectrum in which a plurality of element values of the output vector of each vector neuron included in the partial area are arranged over the plurality of channels along the third axis, (ii) a second type of feature spectrum obtained by multiplying each element value of the first type of feature spectrum by a normalization coefficient corresponding to the vector length of the output vector, (iii) a third type of feature spectrum in which the normalization coefficient is arranged over the plurality of channels along the third axis, and is obtained as any one of them.

10. The method according to claim 9, which depends directly or indirectly on claim 1, wherein the step (b2) is a step of obtaining a plurality of local similarities indicating similarities for each class with respect to the plurality of partial areas of the specific layer by obtaining a local similarity, which is a similarity between the feature spectrum obtained from the output of a specific partial area of the specific layer and all the known feature spectra associated with the specific layer and each class, according to the data to be discriminated, a step of obtaining a similarity for each class by taking a maximum value, an average value, or a minimum value of the plurality of local similarities with respect to the plurality of partial areas for each class, and includes the method.

11. The method according to claim 9, which depends directly or indirectly on claim 1, wherein the step (b2) is a step of obtaining a plurality of local similarities indicating similarities for each class with respect to the plurality of partial areas of the specific layer by obtaining a local similarity, which is a similarity between the feature spectrum obtained from the output of a specific partial area of the specific layer and all the known feature spectra associated with the specific partial area of the specific layer and each class, according to the data to be discriminated, For each class, a step of obtaining the similarity for each class by taking the maximum value, average value, or minimum value of the plurality of local similarities for the plurality of partial regions; A method including this.

12. The method according to claim 9, which is directly or indirectly dependent on claim 1, The step (b2) is A method including a step of obtaining the similarity for each class by respectively obtaining, for each class, the similarity between all the feature spectra obtained from the output of the specific layer and all the known feature spectra associated with the specific layer and each class according to the data to be discriminated.

13. The method according to claim 9, which is directly or indirectly dependent on claim 4, The step (b2) is A step of obtaining a plurality of local similarities indicating the similarity for each class with respect to the plurality of partial regions of the specific layer by obtaining local similarities, which are the similarities between the feature spectra obtained from the output of a specific partial region of the specific layer and all the known feature spectra associated with the specific layer and each class, according to the data to be discriminated; A step of obtaining the discriminant class similarity by taking the maximum value, average value, or minimum value of the plurality of local similarities for the plurality of partial regions for each class; A method including this.

14. The method according to claim 9, which is directly or indirectly dependent on claim 4, The step (b2) is A step of obtaining a plurality of local similarities indicating the similarity for each class with respect to the plurality of partial regions of the specific layer by obtaining local similarities, which are the similarities between the feature spectra obtained from the output of a specific partial region of the specific layer and all the known feature spectra associated with the specific partial region of the specific layer and each class, according to the data to be discriminated; A step of obtaining the discriminant class similarity by taking the maximum value, average value, or minimum value of the plurality of local similarities for the plurality of partial regions for each class; A method including this.

15. The method according to claim 9, which is directly or indirectly dependent on claim 4, The step (b2) is A method including a step of obtaining the discrimination class similarity by calculating, for each class, the similarity between all of the feature spectra obtained from the output of the specific layer and all of the known feature spectra associated with the specific layer and each class, according to the discrimination data.

16. An information processing apparatus that executes discrimination processing for discriminating the class of discrimination data, using M machine learning models of a vector neural network type having a plurality of vector neuron layers, where M is an integer greater than or equal to 1, a memory that stores the M machine learning models, a processor that executes operations using the M machine learning models, comprising: when the number of classes discriminable by the m-th machine learning model among the M machine learning models is Nm which is an integer greater than or equal to 2, and the total number of classes discriminable by the M machine learning models is ΣNm, the processor (a) a process of reading from the memory M sets of known feature spectrum groups associated with the M machine learning models, where the M sets of known feature spectrum groups include known feature spectrum groups obtained from the output of a specific layer among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of the Nm classes are input to each machine learning model, (b) a process of executing class discrimination processing for the discrimination data using the M machine learning models and the M sets of known feature spectrum groups, executing: the process (b) includes (b1) a process of calculating M feature spectra from the output of the specific layer in response to input of the discrimination data to the M machine learning models, (b2) a process of obtaining M sets of class-by-class reliabilities associated with the M machine learning models by calculating, as the similarity between the M feature spectra and the M sets of known feature spectrum groups, the class-by-class similarity for each of the Nm classes in each machine learning model and obtaining the class-by-class reliability depending on the class-by-class similarity, (b3) a process of outputting a discrimination result list in which a plurality of classes, which are at least a part of the ΣNm classes, are arranged in the order of the reliability of each class represented by the M sets of class-by-class reliabilities, An information processing apparatus.

17. When M is an integer of 2 or more, an information processing apparatus that executes a discrimination process for discriminating the class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers, a memory that stores the M machine learning models, a processor that executes operations using the M machine learning models, comprising, the processor, (a) a process of reading out M sets of known feature spectrum groups associated with the M machine learning models from the memory, wherein the M sets of known feature spectrum groups include known feature spectrum groups obtained from outputs of specific layers among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of Nm classes are input to each machine learning model, (b) a process of executing a class discrimination process for the data to be discriminated using the M machine learning models and the M sets of known feature spectrum groups, executing, the process (b) is, (b1) a process of calculating M feature spectra from the output of the specific layer and determining M discrimination classes in response to the input of the data to be discriminated to the M machine learning models, (b2) a process of obtaining a model reliability for each of the M machine learning models by calculating a discrimination class similarity for each of the M discrimination classes as a similarity between the M feature spectra and the M sets of known feature spectrum groups and obtaining a model reliability that depends on the discrimination class similarity, (b3) a process of outputting, as a class discrimination result for the data to be discriminated, the discrimination class obtained from the machine learning model having the highest model reliability among the M discrimination classes, An information processing apparatus including.

18. When M is an integer of 1 or more, a computer program that causes a processor to execute a class discrimination process for discriminating the class of data to be discriminated using M machine learning models of a vector neural network type having a plurality of vector neuron layers, when the number of classes discriminable by the m-th machine learning model among the M machine learning models is Nm which is an integer of 2 or more, and the total number of classes discriminable by the M machine learning models is ΣNm, the computer program, a process of reading out M sets of known feature spectra associated with the M machine learning models from a memory, wherein the M sets of known feature spectra include known feature spectra obtained from outputs of a specific layer among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of Nm classes are input to each machine learning model, and a process of executing a class discrimination process for the data to be discriminated using the M machine learning models and the M sets of known feature spectra, and a computer program causing the processor to execute the above, the process (b) includes a process of calculating M feature spectra from the output of the specific layer in response to the input of the data to be discriminated to the M machine learning models, a process of obtaining M sets of class-specific reliabilities associated with the M machine learning models by calculating class-specific similarities for each of the Nm classes in each machine learning model as similarities between the M feature spectra and the M sets of known feature spectra, and obtaining class-specific reliabilities depending on the class-specific similarities, a process of outputting a discrimination result list in which a plurality of classes, which are at least a part of the ΣNm classes, are arranged in the order of the reliabilities of the classes represented by the M sets of class-specific reliabilities, and a computer program including the above.

19. When M is an integer of 2 or more, a computer program causing a processor to execute a class discrimination process for discriminating a class of data to be discriminated using M vector neural network type machine learning models each having a plurality of vector neuron layers, the computer program including a process of reading out M sets of known feature spectra associated with the M machine learning models from a memory, wherein the M sets of known feature spectra include known feature spectra obtained from outputs of a specific layer among the plurality of vector neuron layers of each machine learning model when a plurality of teacher data for each of Nm classes are input to each machine learning model, and a process of executing a class discrimination process for the data to be discriminated using the M machine learning models and the M sets of known feature spectra, and a computer program causing the processor to execute the above, the process (b) includes (b1) A process of calculating M feature spectra from the output of the specific layer and determining M discrimination classes in response to the input of the data to be discriminated into the M machine learning models; (b2) A process of obtaining the model reliability for each of the M machine learning models by calculating the discrimination class similarity for each of the M discrimination classes as the similarity between the M feature spectra and the M sets of known feature spectra, and obtaining the model reliability depending on the discrimination class similarity; (b3) A process of outputting, as the class discrimination result for the data to be discriminated, the discrimination class obtained from the machine learning model with the highest model reliability among the M discrimination classes; A computer program including the above.

Images