Learning device, inference device, learning method, and program

Abstract

A learning device according to the present invention comprises: an input feature expansion unit that generates, from input features which are inputs from a plurality of data sources, a plurality of similar input features which are similar to the input features; an input feature embedding unit that embeds each of the plurality of similar input features in a low-dimensional vector representation to generate embedded vectors; a dataset feature extraction unit that, for each of the plurality of data sources, extracts a dataset feature by aggregating vector representations of each class calculated from the input features; a class information inference unit that, on the basis of the dataset features and the embedded vectors, infers class information expressing labels of the input features; a mock input feature generation unit that uses a generative model to generate mock input features from the class information; an input feature identification unit that predicts, as identification results, class information having the closest vector representation to the embedded vectors among the class information; and a model training unit that generates a trained model using the embedded vectors, the mock input features, and the identification results as inputs.

Description

Learning device, estimation device, learning method, and program

[0001] The present invention relates to a learning device, an estimation device, a learning method, and a program, and more particularly to a learning device, an estimation device, a learning method, and a program that estimate class information appropriate for each application required in zero-shot learning.

[0002] In recent years, machine learning using neural networks has achieved some success as a technology for identifying image, audio, table data, etc. However, general machine learning models cannot identify classes that do not exist in a given dataset. In reality, data for all classes that one wants to estimate is not always available in advance, so there is a high demand for methods that can learn the relationship between data and classes even when no data for some classes is available.

[0003] One such learning method is called zero-shot learning (ZSL), which assumes that a vector representation of a class can be obtained and uses that vector representation to indirectly learn the relationship between classes and data for which no data has been obtained, as described in Non-Patent Document 1.

[0004] Zero-shot learning assumes that the vector representations of classes are available in advance, so in order to apply zero-shot learning, a technique for acquiring the vector representations of classes is required.

[0005] When the object to be identified is an animal, various vector representations have traditionally been obtained from different modalities by manually defining vectors representing the animal's appearance or by inputting class names into a trained language model to obtain latent representations.

[0006] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In CJ Burges, L.Bottou, M.Welling, Z.Ghahramani, and KQ Weinberger, editors, Advances in Neural Information Processing Systems, Vol.26. Curran Associates, Inc., 2013.

[0007] As mentioned above, zero-shot learning requires a vector representation of the classes we want to identify.

[0008] However, this vector representation differs depending on the area and data to which it is applied, so it is necessary to manually define the vector representation for each area and data, which is a very time-consuming process.

[0009] For example, when identifying emotions, people have different perceptions and impressions of emotions, so the positioning of the class "anger" will differ from person to person. Therefore, it is necessary to define a vector expression for the emotion class for each individual, but the task of defining vector expressions for all the people to whom it is to be applied is extremely time-consuming. In other words, when the application ranges widely, the workload required for the definition work is large, and it is not realistic to perform such manual work for all application targets.

[0010] The present invention has been made in consideration of the above circumstances, and aims to provide a learning device, an estimation device, a learning method, and a program that are capable of automatically estimating vector representations of classes that are appropriate for various application areas, data sources, etc.

[0011] A first aspect of the present invention is a learning device comprising an input feature expansion unit, an input feature embedding unit, a dataset feature extraction unit, a class information estimation unit, a pseudo input feature generation unit, an input feature identification unit, and a model learning unit. The input feature expansion unit generates a plurality of similar input features similar to input features from input features that are input from a plurality of data sources. The input feature embedding unit embeds each of the plurality of similar input features generated by the input feature expansion unit into a low-dimensional vector representation to generate an embedded vector. The dataset feature extraction unit extracts dataset features that are vector representations of the data sources by aggregating vector representations for each class calculated from the input features for each of the plurality of data sources. The class information estimation unit estimates class information that represents the labels of the input features based on the dataset features and the embedding vector. The pseudo input feature generation unit generates pseudo input features from the class information using a generative model prepared in advance. The input feature identification unit predicts, as an identification result, class information having a vector representation closest to the embedding vector from the class information estimated by the class information estimation unit. The model learning unit generates a learning model using the embedding vector, the pseudo input features, and the classification results as inputs.

[0012] A second aspect of the present invention is an estimation device comprising an input feature embedding unit, a dataset feature extraction unit, and a class information estimation unit, each of which has been trained in advance. The input feature embedding unit performs a process of embedding input features, which are inputs from multiple data sources, into a low-dimensional vector representation to generate an embedded vector. The dataset feature extraction unit extracts dataset features, which are vector representations of the data sources, by aggregating vector representations for each class calculated from the input features for each of the multiple data sources. The class information estimation unit estimates class information representing labels of the input features based on the dataset features and the embedded vector.

[0013] A third aspect of the present invention is a learning method implemented by a learning device for model learning, in which a processor of the learning device executes the following steps: a first generation step in which a processor generates a plurality of similar input features similar to input features from input features that are input from a plurality of data sources; an embedding step in which a processor embeds each of the plurality of similar input features generated in the first generation step into a low-dimensional vector representation to generate an embedding vector; an extraction step in which, for each of the plurality of data sources, aggregating vector representations for each class calculated from the input features to extract dataset features that are vector representations of the data sources; an estimation step in which a processor generates pseudo input features from the class information using a generative model prepared in advance; a prediction step in which a class information estimated in the estimation step has a vector representation closest to the embedding vector as an identification result; and a third generation step in which a learning model is generated using the embedding vector, pseudo input features, and the identification result as inputs.

[0014] A fourth aspect of the present invention is a program for causing a computer to function as each unit included in the learning device of the first aspect.

[0015] The learning device, estimation device, learning method, and program of the present invention make it possible to automatically estimate vector representations of classes that are appropriate for various application areas, data sources, etc. This allows learning with a model that can generate class information, thereby reducing the burden of manually defining class information.

[0016] Fig. 1 is a functional block diagram showing an example of the configuration of a learning estimation device to which a learning estimation method according to an embodiment of the present invention is applied. Fig. 2 is a simplified diagram showing the hardware configuration of a general computer. Fig. 3 is a data flow showing the flow of processing by the learning device of the learning estimation device according to this embodiment. Fig. 4 is a flowchart showing the flow of operation of the estimation device of the learning estimation device according to this embodiment. Fig. 5 is a diagram showing an example of application of the estimation device of the learning estimation device according to this embodiment.

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc., are not necessarily the same as those in reality. Furthermore, even when the same part is shown, the dimensions and ratios may be different depending on the drawing. In this specification and each drawing, elements similar to those described above with reference to the previous drawings are designated by the same reference numerals, and detailed and redundant explanations will be omitted as appropriate.

[0018] FIG. 1 is a functional block diagram showing an example of the configuration of a learning estimation device to which a learning estimation method according to an embodiment of the present invention is applied.

[0019] The learning and estimation device 10 includes a learning device 20 and an estimation device 30 .

[0020] The learning estimation device 10 can be realized by a computer such as a PC.

[0021] FIG. 2 is a simplified diagram showing the hardware configuration of a typical computer.

[0022] Computer 100 receives some kind of input from the outside, processes it, and outputs the results to the outside. Input device 120 is responsible for input, and output device 140 is responsible for output. CPU (Central Processing Unit) 110 controls the overall flow of data and processing, and performs calculations and other processes. In Figure 2, solid arrows represent the flow of data, and dashed arrows represent the flow of control.

[0023] A program describing the processing procedure is required for the computer 100 to perform processing. The program contains a series of instructions and data used by the instructions, all of which are temporarily stored in the memory 130.

[0024] The CPU 110 retrieves instructions from the memory 130 one by one, interprets the instructions, and operates in accordance with the instructions.

[0025] Therefore, the learning and estimation device 10 according to this embodiment is realized by the operation of the CPU 110 in accordance with the programs stored in the memory 130 together with the learning device 20 and the estimation device 30 .

[0026] FIG. 3 is a data flow showing the flow of processing by the learning device of the learning estimation device according to this embodiment.

[0027] Before returning to FIG. 1 and describing the details of the learning device 20 and the estimation device 30, the definitions of the terms "input feature," "class information," and "data source" used in this specification will first be explained.

[0028] The learning and estimation device 10 according to this embodiment does not depend on the modality of input data. Therefore, in this specification, the term "input feature" refers to data that belongs to any input feature space, such as an image, a voice, sensor data, table data, or a graph.

[0029] In this specification, "class information" refers to a collection of representative vectors for all classes.

[0030] In this specification, the term "data source" refers to the source of data. For example, in the case of identifying emotions of multiple people using biosignals as input, each individual whose biosignal is measured is considered to be a different data source.

[0031] (Learning Device) As illustrated in FIG. 1, the learning device 20 includes a learning input unit 21, an input feature expansion unit 22, an input feature embedding unit 23, a dataset feature extraction unit 24, a class information estimation unit 25, a pseudo input feature generation unit 26, an input feature identification unit 27, and a model learning unit 28.

[0032] The learning input unit 21 receives input features and their labels from multiple data sources. A label is information indicating the correct answer given to an input feature. For example, when watching a video, the cardiac potential becomes the input feature, and the label becomes the emotion at that time (e.g., "joy," "fear," "anger," etc.). If a horror movie is watched, the label corresponding to the input feature, which is the cardiac potential, becomes "fear." In machine learning processing, the label is expressed as a number, for example, "joy" = 0, "fear" = 1, and "anger" = 2.

[0033] The learning input unit 21 performs processing using a set of input features as input, and generates a set D of input features and labels.ｗ The set of input features and labels is assumed to be given manually. The set of input features and labels is assumed to be given from W different data sources. It is expressed as N ｗ is the number of instances in the wth dataset. ｗｉ is the input feature space The input feature belongs to the label y ｗｉ are different class sets for each dataset. It belongs to where K is the number of classes. That is, for each data set D ｗ does not necessarily cover all instances of the K classes, but shows that all classes are covered when all datasets are taken together.

[0034] The input feature expansion unit 22 receives a set D of input features and labels as input, and ｗｉ The input features generated by performing data augmentation in this manner are referred to as similar input features in the claims.

[0035] Specifically, the input feature extension unit 22 performs two different types of data extension on the input features by well-known techniques such as adding noise and rearranging the data order. ｗｉ respectively In addition, the data augmentation applied to all instances of D is expressed as It is expressed as:

[0036] The input feature embedding unit 23 receives the set of input features and labels D' or D and embeds the input features into the input feature space to lower dimensions and outputs a set ε of embedding vectors and labels.

[0037] The input feature embedding unit 23 embeds the model This model can use any model such as a matrix that performs linear transformation or a neural network that performs linear or nonlinear transformation. The embedding space Embedding into,embedding vector Specifically, we obtain an embedding vector by embedding input features with the same label so that they are closer to each other. By performing this operation on all instances of the set of input features and labels, we obtain the embedding vector and the set of labels. By embedding the input features in this way, it is possible to compress the information by reducing non-essential information from the instance.

[0038] The dataset feature extraction unit 24 extracts dataset features, which are vector representations of the data sources, by aggregating vector representations for each class calculated from the input features for each of the multiple data sources. Specifically, the dataset feature extraction unit 24 receives a set D of input features and labels as input, and extracts the set of input features and labels as a low-dimensional The dataset feature vectors are embedded in Output.

[0039] The dataset feature extraction unit 24 extracts the model This model can use any model such as a matrix that performs linear transformation or a neural network that performs linear or nonlinear transformation. The model H is a function of the input feature x ｗｉ the dataset feature space Embedding into the dataset feature vector By performing this operation on all instances of the set of input features and labels, we obtain the set of dataset feature vectors and labels. For this set, we obtain the data source w with the same label. The representative vector of class k' is obtained by taking the average of the given dataset feature vectors. Then, for each data source w, we get all classes The dataset feature vector obtained by taking the average of the representative vectors for This calculation can be applied to any commutative calculation that does not depend on the amount or order of variables input to the calculation, not just the average value calculation. Then, these dataset feature vectors are collected for all data sources w, Output.

[0040] The class information estimation unit 25 estimates class information, which is a representative expression of the labels of the input features, based on the dataset features extracted by the dataset feature extraction unit 24 and the embedding vectors generated by the input feature embedding unit 23. Specifically, the class information estimation unit 25 estimates class information, which is a representative expression of the labels of the input features, based on the embedding vectors and the set ε of labels, the dataset feature vectors The embedding vector and the dataset feature vector, which are averaged for each class label, are input to the model to obtain class information A. ｗ Estimate.

[0041] The class information estimation unit 25 estimates the model This model can use any model, such as a matrix that performs linear transformation or a neural network that performs linear or nonlinear transformation. First, for a set ε of embedding vectors and labels, The representative embedding vector for class k is obtained by taking the average of the embedding vectors given by Then, we obtain the representative embedding vector and the dataset feature vector h ｗ is given to the model P, and the vector representation of class k is By performing this for all classes k of all data sources w, the class information Output.

[0042] The pseudo-input feature generation unit 26 generates class information A ｗ The generative model generates pseudo-input features from class information, and a set of pseudo-input features and labels is obtained. Output.

[0043] The pseudo-input feature generator 26 generates a generative model This generative model G may be a known generative model that can generate input features using vectors as conditions, such as a conditional generative adversarial network or a conditional diffusion model. This generates a set of pseudo-input features and labels obtained from class information of the data source w. Output. is an arbitrary natural number that indicates the number of pseudo input features generated. is the pseudo input feature Class information used to generate Let k be the class of .

[0044] The input feature identification unit 27 receives a set ε of embedding vectors and labels, a set ε of pseudo input features and labels, , Class Information A ｗ Given the input, the predicted categorical distribution of the embedding vector and the set of labels , a set of predicted categorical distributions and labels for pseudo-input features Output.

[0045] In this way, the input feature identification unit 27 predicts the class information having the vector expression closest to the embedding vector as the identification result. For example, an embedding vector with the label "anger" embedded is identified so as to be close to the class information of the label "anger". The probability that the label of the embedding vector e of the input feature is class k is determined as follows: where d(·) is an arbitrary distance function and Euclidean distance or cosine similarity can be used. This operation is performed on the set of embedding vectors and labels. By iterating over all instances of This operation is also called a set of pseudo input features and labels. By iterating over all instances of , we obtain a set of predicted categorical distributions and labels for the pseudo-input features. Output.

[0046] The model learning unit 28 receives a set ε of embedding vectors and labels, a set ε of pseudo input features and labels, , the predicted categorical distribution of the embedding vectors and the set of labels , a set of predicted categorical distributions and labels for pseudo-input features Using the above as input, a learning model is generated and output by simultaneously performing contrastive learning, generative model learning, and discriminative model learning.

[0047] Here, the objective function is expressed as θ, which is a collective representation of the model parameters of models E, H, P, and G. The model parameters are estimated by optimizing the objective function using data set D. The objective function is calculated according to three rules to be satisfied, which will be described later. Let's say. indicates the expected value. Any hyperparameter may be set for each term of this objective function as a coefficient that determines the weight of each term. The three rules are explained below.

[0048] (Rule 1) When multiple similar input features are given, it is desirable that their embedding vectors are similar. Therefore, the input feature embedding unit 23 is trained by arbitrary contrastive learning. The embedding vector of When written as is minimized. Here, and and teeth Instances of that label is the set of indexes of instances that are the same as is an arbitrary hyperparameter. This loss function is called Supervised Contrastive Loss, and requires that the embedding vectors of the same class remain similar even if noise is added to the input features. Any contrastive learning loss function can be used instead of this loss function.

[0049] (Rule 2) The pseudo-input features generated from class information are trained using a known generative model training technique. Therefore, the known loss function is As a set of pseudo input features and labels, All we need to do is minimize this loss for

[0050] (Rule 3) The embedding vector is trained to be discriminative based on class information. The class with the closest vector representation to the embedding vector is assumed to be the label of the embedding model, and the cross-entropy loss is minimized. This must be satisfied not only for the embedding vectors of the given dataset, but also for the generated pseudo-input features. Therefore, the predicted categorical distribution of the embedding vector and the set of labels are , a set of predicted categorical distributions and labels for pseudo-input features With input, Here, q(·) represents one-hot encoding. This loss function can be not only the cross-entropy function described above, but also the minimization of the distance between the embedding vector and its label, the vector representation of the class, such as the loss of contrastive learning or absolute error.

[0051] Any optimization method can be applied to optimize the objective function, such as the gradient method, the stochastic gradient method, Adam, etc. When using the gradient method, it is sufficient to repeatedly update the parameters according to the following formula at the sth optimization step. where γ ｓ is the learning rate parameter. The gradient of the objective function may be calculated using a calculated function or may be calculated numerically.

[0052] As explained using Figure 2, the operation of each component of the learning device 20 can be constructed as a program, which can be installed and executed on the computer 100 used as the learning device 20, but it can also be distributed via a network.

[0053] 1 , the estimation device 30 includes an input unit 31, an input feature embedding unit 33, a dataset feature extraction unit 34, and a class information estimation unit 35. The input feature embedding unit 33, the dataset feature extraction unit 34, and the class information estimation unit 35 have similar functions to the input feature embedding unit 23, the dataset feature extraction unit 24, and the class information estimation unit 25.

[0054] The input unit 31 performs processing using a set of input features as input, and generates a set D of input features and labels.＊ Output.

[0055] The labels of the input features are given manually. The set of input features and labels is assumed to be given by a data source. where N is the number of instances. is the input feature space The input features belong to the label is a set of classes It belongs to where K is the number of classes. ＊ does not necessarily cover all instances of the K classes.

[0056] At this time, the pre-trained dataset feature extraction unit 34 extracts the feature vector h ＊ Next, the data set provided from the input unit 31 is provided to the trained input feature embedding unit 33 to obtain a set of embedded features and labels. Finally, we obtain the dataset feature vector h ＊ and the set of embedding features and labels ε ＊ The class information is estimated by providing the trained class information estimation unit 35 with the above.

[0057] FIG. 4 is a flowchart showing the flow of operations of the learning and estimation device according to this embodiment.

[0058] FIG. 5 is a diagram showing an application example of the estimation device 30 of the learning estimation device according to this embodiment.

[0059] As shown in FIG. 4, when data and a model are input (S1), the estimation device 30 estimates model parameters based on the input data and model (S2), and outputs the estimated model parameters (S3).

[0060] Such an estimation device 30 can be configured by constructing the operations of each component as a program, which can be installed and executed on a computer used as the estimation device 30, or distributed via a network.

[0061] This application example shows the case where class information of emotions (e.g., joy, sadness, anger, fear, etc.) is estimated when arbitrary input features such as facial expression images, voice, and biological signals are obtained from multiple users 1, 2, ..., W (where W is an arbitrary integer equal to or greater than 3).

[0062] As illustrated in FIG. 5, the estimation device 30 estimates data D １ , D ２ , ...D ｗ and obtain the vector representation V for each user 1, 2, ... W. １ , V ２ , ...V ｗ can be obtained, respectively.

[0063] The class information as exemplified in FIG. 5 is the distribution of emotions of each user 1, 2, . . . W estimated from the data, and can be used to analyze the inner state of a specific user.

[0064] Note that this use example is merely an example, and the application of the estimation device 30 is not limited to estimating class information of human emotions.

[0065] As described above, a learning and estimation device to which the learning and estimation method according to an embodiment of the present invention is applied combines the training of an embedding model that makes input data with the same label similar to each other with a feature extraction model that vectorizes the features of the target itself to train a model that outputs class information suitable for the target. The objective function used in model training is designed under the assumption that if the class information is correct, input data for the target can be generated from that class information, and that the embedding of the input data will be close to the corresponding class information, thereby training a feature extraction model for the dataset and an embedding model for the input data.

[0066] This makes it possible to automatically estimate class information appropriate for each application, which is required for zero-shot learning, eliminating the need for manual work to define class information.Furthermore, it becomes possible to analyze the characteristics of the application from the estimated class information appropriate for the application.

[0067] The present invention is not limited to the above-described embodiments, and can be embodied by modifying the components within the scope of the gist of the present invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0068] 10 Learning and estimation device 20 Learning device 21 Learning input unit 22 Input feature expansion unit 23 Input feature embedding unit 24 Data set feature extraction unit 25 Class information estimation unit 26 Pseudo input feature generation unit 27 Input feature identification unit 28 Model learning unit 30 Estimation device 31 Input unit 33 Input feature embedding unit 34 Data set feature extraction unit 35 Class information estimation unit 100 Computer 120 Input device 130 Memory 140 Output device

Claims

1. A learning device comprising: an input feature expansion unit that generates a plurality of similar input features similar to input features from input features that are input from a plurality of data sources; an input feature embedding unit that performs a process of embedding each of the plurality of similar input features generated by the input feature expansion unit into a low-dimensional vector representation to generate an embedded vector; a dataset feature extraction unit that extracts dataset features that are vector representations of the data sources by aggregating vector representations for each class calculated from the input features for each of the plurality of data sources; a class information estimation unit that estimates class information that represents the labels of the input features based on the dataset features and the embedded vector; a pseudo input feature generation unit that generates pseudo input features from the class information using a generative model prepared in advance; an input feature identification unit that predicts, as an identification result, class information that has a vector representation closest to the embedded vector from the class information estimated by the class information estimation unit; and a model learning unit that generates a learning model using the embedded vectors, the pseudo input features, and the identification result as inputs.

2. An estimation device comprising an input feature embedding unit, a dataset feature extraction unit, and a class information estimation unit, each of which has been trained in advance, wherein the input feature embedding unit performs a process of embedding input features, which are input from multiple data sources, into a low-dimensional vector representation to generate an embedded vector, the dataset feature extraction unit extracts dataset features, which are vector representations of the data sources, by aggregating vector representations for each class calculated from the input features for each of the multiple data sources, and the class information estimation unit estimates class information, which represents the labels of the input features, based on the dataset features and the embedded vector.

3. A learning method implemented by a learning device for model learning, wherein a processor of the learning device executes the following steps: a first generation step of generating a plurality of similar input features similar to input features that are input from a plurality of data sources, from the input features; an embedding step of embedding each of the plurality of similar input features generated by the first generation step into a low-dimensional vector representation to generate an embedding vector; an extraction step of extracting dataset features that are vector representations of the data sources by aggregating vector representations for each class calculated from the input features for each of the plurality of data sources; an estimation step of estimating class information that represents the labels of the input features based on the dataset features and the embedding vector; a second generation step of generating pseudo input features from the class information using a generative model prepared in advance; a prediction step of predicting, as a classification result, class information that has a vector representation closest to the embedding vector from the class information estimated by the estimation step; and a third generation step of generating a learning model using the embedding vector, the pseudo input features, and the classification result as inputs.

4. A program for causing a computer to function as each of the units provided in the learning device according to claim 1.

Images