Data generation method, learning model generation method, information processing method, computer program, and information processing device

Abstract

Provided are a data generation method, a learning model generation method, an information processing method, a computer program, and an information processing device that can make it possible to infer emotions on the basis of multimodal data.　According to one embodiment, a data generation method involves an information processing device generating data for machine learning. The information processing device acquires multimodal data that includes a plurality of pieces of modal data and emotion-related label information that has been associated with the multimodal data, converts each piece of modal data to a first feature quantity, converts the first feature quantities to second feature quantities on the basis of the co-occurrence of the first feature quantities, and generates the data for machine learning on the basis of the first feature quantities and the second feature quantities.

Description

Data generation method, learning model generation method, information processing method, computer program, and information processing device

[0001] The present disclosure relates to a data generation method, a learning model generation method, an information processing method, a computer program, and an information processing device.

[0002] Patent Literature 1 proposes a system for expressing emotions of spectators watching online video game play, which captures interaction data from spectators watching gameplay of a video game, clusters the captured interaction data into various groups according to emotions detected from the spectators' interactions, and transfers graphic interchange format files representing distinct emotions of various spectator groups to spectator client devices for rendering together with the video game content.

[0003] Special Publication No. 2023-551476

[0004] The present disclosure provides a data generation method, a learning model generation method, an information processing method, a computer program, and an information processing device that are expected to realize emotion estimation based on multimodal data.

[0005] In one embodiment, a data generation method is a data generation method in which an information processing device generates data for machine learning, in which the information processing device acquires multimodal data including a plurality of modal data and information on labels related to emotions associated with the multimodal data, converts each of the plurality of modal data into first features, converts the first features into second features based on the co-occurrence of a plurality of the first features, and generates the data for machine learning based on the first features and the second features.

[0006] According to the present disclosure, it is expected that emotion estimation based on multimodal data can be realized.

[0007] FIG. 1 is a schematic diagram illustrating an overview of an information processing system according to the present embodiment. FIG. 1 is a schematic diagram illustrating an overview of an information processing system according to the present embodiment. FIG. 2 is a block diagram illustrating an example configuration of an information processing device according to the present embodiment. FIG. 3 is a schematic diagram illustrating an example configuration of a learning model according to the present embodiment. FIG. 4 is a flowchart illustrating an example procedure of learning data generation processing performed by the information processing device according to the present embodiment. FIG. 5 is a flowchart illustrating an example procedure of learning model generation processing performed by the information processing device according to the present embodiment. FIG. 6 is a flowchart illustrating an example procedure of emotion estimation processing performed by the information processing device according to the present embodiment. FIG. 7 is a schematic diagram illustrating an example configuration of a learning model according to embodiment 2. FIG. 8 is a schematic diagram illustrating an emotion estimation method based on valence and arousal. FIG. 9 is a schematic diagram illustrating an example of a causality graph. FIG. 10 is a schematic diagram illustrating a method of generating a causality graph. FIG. 11 is a schematic diagram illustrating an example of a conversion table.

[0008] Specific examples of information processing systems according to embodiments of the present disclosure will be described below with reference to the drawings. Note that the present disclosure is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope of the claims.

[0009] [First Embodiment] <System Overview> FIGS. 1 and 2 are schematic diagrams illustrating an overview of an information processing system according to this embodiment. The information processing system according to this embodiment is a system that estimates human emotions using a learning model generated by machine learning based on multimodal data including multiple types of data (modal data), such as language data, speech data, and image data. The processing performed by the information processing system according to this embodiment can be broadly divided into two stages: a generation stage in which a learning model is generated by machine learning, and an inference stage in which emotions are estimated using the trained learning model. In this embodiment, the processing of the generation stage and the processing of the inference stage are both performed by a single information processing device 1, but this is not limited thereto, and the processing of the generation stage and the processing of the inference stage may be performed by different devices.

[0010] The multimodal data handled by the information processing device 1 of the information processing system according to this embodiment is obtained, for example, by capturing images of the faces of people who are having a conversation with a camera. The information processing device 1 according to this embodiment separates, for example, moving image data obtained from the camera into image data of the person's face and voice data uttered by the person, and further generates language data by converting the person's speech into text through speech recognition processing of the voice data. The multimodal data is a combination of these three modal data: image data, voice data, and language data.

[0011] The multimodal data is appropriately divided, for example, at fixed time intervals or at variable time intervals depending on the content of the utterance, and an emotion label indicating whether the emotion of the person captured in each divided section is positive or negative is assigned. In this example, the emotion label is a binary label of positive or negative, but this is not limited to this and may include multiple labels such as joy, sadness, anger, surprise, fear, or disgust. The emotion label is assigned in advance to the video data or multimodal data by, for example, a designer of the information processing system according to this embodiment. By dividing one video into multiple sections and assigning emotion labels, multiple pairs of time-series multimodal data and emotion labels can be obtained.

[0012] An information processing device 1 according to this embodiment acquires a set of pre-collected multimodal data and emotion labels and converts each of the language data, speech data, and image data included in the multimodal data into features (first features). The information processing device 1 includes three types of learning models: a learning model pre-machined to convert language data into features, a learning model pre-machined to convert speech data into features, and a learning model pre-machined to convert image data into features. The features output by these three types of learning models are vector data with the same number of dimensions, and the information processing device 1 can treat features based on language data, features based on speech data, and features based on image data as equivalent. In this example, the information processing device 1 obtains 3N first features from N pieces of multimodal data (N is a natural number).

[0013] Next, the information processing device 1 converts each of the multiple first feature quantities (3N in the above example) obtained by converting the multimodal data into a second feature quantity based on the co-occurrence of the multiple first feature quantities. "Co-occurrence" refers to the occurrence of multiple feature quantities together within a specific range, and "co-occurrence" is an index that quantifies the likelihood of a given feature quantity appearing together with another feature quantity. In this example, the information processing device 1 converts the 3N first feature quantities into 3N second feature quantities.

[0014] Next, the information processing device 1 performs dimensionality reduction on the multiple (3N in the above example) second feature quantities obtained based on the co-occurrence, thereby obtaining multiple third feature quantities by reducing the dimension of each second feature quantity. In this embodiment, the information processing device 1 performs dimensionality reduction on the second feature quantities using a method called UMAP (Uniform Manifold Approximation and Projection). However, the information processing device 1 may perform dimensionality reduction using a method other than UMAP, such as PCA (Principal Component Analysis) or t-SNE (t-distributed Stochastic Neighbor Embedding).

[0015] In the present embodiment, the information processing device 1 clusters the third feature values, the dimensions of which have been reduced by the UMAP method, based on a so-called unsupervised machine learning clustering method, such as HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise). The information processing device 1 also receives, from a user, input of meaning information for each of the multiple clusters obtained by the clustering process, and stores the received meaning information.

[0016] The information processing device 1 converts each of the three modal data, namely language data, speech data, and image data, included in the multimodal data into first features, converts the first features into second features based on co-occurrence, and acquires third features by dimensionality reduction of the second features. The information processing device 1 associates three first features and three third features corresponding to the three modal data of the original multimodal data with emotion labels assigned to the original multimodal data, and uses this data as training data for machine learning. The information processing device 1 can generate N sets of training data based on N sets of multimodal data and emotion labels. In the training phase, the information processing device 1 performs so-called supervised machine learning processing using the generated training data to generate a learning model that accepts as input the first and third features corresponding to language data, the first and third features corresponding to speech data, and the first and third features corresponding to image data, and classifies the emotion as positive or negative.

[0017] In the inference stage, the information processing device 1 stores the learning model generated in the learning stage in a storage unit, and acquires video images of a subject in real time, for example, captured by a camera, and estimates the subject's emotions in real time using the stored learning model. The information processing device 1 generates multimodal data by separating the video images acquired from the camera into three modal data: language data, voice data, and image data. The information processing device 1 converts each modal data included in the multimodal data into a first feature using a conversion learning model.

[0018] In the learning model generation stage, the information processing device 1 converts each first feature into a second feature based on the co-occurrence of the first feature obtained by converting each modal data included in the multimodal data. The information processing device 1 stores a conversion rule (parameters such as an arithmetic formula or learning model used for conversion) from the first feature into the second feature determined based on the co-occurrence at this time. In the inference stage, the information processing device 1 converts each first feature obtained by converting the multimodal data into a second feature using the stored conversion rule.

[0019] Furthermore, in the learning model generation stage, the information processing device 1 reduces the dimension of the second feature values ​​using the UMAP method to generate third feature values. The information processing device 1 stores the dimension reduction rules (parameters of the learning model generated using the UMAP method) determined at this time. In the inference stage, the information processing device 1 reduces the dimension of each second feature value obtained by converting the first feature values ​​using the stored dimension reduction rules to generate third feature values.

[0020] Furthermore, in the learning model generation stage, the information processing device 1 clusters the third feature quantities using the HDBSCAN method to classify them into multiple clusters. The information processing device 1 stores the clustering rules (parameters of the learning model generated using the HDBSCAN method) obtained at this time. In the inference stage, the information processing device 1 clusters the third feature quantities, the dimensionality of which has been reduced using the UMAP method, using the stored clustering rules to classify them into multiple clusters.

[0021] The information processing device 1 inputs multiple first feature amounts (in this example, three corresponding to language data, speech data, and image data) and multiple third feature amounts obtained from the multimodal data into a learning model generated in the learning stage, and obtains emotion estimation results output by the learning model. As a result, the information processing device 1 is expected to be able to estimate whether the emotion of a person captured in a video image captured by a camera is positive or negative. The camera continues to capture images, and the information processing device 1 repeatedly acquires video images from the camera and repeatedly performs emotion estimation, thereby enabling real-time emotion estimation of the subject.

[0022] <Device Configuration> Fig. 3 is a block diagram showing an example configuration of an information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment can be realized by installing a predetermined application program or the like in a general-purpose information processing device such as a personal computer or a server computer. The information processing device 1 according to this embodiment is configured to include a processing unit 11, a storage unit 12, a communication unit 13, a display unit 14, an operation unit 15, etc. Note that, in this embodiment, the processing will be described as being performed by a single information processing device 1, but the processing of the information processing device 1 may be distributed among multiple devices.

[0023] The processing unit 11 is configured using an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit) or a quantum processor, a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processing unit 11 reads and executes a program 12a stored in the storage unit 12 to perform various processes, such as generating training data for machine learning, generating a training model by machine learning using the training data, and estimating emotions from multimodal data using the generated training model.

[0024] The storage unit 12 is configured using a large-capacity storage device such as a hard disk or an SSD (Solid State Drive). The storage unit 12 stores various programs executed by the processing unit 11 and various data required for the processing of the processing unit 11. In this embodiment, the storage unit 12 stores a program 12a executed by the processing unit 11. The storage unit 12 also includes a learning data storage unit 12b that stores learning data used for machine learning, and a model information storage unit 12c that stores information related to the learning model 2 generated by the information processing device 1.

[0025] In this embodiment, the program (computer program, program product) 12a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disc, and the information processing device 1 reads the program 12a from the recording medium 99 and stores it in the storage unit 12. However, the program 12a may also be written to the storage unit 12, for example, during the manufacturing stage of the information processing device 1. Alternatively, the program 12a may be distributed by a remote server device or the like and acquired by the information processing device 1 via communication. For example, the program 12a may be read from the recording medium 99 by a writing device and written to the storage unit 12 of the information processing device 1. The program 12a may be provided in a form distributed via a network or in a form recorded on the recording medium 99.

[0026] The training data storage unit 12b stores pre-collected video data or multi-modal data obtained by separating the video data into language data, audio data, and image data, in association with emotion labels previously assigned to these data. The training data storage unit 12b also stores, as training data, data in which three first feature amounts and three third feature amounts generated by the information processing device 1 based on the multi-modal data are associated with the emotion labels previously assigned to the original multi-modal data.

[0027] The model information storage unit 12c stores information about a learning model that has been subjected to machine learning. The information about the learning model may include, for example, information indicating the configuration of the learning model and information such as values ​​of internal parameters determined by machine learning. In the present embodiment, the model information storage unit 12c stores information about the learning model generated by the information processing device 1 through machine learning based on learning data, i.e., the learning model that accepts as input a plurality of first feature amounts (three in this example) and a plurality of third feature amounts obtained from multimodal data and outputs an emotion estimation result of a subject depicted in an original video.

[0028] The model information storage unit 12c also stores information about learning models for converting the language data, voice data, and image data included in the multimodal data into first features. The learning model for converting language data into first features may be an existing learning model such as BERT (Bidirectional Encoder Representations from Transformers) or RoBERTa (a Robustly optimized BERT pretraining approach). The learning model for converting voice data into first features may be an existing learning model such as HuBERT (Hidden unit BERT). The learning model for converting image data into first features may be an existing learning model such as VideoMAE (Video Masked AutoEncoder). The information processing device 1 acquires these existing trained learning models by an appropriate method as learning models for converting modal data into first features, and stores information about the acquired learning models in the model information storage unit 12c. Note that the above-described learning models for converting modal data into first features are merely examples and are not limiting, and any learning model may be employed.

[0029] The model information storage unit 12c also stores information about conversion rules for converting first features determined based on the co-occurrence of multiple first features into second features during the process of generating training data. For example, when converting first features into second features using an arithmetic expression, the information about the conversion rules may include parameters such as coefficients of the arithmetic expression. For example, when converting first features into second features using a learning model, the information about the conversion rules may include internal parameters of the learning model. The information processing device 1 can convert first features obtained in the inference stage into second features based on the information about the conversion rules stored in the model information storage unit 12c.

[0030] In addition, the model information storage unit 12c stores information on rules for reducing the dimension of the second feature values ​​by UMAP during the process of generating training data. Furthermore, in the process of generating training data, the information processing device 1 generates a training model that classifies multiple third feature values ​​into multiple clusters using a so-called unsupervised clustering technique. The model information storage unit 12c stores internal parameters of this training model.

[0031] In the present embodiment, the information processing device 1 receives input of information on meanings or interpretations, etc., from knowledgeable users, etc., for multiple clusters obtained by clustering in the process of generating learning data, and stores the received information (hereinafter referred to as meaning information) together with information on the rules for dimension reduction in the model information storage unit 12 c. The meaning information is, for example, information in natural language, such as a name or description, for each cluster.

[0032] In this embodiment, information about the learning model is stored in the information processing device 1, and processing using the learning model is performed by the information processing device 1, but this is not limited to this. Information about the learning model may be stored in a device different from the information processing device 1, and this device may perform processing using the learning model, and the information processing device 1 may acquire the processing results from this device. Furthermore, the machine learning processing for generating the learning model may be performed by the information processing device 1, or may be performed by a device different from the information processing device 1.

[0033] The communication unit 13 transmits and receives data to and from other devices by performing wired or wireless communication. In this embodiment, the communication unit 13 is connected to a camera via a communication line, for example, receives image (moving image) data captured and transmitted by the camera, and provides the received data to the processing unit 11. The communication unit 13 may also transmit data such as control commands provided by the processing unit 11 to other devices such as cameras.

[0034] The display unit 14 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 11. In the present embodiment, the display unit 14 displays, for example, image data and language data collected for machine learning, information regarding the progress of machine learning processing for generating a learning model, and estimation results of emotions of a subject photographed by a camera.

[0035] The operation unit 15 accepts user operations and notifies the processing unit 11 of the accepted operations. For example, the operation unit 15 accepts user operations using input devices such as mechanical buttons or a touch panel provided on the surface of the display unit 14. Furthermore, for example, the operation unit 15 may be input devices such as a mouse and a keyboard, and these input devices may be configured to be detachable from the information processing device 1.

[0036] The storage unit 12 may be an external storage device connected to the information processing device 1. The information processing device 1 may be a multi-computer including multiple computers, or may be a virtual machine virtually constructed by software. The information processing device 1 is not limited to the above configuration, and may not include, for example, the display unit 14 and the operation unit 15.

[0037] Furthermore, in the information processing device 1 according to this embodiment, the processing unit 11 reads and executes a program 12a stored in the storage unit 12, thereby realizing a learning data generation unit 11a, a learning processing unit 11b, a feeling estimation unit 11c, a display processing unit 11d, etc. as software functional units in the processing unit 11. Note that in the figure, functional units related to the feeling estimation process are shown as functional units of the processing unit 11, and functional units related to other processes are not shown.

[0038] The training data generation unit 11a performs a process of generating training data for machine learning to generate a training model that estimates human emotions, based on video data of a person or multimodal data including multiple modal data such as language data, image data, and audio data separated from the video. In this embodiment, the video data or multimodal data is collected in advance, and annotation work is performed in advance to assign emotion labels to the collected data. Data associating the video data or multimodal data with the emotion labels is stored in advance in the training data storage unit 12b of the information processing device 1. Note that this data associating the video data or multimodal data with the emotion labels can be provided as (broadly defined) training data generally used for machine learning of a training model that estimates emotions. The training data generation unit 11a performs a process of generating (narrowly defined) training data suitable for machine learning to generate a training model for emotion estimation according to this embodiment, based on this broadly defined training data. In this embodiment, the training data generation unit 11a converts each of multiple modal data included in the multimodal data into a first feature, converts the first feature into a second feature based on the co-occurrence of the multiple first feature, and generates a third feature by reducing the dimension of the second feature. The training data generation unit 11a associates three first feature and three third feature corresponding to three modal data included in the original multimodal data with the emotion labels assigned to the original multimodal data, and generates training data.

[0039] The learning processing unit 11b performs machine learning using the learning data generated by the learning data generation unit 11a to generate a learning model that estimates human emotions. In this embodiment, the learning model generated by the learning processing unit 11b receives as input first and third feature amounts based on language data included in the multimodal data, first and third feature amounts based on image data, and first and third feature amounts based on audio data, and outputs data indicating an estimation result of the emotion of the person captured in the original multimodal data. The learning processing unit 11b inputs the three first feature amounts and three third feature amounts included in the learning data to the learning model, obtains an estimation result of the emotion output by the learning model in response to the input first and third feature amounts, calculates an error between the obtained estimation result and an emotion label associated with the input first and third feature amounts, and updates the internal parameters of the learning model based on the calculated error. The learning processing unit 11b repeatedly performs the above process using the learning data to repeatedly update the internal parameters, thereby generating a learning model that estimates emotions with a desired accuracy. The learning processing unit 11b stores information about the internal parameters of the learning model finally obtained by machine learning in the model information storage unit 12c.

[0040] The emotion estimation unit 11c performs a process of estimating the emotion of a person captured by a camera using a learning model generated by the learning processing unit 11b through machine learning. The emotion estimation unit 11c acquires video images of a person captured by a camera via the communication unit 13, separates the acquired video images into image data and audio data, and converts the audio data into language data through speech recognition processing to generate multimodal data including three modal data: language data, image data, and audio data. The emotion estimation unit 11c sequentially converts the multimodal data generated based on the video image data into first feature amounts, second feature amounts, and third feature amounts using information on conversion rules stored in the model information storage unit 12c. The emotion estimation unit 11c inputs the obtained first feature amounts and third feature amounts into an emotion estimation learning model stored in the model information storage unit 12c and acquires an emotion estimation result output by the learning model in response to the input, thereby estimating the emotion of the person captured in the video.

[0041] The display processing unit 11d performs processing to display various characters, images, and the like on the display unit 14. In the present embodiment, for example, during the learning stage of the learning model, the display processing unit 11d displays information such as an error or evaluation value related to the prediction result of the learning model as the progress status of the machine learning process performed by the learning processing unit 11b on the display unit 14. Furthermore, for example, during the inference stage using the trained learning model, the display processing unit 11d displays information such as image data or language data included in a moving image acquired from a camera, and an emotion estimation result estimated by the emotion estimation unit 11c based on the moving image on the display unit 14. Note that the display processing unit 11d may display various information other than the above on the display unit 14.

[0042] <Learning Data Generation Process and Learning Model Generation Process> In the information processing system according to the present embodiment, video data of a person is collected in advance, the collected video data is separated into image data and audio data, and the audio data is converted into language data by speech recognition processing, thereby generating multimodal data that is a combination of three modal data: image data, audio data, and language data. Note that the process of collecting video data and the process of generating multimodal data from the video data may be performed by the information processing device 1, or may be performed in advance by a device different from the information processing device 1. Furthermore, the process of generating multimodal data from video data may be performed in advance before the information processing device 1 performs the learning data generation process, or may be performed by the information processing device 1 during the learning data generation process.

[0043] In addition, in the information processing system according to this embodiment, collected video data or multimodal data generated from this video data is appropriately divided into sections, for example, at fixed time intervals or at variable time intervals depending on the content of the speech, and an emotion label indicating the emotion of the person depicted in each section is assigned. In this embodiment, two types of emotion labels, positive and negative, are used, but this is not limited thereto. Three or more types, such as joy, sadness, anger, surprise, fear, or disgust, may also be used. Furthermore, the task of assigning emotion labels to video data or multimodal data, i.e., the so-called annotation task, is performed in advance by, for example, a designer of the information processing system according to this embodiment. The annotation task may be performed by the information processing device 1 or by a device different from the information processing device 1. The information processing device 1 stores data in advance in the training data storage unit 12b, in which the video data or multimodal data is associated with emotion labels.

[0044] An information processing device 1 according to this embodiment performs processing to generate training data for machine learning that generates a learning model that estimates human emotions, based on pre-stored data associating video data or multimodal data with emotion labels. The pre-collected data associating video data or multimodal data with emotion labels can generally be provided as training data (in a broad sense) used for machine learning of a learning model that estimates emotions. Based on this training data in a broad sense, the information processing device 1 performs processing to generate training data (in a narrow sense) suitable for machine learning that generates a learning model for emotion estimation according to this embodiment.

[0045] 4 is a schematic diagram illustrating an example configuration of the learning model 2 according to the present embodiment. The learning model 2 according to the present embodiment is a learning model that has undergone machine learning in advance to receive as input first and third feature amounts based on language data included in multimodal data, first and third feature amounts based on image data, and first and third feature amounts based on speech data, and to output information on the result of emotion estimation of the subject related to the original multimodal data. In the present embodiment, the subject's emotion estimated by the learning model 2 can be either positive or negative. The learning model 2 outputs, for example, two numerical values, one indicating the likelihood that the subject's emotion is positive and the other indicating the likelihood that the subject's emotion is negative.

[0046] When the pre-collected data is video data, the information processing device 1 according to this embodiment separates the video data into image data and audio data, and further converts the audio data into language data (text data) through speech recognition processing, thereby generating multimodal data including three modal data: language data, image data, and audio data. Note that the technology for generating multimodal data including language data, image data, and audio data based on video data is already known, so a detailed description thereof will be omitted. When the collected data is multimodal data, the training data generation unit 11a may omit this process.

[0047] 5 is a flowchart showing an example of the procedure of the training data generation process performed by the information processing device 1 according to this embodiment. The training data generation unit 11a of the processing unit 11 of the information processing device 1 according to this embodiment reads out multiple sets of multimodal data and emotion labels that are previously associated and stored in the training data storage unit 12b (step S1).

[0048] Next, the training data generator 11a converts the three modal data, language data, image data, and voice data, contained in the multimodal data read in step S1, into first features (step S2). The information processing device 1 stores information about multiple learning models that convert modal data into first features appropriate for the data format in the model information storage unit 12c. Based on the information stored in the model information storage unit 12c, the training data generator 11a converts the language data contained in the multimodal data into first features using a learning model that converts language data into first features, converts the image data contained in the multimodal data into first features using a learning model that converts image data into first features, and converts the voice data contained in the multimodal data into first features using a learning model that converts voice data into first features. Note that in this embodiment, the first features converted from the language data, the first features converted from the image data, and the first features converted from the voice data are assumed to be vectors of the same dimension. The learning data generating unit 11a repeatedly performs a conversion process on a plurality of pieces of multimodal data collected in advance, and obtains a plurality of first feature amounts.

[0049] Next, the training data generation unit 11a converts each of the first feature quantities obtained from the plurality of multimodal data into a second feature quantity based on the co-occurrence of the first feature quantities. To this end, the training data generation unit 11a calculates the Euclidean distance between each of the first feature quantities and other first feature quantities based on the plurality of first feature quantities obtained in step S2 (step S3). The training data generation unit 11a performs a k-nearest neighbor search based on the Euclidean distances calculated in step S3 (step S4) to identify a predetermined number of other first feature quantities that are close in Euclidean distance to each first feature quantity. In this embodiment, the predetermined number of other first feature quantities identified in step S4 are considered to co-occur with a given first feature quantity. The predetermined number is determined as appropriate by the designer of the information processing system according to this embodiment.

[0050] Based on the predetermined number of other first feature quantities identified by the k-nearest neighbor search in step S4, the learning data generation unit 11a determines weights w of a predetermined number of other first feature quantities neighboring each first feature quantity, using the following formula (1) (step S5). In formula (1), K (where K is a natural number) neighboring first feature quantities f are defined as fj (j = 1, 2, ..., K), the weights of the other first feature quantities fj with respect to the first feature quantity f are defined as wj (j = 1, 2, ..., K), the Euclidean distances between the first feature quantity f and the other first feature quantities fj are defined as dj (j = 1, 2, ..., K), and the minimum of the K Euclidean distances dj is defined as p. In formula (1), α is a hyperparameter, and an appropriate value is set in advance by, for example, a designer of the information processing system according to this embodiment.

[0051]

[0052] The training data generation unit 11a converts each first feature into a second feature based on the weights determined in step S5 and equation (2) below (step S6). In equation (2), a second feature g is obtained by converting a certain first feature f based on multiple weights wj. According to equation (2), the second feature g for a certain first feature f is obtained by reconstructing the first feature f using a weighted sum (weighted sum, linear sum) of multiple other first feature fj that exist in the vicinity of the first feature f (i.e., have co-occurrence). The training data generation unit 11a converts all first feature values ​​obtained in step S2 into second feature values.

[0053]

[0054] Next, the learning data generating unit 11a converts the plurality of second feature quantities obtained by the conversion in step S6 into third feature quantities by reducing the dimensions using the UMAP method (step S7).

[0055] Next, the training data generator 11a clusters the multiple third feature quantities obtained by the conversion in step S7 based on the HDBSCAN method (step S8). Although detailed description of this method is omitted because it is an existing technology, clustering using the HDBSCAN method classifies the multiple third feature quantities into multiple clusters. The training data generator 11a outputs, for example, some of the original multimodal data of the second feature quantities classified into each cluster as samples, and receives input of semantic information for each cluster from a designer or the like of the information processing system according to this embodiment (step S9). The semantic information may be text information written in natural language by a designer or the like, such as a description of each cluster. Note that the input of semantic information in step S9 may be received at a different time, such as after the generation of the training data.

[0056] Next, the training data generator 11a generates training data by associating the first and third feature amounts corresponding to the language data of the original multimodal data, the first and third feature amounts corresponding to the image data, and the first and third feature amounts corresponding to the speech data with the emotion labels corresponding to the original multimodal data read out in step S1, for the plurality of first feature amounts obtained by the conversion in step S2 and the plurality of third feature amounts obtained by the conversion in step S7 (step S10). The training data generator 11a stores the training data generated in step S10 in the training data storage unit 12b of the storage unit 12 (step S11), and ends the process.

[0057] In step S10, the training data generation unit 11a stores information such as conversion rules for converting first features into second features, conversion rules for converting second features into third features, rules for clustering the third features, and semantic information input in step S9 in the model information storage unit 12c. The information on the conversion rules for converting first features into second features is information necessary for performing the calculations of the above-described equations (1) and (2), and in this embodiment may include information such as the multiple first features obtained in step S2 and the hyperparameter α of equation (2). In this embodiment, the information on the conversion rules for converting second features into third features may include information on a model for performing dimensionality reduction using the UMAP method. In this embodiment, the information on rules for clustering the third features may include information on a model obtained as a result of clustering using the HDBSCAN method.

[0058] 6 is a flowchart showing an example of the procedure of a learning model generation process performed by the information processing device 1 according to this embodiment. The learning processing unit 11b of the processing unit 11 of the information processing device 1 according to this embodiment acquires information such as the configuration of the learning model to be generated and initial values ​​of internal parameters based on input from a user such as a designer, or by reading a setting file or the like stored in the storage unit 12 (step S21). The learning processing unit 11b also reads the learning data generated by the above-mentioned learning data generation process from the learning data storage unit 12b of the storage unit 12 (step S22).

[0059] The learning processing unit 11b inputs the first and third feature quantities of the multimodal data (language data, image data, and speech data) included in the training data read in step S22 into the training model acquired in step S21 (step S23), and acquires information (emotion estimation results) output by the training model in response to this (step S24). The learning processing unit 11b calculates the error between the emotion estimation result acquired in step S24 and the emotion label included in the training data acquired in step S22 (step S25). Based on the error calculated in step S25, the learning processing unit 11b updates the internal parameters of the training model using, for example, an error backpropagation algorithm (step S26). The learning processing unit 11b performs the processes of steps S23 to S26 for all pairs of the first and third feature quantities of the multimodal data included in the training data read in step S22 and emotion labels.

[0060] The learning processing unit 11b determines whether the error calculated in step S25 is smaller than a threshold value previously determined by the designer or the like (step S27). If the error is larger than the threshold value (S27: NO), the learning processing unit 11b returns to step S23 and repeatedly updates the internal parameters of the learning model to improve prediction accuracy. If the error is smaller than the threshold value (S27: YES), the learning processing unit 11b stores information about the learning model, such as the internal parameters at that time, in the model information storage unit 12c (step S28), and ends the learning model generation process.

[0061] <Emotion Estimation Processing> In the information processing system according to the present embodiment, a camera captures the face of a subject whose emotion is to be estimated, and the information processing device 1 acquires video data captured by the camera. The information processing device 1 then performs a process of estimating the emotion of the subject using a learning model generated by the above-described generation method based on the acquired video data. The information processing device 1 converts the video data acquired from the camera into first, second, and third feature quantities in the same order as when generating the training data. To achieve this, the information processing device 1 must store information regarding the feature quantity conversion rules determined during the learning phase in the model information storage unit 12c. The information processing device 1 inputs the converted first and third feature quantities into the learning model and acquires the emotion estimation results output by the learning model in response to the input. This allows the information processing device 1 to estimate the emotion of the subject.

[0062] 7 is a flowchart showing an example of the procedure of emotion deduction processing performed by the information processing device 1 according to the present embodiment. The emotion deduction unit 11c of the processing unit 11 of the information processing device 1 according to the present embodiment transmits and receives data to and from the camera via the communication unit 13, thereby acquiring video data of a subject captured by the camera (step S41). The emotion deduction unit 11c separates the video data acquired in step S41 into image data and audio data (step S42). Furthermore, the emotion deduction unit 11c performs speech recognition processing on the audio data separated in step S42, thereby converting the audio data into language data (step S43). In this way, the emotion deduction unit 11c can convert the video data into multimodal data.

[0063] The feeling deduction unit 11c converts the language data, image data, and speech data obtained in Steps S42 and S43 into first feature quantities using the conversion learning models stored in advance in the model information storage unit 12c (Step S44). Through the processing of Step S44, the feeling deduction unit 11c obtains three first feature quantities from one piece of multimodal data. The feeling deduction unit 11c reads information on the conversion rules determined in the learning stage from the model information storage unit 12c, and converts the first feature quantities obtained in the processing of Step S44 into second feature quantities that take co-occurrence into consideration (Step S45).

[0064] Next, the emotion deduction unit 11c reads information about the conversion rules determined in the learning stage from the model information storage unit 12c, and converts the second feature obtained by the processing of step S45 into a third feature obtained by dimension reduction based on the UMAP method (step S46). At this time, the emotion deduction unit 11c also obtains the result of clustering the third feature based on the HDBSCAN method, i.e., information about which cluster the second feature belongs to (step S47).

[0065] Next, the feeling estimation unit 11c inputs the first feature amount obtained by the processing of step S44 and the third feature amount obtained by the processing of step S46 (the first feature amount and third feature amount corresponding to the language data, the first feature amount and third feature amount corresponding to the image data, and the first feature amount and third feature amount corresponding to the audio data) to a learning model generated by machine learning in the learning stage (step S48). The feeling estimation unit 11c acquires information on the feeling estimation result output by the learning model in response to the input of the third feature amount in step S48 (step S49).

[0066] The display processing unit 11d of the processing unit 11 displays the emotion estimation result acquired in step S49 on the display unit 14 (step S50). At this time, the display processing unit 11d can display the emotion estimation result together with, for example, a moving image based on the moving image data acquired in step S41. Furthermore, based on the clustering result acquired in step S47, the display processing unit 11d reads semantic information for the cluster to which the multimodal data belongs from the model information storage unit 12c, displays the read semantic information as grounds information for the emotion estimation result (step S51), and terminates the emotion estimation process. Note that by repeatedly performing the above-described process, the emotion estimation unit 11c can repeatedly estimate the emotion of a subject captured in a moving image captured by a camera in real time.

[0067] <Summary> In the information processing system according to the present embodiment configured as described above, emotion labels are assigned in advance to multimodal data including multiple modal data such as language data, image data, and audio data. The information processing device 1 acquires multiple sets of multimodal data and emotion labels, converts each of the multiple modal data into first feature quantities, converts each of the first feature quantities into second feature quantities based on co-occurrence among the multiple first feature quantities, and generates training data based on the converted first and second feature quantities and the emotion labels. As a result, the information processing system according to the present embodiment generates a training model through machine learning using training data generated in consideration of co-occurrence, and is expected to achieve highly accurate emotion estimation based on multimodal data using the generated training model.

[0068] Furthermore, in the information processing system according to this embodiment, the information processing device 1 calculates the distance (e.g., Euclidean distance) between multiple first feature quantities, and acquires multiple other first feature quantities that exist near one first feature quantity as other first feature quantities that have co-occurrence, for example, by performing a k-nearest neighbor search based on the calculated distance. The information processing device 1 calculates weights for each of the multiple other first feature quantities with respect to the one first feature quantity based on the above-described formula (1), and converts the one first feature quantity into a second feature quantity based on the calculated weights and formula (2). This allows the information processing device system according to this embodiment to obtain second feature quantities that take into account the co-occurrence of the first feature quantities.

[0069] In the information processing system according to the present embodiment, the information processing device 1 converts the second feature obtained by converting the first feature into a third feature with reduced dimensionality, and generates training data in which emotion labels are attached to the first feature and the third feature. The information processing device 1 also clusters multiple third feature quantities using a method such as HDBSCAN, accepts input of meaning information for each cluster, and stores it in a storage unit. This allows the information processing system according to the present embodiment to reduce the size of the training data by using third feature quantities obtained by reducing the dimensionality of the second feature quantities, which is expected to speed up machine learning using this training data and reduce the size of the training model generated using this training data.

[0070] Furthermore, in the information processing system according to this embodiment, the information processing device 1 performs machine learning using the training data generated by the above-described processing to generate a training model that estimates the emotions of a subject related to multimodal data. As a result, the information processing device system according to this embodiment can generate a training model by machine learning using training data generated in consideration of co-occurrence, and is expected to realize accurate emotion estimation based on multimodal data using the generated training model.

[0071] Furthermore, in the information processing system according to the present embodiment, the information processing device 1 acquires multimodal data including a plurality of modal data, such as language data, image data, and audio data, based on video images of a subject captured with a camera, converts each of the modal data into first feature quantities, converts the first feature quantities into second feature quantities based on the co-occurrence of the plurality of first feature quantities, reduces the dimensions of the second feature quantities and converts them into third feature quantities, inputs the first feature quantities and the third feature quantities into a learning model, acquires an emotion estimation result output by the learning model, and outputs the acquired emotion estimation result. As a result, the information processing device system according to the present embodiment can infer emotions taking into account the co-occurrence of a plurality of modal data included in the multimodal data, and is expected to achieve highly accurate emotion estimation based on multimodal data.

[0072] Furthermore, in the information processing system according to the present embodiment, the information processing device 1 classifies the second feature amounts, acquires meaning information set for each class, and outputs the meaning information together with the emotion estimation result. As a result, the information processing device system according to the present embodiment can be expected to output information that serves as the basis for the estimation together with the emotion estimation result.

[0073] In the present embodiment, the information processing device 1 reduces the dimension of the second feature quantity and converts it into a third feature quantity, and associates the first and third feature quantities with emotion labels to generate training data. However, this is not limited to this. The information processing device 1 may also associate the first and second feature quantities with emotion labels to generate training data without performing dimensionality reduction on the second feature quantity. Furthermore, in the information processing system according to the present embodiment, multimodal data including three modal data types, namely language data, image data, and voice data, is processed. However, the multiple modal data included in the multimodal data are not limited to these three. The multimodal data may include modal data such as a subject's heart rate, blood pressure, or body temperature, or any other modal data.

[0074] [Embodiment 2] Fig. 8 is a schematic diagram illustrating an example of the configuration of a learning model according to embodiment 2. A learning model 202 according to embodiment 2 receives as input first and third feature amounts based on language data included in multimodal data, first and third feature amounts based on image data, and first and third feature amounts based on speech data. In response to the input of this information, the learning model 202 is pre-machined to estimate two numerical values, namely, the valence and arousal level, of the subject's emotion related to the original multimodal data.

[0075] Valence is an evaluation index that indicates whether a person's emotion is positive (pleasant) or negative (unpleasant). In the second embodiment, valence is given as a numerical value (decimal value) between 0.0 and 1.0, and for example, a valence of 1.0 can be determined as positive, a valence of 0.0 as negative, and a valence of 0.5 as neutral.

[0076] The arousal level is an evaluation index that indicates whether a person is feeling aroused or calm. In the second embodiment, the arousal level is given as a numerical value between 0.0 and 1.0. For example, arousal level of 1.0 can be determined as arousal, and arousal level of 0.0 can be determined as calm.

[0077] In the information processing system according to the second embodiment, as training data used in machine learning to generate the learning model 202 shown in Fig. 8, the information processing system generates training data in which first and third feature amounts obtained by converting language data, image data, and audio data extracted from multimodal data are associated with two labels, valence and arousal, that have been assigned to the multimodal data in advance. The information processing device 1 performs machine learning processing using the generated training data, thereby generating the learning model 202 shown in Fig. 8.

[0078] The information processing device 1 according to the second embodiment uses the generated learning model 202 to estimate human emotions based on multimodal data. The information processing device 1 converts language data, image data, and audio data included in given multimodal data (or multimodal data converted from given video data) into first, second, and third feature quantities, respectively. Note that the process of converting the multimodal data into first feature quantities, the process of converting the first feature quantities into second feature quantities based on co-occurrence, and the process of converting the second feature quantities into third feature quantities based on clustering results are the same as those described in the first embodiment, and therefore detailed descriptions thereof will be omitted. The information processing device 1 inputs the first and third feature quantities of the image data, the first and third feature quantities of the image data, and the first and third feature quantities of the audio data into the learning model 202, and obtains two numerical values, valence and arousal, output by the learning model 202.

[0079] The information processing device 1 according to the second embodiment can perform more detailed emotion estimation based on two numerical values, valence and arousal, output by the learning model 202. FIG. 9 is a schematic diagram illustrating an emotion estimation method based on valence and arousal. This diagram shows a two-dimensional graph with valence on the horizontal axis and arousal on the vertical axis. The horizontal axis indicates pleasure toward the right and discomfort toward the left. The vertical axis indicates arousal toward the top and sleepiness toward the bottom. The graph also plots multiple points corresponding to 28 detailed emotions: "surprised," "excited," "pleasant," "happy," "happy," "satisfied," "fulfilled," "calm," "comfortable," "relaxed," "sleepy," "fatigued," "bored," "sad," "depressed," "depressed," "miserable," "frustrated," "unpleasant," "worried," "anger," "tension," "fear," "alert," and "awakened." The positions of the points corresponding to each emotion are predetermined.

[0080] Based on the two numerical values ​​of valence and arousal output by the learning model 202, a single point corresponding to the emotion estimation result can be plotted in the graph shown. In this figure, an example of the estimation result of the learning model 202 is indicated by a star plot. The information processing device 1 compares the positions of the 28 types of detailed emotions plotted in the graph with the position of the emotion estimation result of the learning model 202, and selects the detailed emotion that is closest to the position of the emotion estimation result. The information processing device 1 outputs the selected single detailed emotion as the final emotion estimation result. In the example shown, the information processing device 1 can set, for example, "relaxation" as the final emotion estimation result.

[0081] The information processing device 1 also calculates the distance from the origin of the graph for the estimation results of the learning model 202 plotted on the graph shown in the figure. The calculated distance serves as an index showing the strength of the estimated emotion. For example, the information processing device 1 compares the calculated distance with an appropriate threshold to calculate an index such as an emotion intensity level on a three- or five-level scale and presents it to the user along with the emotion estimation result. The information processing device 1 may also generate the graph shown in FIG. 9 and display it on the display unit 14.

[0082] Furthermore, in order to assist a user in interpreting the causal relationships of emotion estimation by the learning model 202, the information processing device 1 according to embodiment 2 can generate a causal relationship graph in which the correspondence between multiple feature amounts obtained from multimodal data and the emotions estimated by the learning model 202 based on these feature amounts is mapped onto a two-dimensional plane, and can display the generated graph on the display unit 14.

[0083] Fig. 10 is a schematic diagram showing an example of a causal relationship graph. Fig. 11 is a schematic diagram for explaining a method for generating a causal relationship graph. The causal relationship graph displayed by the information processing device 1 according to embodiment 2 is a graph in which the feature amounts of the language data, image data, and audio data included in the multimodal data and the emotions estimated by the learning model 202 based on these feature amounts are mapped on the same two-dimensional plane.

[0084] In this embodiment, the language data included in the multimodal data is, for example, text information obtained by converting speech uttered by a subject in a video image into text. The information processing device 1 extracts one or more words from the text information of the language data included in the multimodal data using an appropriate method. The information processing device 1 converts a feature (third feature) obtained based on the language data into a two-dimensional feature, for example, by dimension reduction, and places the words extracted from the text information at coordinates in the graph corresponding to the converted feature. In the causal relationship graph shown in FIG. 10 , character strings other than those written within ellipses, such as "good story," "think," "say," and "depressing," are words extracted from the language data.

[0085] The information processing device 1 converts the features (third features) obtained by performing the conversion process using the above-mentioned procedure on the image data and audio data contained in the multimodal data into text information by referring to a pre-stored conversion table.

[0086] 12 is a schematic diagram showing an example of a conversion table. The conversion table shown is for converting the feature quantities of image data into text information, and the feature quantities of the image data are assumed to be n-dimensional (n is a natural number). The conversion table stores conditions defined for n numerical values ​​f1, f2, ..., fn included in the n-dimensional feature quantities, and text information that is the conversion result when these conditions are satisfied, in association with each other.

[0087] In the illustrated example, the conversion table stores a correspondence such that, for example, if the first feature value f1 exceeds 0.1, the second feature value f2 is less than 0.02, and so on, and the nth feature value fn exceeds 0.18, the feature is converted into text information such as "downturned corners of the mouth." Also, the conversion table stores a correspondence such that, for example, if the first feature value f1 is less than 0.2, the second feature value f2 is less than 0.05, and so on, and the nth feature value fn exceeds 0.12, the feature is converted into text information such as "wide open mouth." Also, the conversion table stores a correspondence such that, for example, if the first feature value f1 is less than 0.3, the second feature value f2 is less than 0.11, and so on, and the nth feature value fn exceeds 0.7, the feature is converted into text information such as "pointed mouth."

[0088] 12 shows a conversion table for converting the feature quantities of image data into text information, but a conversion table for converting the feature quantities of audio data into text information can also be realized with a similar configuration. The conditions and text information stored in these conversion tables are determined in advance by, for example, a designer or administrator of the information processing system according to this embodiment, and are stored in the storage unit 12 of the information processing device 1.

[0089] The information processing device 1 converts the features (third features) obtained based on the image data and audio data into two-dimensional features by, for example, dimension reduction, and places text information obtained by converting the features using a conversion table at coordinates in the graph corresponding to the converted features. In the causal relationship graph shown in Fig. 10 , character strings written in black letters within ovals, such as "voice pitch: low," "eyes: corners of eyes drooping," "mouth: corners of mouth drooping," "speaking rate: slow," "voice volume: quiet," and "eyes: downcast," represent text information obtained by converting the features.

[0090] 11 , the information processing device 1 can obtain three pieces of text information corresponding to each modal data and an emotion estimation result by the learning model 202 based on the given multimodal data. In this example, the information processing device 1 acquires text information of "boredom" from the language data included in the multimodal data, acquires text information of "drooping mouth corners" from the image data, and acquires text information of "high-pitched voice" from the audio data, and also acquires the emotion estimation result of "anger" by the learning model 202 based on the feature amounts (first feature amount and third feature amount) of each modal data. In this example, the learning model 202 outputs two values, valence and arousal, and the information processing device 1 determines a final emotion estimation result based on these two values ​​using the graph shown in FIG. 9 .

[0091] The information processing device 1 arranges the obtained three pieces of text information in a causal relationship graph at positions according to the feature amounts, and displays the three pieces of text information in colors according to the emotion estimation results. However, since FIG. 10 is a monochrome image, the color coding is omitted. The information processing device 1 similarly acquires three pieces of text information for each of multiple modal data and plots them on a causal relationship graph.

[0092] After plotting text information on a causal relationship graph for multiple pieces of multimodal data for which emotion estimation has been performed, the information processing device 1 plots and displays labels indicating emotion estimation results by the learning model 202 on the causal relationship graph. In the causal relationship graph shown in FIG. 10 , character strings written in white letters within ovals on a black background, such as "surprise," "anger," "joy," "fun," "sadness," and "fear," are labels indicating emotion estimation results. For example, the information processing device 1 acquires the coordinates of one or more pieces of text information corresponding to the emotion estimation result of "anger" from the multiple pieces of text information plotted on the causal relationship graph, and places the label "anger" near the location where this text information is placed. The same applies to other emotion estimation results.

[0093] The information processing device 1 according to the present embodiment converts multiple modal data included in multimodal data into feature quantities taking co-occurrence into consideration. The modal data mapped onto the causal graph based on these feature quantities are arranged so that data with high co-occurrence are located close to each other.

[0094] In the information processing system according to the second embodiment configured as described above, the information processing device 1 acquires information on valence (first label) and arousal (second label) associated with multimodal data, and generates training data in which the acquired valence and arousal information is associated with first and third feature amounts obtained by converting multiple modal data included in the multimodal data. The information processing device 1 performs machine learning using the generated training data to generate a learning model 202 that estimates valence and arousal based on the feature amounts (first and third feature amounts) generated from the multiple modal data included in the multimodal data. As a result, the information processing device 1 according to the second embodiment is expected to estimate more detailed emotions based on the two numerical values ​​of valence and arousal estimated by the generated learning model 202.

[0095] Furthermore, in the information processing system according to the second embodiment, the information processing device 1 converts each of multiple modal data included in the multimodal data into linguistic information, and displays a causal relationship graph in which the converted linguistic information is arranged at coordinates according to the feature quantities of the corresponding modal data. The information processing device 1 also displays emotion estimation results by the learning model 202 near the linguistic information. As a result, the information processing system according to the present embodiment is expected to assist the user in estimating causal relationships regarding the emotion estimation results by the learning model 202, and to improve the interpretability of emotion estimation by the learning model 202.

[0096] Furthermore, the learning model 2, 202 generated by machine learning using the learning data generated by the information processing device 1 in the first and second embodiments is expected to have a smaller data size and to be able to operate on a low-spec information processing device, compared to, for example, a large-scale language model that can handle multimodal data. Therefore, even in an environment where communication with a high-performance server device is impossible or where sufficient communication speed is not available, it is expected that the learning model 2, 202 can be operated on a low-spec information processing device to perform emotion estimation. For example, it is expected that the learning model 2, 202 can estimate the driver's emotion based on video images of the driver captured by an on-board camera installed near the driver's seat of the vehicle, and output a warning if drowsiness is detected, and other processing can be performed by a low-spec information processing device that can be installed in the vehicle.

[0097] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0098] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the reference format. Furthermore, the claims use a format in which a claim references two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that references at least one other multiple claim may also be used.

[0099] REFERENCE SIGNS LIST 1 Information processing device (computer) 2 Learning model 11 Processing unit 11a Learning data generation unit 11b Learning processing unit 11c Emotion estimation unit 11d Display processing unit 12 Storage unit 12a Program (computer program) 12b Learning data storage unit 12c Model information storage unit 13 Communication unit 14 Display unit 15 Operation unit 202 Learning model

Claims

1. A data generation method in which an information processing device generates data for machine learning, the data generation method comprising the steps of: acquiring multimodal data including a plurality of modal data and information on labels related to emotions associated with the multimodal data; converting each of the plurality of modal data into a first feature; converting the first feature into a second feature based on the co-occurrence of a plurality of the first feature; and generating the data for machine learning based on the first feature and the second feature.

2. The data generation method according to claim 1, further comprising: calculating a distance between a plurality of said first features; acquiring a plurality of other first features that co-occur with one first feature based on the calculated distance; calculating weights of the other first features with respect to the one first feature based on the acquired plurality of other first features; and converting the one first feature into the second feature based on the calculated weights.

3. The data generation method according to claim 1, further comprising: converting the converted second feature into a third feature with reduced dimensions; and generating data in which the labels are assigned to the first feature and the third feature as data for machine learning.

4. The data generation method according to claim 3, further comprising: clustering the third feature amount; accepting input of meaning information for each cluster; and storing the accepted meaning information in a storage unit.

5. The data generation method of claim 3, wherein the information processing device acquires information on a first label and a second label associated with the multimodal data, and generates data in which the first label and the second label are assigned to the first feature and the third feature as data for machine learning, wherein the first label is numerical information indicating emotional valence, and the second label is numerical information indicating arousal level.

6. A method for generating a learning model in which an information processing device generates a learning model that estimates human emotions based on multimodal data, wherein the information processing device: acquires multimodal data including a plurality of modal data and information on labels related to emotions associated with the multimodal data; converts each of the plurality of modal data into a first feature; converts the first feature into a second feature based on co-occurrence of a plurality of the first feature; generates data for machine learning based on the first feature and the second feature; and generates the learning model by machine learning using the generated data for machine learning.

7. An information processing method in which an information processing device estimates human emotions based on multimodal data, the information processing device: acquires multimodal data including a plurality of modal data; converts each of the plurality of modal data into a first feature; converts the first feature into a second feature based on the co-occurrence of a plurality of the first feature; estimates the human emotion related to the multimodal data based on the first feature and the second feature and a learning model that has been previously subjected to machine learning to estimate human emotions; and outputs the estimation result.

8. The information processing method according to claim 7, further comprising: converting the converted second feature into a third feature with reduced dimensionality; inputting the first feature and the third feature into the learning model that has undergone machine learning to accept the first feature and the third feature as input and output an emotion estimation result; obtaining the emotion estimation result output by the learning model; and outputting the obtained estimation result.

9. The information processing method according to claim 8, wherein the learning model is generated by: acquiring multimodal data including a plurality of modal data and information on labels related to emotions associated with the multimodal data; converting each of the plurality of modal data into first features; converting the first features into second features based on co-occurrence of a plurality of the first features; converting the converted second features into third features with reduced dimensions; generating data for machine learning based on the first features, the third features, and the labels; and performing machine learning using the generated data for machine learning.

10. The information processing method according to claim 9, further comprising: clustering the third feature amount to obtain semantic information set for each cluster; and outputting the obtained semantic information together with the estimation result.

11. The information processing method of claim 8, wherein the learning model is generated by: acquiring multimodal data including a plurality of modal data, and information on first labels and second labels related to emotions associated with the multimodal data; converting each of the plurality of modal data into a first feature; converting the first feature into a second feature based on co-occurrence of a plurality of the first feature; converting the converted second feature into a third feature with reduced dimension; generating data for machine learning based on the first feature, the third feature, the first label, and the second label; and performing machine learning using the generated data for machine learning, wherein the first label is numerical information indicating emotional valence, and the second label is numerical information indicating arousal level.

12. The information processing method according to claim 8, further comprising converting each of a plurality of modal data included in the multimodal data into linguistic information, and displaying a graph in which the linguistic information converted from each modal data is arranged at coordinates corresponding to the third feature amount of the modal data.

13. The information processing method according to claim 12, wherein the emotion estimation result obtained by the learning model is displayed near the linguistic information.

14. A computer program that causes a computer to execute a process of generating data for machine learning, the computer executing the following processes: acquiring multimodal data including a plurality of modal data and information on labels related to emotions associated with the multimodal data; converting each of the plurality of modal data into first feature quantities; converting the first feature quantities into second feature quantities based on co-occurrence of a plurality of the first feature quantities; and generating the data for machine learning based on the first feature quantities and the second feature quantities.

15. A computer program that causes a computer to execute processes for generating a learning model that estimates human emotions based on multimodal data, the computer program causing the computer to execute processes of: acquiring multimodal data including a plurality of modal data and information on labels related to emotions associated with the multimodal data; converting each of the plurality of modal data into first features; converting the first features into second features based on co-occurrence of a plurality of the first features; generating data for machine learning based on the first features and the second features; and generating the learning model by machine learning using the generated data for machine learning.

16. A computer program that causes a computer to execute a process of estimating human emotions based on multimodal data, the computer executing the following processes: acquiring multimodal data including a plurality of modal data; converting each of the plurality of modal data into a first feature; converting each of the first feature into a second feature based on co-occurrence of a plurality of the first feature; estimating human emotions related to the multimodal data based on the first feature and the second feature and a learning model that has been previously subjected to machine learning to estimate human emotions; and outputting the estimation result.

17. An information processing device comprising a processing unit that generates data for machine learning, wherein the processing unit: acquires multimodal data including a plurality of modal data and information on labels related to emotions associated with the multimodal data; converts each of the plurality of modal data into a first feature; converts the first feature into a second feature based on co-occurrence of a plurality of the first feature; and generates the data for machine learning based on the first feature, the second feature, and the label.

18. An information processing device comprising a processing unit that generates a learning model that estimates human emotions based on multimodal data, wherein the processing unit: acquires multimodal data including a plurality of modal data and information on labels related to emotions associated with the multimodal data; converts each of the plurality of modal data into a first feature; converts the first feature into a second feature based on co-occurrence of a plurality of the first feature; generates data for machine learning based on the first feature, the second feature, and the label; and generates the learning model by machine learning using the generated data for machine learning.

19. An information processing device comprising a processing unit that estimates human emotions based on multimodal data, wherein the processing unit: acquires multimodal data including a plurality of modal data; converts each of the plurality of modal data into a first feature; converts the first feature into a second feature based on co-occurrence of a plurality of the first feature; estimates the human emotion associated with the multimodal data based on the first feature and the second feature and a learning model that has been previously machine-learned to estimate human emotions; and outputs the estimation result.

Images