Image generation device and image generation method

Abstract

This image generation device estimates a first feature amount (feature amount of an inner face state) indicating an inner state of a user from brain wave data of the user, and estimates a second feature amount (feature amount of an action unit) indicating presence / absence of each movement of the face of the user from the brain wave data. Thereafter, the image generation device estimates a movement amount for each movement of the face of the user by using an exercise amount estimation model trained to output an estimation result of the movement amount for each movement of the face of the user, with the first feature amount and the second feature amount of the user as an input. Then, the image generation device generates and outputs an image of the face of the user using the estimation result of the movement amount for each movement of the face of the user.

Description

Image generating device and image generating method

[0001] The present invention relates to an image generating device and an image generating method for generating an image of a user's internal state and facial expression using the user's electroencephalogram.

[0002] Conventionally, a communication support system using a BMI (Brain Machine Interface) has been proposed (see Non-Patent Document 1).

[0003] Ryohei Hasegawa, Communication Strategies in ALS Patients: Current Status and Prospects of BMI, Clinical Neurology, Vol. 53, No. 11 (2013: 11)

[0004] However, existing BMIs require command input, making it impossible to transmit emotions in a timely manner. They also cannot express emotions unconsciously. Furthermore, existing BMIs can only select from a set of discrete emotions, and are unable to express emotions as flexibly as facial expressions.

[0005] Therefore, an object of the present invention is to solve the above-mentioned problems, generate a facial image of a user with a natural expression from the user's electroencephalogram, and convey emotions in a timely manner.

[0006] In order to solve the above-mentioned problems, the present invention is characterized by comprising a first estimation unit that estimates a first feature amount that indicates the internal state of the user from the electroencephalogram data of the user; a second estimation unit that estimates a second feature amount that indicates the presence or absence of each facial movement of the user from the electroencephalogram data; a third estimation unit that estimates the amount of movement of each facial movement of the user from the estimation results of the first feature amount and the second feature amount of the user using a movement amount estimation model that receives the first feature amount and the second feature amount of the user as input and is trained to output an estimation result of the amount of movement for each facial movement of the user; and an image generation unit that generates and outputs an image of the facial expression of the user using the estimation result of the amount of movement for each facial movement of the user.

[0007] According to the present invention, a facial image with a natural expression can be generated from the user's electroencephalogram, and emotions can be conveyed in a timely manner.

[0008] Fig. 1 is a diagram for explaining an overview of an image generation device. Fig. 2 is a diagram showing an example of the configuration of an image generation device. Fig. 3 is a flowchart showing an example of a processing procedure executed by the image generation device. Fig. 4 is a diagram showing an example of a computer that executes an image generation program.

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a description will be given of an embodiment of the present invention with reference to the drawings, but the present invention is not limited to the embodiment.

[0010] In the following, a case will be described in which the user's facial movements (movements of facial muscles) are expressed by at least one of the action units (AUs) used in the FACS (Facial Action Coding System). An action unit is a component of various facial movements when facial expressions are coded according to the FACS.

[0011] [Overview] An overview of the image generating device of this embodiment will be described using Fig. 1. For example, as shown in Fig. 1, the image generating device estimates the presence or absence of movement of each action unit of the user's face from the user's electroencephalogram data, and also estimates the user's internal state. The image generating device then combines these estimation results to estimate the amount of movement (amount of movement) for each action unit of the user's face. Thereafter, the image generating device generates an image of the user's face (image of facial expression) using the estimation results of the amount of movement for each action unit.

[0012] In this way, the image generating device can generate and output a facial image with a natural expression based on the user's electroencephalogram data, thereby enabling the user to convey their emotions in a natural and timely manner.

[0013] [Configuration Example] Next, a configuration example of the image generating device 10 will be described with reference to Fig. 2. The image generating device 10 includes an electroencephalogram acquiring unit 111, an inner state estimating unit (first estimating unit) 121, an action unit estimating unit (second estimating unit) 122, an activity amount estimating unit (third estimating unit) 123, a facial feature setting unit 131, a facial image generating unit 132, an output control unit 133, and an output unit 134.

[0014] The electroencephalogram acquisition unit 111 acquires the user's electroencephalogram data (for example, time-series electroencephalogram data) and outputs it to the inner state estimation unit 121 and the action unit estimation unit 122 .

[0015] The internal state estimation unit 121 estimates feature quantities (internal state feature quantities, first feature quantities) of the internal state (emotional state) of the user from the user's electroencephalogram data. For example, the internal state estimation unit 121 estimates the feature quantities of the user's internal state from the electroencephalogram data using an electroencephalogram-internal state model (details of which will be described later). Then, the internal state estimation unit 121 outputs the estimation results of the feature quantities of the user's internal state to the exercise amount estimation unit 123.

[0016] The feature quantity of the internal state is expressed, for example, as a one-hot vector or M-hot vector (M≦N) of N-dimensional emotion categories, or a likelihood vector. Note that, for example, if it is assumed that a user has an emotion that is a mixture of emotions from multiple categories (mixed emotion), it is possible to express any mixed emotion as a one-hot vector by treating the mixed emotion as a single emotion category.

[0017] Furthermore, when expressing the inner state as two-dimensional information of Valence-Arousal, each axis may be classified into several divisions to generate N categories, and the information indicating the category to which the inner state belongs as an N-dimensional vector may be used as the feature of the inner state.

[0018] The electroencephalogram data used by the internal state estimation section 121 may be the electroencephalogram time series data itself acquired by the electroencephalogram acquisition section 111, or may be electroencephalogram time series data that has been preprocessed by noise removal, etc. Furthermore, the electroencephalogram data used by the internal state estimation section 121 may be, for example, data obtained by converting the electroencephalogram time series data or preprocessed electroencephalogram time series data into electroencephalogram feature quantities such as power spectral density and instantaneous phase.

[0019] The electroencephalogram-internal state model is an estimation model trained to receive electroencephalogram data as input and output estimation results of feature quantities of the internal state of the subject (user) based on the electroencephalogram data. The electroencephalogram-internal state model is an estimation model trained, for example, by the process shown in FIG.

[0020] (1) EEG data is recorded when the user is induced into a specific emotional state. (2) Training data is created using the features of the induced emotional state as training labels for the features of the internal state corresponding to the EEG data. (3) Using the training data, an EEG-internal state model is trained to estimate the features of the user's internal state from the EEG data.

[0021] Alternatively, EEG data obtained when the user is induced into various emotional states may be recorded in advance, and an estimation model may be used as the EEG-internal state model, which outputs a vector (or a vector obtained by dimensionally compressing the vector) indicating the expression similarity between the input EEG data and the pre-recorded EEG data as a feature of the internal state.

[0022] Furthermore, an inference model that estimates the feature amount of the internal state expressed as the emotional state feature amount for the induced emotional state from the expression similarity vector may be used as the electroencephalogram-internal state model.

[0023] The action unit estimation unit 122 estimates, from the electroencephalogram data of the user, features (action unit features, second features) indicating the presence or absence of movements of each action unit on the face of the user. For example, the action unit estimation unit 122 estimates the features of the action units of the user from the electroencephalogram data of the user using an electroencephalogram-action unit model (details of which will be described later).

[0024] The feature amount of an action unit is, for example, an M-dimensional binary vector in which, for M action units, 1 is assigned if the action of that action unit is present at a predetermined value or more, and 0 is assigned otherwise. Alternatively, the feature amount of an action unit may be, for example, an M-dimensional real vector in which the probability or likelihood that the action of that action unit is present at a predetermined value or more is assigned.

[0025] The EEG-action unit model is an estimation model trained to input a user's EEG data and output an estimation result of the feature quantities of the user's action units. The EEG-action unit model is an estimation model trained, for example, by the processes shown in (1) to (3) below.

[0026] (1) Record facial expressions and EEG data when specific facial expressions are induced in the user. (2) Prepare training data using the feature quantities of the action units created by estimating the action units corresponding to the above facial expressions according to FACS as training labels. (3) Use the above training data to train an EEG-action unit model that estimates the feature quantities of the user's action units from EEG data.

[0027] In (1) above, methods for inducing a specific facial expression in the user include, for example, consciously making the user make a specific facial expression, showing the user the facial expression, having the user imagine the facial expression, or providing some kind of stimulus to naturally bring out the facial expression.

[0028] Furthermore, when showing facial expression images to a user, for example, facial expression images generated by specifying an action unit may be shown. In this case, for example, the specified action unit may be used as a feature of the action unit to train an electroencephalogram-action unit model.

[0029] The exercise amount estimation unit 123 estimates the exercise amount for each action unit of the user's face using the estimation results of the feature amounts of the user's inner state and the estimation results of the feature amounts of the user's action units.

[0030] The action unit for which the amount of exercise is to be estimated may be the same as or different from the action unit for which the feature of the above action unit is to be estimated. For example, the action unit for which the amount of exercise is to be estimated may be any action unit used in FACS, or may be an action unit that can be operated by the face image generation unit 132 (described later).

[0031] The momentum estimation unit 123 receives the feature amounts of the user's inner state and the feature amounts of the action units as input, and estimates the momentum of the user for each action unit using a momentum estimation model trained to output an estimation result of the momentum for each action unit of the user's face. The momentum estimation model is, for example, an estimation model trained by the following processes (1) to (3).

[0032] (1) For a facial expression image whose emotional state is known, the feature amount of the inner state of the person in the facial expression image, the feature amount of the action unit, and the amount of movement of the action unit are estimated.

[0033] The feature amounts of the inner state, the feature amounts of the action units, and the momentum amounts of the action units may be estimated by a human being.

[0034] Furthermore, the above-mentioned feature amounts of the internal state may be obtained, for example, by having an arbitrary machine learning model that estimates the emotion of a person in a facial image from the image output the likelihood of each emotion category for the facial expression image.

[0035] Furthermore, the feature amounts of the above action units may be generated from a face image using, for example, OpenFace etc. For example, using the above OpenFace etc., the presence or absence, likelihood, and strength of the movement of each action unit are calculated from the face image, and the calculation results are used as the feature amounts of the action unit.

[0036] Furthermore, facial expression images with known emotional states can be collected in any way. For example, facial expression images can be naturally captured images of faces, or images generated by an image generation AI or the like that specifies either or both of the emotional state and the action unit's movement.

[0037] (2) Prepare training data in which the momentum of the action unit is assigned as a training label for the feature amount of the inner state and the feature amount of the action unit.

[0038] (3) Using the training data, a model (momentum estimation model) that estimates the momentum of an action unit from the feature amounts of the inner state and the feature amounts of the action unit is trained.

[0039] The momentum of an action unit may be estimated, for example, as follows: For example, from a collection of facial expression images whose emotional states and action units are known, a facial expression image whose matching rate between the feature amounts of a given internal state and the feature amounts of an action unit is equal to or exceeds a predetermined value is selected, and the momentum of the action unit of the selected facial expression image is then taken as the estimation result.

[0040] The matching rate may be calculated by, for example, calculating a correlation coefficient between the feature of the internal state or the feature of the action unit and the emotional state or action unit of each facial expression image. Alternatively, the feature expressed by likelihood may be regarded as the probability of existence of the emotional state or action unit, and the matching rate may be calculated by calculating the probability that a combination of the emotional state of the facial expression image and the action unit occurs.

[0041] The facial feature setting unit 131 sets facial feature amounts representing a base facial image (for example, an image of the user's face) in the facial image generation unit 132. For example, the facial feature setting unit 131 may convert an input facial image into facial feature amounts, or may set the facial feature amounts by directly or indirectly manipulating the facial feature amounts.

[0042] The facial image generation unit 132 generates a facial image based on the facial feature amounts set by the facial feature setting unit 131 and the amounts of movement of the action units estimated by the movement amount estimation unit 123. For example, the facial image generation unit 132 inputs the facial feature amounts and the amounts of movement of the action units to an image generation decoder to generate a facial image. Then, the facial image generation unit 132 outputs the generated facial image to the output control unit 133.

[0043] The image generation decoder uses a model that has been trained in advance to generate an image from the latent space of facial features. If the facial features represent key points of facial features, the facial image generation unit 132 may generate a facial image by having the image generation decoder morph each facial feature using the momentum of the action unit.

[0044] The output control unit 133 outputs the face image output from the face image generation unit 132 to the output unit 134. The output unit 134 presents the face image.

[0045] [Example of Processing Procedure] Next, an example of processing procedure executed by the image generating device 10 will be described with reference to Fig. 3. It is assumed that the facial feature setting unit 131 of the image generating device 10 has already set facial features representing a base face image in the facial image generating unit 132.

[0046] First, the electroencephalogram acquisition unit 111 of the image generating device 10 acquires electroencephalogram data of the user (S1). Then, the internal state estimation unit 121 estimates features of the internal state from the electroencephalogram data acquired in S1 (S2). Furthermore, the action unit estimation unit 122 estimates features of action units from the electroencephalogram data acquired in S1 (S3).

[0047] Thereafter, the exercise amount estimation unit 123 estimates the exercise amount for each action unit using the estimation result of the feature amount of the inner state obtained in S2 and the estimation result of the feature amount of the action unit obtained in S3 (S4).

[0048] After S4, the facial image generation unit 132 generates a facial image using the estimated results of the amount of movement for each action unit obtained in S4 (S5).Then, the output control unit 133 outputs the image generated in S5 to the output unit 134 (S6).

[0049] According to the image generating device 10 described above, when generating a facial image that expresses the emotions of a user using the user's electroencephalogram, the amount of movement for each action unit of the user's facial expression is estimated based on a combination of the features of the user's action units and the features of the user's internal state.

[0050] As a result, even if only rough information about the user's facial expression (e.g., the presence or absence of an action unit) can be obtained from the user's brain waves, the image generating device 10 can generate a facial image with a natural expression and convey emotions in a natural and timely manner.

[0051] In other words, because EEG data can only capture global brain activity, conventional technology has made it difficult to estimate the magnitude of a user's movement (amount of movement) for each action unit from EEG data.

[0052] Meanwhile, the image generating device 10 learns in advance the relationship between the user's internal state and the magnitude of the movement (movement of facial muscles) for each action unit. This allows the image generating device 10 to estimate the magnitude of the movement of an action unit based on the user's internal state and the presence or absence of the movement of the action unit, which are estimated from the electroencephalogram data. As a result, the image generating device 10 can generate a stable and highly accurate facial expression image from the user's electroencephalogram.

[0053] Furthermore, the image generating device 10 can indirectly express the intensity of emotions and the magnitude of actions of action units by using the presence or absence of each emotion and the likelihood of actions of action units as features of the internal state and action units. As a result, the image generating device 10 can improve the accuracy of estimating the magnitude of actions of action units.

[0054] [System Configuration, etc.] The components of each unit shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0055] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0056] [Program] The image generating device 10 can be implemented by installing a program (image generating program) as package software or online software on a desired computer. For example, by executing the program on an information processing device, the information processing device can function as the image generating device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as terminals such as PDAs (Personal Digital Assistants).

[0057] 4 is a diagram showing an example of a computer that executes an image generation program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0058] The memory 1010 includes a read-only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input / output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0059] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the image generation device 10 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configuration of the image generation device 10 are stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0060] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0061] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0062] REFERENCE SIGNS LIST 10 Image generating device 111 Brain wave acquiring unit 121 Inner state estimating unit (first estimating unit) 122 Action unit estimating unit (second estimating unit) 123 Momentum estimating unit (third estimating unit) 131 Facial feature setting unit 132 Facial image generating unit 133 Output control unit 134 Output unit

Claims

1. An image generation device comprising: a first estimation unit that estimates a first feature amount indicating the internal state of the user from the user's electroencephalogram data; a second estimation unit that estimates a second feature amount indicating the presence or absence of each facial movement of the user from the electroencephalogram data; a third estimation unit that estimates the amount of movement of the user for each facial movement from the estimation results of the first feature amount and the second feature amount of the user using a movement amount estimation model that receives the user's first feature amount and second feature amount as input and is trained to output an estimation result of the movement amount for each facial movement of the user; and an image generation unit that generates and outputs an image of the user's facial expression using the estimation result of the movement amount for each facial movement of the user.

2. The image generating device described in claim 1, characterized in that the first feature is a feature indicating the likelihood of each emotion category of the user, and the second feature is a feature indicating the likelihood of the presence or absence of each facial movement of the user.

3. The image generating device according to claim 1, wherein each of the facial movements is a movement of at least one of the action units in a FACS (Facial Action Coding System).

4. An image generation method executed by an image generation device, comprising: a step of estimating a first feature amount indicating the internal state of the user from the user's electroencephalogram data; a step of estimating a second feature amount indicating the presence or absence of each facial movement of the user from the electroencephalogram data; a step of estimating the amount of movement for each facial movement of the user from the estimation results of the first feature amount and the second feature amount of the user using a momentum estimation model that receives the first feature amount and the second feature amount of the user as input and is trained to output an estimation result of the amount of movement for each facial movement of the user; and a step of generating and outputting an image of the user's facial expression using the estimation result of the amount of movement for each facial movement of the user.

Images