Emotion determination device and model learning device, and their programs

Abstract

To provide an emotion determination device capable of accurately determining the emotion of a person shown in an image.SOLUTION: An emotion determination device 1 comprises: three-dimensional facial feature point extraction means 21 for extracting positions of three-dimensional facial feature points from an input facial image using a prior-trained three-dimensional facial feature point extraction model; face image feature extraction means 22 for extracting a facial image feature vector, which is a facial feature quantity, from the input facial image using a prior-trained facial image feature extraction model; feature amount combining means 23 for generating a feature vector by concatenating the positions of the three-dimensional facial feature points with the facial image feature vector; and determination means 24 for determining a label of a predetermined emotion from the feature vector using a prior-trained emotion determination model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to an emotion determination device and a model learning device for determining a person's emotions from a facial image, as well as programs for these devices. [Background technology]

[0002] In recent years, expanding universal services has become an important societal initiative. Many similar efforts are being made in broadcast content, including subtitled broadcasts and sign language commentary. Furthermore, in recent years, research and development of technologies aimed at realizing video media that can be viewed with a greater sense of realism by all people, including those with visual impairments. For example, by acquiring match events such as "scores" from competition footage in real time and presenting haptic vibrations corresponding to those match events on a haptic feedback device, it is possible to provide viewers with a greater sense of realism. Furthermore, presenting the emotions of the characters within broadcast content also contributes to providing viewers with a high level of immersion.

[0003] Numerous machine learning-based methods have been studied for determining emotions from images of people within such content. For example, a method has been disclosed that uses a convolutional neural network to capture facial features and determine emotions (see Patent Document 1 and Non-Patent Document 1). Furthermore, in broadcast footage, people are not always facing forward. Therefore, a method has been disclosed that uses an attention network to determine facial expressions from facial images that include difficult conditions such as head tilt and obstacles (for example, covering the face with hands, wearing glasses) (see Non-Patent Document 2). [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2018-62313 [Non-patent literature]

[0005] [Non-Patent Document 1] "Acquisition of Facial Expression Representation Using Convolutional Neural Networks," Daiki Nishime et al., 2016 Annual Conference of the Japanese Society for Artificial Intelligence, 4L1-5in1, General Presentation, June 9, 2016. [Non-Patent Document 2] Wang, K.; Peng, X.; Yang, J. "Region attention networks for pose and occlusion robust facial expression recognition." IEEE Trans. Image Process. 2020, 29, 4057-4069. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] However, the methods described in Patent Document 1 and Non-Patent Document 1 have the problem that, because they determine emotions from the features of a facial image, the accuracy of the determination becomes low when the person is not facing forward, or when there are difficult conditions such as the tilt of the face or obstacles. Furthermore, the method described in Non-Patent Document 2 only considers the facial region visible in the image and does not consider facial information not visible in the image when determining emotion, so further improvement in estimation accuracy was desired.

[0007] This invention has been made in view of the above problems, and aims to provide an emotion determination device and a model learning device, as well as programs thereof, that can accurately determine the emotions of a person in an image by using facial information not visible in the image as auxiliary information. [Means for solving the problem]

[0008] In order to solve the above problems, an emotion determination device according to the present invention is an emotion determination device that determines the emotion of a person shown in a face image, and includes a three-dimensional face feature point extraction unit, a face image feature extraction unit, a feature quantity combination unit, and a determination unit.

[0009] In such a configuration, the emotion determination device extracts the positions of the three-dimensional feature points of the face from the input face image using a three-dimensional face feature point extraction model learned in advance by the three-dimensional face feature point extraction unit. As a result, the emotion determination device can extract the positions of the three-dimensional feature points of the face, which are information of the face not shown in the two-dimensional image. Then, the emotion determination device extracts a face image feature vector, which is a feature quantity of the face, from the input face image using a face image feature extraction model learned in advance by the face image feature extraction unit.

[0010] Then, the emotion determination device generates a feature vector by connecting the positions of the three-dimensional feature points of the face and the face image feature vector by the feature quantity combination unit. This feature vector is information in which, in addition to the features of the face image, the positions of the three-dimensional feature points of the face that specify the shape of the face and the shapes of the parts of the face (eyes, nose, mouth, etc.) are added as auxiliary information. Then, the emotion determination device determines a predetermined emotion label from the feature vector using an emotion determination model learned in advance by the determination unit. Note that the emotion determination device can be operated by a program for causing a computer to function as each of the above-described units.

[0011] In order to solve the above problems, a model learning device according to the present invention is a model learning device that learns a face image feature extraction model that inputs a face image and outputs a face image feature vector, which is a feature quantity of the face, and an emotion determination model that inputs a feature vector obtained by connecting the positions of the three-dimensional feature points of the face to the face image feature vector and outputs a predetermined emotion label as a determination result, and includes a face image feature extraction model learning unit, a three-dimensional face feature point extraction unit, a face image feature extraction unit, a feature quantity combination unit, and an emotion determination model learning unit.

[0012] In this configuration, the model learning device uses a face image feature extraction model learning means to input training face images into the face image feature extraction model, input the output face image feature vector as a vector of a predetermined length into the emotion determination model, and learns the face image feature extraction model so that it outputs the correct emotion label corresponding to the training face images. The model learning device then uses a pre-trained 3D facial feature point extraction model, along with a 3D facial feature point extraction means, to extract the positions of 3D facial feature points from training facial images. Then, the model learning device uses a face image feature extraction means to extract face image feature vectors, which are facial features, from the training face images using a face image feature extraction model.

[0013] Furthermore, the model learning device generates a feature vector by concatenating the positions of the 3D feature points of the face with the face image feature vector using a feature coupling means. The model learning device then uses the emotion determination model learning means to learn an emotion determination model by inputting feature vectors and outputting the correct emotion label corresponding to the training face image. In this way, the emotion determination model learning means learns the emotion determination model using the positions of the 3D feature points of the face as auxiliary information, in addition to the features of the face image. Furthermore, the model learning device can be operated using a program that causes the computer to function as each of the aforementioned means. [Effects of the Invention]

[0014] According to the present invention, by utilizing three-dimensional facial feature points that are not visible in a two-dimensional image as auxiliary information, it is possible to accurately determine the emotions of a person depicted in an image. Furthermore, according to the present invention, by utilizing three-dimensional facial feature points that are not visible in two-dimensional images as auxiliary information, it is possible to train a model that accurately determines the emotions of a person depicted in an image. [Brief explanation of the drawing]

[0015] [Figure 1] This is a block diagram showing the configuration of an emotion determination device according to the first embodiment of the present invention. [Figure 2] This is an explanatory diagram illustrating the overall structure of the linked machine learning models used in the emotion determination device according to the first embodiment of the present invention. [Figure 3] This is an explanatory diagram illustrating the input and output data of a 3D facial feature point extraction model. [Figure 4] This is an explanatory diagram illustrating the input and output data of a facial image feature extraction model. [Figure 5] This is an explanatory diagram illustrating the input and output data of an emotion classification model. [Figure 6] This is an explanatory diagram illustrating the content of the merging process in the feature merging mechanism. [Figure 7] This figure shows the flow of input and output data for a machine learning model used in an emotion determination device according to the first embodiment of the present invention. [Figure 8] This is a flowchart illustrating the operation of the emotion determination device according to the first embodiment of the present invention. [Figure 9] This is a block diagram showing the configuration of a model learning device according to a second embodiment of the present invention. [Figure 10] This is a flowchart illustrating the operation of a model learning device according to a second embodiment of the present invention. [Figure 11] This is a block diagram showing the configuration of an emotion determination device according to the third embodiment of the present invention. [Modes for carrying out the invention]

[0016] Embodiments of the present invention will be described below with reference to the drawings. <<Configuration of the Emotion Judgment Device>> First, with reference to Figure 1, the configuration of the emotion determination device 1 according to the first embodiment of the present invention will be described. The emotion determination device 1 determines the emotions of the person depicted in the facial image. As shown in Figure 1, the emotion determination device 1 comprises a storage unit 10 and a control unit 20.

[0017] The memory unit 10 is a common storage medium such as a hard disk or semiconductor memory. The memory unit 10 includes a 3D facial feature point extraction model storage means 11, a facial image feature extraction model storage means 12, and an emotion determination model storage means 13. Each storage means may be configured to store data in separate areas within the same storage medium, or they may be configured to store data in different storage media.

[0018] Each model stored in this memory unit 10 is linked together as a machine learning model as shown in Figure 2, resulting in a model that determines a predetermined emotion label (emotion label L) from a face image F. Specifically, the emotion determination device 1 extracts 3D facial feature points from the facial image F using the 3D facial feature point extraction model M1. The emotion determination device 1 also extracts facial image features from the facial image F using the facial image feature extraction model M2. Then, the emotion determination device 1 uses the emotion determination model M3 to determine the emotion label L from the 3D facial feature points and facial image features. The emotion label L corresponds to each component of the vector output from the output layer of the emotion judgment model M3, and is pre-assigned to emotions such as "happy," "angry," "sad," and "surprised." Returning to Figure 1, we will now specifically describe each model stored in the memory unit 10.

[0019] The 3D facial feature point extraction model storage means 11 stores a 3D facial feature point extraction model M1 (specifically, the structure of the neural network, weight coefficients, etc.) which is composed of a pre-trained neural network that extracts the positions of 3D feature points of a face from a facial image. For example, the 3D facial feature point extraction model M1 can use existing pre-trained models such as "FaceAlignment" (reference 1), which estimates 68 3D keypoints from a facial image, or "Face Mesh" (reference 2), which estimates 468 3D facial landmarks from a facial image.

[0020] (Reference 1) “Adrian et al.: “How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks) “International Conference on Computer Vision, 2017”, “URL: https: / / github.com / 1adrianb / face-alignment” (Reference 2) "MediaPipe Face Mesh", "URL: https: / / google.github.io / mediapipe / solutions / face_mesh"

[0021] The positions of three-dimensional feature points on a face are coordinate positions in three-dimensional space that identify the shape of the face, the shape of facial features (eyes, nose, mouth, etc.), etc. Here, the 3D facial feature point extraction model M1 takes a facial image F as input and, as shown in Figure 3, extracts 3D facial feature points C represented by 3D coordinates (X,Y,Z). 3D This is a model that outputs the following. Note that one coordinate (X,Y,Z) corresponds to one feature point of the face. This 3D facial feature point extraction model M1 allows for the estimation of the position of the 3D facial feature points from a facial image, even when parts of the face are not visible in a 2D image, such as when the face is turned to the side or partially obscured by an obstacle. This is because the model has already learned the positions of the 3D facial feature points as their positions in 3D space.

[0022] The facial image feature extraction model storage means 12 stores a facial image feature extraction model M2 (specifically, the structure of the neural network, weight coefficients, etc.) which is composed of a neural network that has been trained in advance using supervised learning to extract facial features (feature quantities) from a facial image. For example, the facial image feature extraction model M2 can use common image recognition models such as convolutional neural networks (CNNs) and vision transformers. However, the facial image feature extraction model M2 is not limited to neural networks; it can also use general supervised machine learning models.

[0023] Here, the facial image feature extraction model M2 takes a facial image F as input and extracts a one-dimensional facial image feature vector V, as shown in Figure 4. 2D This is a model that outputs [this]. The facial image feature extraction model M2 can be trained using the model learning device 2 (see Figure 9), which will be described later. This training method will be explained in the description of the model learning device 2.

[0024] The emotion determination model memory means 13 stores an emotion determination model M3 (specifically, the structure of the neural network, weight coefficients, etc.) which is composed of a neural network that has been trained in supervised learning beforehand to determine an emotion label from face image features (face image feature vectors) and 3D face feature points (3D face feature point vectors). For example, the sentiment classification model M3 can use a feed-forward neural network (FFNN). However, the sentiment classification model M3 is not limited to neural networks and may also use general supervised machine learning models. For example, it may use a support vector machine (SVM), a conditional random field (CRF), etc.

[0025] Here, the emotion determination model M3 uses the facial image feature vector V, as shown in Figure 5. 2D and 3D facial feature point vector V 3D This model takes a one-dimensional vector (feature vector) formed by concatenating two elements as input and outputs an emotion label L as the result of the judgment. The emotion determination model M3 outputs probability values in the range from 0 to 1 for each predetermined emotion label as the determination result. That is, the emotion label with the maximum probability value becomes the label indicating the emotion of the determination result.

[0026] Note that the face image feature vector V 2D is the vector output by the face image feature extraction model M2 described in FIG. 4. The 3D face feature point vector V 3D is the 3D face feature point C output by the 3D face feature point extraction model M1 described in FIG. 3 3D normalized and made into a 1D vector. For example, the 3D face feature point vector V 3D is a 1D vector obtained by normalizing each coordinate value of the 3D face feature point C 3D so that the maximum value is "1" and the minimum value is "0", and all coordinate values are in the range of 0 to 1 (Min - Max method).

[0027] That is, as shown in FIG. 6, the feature vector V F input to the emotion determination model M3 is the 3D face feature point vector V 3D obtained by normalizing and making into a 1D vector from the 3D face feature point C 3D concatenated with the face image feature vector V 2D Note that the learning of the emotion determination model M3 can be performed by the model learning device 2 (see FIG. 9) described later. The description of this learning method will be given in the description of the model learning device 2.

[0028] FIG. 7 shows a more specific diagram of the input / output between the models for determining the emotion label L from the face image F shown in FIG. 2. As shown in FIG. 7, the feature vector V F input to the emotion determination model M3 includes the face image feature vector V 2D extracted from the 2D face image, to which the 3D face feature point vector V 3D is added as auxiliary information, where the positions of the 3D feature points of the face are added. ​As a result, the resulting emotion label L takes into account not only the facial features in the 2D image but also the positions of the 3D facial feature points. Returning to Figure 1, we will continue our explanation of the configuration of the emotion determination device 1.

[0029] The control unit 20 controls the entire emotion determination device 1. The control unit 20 loads the program (emotion determination program) stored in the hard disk, ROM, etc., into memory, and the computer (CPU) reads the program and makes it function as one of the means described below. The control unit 20 includes a 3D facial feature point extraction means 21, a facial image feature extraction means 22, a feature quantity combining means 23, and a determination means 24.

[0030] The 3D facial feature point extraction means 21 extracts the positions of 3D facial feature points from an input facial image using a pre-trained 3D facial feature point extraction model stored in the 3D facial feature point extraction model storage means 11. As shown in Figure 3, the 3D facial feature point extraction means 21 takes a facial image F as input and performs calculations on the neural network of the 3D facial feature point extraction model M1 to extract the 3D coordinate positions of multiple predetermined locations (landmarks, keypoints) on the face as 3D facial feature points C. 3D Extract it as follows. The 3D facial feature point extraction means 21 extracts the 3D facial feature points C 3D The output is sent to the feature concatenation means 23.

[0031] The face image feature extraction means 22 uses a pre-trained face image feature extraction model stored in the face image feature extraction model storage means 12 to extract face image feature vectors, which are facial features (feature quantities), from the input face image. As shown in Figure 4, the facial image feature extraction means 22 takes a facial image F as input and performs calculations on the neural network of the facial image feature extraction model M2 to extract facial features into a facial image feature vector V. 2D Extract it as follows. The facial image feature extraction means 22 extracts the facial image feature vector V 2DThe output is sent to the feature concatenation means 23.

[0032] The feature merging means 23 generates a feature vector by concatenating the positions of the 3D feature points of the face extracted by the 3D face feature point extraction means 21 with the face image feature vector extracted by the face image feature extraction means 22. The feature merging means 23 normalizes and converts the positions of the 3D feature points of the face to one dimension before merging them with the face image feature vector. The feature merging means 23, for example, as shown in Figure 6, includes 3D facial feature points C 3D Each coordinate value is normalized to a range of 0 to 1, such that the maximum value is "1" and the minimum value is "0". Then, the feature concatenation means 23 arranges the normalized X-coordinate numerical sequence, the normalized Y-coordinate numerical sequence, and the normalized Z-coordinate numerical sequence to form a one-dimensional numerical sequence, which is a three-dimensional face feature point vector V. 3D The feature concatenation means 23 generates the normalized and one-dimensionalized 3D facial feature point vector V. 3D Face image feature vector V 2D Concatenated to, feature vector V F Generates. The feature concatenation means 23 combines the feature vector V F The result is output to the determination means 24.

[0033] The determination means 24 uses a pre-trained emotion determination model stored in the emotion determination model storage means 13 to determine the emotion label from the feature vector (face image feature vector + 3D face feature point vector) generated by the feature concatenation means 23. As shown in Figure 5, the determination means 24 uses the face image feature vector V 2D and 3D facial feature point vector V 3D The feature vector V formed by concatenating the two. F By inputting this data and performing calculations on the neural network of the emotion classification model M3, the probability value for each emotion label L is calculated. The determination means 24 then outputs the emotion label L with the highest probability value as the determination result.

[0034] With the configuration described above, the emotion determination device 1 utilizes not only the facial features captured in the facial image but also the positions of the three-dimensional feature points of the face as auxiliary information, thereby enabling it to determine a person's emotions with greater accuracy than conventional methods.

[0035] ≪Operation of the Emotion Assessment Device≫ Next, with reference to Figure 8 (and Figure 1 as appropriate for the configuration), the operation of the emotion determination device 1 according to the first embodiment of the present invention will be described. In step S1, the emotion determination device 1 receives a facial image from an external source via an input means (not shown in the illustration). In step S2, the emotion determination device 1 uses the 3D facial feature point extraction means 21 to extract the positions of 3D facial feature points (3D facial feature points) from the face image input in step S1, using the 3D facial feature point extraction model stored in the 3D facial feature point extraction model storage means 11. In step S3, the emotion determination device 1 uses the face image feature extraction means 22 to extract facial features as face image feature vectors from the face image input in step S1, using the face image feature extraction model stored in the face image feature extraction model storage means 12. Steps S2 and S3 may be operated in parallel, as shown in Figure 8, or they may be operated in the order of S2, S3, or S3, S2.

[0036] In step S4, the emotion determination device 1 generates a 3D facial feature point vector by normalizing and converting the coordinate values ​​of the 3D facial feature points extracted in step S2 into a 1D shape using the feature summarization means 23. In step S5, the emotion determination device 1 uses the feature concatenation means 23 to concatenate the face image feature vector extracted in step 3 with the 3D face feature point vector generated in step S4, thereby generating a feature vector.

[0037] In step S6, the emotion determination device 1 uses the emotion determination model stored in the emotion determination model storage means 13 to determine the emotion label from the feature vector generated in step S5, using the determination means 24. In step S7, the emotion determination device 1 outputs the emotion label determined in step S6 to the outside as a determination result using the determination means 24.

[0038] In step S8, the emotion determination device 1 determines whether or not a face image to be determined is input via an input means (not shown in the diagram) to determine the end of the determination process. If a face image is still input at this point (No in step S8), the emotion determination device 1 returns to step S1 and continues its operation. On the other hand, if no facial image is input (Yes in step S8), the emotion determination device 1 terminates its operation. Through the above operations, the emotion determination device 1 can determine the emotion of the person depicted in the facial image.

[0039] <<Configuration of the model learning device>> Next, with reference to Figure 9, the configuration of the model learning device 2 according to the second embodiment of the present invention will be described.

[0040] Model learning device 2 learns a face image feature extraction model that takes a face image as input and outputs a face image feature vector, which is a set of facial features, from the face image, and an emotion determination model that takes a feature vector, which is created by concatenating the positions of three-dimensional feature points of the face, as input and outputs a predetermined emotion label as the determination result. In other words, the model learning device 2 learns the face image feature extraction model and the emotion determination model used in the emotion determination device 1. The model learning device 2 takes face images with known emotion labels (training face images) and corresponding emotion labels (ground truth emotion labels) as training data and learns a face image feature extraction model and an emotion determination model. As shown in Figure 9, the model learning device 2 comprises a storage unit 10 and a control unit 20B.

[0041] The memory unit 10 includes a 3D facial feature point extraction model storage means 11, a facial image feature extraction model storage means 12, and an emotion determination model storage means 13, which are the same configuration as the emotion determination device 1 described in Figure 1.

[0042] In other words, the 3D facial feature point extraction model storage means 11 stores a 3D facial feature point extraction model, which is composed of a pre-trained neural network that extracts the positions of 3D feature points of a face from a facial image. Furthermore, the facial image feature extraction model storage means 12 stores pre-trained models (such as convolutional neural networks [CNN] and Vision Transformer) that are used for general image recognition as facial image feature extraction models. Furthermore, the emotion determination model memory means 13 stores a model that estimates predetermined labels from a one-dimensional vector, such as a feedforward neural network (FFNN) model, as an emotion determination model.

[0043] The facial image feature extraction model stored in the facial image feature extraction model storage means 12 and the emotion determination model stored in the emotion determination model storage means 13 are the models that are trained.

[0044] The control unit 20B controls the entire model learning device 2. The control unit 20B loads the program (emotion judgment program) stored in the hard disk, ROM, etc., into memory, and the computer (CPU) reads the program and makes it function as one of the means described below.

[0045] The control unit 20B includes a 3D facial feature point extraction means 21, a facial image feature extraction means 22, a feature quantity combining means 23, a facial image feature extraction model learning means 25, and an emotion determination model learning means 26. The 3D facial feature point extraction means 21, the facial image feature extraction means 22, and the feature quantity merging means 23 are the same configuration as the emotion determination device 1 described in Figure 1. However, the facial images input to the 3D facial feature point extraction means 21 and the facial image feature extraction means 22 are training facial images.

[0046] The facial image feature extraction model learning means 25 inputs a training facial image into the facial image feature extraction model, inputs the output facial image feature vector as a vector of a predetermined length into the emotion judgment model, and trains the facial image feature extraction model to output a correct emotion label corresponding to the training facial image.

[0047] In other words, as shown in Figure 7, the facial image feature extraction model learning means 25 inputs a training facial image (facial image F) to the facial image feature extraction model M2 and outputs a facial image feature vector V. 2D The parameters of the face image feature extraction model M2 are trained so that the face image feature vector V is input to the emotion determination model M3 and outputs the correct emotion label (emotion label L). 2D When the facial image feature vector V is input to the emotion determination model M3, the facial image feature extraction model learning means 25 uses the facial image feature vector V 2D V, a 3D facial feature point vector 3D Empty data is added to create a vector of a predetermined length that matches the input data length of the emotion judgment model M3.

[0048] For this learning process, methods such as backpropagation can be used. Although the facial image feature extraction model learning method 25 uses an emotion determination model M3 in addition to the facial image feature extraction model M2, here it is sufficient to focus the learning on only the facial image feature extraction model M2. As a result, the facial image feature extraction model M2 generates features (facial image feature vector V) corresponding to the number of classifications of emotion labels L. 2D The system will be trained to output ). The facial image feature extraction model learning means 25 stores the learned parameters in the facial image feature extraction model storage means 12.

[0049] The facial image feature extraction model learning means 25 sequentially inputs training facial images and correct emotion labels for a predetermined number of times, or until the parameter error converges to a predetermined range, and performs training. When the facial image feature extraction model learning means 25 inputs a new training facial image, the 3D facial feature point extraction means 21 inputs the same training facial image in synchronization with it. The facial image feature extraction model learning means 25 outputs the training facial images to the facial image feature extraction means 22 once it has completed training using a pair of training facial images and correct emotion labels.

[0050] The emotion determination model learning means 26 takes the feature vectors combined by the feature coupling means 23 as input and learns an emotion determination model that outputs emotion labels. Here, the emotion determination model learning means 26 inputs the feature vectors combined by the feature coupling means 23 into the emotion determination model and trains the emotion determination model to output the correct emotion label (correct emotion label).

[0051] In other words, as shown in Figure 7, the emotion determination model learning means 26 uses the face image feature vector V, which is the output of the face image feature extraction model M2. 2D And the 3D facial feature points C extracted by the 3D facial feature point extraction model M1 3D Normalized and 1Dized 3D facial feature point vector V 3D The feature vector V formed by concatenating the two. F The parameters of the emotion classification model M3 are trained so that it takes input and outputs the correct emotion label (emotion label L). For training, for example, backpropagation can be used.

[0052] With the configuration described above, the model learning device 2 can learn a neural network model (face image feature extraction model and emotion determination model) for determining emotions by using the feature vectors combined by the feature coupling means 23, and by using not only the facial features captured in the face image but also the positions of the 3D feature points of the face as auxiliary information.

[0053] ≪Operation of the Model Learning Device≫ Next, with reference to Figure 10 (and Figure 9 as appropriate for the configuration), the operation of the model learning device 2 according to the second embodiment of the present invention will be described. In step S10, the model learning device 2 receives learning face images and correct emotion labels from an external source via input means (not shown in the illustration). In step S11, the model learning device 2 uses the 3D facial feature point extraction means 21 to extract the positions of 3D facial feature points (3D facial feature points) from the training face image input in step S10, using the 3D facial feature point extraction model stored in the 3D facial feature point extraction model storage means 11.

[0054] In step S12, the model learning device 2 learns the face image feature extraction model stored in the face image feature extraction model storage means 12 using the face image feature extraction model learning means 25. Here, the face image feature extraction model learning means 25 learns the parameters of the face image feature extraction model so that it inputs the training face image input in step S1 into the face image feature extraction model, outputs the resulting face image feature vector to the emotion determination model, and outputs the correct emotion label (emotion label L). Then, the facial image feature extraction model learning means 25 stores the learned parameters in the facial image feature extraction model storage means 12 (step not shown).

[0055] In step S13, the model learning device 2 uses the face image feature extraction means 22 to extract facial features as face image feature vectors from the training face image input in step S1, using the face image feature extraction model learned in step S12. Steps S11 and S12 and S13 may be operated in parallel, as shown in Figure 10, or they may be operated in the order of S11, S12, S13, or S12, S13, S11.

[0056] In step S14, the model learning device 2 generates a 3D facial feature point vector by normalizing and converting the coordinate values ​​of the 3D facial feature points extracted in step S11 into one dimension, using the feature coupling means 23. In step S15, the model learning device 2 uses the feature coupling means 23 to concatenate the face image feature vector extracted in step 13 with the 3D face feature point vector generated in step S14, thereby generating a feature vector.

[0057] In step S16, the model learning device 2 uses the emotion determination model learning means 26 to learn the emotion determination model stored in the emotion determination model storage means 13. Here, the emotion determination model learning means 26 inputs the feature vectors combined in step S15 into the emotion determination model and learns the emotion determination model to output the correct emotion label (correct emotion label) input in step S1. Then, the face image feature extraction model learning means 25 stores the learned parameters in the face image feature extraction model storage means 12. Then, the emotion judgment model learning means 26 stores the learned parameters in the emotion judgment model storage means 13 (step not shown).

[0058] In step S17, the model learning device 2 determines whether the learning process has been completed by the face image feature extraction model learning means 25, based on the termination conditions of a predetermined number of times or when the parameter error converges to a predetermined range. If learning is not yet complete (No in step S17), the model learning device 2 returns to step S10 and continues its operation. On the other hand, if learning is complete (Yes in step S17), the model learning device 2 terminates its operation.

[0059] Through the above operations, the model learning device 2 can learn a neural network model (a facial image feature extraction model and an emotion determination model) that can determine the emotions of a person depicted in a facial image.

[0060] The configuration and operation of the emotion determination device 1 and the model learning device 2 according to embodiments of the present invention have been described above, but the present invention is not limited to these embodiments.

[0061] Here, an emotion determination device 1, which determines emotions from facial images, and a model learning device 2, which learns a model for determining emotions from facial images, were configured as separate devices. However, the present invention may also be configured in which the emotion determination device 1 and the model learning device 2 operate in a single device by changing the operating mode (third embodiment). The specific configuration is shown in Figure 11.

[0062] The emotion determination device 1B shown in Figure 11 operates in two modes: a learning mode and a determination mode. These modes can be switched using an external switch, an operation screen, or other means. When operating in learning mode, the emotion determination device 1B uses training face images (training face images) and corresponding correct emotion labels (correct emotion labels) as training data to train a model for determining emotions (face image feature extraction model and emotion determination model). Furthermore, when the emotion determination device 1B operates in determination mode, it determines an emotion label (determined emotion label) from a face image for determination (face image for determination). As shown in Figure 11, the emotion determination device 1B comprises a storage unit 10 and a control unit 20C.

[0063] The memory unit 10 includes a 3D facial feature point extraction model storage means 11, a facial image feature extraction model storage means 12, and an emotion determination model storage means 13, which are the same configurations as the emotion determination device 1 described in Figure 1 and the model learning device 2 described in Figure 9. The contents stored in the 3D facial feature point extraction model storage means 11, the facial image feature extraction model storage means 12, and the emotion determination model storage means 13 are the same as those stored in the emotion determination device 1 and the model learning device 2, so their explanation is omitted.

[0064] The control unit 20C controls the entire emotion determination device 1B. The control unit 20C loads the program (emotion determination program) stored in the hard disk, ROM, etc., into memory, and the computer (CPU) reads the program and makes it function as the parts described below. The control unit 20C includes a 3D facial feature point extraction means 21, a facial image feature extraction means 22, a feature quantity combining means 23, a determination means 24, a facial image feature extraction model learning means 25, and an emotion determination model learning means 26. Since each of these means has the same function as the emotion determination device 1 and the model learning device 2, they are given the same reference numerals and their descriptions are omitted.

[0065] Each of these means operates differently depending on the operating mode, and the inputs and outputs vary accordingly. Specifically, in learning mode, the 3D facial feature point extraction means 21, the facial image feature extraction means 22, the feature quantity merging means 23, the facial image feature extraction model learning means 25, and the emotion determination model learning means 26 operate.

[0066] In the judgment mode, the 3D facial feature point extraction means 21, the facial image feature extraction means 22, the feature quantity merging means 23, and the judgment means 24 are all in operation. In the judgment mode, the facial image feature extraction model learning means 25 outputs the input facial image for judgment directly to the facial image feature extraction means 22. The feature concatenation means 23 outputs the concatenated feature vector to the emotion judgment model learning means 26 in learning mode, and to the judgment means 24 in judgment mode.

[0067] With the configuration described above, the emotion determination device 1B can learn a neural network model that utilizes not only the facial features captured in a facial image, but also the positions of the three-dimensional feature points of the face as auxiliary information. Furthermore, using this model, it can determine a person's emotions with greater accuracy than conventional methods.

[0068] Furthermore, since the emotion determination device 1B can be switched to learning mode at any time to perform additional model training, the accuracy of the model can be improved, and the accuracy of emotion determination can be increased. Note that the operation of emotion determination device 1B is the same as that of model learning device 2 described in Figure 10 in learning mode, and the same as that of emotion determination device 1 described in Figure 8 in determination mode, so the explanation is omitted.

[0069] As explained above, the emotion determination device 1,1B can accurately determine emotions even when part of the face is obscured. For example, the emotion determination device 1,1B can determine the emotions of a character from their facial image in broadcast content. This makes it possible to convey the emotions of the characters in broadcast content through explanatory audio, tactile stimuli, etc., providing a greater sense of realism to everyone, including those with visual impairments. [Explanation of Symbols]

[0070] 1,1B Emotion determination device 2 Model Learning Devices 10 Storage section 11. 3D facial feature point extraction model storage means 12. Memory for facial image feature extraction model 13. Emotion Judgment Model Memory Means 20, 20B, 20C Control Unit 21. Three-dimensional facial feature point extraction means 22. Face image feature extraction means 23 Feature Combination Means 24 Judgment means 25. Method for training a facial image feature extraction model. 26. Methods for Learning Emotion Classification Models

Claims

1. An emotion determination device that determines the emotions of a person captured in a facial image, A 3D facial feature point extraction means that extracts the positions of 3D facial feature points from an input facial image using a pre-trained 3D facial feature point extraction model, A face image feature extraction means that extracts face image feature vectors, which are facial features, from the input face image using a pre-trained face image feature extraction model, A feature vector concatenation means for generating a feature vector by concatenating the positions of the three-dimensional feature points of the face and the face image feature vector, A determination means that determines a predetermined emotion label from the feature vector using a pre-trained emotion determination model, An emotion determination device characterized by being equipped with the following features.

2. A face image feature extraction model learning means learns the face image feature extraction model by inputting a face image for training into the face image feature extraction model, inputting the output face image feature vector as a vector of a predetermined length into the emotion determination model, and outputting the correct emotion label corresponding to the face image for training. A sentiment determination model learning means for learning the sentiment determination model so as to take the feature vector as input and output the correct sentiment label corresponding to the facial image used for training, The emotion determination device according to claim 1, further comprising the features described above.

3. A program for causing a computer to function as an emotion determination device according to claim 1 or claim 2.

4. A model learning device that learns a face image feature extraction model that takes a face image as input and outputs a face image feature vector, which is a set of facial features from the face image, and an emotion determination model that takes a feature vector, which is obtained by concatenating the positions of three-dimensional feature points of the face to the face image feature vector, as input and outputs a predetermined emotion label as the determination result, A face image feature extraction model learning means learns the face image feature extraction model by inputting a face image for training into the face image feature extraction model, inputting the output face image feature vector as a vector of a predetermined length into the emotion judgment model, and outputting the correct emotion label corresponding to the face image for training. A 3D facial feature point extraction means extracts the positions of 3D facial feature points from the training facial image using a pre-trained 3D facial feature point extraction model, A face image feature extraction means that extracts face image feature vectors, which are facial features, from the training face images using the face image feature extraction model, A feature vector concatenation means for generating a feature vector by concatenating the positions of the three-dimensional feature points of the face and the face image feature vector, A sentiment determination model learning means for learning the sentiment determination model so as to take the feature vector as input and output the correct sentiment label corresponding to the facial image used for training, A model learning device characterized by comprising the following features.

5. A program for causing a computer to function as a model learning device as described in claim 4.

Images