Emotion estimation device, emotion estimation program, training data generation method, and training data generation program

Abstract

To provide an emotion estimation device, an emotion estimation program, a training data generation method, and a training data generation program that can improve the accuracy of emotion estimation. [Solution] The emotion estimation device according to the embodiment is an emotion estimation device that estimates the emotion information of a vehicle driver based on a facial image of the driver, and has a controller. The controller acquires information on external factors that affect the facial image, and estimates the emotion information based on the facial image and the information on external factors.

Description

Technical Field

[0001] The present invention relates to an emotion estimation device, an emotion estimation program, a learning data generation method, and a learning data generation program.

Background Art

[0002] Conventionally, there has been a technique for estimating the emotion of a driver from the facial expression or the like of the driver in a vehicle. In such a technique, a technique for estimating the emotion of a driver from a facial image of the driver using an AI (Artificial Intelligence) model has been proposed. For example, Patent Document 1 discloses a technique for specifying the orientation of a face from a facial image and estimating an emotion based on the orientation of the face.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, there was room for further improvement in terms of improving the accuracy of emotion estimation. Specifically, in addition to the case where the driver changes the orientation of the face due to a change in emotion, the driver may change the orientation of the face due to external factors such as checking for blind spots when turning right or left. Therefore, if emotion estimation is performed on the premise that a change in the orientation of the face is caused by a change in emotion, the accuracy of emotion estimation may decrease. In addition, although changes in facial expressions and the like also occur due to changes in emotion, changes in facial expressions and the like can also occur due to external factors other than emotion, such as the occurrence of a dazzling situation due to external light (such as sunlight), and as a result, the accuracy of emotion estimation may decrease.

[0005] The present invention has been made in view of the above, and aims to provide an emotion estimation device, an emotion estimation program, a learning data generation method, and a learning data generation program that can improve the accuracy of emotion estimation. [Means for solving the problem]

[0006] To solve the above-mentioned problems and achieve the objective, the emotion estimation device according to the present invention is an emotion estimation device that estimates the emotion information of a vehicle driver based on a facial image of the driver, and includes a controller. The controller acquires information on external factors that affect the facial image, and estimates the emotion information based on the facial image and the external factor information. [Effects of the Invention]

[0007] According to the present invention, by estimating emotions based on a facial image and external factors that influence facial changes, it is possible to estimate emotions while taking into account the possibility that facial changes may be caused by external factors, thereby improving the accuracy of emotion estimation. [Brief explanation of the drawing]

[0008] [Figure 1A] Figure 1A is an explanatory diagram of the emotion estimation process according to the embodiment. [Figure 1B] Figure 1B is an explanatory diagram of the emotion estimation process according to the embodiment. [Figure 2] Figure 2 shows an example of the configuration of an emotion estimation device according to an embodiment. [Figure 3] Figure 3 shows an example of training data generated during the learning process. [Figure 4] Figure 4 shows an example of inference input data generated during the inference process. [Figure 5] Figure 5 is a flowchart showing the processing steps of the learning process performed by the emotion estimation device according to the embodiment. [Figure 6] Figure 6 is a flowchart showing the processing steps of the inference process performed by the emotion estimation device according to the embodiment. [Modes for carrying out the invention]

[0009] The emotion estimation device, emotion estimation program, learning data generation method, and learning data generation program according to the following embodiments will be described in detail with reference to the attached drawings. However, the present invention is not limited to the embodiments described below.

[0010] First, the emotion estimation process according to the embodiment will be explained using Figures 1A and 1B. Figures 1A and 1B are explanatory diagrams of the emotion estimation process according to the embodiment. The emotion estimation process according to the embodiment is performed by the emotion estimation device 1.

[0011] As shown in Figure 1A, the emotion estimation process involves inputting inference input data into an emotion estimation model and obtaining emotion information output from the emotion estimation model. The inference input data includes, for example, a facial image of the driver captured by an in-vehicle camera. The emotion information obtained through the emotion estimation process is output to various in-vehicle devices and vehicle control devices and used for driver assistance, such as providing driving advice and vehicle control.

[0012] Next, using Figure 1B, we will explain the training process of the emotion estimation model and the inference process using the trained emotion estimation model. The training and inference processes are performed by the emotion estimation device 1 according to the training mode and estimation mode set, for example, by user operation. Note that the training process may be handled by the training device and the inference process by the emotion estimation device, or each process may be handled by separate devices. In the following, we will give an example in which the emotion estimation device 1 performs both the training and inference processes.

[0013] (Learning process) First, the learning process will be explained using Figure 1B. The learning process is executed when the learning mode is set by the user's operation. Specifically, as shown in Figure 1B, the learning process generates emotion information D5 by performing emotion estimation processing C2 based on the driver's biometric information (such as brain waves and heart rate) D1 detected by the biosensor C1. Emotion estimation processing C2 estimates two emotion indices that indicate the driver's mental and physical state based on biometric information such as brain waves and heart rate. Specifically, the emotion indices are the level of arousal of the central nervous system and the activity level of the autonomic nervous system. For example, the level of arousal is calculated from the β / α waves of the brain wave, and the activity level is calculated from the standard deviation of the heart rate LF (Low Frequency) component (low-frequency component of the heart rate waveform signal). Then, emotion (type) information is estimated from the level of arousal and the activity level. Specifically, a matrix table is constructed with arousal level and activity level as the two axes, and matrix table data is created that stores the emotion information corresponding to each cell defined by the level of arousal level and activity level in the matrix table. Then, the matrix table data is searched using the arousal and activity levels calculated based on the biological information D1, and the search results are determined as emotional information D5.

[0014] An external factor determination process C6 is performed based on the in-vehicle camera image (driver's face image) D2 captured by the in-vehicle camera C3, the exterior camera image (vehicle surroundings image) D3 captured by the exterior camera C4, and the vehicle state information (other factor information) D4 detected by the on-board sensor C5. The external factor determination process C6 determines whether the driver's face image has changed due to factors other than the driver's emotions. For example, it determines whether an event has occurred that would cause the driver's facial expression to change based on the in-vehicle camera image D2 (for example, direct sunlight hitting the driver's face), whether an event has occurred around the vehicle that would cause the driver's facial expression to change based on the exterior camera image D3, and whether a vehicle driving condition has occurred that would cause the driver's facial expression to change based on the vehicle state information (other factor information) D4. Based on these conditions, it determines whether an external factor has occurred that would cause the driver's face image to change (external factor determination information D6 is output). Other factor information specifically includes information about vehicle driving conditions, such as driving operation information (turn signal operation, accelerator operation, brake operation, steering operation, etc.) and route guidance information from the navigation system. In addition, other factor information includes information about vehicle equipment, such as the position of warning indicators on the instrument panel, vehicle height (viewpoint height), and mirror position.

[0015] Then, based on the emotion information D5, the in-car camera image D2, and the external factor judgment information D6, the training data generation process C7 is executed, and training data D7 is generated. Specifically, training data D7 is generated using the in-car camera image D2 and the external factor judgment information D6 as input data and the emotion information D5 as the ground truth data. Then, the (pre-training) training model MB is trained using the large amount of training data D7 generated in this way. Through this training, the training model MB becomes the (trained) training model MA, which is then used in emotion estimation scenarios.

[0016] For example, in the external factor determination process C6, when it is detected that the vehicle is turning left based on an external view image of the vehicle, a wink operation, route guidance information, etc., it is determined that an external factor that affects the change in the driver's face has occurred. In this case, in the external factor determination process C6, external factor determination information D6 indicating a face change factor code "1" indicating that an external factor that affects the face change has occurred is generated. Note that in the external factor determination process C6, when it is determined that no external factor has occurred, external factor determination information D6 indicating a face change factor code "0" is generated. Then, in the learning data generation process C7, learning data is generated with the emotion information as the correct answer data and the face image and the face change factor code as the input data, and a learning data set composed of a plurality of pieces of learning data is generated. Then, based on the learning data set, the emotion estimation device 1 learns an emotion estimation model with the emotion information of the input data as the target variable and the face image of the correct answer data and the external factor determination information D6 (face change factor code) as the explanatory variables.

[0017] For example, when the driver's face is turned to the left, as the cause of the face orientation change, an emotion change and an external factor such as the vehicle turning left can be considered. In the former case, the learning data obtained in that situation is the face image D2 indicating that the driver's face is turned to the left, the external factor determination information D6 (face change factor code "0") indicating that no external factor has occurred, and the emotion information D5 (emotion type). Also, in the latter case, the learning data obtained in that situation is the face image D2 indicating that the driver's face is turned to the left, the external factor determination information D6 (face change factor code "0") indicating that no external factor has occurred, and the emotion information D5 (emotion type). That is, the learning data includes data indicating the presence or absence of an external factor, and the learning model for estimating emotions becomes a learning model that estimates (as an input) based on the driver's face image D2 and the external factor determination information D6, and it becomes possible to perform emotion estimation considering the external factor determination information D6. Note that the algorithm of the emotion estimation model is, for example, a neural network, but is not limited to this, and any machine learning algorithm can be used.

[0018] (Inference process) Next, the inference process will be described using FIG. 1B. The inference process is executed when the inference mode is set by a user's operation. Specifically, in the inference process, emotion information is estimated using the (trained) learning model MA (emotion estimation model) generated in the learning process. Specifically, in the external factor determination process C6, based on the in-vehicle camera image (driver's face image) D2 captured by the in-vehicle camera C3, the out-of-vehicle camera image D3 captured by the out-of-vehicle camera C4, and other factor information D4 detected by the vehicle-mounted sensor C5 and the like, external factor determination information D6 is generated. Then, in the input data generation process C8, input data D8 for input to the learning model MA (corresponding to the input data (explanatory variables) of the training data D7), which is composed of the in-vehicle camera image (driver's face image) D2 and the external factor determination information D6, is generated and output to the learning model MA. Thereby, the learning model MA estimates the emotion information D9 and outputs it to a device that uses the emotion information for control, such as a driving control device.

[0019] Note that the biometric information D1, the in-vehicle camera image D2, the out-of-vehicle camera image D3, and the other factor information D4 used in the learning process and the estimation process are time-series data (time-axis waveform data of biometric signals, videos, etc.) of a suitable time length within a period length suitable for preset emotion estimation. Using these time-series data, emotion information for the corresponding period will be estimated. Note that for the other factor information D4, in some cases, a representative value (instantaneous value at a certain timing) in the said period may be appropriate depending on the data type.

[0020] That is, in the present disclosure, in the learning process, the model is learned including external factor information indicating whether a change in the face is due to an external factor, and in the inference process, the external factor information indicating whether a change in the face is due to an external factor is included in the input data to the model. Thereby, since the emotion estimation device 1 can estimate emotions considering the influence of external factors on the facial expression of the estimation target person (such as a driver), the accuracy of emotion estimation can be improved.

[0021] The external factor information (external factor determination information D6) mentioned above is data indicating the presence or absence of external factors, but it can also be data such as the level of occurrence of external factors (degree of impact on emotions) or the type of external factor. In this case, the input to the learning model during inference will be the face image and the level of occurrence of external factors or the type of external factor.

[0022] Furthermore, if external factors are present, a method can be applied to train the model using only training data without external factors, excluding the relevant training data. In this case, the input to the training model during inference will be a face image, and emotion estimation will be performed only when there are no external factors (the decision to perform or not perform emotion estimation is made based on the result of the external factor determination process C6).

[0023] Next, an example of the configuration of the emotion estimation device 1 according to the embodiment will be described using Figure 2. Figure 2 is a diagram showing an example of the configuration of the emotion estimation device 1 according to the embodiment. As shown in Figure 2, the emotion estimation device 1 has a controller 2 and a storage unit 3. The emotion estimation device 1 is also connected to an in-vehicle camera 100, an out-of-vehicle camera 110, various sensors 120, and an in-vehicle device 130.

[0024] The in-vehicle camera 100 is a camera that captures images of the driver inside the vehicle. Specifically, the in-vehicle camera 100 captures images of the driver's face. The exterior camera 110 is a camera that captures images of the area around the vehicle. Specifically, the exterior camera 110 captures images of the front, rear, and sides of the vehicle. The various sensors 120 are a group of sensors that detect the vehicle's driving state, such as a vehicle speed sensor, acceleration sensor, steering sensor, etc. The in-vehicle devices 130 are various electronic devices mounted on the vehicle, such as a vehicle control device, navigation device, turn signal, etc.

[0025] Controller 2 is implemented, for example, by a processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs stored in the memory device inside the emotion estimation device 1 using RAM or the like as a working area. Alternatively, Controller 2 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or GPGPU (General Purpose Graphic Processing Unit).

[0026] The memory unit 3 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) or flash memory, or by a storage device such as a hard disk or optical disc. As shown in Figure 2, the memory unit 3 stores model information 31, a mode program 32, and learning data information 33.

[0027] Model information 31 is information about the emotion estimation model described above. Model information 31 includes information about the parameters (weight values, etc.) of the emotion estimation model. Controller 2 uses model information 31 to construct the emotion estimation model and implements the emotion estimation process.

[0028] Mode program 32 contains information about the learning mode and inference mode programs. Controller 2 reads and executes the mode program of the learning mode or inference mode program set by the user's operation, and implements the operation in the learning mode and inference mode. In other words, controller 2 executes a program that implements the learning process or the inference process in the sentiment estimation process, depending on the operation mode (learning mode and estimation mode).

[0029] The training data information 33 is information about the training data generated by the training process. For example, the training data includes, as described above, face images (or face features described later), emotion information (or emotion indicators), and external factor information. The controller 2 uses the training data information 33 to train the emotion estimation model and stores model generation information, which includes information about the parameters (weight values, etc.) of the trained emotion estimation model, in the storage unit 3 as model information 31.

[0030] (Learning process) Controller 2, during the learning process, collects each data item that constitutes the training data for the emotion estimation model and generates a training data set of a sufficient amount for learning. Specifically, Controller 2 collects time-series data of biometric information, face images, and external factor information over a predetermined time period appropriate for emotion estimation (set by the designer / developer based on experiments, etc.). In other words, Controller 2 collects time-series biometric information, time-series face images (i.e., videos), and time-series external factor information over a predetermined time period. Then, Controller 2 estimates the emotion based on the biometric information and generates one training data item using the face image corresponding to the estimated emotion and the external factor information. It is preferable that the external factor information be a representative value (such as an instantaneous value of a feature point) over a predetermined long period of time, depending on its information characteristics. Furthermore, Controller 2 generates a training data set from the multiple training data items it has generated.

[0031] Although the example given for Controller 2 uses facial images (videos) of a predetermined duration as training data, image data obtained by converting facial images into numerical data (hereinafter referred to as facial features) may also be used as training data. Below, an example of using facial features as training data will be given.

[0032] Facial features are extracted from the driver's face image captured by the in-car camera 100. Facial features include, for example, information about the driver's facial expression, face orientation, and gaze. For example, controller 2 obtains facial features by converting facial expression, face orientation, gaze (direction), etc., into numerical data. For example, the face image is converted into positional coordinate values ​​of each facial part such as the eyes, nose, and mouth (for example, coordinate values ​​in the face plane coordinate system) through image analysis processing of the face image, becoming numerical data of the facial expression. Similarly, face orientation and gaze (direction) are converted into numerical data such as vector data perpendicular to the face plane (for example, coordinate values ​​in an appropriately set in-car spatial coordinate system) and vector data in the direction of gaze (for example, coordinate values ​​in an appropriately set in-car spatial coordinate system) through image analysis processing of the face image. These numerical processing operations are performed on each still image in the video (time-series of still images), and the numerical data calculated for each still image are arranged in chronological order to form time-series image numerical data.

[0033] External factor information is information about external factors that affect changes in the driver's face, and is generated based on external video from the external camera 110, sensor information from various sensors 120, and information output from the in-vehicle device 130. Furthermore, the controller 2 generates a face change factor code "1" if an external factor affecting changes in the face occurs based on this information, and a face change factor code "0" if no external factor affecting changes in the face occurs.

[0034] For example, if controller 2 detects that the driver has turned left based on the facial orientation, which is a facial feature, and the driver is instructed to turn left by operating the turn signal, controller 2 generates a facial change factor code "1", which is external factor information indicating that the change in facial orientation was caused by a left turn during driving (performed out of necessity during driving) and not by emotion (mental state). In other words, controller 2 generates external factor information indicating that the change in facial orientation was caused by driving and not by emotion, that is, that an external factor influencing the change in facial orientation occurred. To put it another way, if controller 2 acquires the vehicle's turning state and the turning state (facing left) of the driver's facial orientation based on the facial image is equivalent to the vehicle's turning state (turning left) (facing the same left side), controller 2 generates external factor information indicating that an external factor (a factor other than emotion) occurred as a cause of the state related to the driver's facial orientation. Note that controller 2 may also detect a left turn based on steering operations, route guidance information, etc., in addition to turn signal operation.

[0035] Furthermore, when Controller 2 detects that the driver is not looking straight ahead (turning left or right) based on the facial features, specifically the orientation of the face, and the driver is driving on a highway (when no driving-related situations occur that cause the driver to look away from the road), it generates a face change factor code "0," which is external factor information indicating that the behavior is due to emotion, such as distracted driving, and not due to external factors (such as a change in facial orientation to check for an obstacle). In other words, it indicates that no external factors are causing the state in which the driver's face orientation is in question. Specifically, Controller 2 generates external factor information indicating that the fact that the driver's face is not facing straight ahead is not due to a driving necessity, but rather to a change in emotion, meaning that no external factors influencing the change in face are present.

[0036] Furthermore, when a warning indicator from ADAS (Advanced Driver-Assistance Systems) lights up on the meter panel, if the facial feature of the gaze is directed downwards, Controller 2 generates a facial change factor code "1," which is external factor information indicating that the change in gaze is not due to emotion but rather to driving-related causes such as checking the warning indicator while driving. In other words, Controller 2 generates external factor information indicating that the change in gaze is not due to emotion but rather to driving, that is, that an external factor affecting the change in face has occurred.

[0037] Furthermore, when the driver's gaze is directed downwards and the ADAS warning indicator is not illuminated, Controller 2 generates a facial change factor code "0," which is external factor information indicating that the driver's behavior is emotionally driven, such as looking away, and not due to external factors; in other words, it indicates that no external factors are present. Specifically, Controller 2 generates external factor information indicating that the downward gaze was not caused by a necessity in driving, but rather by an emotional change; in other words, it indicates that no external factors influencing facial changes are present.

[0038] Controller 2 then generates training data during the learning process, consisting of emotion information, facial features, and external factor information for the corresponding (same) detection timing. It then generates a training dataset using multiple training datasets and uses this dataset to train an emotion estimation model. Specifically, for each training dataset in the training dataset, Controller 2 inputs the emotion information as the target variable and the facial features and external factor information as explanatory variables into the pre-trained emotion estimation model, and trains the emotion estimation model using a learning method such as backpropagation. An example of a training dataset in the learning process is explained using Figure 3.

[0039] Figure 3 shows an example of a training dataset (data table) generated during the learning process. As shown in Figure 3, the training dataset is a collection of training data consisting of data of each data type for each time period (duration). The data type is information indicating the type of data included in the training data. The time is the data collection time (time) for each training data, and from that time, time series data or representative values ​​(values ​​calculated based on the time series data for that period, singular values ​​of various sensors for that period, etc.) for a period of time suitable for emotion estimation (set by the designer / developer based on experiments, etc.) are stored. This time data also serves as the primary key (identification code) for each training data.

[0040] In the example shown in Figure 3, the data types include arousal and activity levels, which are the raw data for estimating emotional information; facial features, such as facial expressions, gaze, and face orientation; and external factors, such as the presence or absence of external factors. In other words, Controller 2 generates training data by combining emotional information (arousal and activity levels), facial features, and external factors over time. It then generates a training dataset by collecting an appropriate amount of training data suitable for learning. By using this training dataset, Controller 2 can perform model learning that takes external factors into account.

[0041] In the example shown in Figure 3, based on data acquired during the period from time t1 to time t2, training data is generated in which the arousal level is a1, activity is b1, facial features are c11, c12, c13... (values ​​at each time point during the period from time t1 to time t2 (for example, at predetermined equally spaced time points) (gaze and face orientation data are similar)), gaze is d11, d12, d13..., face orientation is e11, e12, e13..., and external factor information is "0". Furthermore, based on data acquired during the period from time t2 to time t3, training data is generated in which the arousal level is a2, activity is b2, facial features are c21, c22, c23..., gaze is d21, d22, d23..., face orientation is e21, e22, e23..., and external factor information is "1".

[0042] In the example in Figure 3, the training data for the period from time t2 to time t3 is arousal level a2, activity level b2, facial features c21, c22, c23..., gaze direction d21, d22, d23..., and face orientation e21, e22, e23..., with an external factor (face change factor code) of "1". Through training with this training data, the training model learns that face orientation e21, e22, e23... is due to an external factor, in other words, it has little correlation with emotion (in this training data, arousal level and activity level, which are emotion indices), and it becomes easier to determine that the emotion is unknown (or there is no change in emotion) when face orientation is e21, e22, e23.... In other words, in the inference process described later, even if the face orientation in the input data for inference that is detected is e21, e22, e23... (or a similar pattern), if the external factor (face change factor code) is "1", the emotion estimation model will output (or is more likely to output) that the emotion is unknown (or that there is no change in emotion). This allows the controller 2 to reduce the likelihood of making incorrect emotion estimations based on face changes in situations where external factors affecting face changes occur.

[0043] (Inference process) In the inference process, Controller 2 estimates the driver's emotional information using the trained emotion estimation model generated in the learning process. In the inference process, the explanatory variables of the training data from the learning process are used as input data to the emotion estimation model (hereinafter referred to as inference input data). Specifically, Controller 2 generates inference input data using facial features based on facial images from the in-vehicle camera 100 and external factor information calculated (converted) based on various sensor output data as described above. The method for generating facial features and external factor information is the same as in the learning process described above, so the explanation is omitted. An example of inference input data in the inference process will be described later in Figure 4.

[0044] Controller 2 inputs the generated inference input data into the emotion estimation model and obtains emotion information (in the case of the training data example in Figure 3 and the inference input data in Figure 4, the arousal and activity levels of the emotion indicators) as the output of the emotion estimation model. Controller 2 outputs the emotion information obtained from the emotion estimation model (for example, the emotion type estimated from the arousal and activity levels of the emotion indicators) to the in-vehicle device or vehicle control device. Here, an example of the inference input data in the inference process will be explained using Figure 4.

[0045] Figure 4 shows an example of inference input data (data table) generated in the inference process. As shown in Figure 4, the inference input data consists of data of a predetermined data type that serves as input to the emotion estimation model detected during the emotion estimation target period. The data type indicates the type of data required as inference input data. In Figure 4, the data of the predetermined data type is shown for each emotion estimation target period (T1 (T1~T2), T3 (T3~T4)). It is desirable that each emotion estimation target period has the same duration as the data collection time in the learning process.

[0046] In the example shown in Figure 4, the data types include facial features (facial expressions, gaze, and face orientation) and external factors (external factors), which correspond to the explanatory variables in the training data. In other words, Controller 2 generates inference input data consisting of facial features and external factors from the corresponding (same) detection timings. For example, when Controller 2 estimates emotion information for the period from time T1 to time T2, it uses the data from time T1 to time T2. Specifically, Controller 2 inputs inference input data where facial features are c31, c32, c33..., gaze is d31, d32, d33..., face orientation is e31, e32, e33..., and external factors information is "0" into the emotion estimation model to estimate the driver's emotion information for the period from time T1 to time T2. This allows Controller 2 to perform emotion estimation that takes external factors into account.

[0047] In Figures 3 and 4, the controller 2 generates a face change factor code by combining (comprehensively judging) various types of information (output data from each sensor) as external factor information, and uses this as training data and inference input data. However, these data can be used as other external factor information. For example, the controller 2 may use image features extracted from the external video feed (such as whether the vehicle is going straight or turning left or right, and whether there are obstacles in the surroundings), sensor information from various sensors 120, and route guidance information from the navigation system (these data can be binarized as described above) as training data and inference input data.

[0048] Next, the processing procedure of the emotion estimation device 1 according to the embodiment will be described using Figures 5 and 6. Figure 5 is a flowchart showing the processing procedure of the learning process performed by the emotion estimation device 1 according to the embodiment. Note that the learning process shown in Figure 5 is executed when the learning mode is set by an operation by the administrator (designer) of the emotion estimation model.

[0049] As shown in Figure 5, the controller 2 acquires the driver's facial images from the in-vehicle camera 100 over a predetermined period of time suitable for emotion estimation, as well as external video and other factor information from the external camera 110, various sensors 120, and in-vehicle devices 130, and also the driver's biometric information from biosensors (step S101). Specifically, the controller 2 acquires time-series facial images (i.e., video) over a predetermined period of time.

[0050] Next, Controller 2 extracts time-series facial features of the driver based on the time-series facial images (step S102). For example, Controller 2 extracts (transforms) from the facial images numerical data such as the driver's facial expression, face orientation, and gaze at predetermined time intervals (i.e., based on each corresponding frame image) that are suitable for emotion estimation.

[0051] Next, the controller 2 determines whether or not external factors affecting facial changes have occurred based on external video footage and other factor information acquired in step S101 from the external camera 110, various sensors 120, and in-vehicle device 130 (step S103).

[0052] Next, controller 2 generates external factor information based on the determination result of the external factor (step S104). Specifically, if an external factor occurs, controller 2 generates external factor information indicating face change factor code "1", and if no external factor occurs, it generates external factor information indicating face change factor code "0".

[0053] Next, the controller 2 estimates the driver's emotional information based on the driver's biometric information acquired in step S101 (step S105). Specifically, the controller 2 calculates the level of arousal and activity from the driver's biometric information, which is brain waves and heart rate, and estimates the emotion (type) based on the level of arousal and activity.

[0054] Next, controller 2 generates training data consisting of generated and estimated facial features, external factor information, and emotion information (step S106).

[0055] Next, Controller 2 determines whether the training data included in the dataset has accumulated to a certain amount or more necessary for training (step S107). That is, Controller 2 determines whether the amount of data in the training dataset, which is a collection of training data, has accumulated to a sufficient amount for training the sentiment estimation model. If Controller 2 determines that the training data has accumulated to a certain amount or more (step S107: Yes), it trains the sentiment estimation model using the training dataset (step S108) and terminates the training process. On the other hand, if Controller 2 determines that the dataset has not accumulated to a certain amount or more (step S107: No), it returns to step S101 and generates training data for the next period.

[0056] Figure 6 is a flowchart showing the processing procedure of the inference process performed by the emotion estimation device 1 according to the embodiment. The inference process shown in Figure 6 is repeatedly performed at a predetermined appropriate emotion estimation interval from the time the vehicle ignition is turned on until it is turned off, while the inference mode is set. Alternatively, the inference process is performed when a command to start emotion estimation is received from another device or the like.

[0057] As shown in Figure 6, the controller 2 acquires the driver's face image from the in-vehicle camera 100, and also acquires external video and other factor information from the external camera 110, various sensors 120, and in-vehicle device 130 (step S201). Specifically, the controller 2 acquires time-series face images (i.e., video), time-series sensor signals, etc., over a predetermined period set to a time length appropriate for emotion estimation.

[0058] Next, Controller 2 extracts time-series facial features of the driver based on the acquired time-series facial images (step S202). For example, Controller 2 extracts (converts) facial features, which are numerical data such as the driver's facial expression, face orientation, and gaze.

[0059] Next, the controller 2 determines whether or not external factors affecting facial changes have occurred based on external video footage and other factor information from the external camera 110, various sensors 120, and in-vehicle device 130 (step S203). Specifically, the controller 2 determines whether or not external factors have occurred at the predetermined period lengths mentioned above.

[0060] Next, the controller 2 generates external factor information for the predetermined period based on the determination result of the external factors (step S204). Specifically, if an external factor occurs, the controller 2 generates a face change factor code "1" as external factor information, and if no external factor occurs, it generates a face change factor code "0".

[0061] Next, Controller 2 generates inference input data using the generated facial features and the generated external factor information (step S205). Then, Controller 2 inputs the generated inference input data into the emotion estimation model to estimate emotion information (step S206), and terminates the inference process.

[0062] As described above, the emotion estimation device 1 according to the embodiment is an emotion estimation device that estimates the driver's emotion information based on the driver's facial image, and has a controller 2. The controller 2 acquires external factor information that affects changes in the driver's face in the facial image, and estimates the emotion information based on the facial image and the external factor information.

[0063] According to this disclosure, by estimating emotions based on facial images and external factors that influence facial changes, it is possible to estimate emotions while taking into account that facial changes may be caused by external factors that are only loosely related to emotions, thereby improving the accuracy of emotion estimation.

[0064] In the embodiment described above, an example was shown in which a model that estimates emotion (type) is used as the emotion estimation model. However, the model is not limited to this, and a model that estimates emotion indicators may also be used. In this case, during the learning process, the controller 2 learns an emotion indicator estimation model using emotion indicators, face images (or face features), and external factor information as learning data. Then, during the inference process, the controller 2 inputs the face images (or face features) and external factor information as inference input data to the emotion indicator estimation model, and obtains emotion indicators as the output of the emotion indicator estimation model. The controller 2 then estimates emotion (type), which is emotion information, based on the obtained emotion indicators.

[0065] Further effects and modifications can be readily derived by those skilled in the art. Therefore, broader aspects of the present invention are not limited to the specific details and representative embodiments expressed and described above. Accordingly, various modifications are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents. [Explanation of Symbols]

[0066] 1 Emotion estimation device 2 Controllers 3 Storage section 100 In-car cameras 110 Exterior car camera 120 sensors 130 In-vehicle equipment

Claims

1. An emotion estimation device that estimates the emotional information of a vehicle driver based on the facial image of the driver, It has a controller, The aforementioned controller, External factor information influencing the facial image is acquired, and the emotional information is estimated based on the facial image and the external factor information. Emotion estimation device.

2. The aforementioned external factor information is encoded information indicating whether or not an external factor affecting the facial image has occurred. The emotion estimation device according to claim 1.

3. The aforementioned controller, The emotional information is estimated based on the facial feature quantities, which are numerical representations of the features in the aforementioned facial image. The emotion estimation device according to claim 1.

4. The aforementioned controller, If the external factor information indicates that an external factor affecting the facial image has occurred, it is presumed that there has been no change in the driver's emotions. The emotion estimation device according to claim 1.

5. The aforementioned controller, The turning state of the aforementioned vehicle is acquired, If the turning state of the driver's face, based on the facial image, is equivalent to the turning state of the vehicle, then the external factor information indicates that an external factor has occurred. The emotion estimation device according to claim 4.

6. An emotion estimation program executed by an emotion estimation device that estimates the emotion information of a vehicle driver based on the facial image of the driver, External factor information influencing the facial image is acquired, and the emotional information is estimated based on the facial image and the external factor information. The emotion estimation program executed by the controller.

7. A method for generating learning data, performed by a learning data generation device that generates learning data for a learning model that estimates the emotional information of a vehicle driver based on the facial image of the vehicle driver, The system acquires information on external factors that affect the facial image, and generates the training data based on the facial image and the external factor information. The method used by the controller to generate training data.

8. A learning data generation program executed by a learning data generation device that generates learning data for a learning model that estimates the emotional information of a vehicle driver based on the facial image of the vehicle driver, Emotional information is estimated based on the subject's biometric information. The facial image of the subject is obtained, We obtain information on external factors that affect the facial image of the subject, The training data is generated with the aforementioned emotional information as the dependent variable and the aforementioned facial image and the aforementioned external factor information as independent variables. The training data generation program executed by the controller.

Images