Learning data creation method, learning data creation program, learning data creation device, learning data creation system, learning method, learning device, and emotion estimation device

Abstract

To improve accuracy of emotion estimation by creating a larger number of pieces of learning data necessary for learning of an AI model in a short period.SOLUTION: A learning data creation method for creating learning data of an AI model that estimates biological information regarding an emotion of an object person on the basis of appearance information of the object person of emotion estimation, includes: acquiring emotion representing biological information being the biological information at the time of representation of the emotion of a subject for creating the learning data; acquiring emotion representation appearance information being the appearance information in which the emotion is known; and associating the emotion representation biological information and the emotion representation appearance information having the same emotion to create the learning data.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to a learning data creation method, a learning data creation program, a learning data creation device, a learning data creation system, a learning method, a learning device, and an emotion estimation device related to an emotion estimation AI. [Background technology]

[0002] 2. Description of the Related Art There is a conventional emotion estimation device that estimates an emotion, including a calm state, of a user based on a facial image of the user (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-3497 Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, artificial intelligence (AI) is becoming more sophisticated and widespread, and the application of AI to emotion estimation devices is also being considered. In order to create an AI model to be used in emotion estimation devices and to improve its accuracy, it is necessary to sample various data related to emotion estimation, such as appearance information and biometric information, from many people. Furthermore, it is necessary to create supervised learning data based on this sampling data and have the pre-learning AI model learn from it.

[0005] However, training an AI model requires a large amount of training data that correlates a person's facial expressions (appearance information) with biological signals (biological information) such as brain waves and heart rate, and creating this training data requires a huge amount of labor, which has been an issue.

[0006] In view of the above problems, the present invention aims to provide a technique that can generate a large amount of learning data required for training an AI model in a short period of time, thereby improving the accuracy of emotion estimation. [Means for solving the problem]

[0007] An exemplary training data creation method of the present invention is a training data creation method for creating training data for an AI model that estimates biometric information related to the emotions of a subject based on appearance information of the subject for emotion estimation, the method comprising the steps of: obtaining emotion-expression biometric information, which is the biometric information, when the subject expresses the emotion for creating the training data; obtaining emotion-expression appearance information, which is the appearance information in which the emotion is known; and associating the emotion-expression biometric information and the emotion-expression appearance information, which are the same emotion, to create the training data. Effect of the Invention

[0008] According to the present invention, biosignals (biological information) of brain waves and heartbeats that are the same emotion but are acquired (measured, photographed) at different times are associated with image data (appearance information) that records a person's facial expression, and these are used as learning data. This allows the biosignals (biological information) of brain waves and heartbeats, which require the subject to wear contact sensors such as an brain wave sensor and a heartbeat sensor during measurement, to be associated with a large amount of image data (appearance information). As a result, it becomes possible to create a large amount of learning data required for training an AI model in a short period of time. By learning using a large amount of learning data, it is expected that the accuracy of emotion estimation by the AI ​​model will improve. [Brief description of the drawings]

[0009] [Figure 1] A conceptual explanatory diagram showing emotion estimation processing by an emotion estimation device. [Diagram 2] A diagram showing an example of a multi-dimensional (2-dimensional) model (mental plane) for emotion estimation. [Diagram 3] Block diagram showing the configuration of a vehicle control system [Figure 4]A flowchart showing an emotion estimation process executed by a controller of the emotion estimation device in FIG. [Diagram 5] Conceptual diagram showing the learning process of emotion estimation AI using a learning device [Figure 6] Block diagram showing the configuration of a learning device [Figure 7] Schematic diagram showing time changes in the subject's biosignals (emotion index values) [Figure 8] A diagram showing an example of a task table. [Figure 9] Enlarged schematic diagram showing time changes in the subject's biosignals (emotion index values) [Figure 10] An example of a data type table [Figure 11] A flowchart showing the learning process of the emotion estimation AI executed by the controller of the learning device of FIG. [Figure 12] Block diagram showing the configuration of the learning data creation system [Figure 13] A flowchart showing a learning data creation process executed by a controller of the learning data creation device of FIG. 12. [Figure 14] A flowchart showing the learning process of the emotion estimation AI executed by the controller of the learning device of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the contents of the embodiments described below.

[0011] <1. Conceptual structure of emotion estimation device> First, the feeling estimation device 20 will be described.

[0012] The emotion estimation model 222m of the emotion estimation device 20 estimates emotions based on two emotion index values, which are indexes indicating mental and physical states related to emotions. One of the emotion index values ​​used in this embodiment is the central nervous system arousal level (hereinafter referred to as arousal level), and its index value can be calculated from "β waves / α waves of electroencephalograms." In addition, the other emotion index value is the autonomic nervous system activity level (hereinafter referred to as activity level), and its index value can be calculated from "the standard deviation of the heartbeat LF (Low Frequency) component (the low frequency component of the heartbeat waveform signal)."

[0013] The emotion estimation model 222m is configured by a model (calculation formula or conversion data table) for emotion estimation based on arousal and activity. The emotion estimation model 222m is configured by a multidimensional model (here, a two-dimensional model with arousal and activity as two axes) that estimates emotion using arousal and activity as parameters. The two-dimensional model is created based on medical evidence (papers, etc.) showing the relationship between each of a plurality of indexes and emotion (relationship between arousal and activity and emotion). Alternatively, the two-dimensional model is created based on the results of a questionnaire by many subjects (data consisting of emotion declaration by subjects and arousal and activity at that time (based on electroencephalogram and heart rate measurement values)). The emotion estimation model 222m can be not only a two-dimensional model, but also a multidimensional model of three or more dimensions.

[0014] FIG. 2 is a diagram showing an example of a multi-dimensional (two-dimensional) model (psychological plane) for emotion estimation. According to various medical evidence related to psychology, psychology can be estimated based on two types of indicators (emotion index values) that indicate physical conditions. In the psychological plane shown in FIG. 2, the vertical axis is "arousal level (arousal-unaroused)" and the horizontal axis is "autonomic nervous system activity (sympathetic nervous activity (strong emotion)-parasympathetic nervous activity (weak emotion)").

[0015] In this psychological plane, each of the four quadrants separated by the vertical and horizontal axes is assigned to a corresponding type of emotion. The distance from each axis indicates the intensity of the corresponding emotion. The emotions of "fun, joy, anger, and sadness" are assigned to the first quadrant. The emotion of "melancholy" is assigned to the second quadrant. The emotions of "relaxation and calm" are assigned to the third quadrant. The emotions of "anxiety, fear, and discomfort" are assigned to the fourth quadrant. The positions of the axes are appropriately set based on experiments, for example, by measuring the subject's level of arousal and activity and performing statistical processing.

[0016] Then, two types of emotion index values ​​(arousal and activity) obtained based on the biosignal are plotted on a psychological plane to obtain coordinates, from which emotion can be estimated. Specifically, emotion and its intensity can be estimated based on which quadrant of the psychological plane the plotted coordinates are in, where they are located within the quadrant, and how far they are from the origin. Note that the emotion estimation model shown in Figure 2 is a two-dimensional plane, but it can become a multidimensional space of three or more dimensions depending on the number of indices used.

[0017] However, since estimating the intensity of emotions is difficult and involves a relatively large error, and the applications of such estimation are limited, a practical and useful method is to determine the emotion based on which quadrant of the psychological plane the emotion index value is in, and to use the determined emotion.

[0018] In the emotion estimation device 20 shown in Fig. 1, the vertical axis "arousal level" of the emotion estimation model 222m is configured with two values ​​(two values ​​with the axis as a boundary), and the horizontal axis "activity level" is also configured with two values. The emotion estimation device 20 estimates emotions by fitting (plotting as coordinates) the input emotion indicators "arousal level" and "activity level" to the emotion estimation model 222m shown on such a psychological plane.

[0019] The emotion estimation model 222m of the emotion estimation device 20 uses the arousal and activity of emotion indicators calculated based on electroencephalogram data and heartbeat data. Note that in this embodiment, the emotion estimation device 20 inputs to the emotion estimation model 222m the arousal and activity of emotion indicators estimated by the first AI model 121mA and the second AI model 121mB from the gaze, facial direction, facial expression, and the like, instead of the electroencephalogram data and heartbeat data detected by an electroencephalogram sensor and a heartbeat sensor.

[0020] This technology eliminates the need to equip the subject of emotion estimation with contact sensors such as an EEG sensor and a heart rate sensor. This technology uses non-contact sensors such as a camera to collect data (appearance information) such as gaze, facial direction, and facial expression, and inputs this data into an AI model to estimate EEG data and heart rate data. For example, when estimating the emotions of a vehicle driver and using them for vehicle driving control, it is practically difficult to equip the driver with contact sensors, making this technology particularly useful.

[0021] In other words, calculation of emotion indices usually requires biosignals, but most biosignals are obtained from contact-type sensors, which limits their applications. For this reason, it is desirable to use sensors that detect information based on the external appearance of the emotion estimation target (subject), which can be detected by non-contact sensors.

[0022] In the use example of the feeling estimation device 20 shown in FIG. 1, a user U2 (a target of feeling estimation) is a driver of a vehicle V2. The first AI model 121mA and the second AI model 121mB used in the estimation method are used to estimate the feeling of the driver (user U2). The first AI model 121mA and the second AI model 121mB are separately trained, provided as trained models, and loaded into the feeling estimation device 20. Details of the feeling estimation device 20 will be described later.

[0023] The feeling estimation device 20 can be realized by a computer device mounted on the vehicle V2, or by a server connected to the vehicle V2 via a network. The server may be a physical server or a virtual server.

[0024] The vehicle V2 is equipped with a camera C as an on-board sensor. The camera C captures an image of the user U2 (driver) and outputs a captured image (a facial image including the line of sight and facial direction) relating to the movement of the user U2. Then, the line of sight data and facial direction data of the user U2 (driver) based on the captured image of the camera C are input to the first AI model 121mA and the second AI model 121mB.

[0025] In the illustrated example, the image captured by the camera C is processed by image recognition processing or the like to be processed into gaze data (image of the eyeball or data processed into text and numerical data indicating gaze (direction)) and facial direction data (image of the face or data processed into text and numerical data indicating the facial direction) and input to the first AI model 121mA and the second AI model 121mB, but the image captured by the camera C may be input. The first AI model 121mA and the second AI model 121mB (AI models designed and trained according to the input data type) corresponding to the input data type are mounted on the emotion estimation device 20.

[0026] Since gaze and facial movement are important in emotion estimation, gaze data and facial direction data are preferably video data for a period that matches the emotion estimation timing, for example, a video of a predetermined length just before the emotion estimation timing. For example, if the emotion estimation time length is 10 seconds, gaze data and facial direction data based on facial video data from time Ts-10 seconds (start point) to time Ts (end point) are input to the first AI model 121mA and the second AI model 121mB at the emotion estimation timing Ts. The first AI model 121mA and the second AI model 121mB are trained using learning input data in this data format.

[0027] The first AI model 121mA is an AI model that inputs gaze data and face direction data of the user U2 and outputs an emotional index value of arousal level, and is designed to have a structure suitable for the input and output. The first AI model 121mA is trained with a large number of learning data sets in which gaze data and face direction data are input and the arousal level corresponding to the input data is used as the correct answer data.

[0028] The second AI model 121mB is an AI model that inputs gaze data and face direction data of the user U2 and outputs an emotion index value of activity, and is designed to have a structure suitable for the input and output. The second AI model 121mB is trained with a large number of learning data sets in which gaze data and face direction data are input and activity corresponding to the input data is used as correct answer data.

[0029] The learning method of the first AI model 121mA and the second AI model 121mB will be described later in detail.

[0030] The first AI model 121mA and the second AI model 121mB output the two types of emotion indexes, ie, arousal level and activity level data, estimated (output) based on the gaze data and face direction data of the user U2 to the emotion estimation model 222m. The emotion estimation model 222m estimates the emotion based on the input arousal level and activity level.

[0031] <2. Vehicle control system> Next, a vehicle control system that uses estimated emotion data to control a vehicle will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of a vehicle control system 40. Fig. 3 shows components necessary for explaining the features of this embodiment, and omits the description of general components.

[0032] 3, the vehicle control system 40 includes the feeling estimation device 20, the vehicle control device 30, an actuator unit 41, and a notification unit 42. Although not shown in the figure, the vehicle control system 40 includes input devices such as a keyboard and a touch panel, and an output device such as a display.

[0033] <2-1. Emotion estimation device> The feeling estimation device 20 includes a communication unit 21, a storage unit 22, and a controller 23. The feeling estimation device 20 is generally required to have quick response in control, and is therefore mounted on the vehicle V2 as in this example, but may also be configured as a server connected to the vehicle V2 via a network.

[0034] The storage unit 22 is provided with a first AI model storage unit 221A and a second AI model storage unit 221B that store the trained first AI model 121mA and the trained second AI model 121mB shown in FIG.

[0035] The first AI model 121mA and the second AI model 121mB are stored in the first AI model storage unit 221A and the second AI model storage unit 221B, for example, in the following manner.

[0036] The creator of the vehicle control system 40, when assembling the system, etc., installs memories in which the trained first AI model 121mA and the second AI model 121mB are written as the first AI model storage unit 221A and the second AI model storage unit 221B. Alternatively, the creator of the vehicle control system 40 connects an external storage device in which the trained first AI model 121mA and the second AI model 121mB are stored to the vehicle control system 40 (emotion estimation device 20), reads the first AI model 121mA and the second AI model 121mB, and writes them to the first AI model storage unit 221A and the second AI model storage unit 221B. Alternatively, the creator of the vehicle control system 40 receives the trained first AI model 121mA and the second AI model 121mB from the learning device 10 via a network, and writes them to the first AI model storage unit 221A and the second AI model storage unit 221B.

[0037] The storage unit 22 is also provided with an emotion estimation model storage unit 222 that stores the emotion estimation model 222m shown in Fig. 1. The emotion estimation model storage unit 222 stores various data necessary for emotion estimation, such as a psychological plane table 224 that is data for estimating emotions from emotion index values ​​shown in Fig. 2.

[0038] The psychological plane table 224 is data forming the emotion estimation model (psychological plane) shown in Fig. 2, and is a data group that associates two types of emotion index values ​​(arousal level, activity level) with the corresponding emotion information. In addition, such emotion estimation model (various data constituting the emotion estimation model) is also stored in the storage unit 22 (emotion estimation model storage unit 222) at the time of designing or assembling the emotion estimation device 20 in a similar manner to the first AI model 121mA and the second AI model 121mB.

[0039] In addition, the psychological plane table 224 can be replaced with data of an arithmetic (logical) processing formula for calculating emotions from two types of emotion index values ​​(arousal level, activity level). In that case, the controller 23 described later will calculate emotions by arithmetic processing based on the arithmetic (logical) processing formula.

[0040] The controller 23 is a device that controls various operations of the feeling estimation device 20, includes a processor that performs arithmetic processing, and is configured by, for example, a CPU. The controller 23 has, as its functions, an acquisition unit 231, a behavior determination unit 232, a feeling estimation unit 233, and a providing unit 234. The functions of the controller 23 are realized by the processor executing arithmetic processing in accordance with a program stored in the storage unit 22.

[0041] The acquisition unit 231 acquires various appearance information (image information) of the user U2, who is a target of emotion estimation, photographed by the camera C, via the communication unit 21. The acquisition unit 231 stores the acquired various appearance information in a data table formed in the storage unit 22 as necessary for subsequent processing. The acquisition unit 231 acquires various appearance information at substantially the same time and stores it in the data table as one data set. These data are then used to estimate the behavior state and emotion at that time.

[0042] The behavior determination unit 232 determines predetermined types of behavior such as the gaze, facial direction, and facial expression of the user U2 by performing an analysis process on the image information of the user U2 acquired by the acquisition unit 231. The predetermined types of behavior are behaviors corresponding to the types of data input to the first AI model 121mA and the second AI model 121mB, and in this embodiment, specifically, the data are the gaze, facial direction, and facial expression.

[0043] Therefore, the behavior determination unit 232 performs various processes according to the input specifications (determined by design conditions, learning conditions, etc.) of the first AI model 121mA and the second AI model 121mB. The various processes include, for example, a process of cutting out image (a group of still images or a video) data of a processing target time length, a process of cutting out the face and eyeballs of the user U2 from an image captured by the camera C, and a process of detecting the orientation.

[0044] The first AI model 121mA and the second AI model 121mB output appropriate estimated (correct) data in response to input of data of the same type (content, format) as the input data during learning. For this reason, the type (content, format) of the output data from the behavior determination unit 232 needs to be the same as the input data during learning of the first AI model 121mA and the second AI model 121mB. The behavior determination unit 232 performs such data processing. Programs and data for realizing these operations of the behavior determination unit 232 are stored in the memory unit 22.

[0045] The emotion estimation unit 233 estimates the emotion of the user U2 based on the behavior data determined by the behavior determination unit 232. In detail, the emotion estimation unit 233 inputs the behavior data determined by the behavior determination unit 232 to the first AI model 121mA and the second AI model 121mB. As a result, the first AI model 121mA and the second AI model 121mB output the emotion index value (arousal level, activity level) of the user U2 estimated based on the behavior information (appearance information) related to the line of sight, face direction, facial expression, etc. of the user U2. Then, the estimated emotion index value (arousal level, activity level) of the user U2 output by the first AI model 121mA and the second AI model 121mB is input to the emotion estimation model 222m. Then, the emotion estimation model 222m estimates the emotion of the user U2 based on the estimated emotion index value (arousal level, activity level) of the user U2 and outputs it.

[0046] The estimated emotion index value used in the emotion estimation described above may be estimated using data obtained by performing statistical processing (average processing, low-pass filter processing, etc.) on a plurality of pieces of data for an appropriate time length. In this case, the statistical processing is performed by the emotion estimation unit 233.

[0047] The providing unit 234 provides the emotion of the user U2 estimated by the emotion estimation unit 233 to the vehicle control device 30. This enables the vehicle control device 30 to control the vehicle V2 based on the emotion of the user U2.

[0048] <2-2. Vehicle control device and other components> The vehicle control device 30 is, for example, an ECU (Electronic Control Unit) for vehicle control, and is mounted on the vehicle V2. The vehicle control device 30 includes a controller 31, a storage unit 32, and a communication unit 33.

[0049] The controller 31 performs various vehicle controls, such as control of the direction and speed of the vehicle V2, based on driving operations by the user U2 who is the driver of the vehicle V2 and information from various connected sensors (not shown), such as a vehicle speed sensor, an air-fuel ratio sensor, and a steering angle sensor. Furthermore, the controller 31 receives information related to the emotion of the user U2 (estimated emotion) from the emotion estimation device 20 via the communication unit 33. The controller 31 then performs vehicle control using the received estimated emotion information, such as control of the speed, accelerator sensitivity, and brake sensitivity. Specifically, when the driver is excited, the controller 31 performs driving control that is on the safe side, that is, that suppresses the influence of the driver's excitement, by lowering the accelerator sensitivity (to make it harder to accelerate), increasing the brake sensitivity (to make it easier to stop), and lowering the upper limit of the maximum speed control.

[0050] In addition, the controller 31 notifies the driver of the estimated emotion information itself and notifies the driver according to the estimated emotion information. Specifically, when the driver is excited, the controller 31 displays and provides voice guidance such as "calm down," and controls the quality of the voice guidance to be calm in speech and expression.

[0051] The storage unit 32 is configured with various memories like the storage unit 12, and stores data used by the controller 31 in processing, such as programs, processing coefficient data, temporary storage data during processing, etc. The storage unit 32 is provided with a plurality of data tables for various processing in which information related to emotions received from the emotion estimation device 20 is associated with control signals for the vehicle V2 and notification information for the user U2 in the vehicle cabin.

[0052] The communication unit 33 is an interface for communicating data among the feeling estimation device 20, the actuator unit 41, and the notification unit 42, and is, for example, a Controller Area Network (CAN) interface.

[0053] The actuator unit 41 is composed of various driving components such as a motor that realizes various operations of the vehicle, and the operations are driven and controlled by the vehicle control device 30. Specifically, the actuator unit 41 is, for example, an engine and a motor that generate a driving force in the vehicle V2, a steering actuator that drives the steering of the vehicle V2, a brake actuator that drives the brakes of the vehicle V2, and the like.

[0054] The notification unit 42 notifies the user U2 in the vehicle cabin of the notification information from the vehicle control device 30 by a visual, auditory, or other means. Specifically, for example, the notification unit 42 is composed of a liquid crystal display that conveys the notification information to the user U2 by characters, images, videos, or the like, and a speaker that conveys the notification information by voice, warning sound, or the like.

[0055] The vehicle control device 30 generates and outputs a control signal for the vehicle V2 based on information from various sensors and the emotion of the user U2 provided by the emotion estimation device 20. Specifically, for example, when the user U2 is excited or scared, the vehicle control device 30 displays and sounds a message such as "calm down" via a display unit or a speaker. In addition, the vehicle control device 30 controls the speed of the vehicle V2 based on information from a vehicle speed sensor, an obstacle sensor (radar, etc.), etc., and when the user U2 is excited, the vehicle control device 30 controls the speed to be slower than when the user U2 is in a normal emotion.

[0056] That is, the vehicle control system 40 estimates the emotion of the user U2 who is the driver of the vehicle V2, and performs the operation of the vehicle V2, for example, driving control, taking into consideration the estimated emotion. With this configuration, the vehicle V2 can be controlled according to the emotion of the user U2. Therefore, according to this embodiment, it is possible to control the vehicle V2 according to the emotion of the driver that affects safe driving, which can contribute to safe driving of the vehicle V2.

[0057] <2-3. Processing of emotion estimation device> Fig. 4 is a flowchart showing emotion estimation processing executed by the controller 23 of the emotion estimation device 20 in Fig. 3. This flowchart shows the technical contents of a computer program that causes a computer device to realize the emotion estimation processing. In addition, the computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0058] The process shown in FIG. 4 is initiated when the vehicle V2 starts and various controls based on the estimated emotion are initiated, and is then repeatedly executed at appropriate timing for the various controls, that is, at timings when emotion estimation is required, or over a time period when emotion estimation is required.

[0059] In step S101, the controller 23 (acquisition unit 231) acquires data (appearance information) indicating the behavior of the user U2, specifically, an image captured by a camera of the user U2, and proceeds to step S102. The acquired various pieces of appearance information are stored in a data table of the storage unit 22 as necessary.

[0060] In step S102, the controller 23 (behavior determination unit 232) determines the behavior of the user U2, specifically, the line of sight, facial direction, and facial expression, based on the various data (camera captured images) acquired by the acquisition unit 231 in step S101, and proceeds to step S103.

[0061] In step S103, the controller 23 (emotion estimation unit 233) inputs each piece of data based on the behavior judgment result by the behavior judgment unit 232 judged in step S102 as input values ​​to the first AI model 121mA and the second AI model 121mB, and has the first AI model 121mA and the second AI model 121mB estimate the emotion index values ​​of alertness and activity, respectively, and proceeds to step S104.

[0062] In step S104, the controller 23 (emotion estimation unit 233) applies (inputs) the two estimated emotion index values ​​(arousal level, activity level) to the emotion estimation model 222m (data of the psychological plane table 224) to estimate the emotion of the user U2, and proceeds to step S105.

[0063] In step S105, the controller 23 (providing unit 234) provides (outputs) the estimated emotion information of the user U2 estimated by the emotion estimation unit 233 to the vehicle control device 30, and ends the process.

[0064] Since each estimated emotion is estimated to be adversely affected (misjudgment) due to noise in the sensor output, it is preferable to adopt an estimated emotion that has been subjected to statistical processing, such as adopting an average or mode over a predetermined period. For this reason, it is preferable for the controller 23 (providing unit 234) to add a timestamp to the estimated emotion, store it, and provide the estimated emotion that has been subjected to statistical processing to the vehicle control device 30 (step S105). It is also possible for the vehicle control device 30 to store estimated emotion information, perform statistical processing, and use the results for control.

[0065] <3. Learning method of emotion estimation AI> Next, the learning method of the first AI model 121ma and the second AI model 121mb will be described. In order to easily distinguish between the models before and after learning, the models before learning are referred to as the first AI model 121ma and the second AI model 121mb, and the models after learning are referred to as the first AI model 121mA and the second AI model 121mB, and are written so that the models before and after learning can be distinguished by the last character of the code. Figure 5 is a conceptual explanatory diagram showing the learning process of the emotion estimation AI by the learning device 10.

[0066] In this embodiment, the user U1 is a subject for creating learning data, and the subject is a member of the development team of the emotion estimation device 20. In the case of a personal emotion estimation device 20 in which the user is limited, the subject is preferably the user, but since there is similarity even if the subject is a different person, the subject may be the user and a different person. In the case of a general-purpose emotion estimation device 20 in which the user is not limited, the subject is the user and a different person, but if learning data is created by multiple subjects, learning data by subjects with various characteristics can be obtained. As a result, the emotion estimation device 20 is expected to be a highly versatile device. In this embodiment, since the emotion estimation device 20 is assumed to be mounted on a vehicle, learning is performed on an in-vehicle device in a similar environment, and the subject user U1 is the driver of the vehicle V1.

[0067] The appropriate learning data used to train an AI model is determined by the purpose of the AI ​​model. For example, in the case of an AI-applied device for a specific driver, the appropriate information is information on that driver, and in the case of an AI-applied device for various drivers, the appropriate information is information on a wide variety of drivers. In addition, in the case of an AI-applied device for a driver, the appropriate information is information on the driver, and in the case of an AI-applied device for each vehicle occupant other than the driver, the appropriate information is information on various occupants other than the driver. In order to increase the accuracy of emotion estimation by the AI ​​model, a large amount of information is required as learning data.

[0068] In this embodiment, the AI ​​application device is applicable to drivers in general (not specific individuals, but general drivers). In order to make the device control the vehicle based on the emotions of the drivers, it is preferable that the experimental test run for learning the AI ​​is an experimental test run with various patterns by multiple subjects of various types. Since the learning by each subject is similar, the learning by one subject (driver) will be described.

[0069] Furthermore, for example, if the AI-applied device is a medical treatment device in a medical institution, the user U1 suitable for the subject (information collection target) will be a patient or a doctor in the medical institution. If the AI-applied device is an educational instruction device in an educational institution, the user U1 suitable for the subject will be a student or a teacher. If the AI-applied device is an e-sports-related device or an entertainment content-related device, the user U1 suitable for the subject will be an e-sports player or a content viewer.

[0070] It is also possible to mount the learning device 10 on the vehicle V1 and have it perform learning, or to install the learning device 10 in a location other than the vehicle V1, such as a research and development laboratory, and have it perform learning. In the latter case, each biological data (measurement data such as brain waves, heart rate, images, etc.) of the subject user U1 who is riding in the vehicle V1 is input to the learning device 10 via a recording medium or a communication line. In this embodiment, an example will be described in which the learning device 10 is mounted on the vehicle V1 and has it perform learning.

[0071] A storage unit for the AI ​​models is provided in a storage device (memory, etc.) of the learning device 10 mounted on the vehicle V1, and a first AI model 121ma and a second AI model 121mb before learning are stored. Then, the learning device 10 executes a learning process described below, whereby learning of the AI ​​models before learning is performed, and a trained AI model, that is, a first AI model 121mA and a second AI model 121mB to be implemented in the AI ​​application device, is generated.

[0072] The trained first AI model 121mA and the trained second AI model 121mB are used in the emotion estimation device 20 shown in Figs. 1 and 3 described above. Therefore, the training data of the first AI model 121ma is behavior information such as the gaze, face direction, and facial expression of the user U1 (appearance information, data of the same type and style as the input data of the first AI model 121mA of the emotion estimation device 20), and the correct answer data is an emotion index value representing the arousal level. The training data of the second AI model 121mb is behavior information such as the gaze, face direction, and facial expression of the user U1 (appearance information, data of the same type and style as the input data of the second AI model 121mB of the emotion estimation device 20), and the correct answer data is an emotion index value representing the activation level.

[0073] For this reason, the vehicle V1 on which the learning device 10 is mounted is provided with various on-board sensors that output data used by (input to) the emotion estimation device 20. In this embodiment, specifically, the on-board sensors include a camera C. The camera C outputs captured image information showing the face of the driver U1 so that data on the line of sight and facial direction of the driver U1 can be obtained. The camera C is installed, for example, near the windshield or dashboard of the vehicle V1 so that the direction of the driver U1 is the direction of the image capture.

[0074] The output from camera C is then processed into behavior data by behavior determination unit 133, which has a configuration similar to that of behavior determination unit 232 in Fig. 3, and is processed into behavior data on gaze, facial direction, and facial expression. These behavior data are then input as input values ​​to first AI model 121ma and second AI model 121mb of learning device 10.

[0075] A biosensor is attached to the driver U1. In this embodiment, the biosensors are an electroencephalogram sensor ES for detecting an electroencephalogram and a heartbeat sensor HS for detecting a heartbeat. For example, a headgear-type electroencephalogram sensor is used as the brainwave sensor ES. For example, a chest belt-type electrocardiogram-type heartbeat sensor is used as the heartbeat sensor HS. Note that the biosensor may be replaced or replaced with other sensors depending on the bioinformation to be acquired, wearability, and the like. Examples of the other biosensors may include an optical heartbeat (pulse) sensor, a blood pressure monitor, or a NIRS (Near Infrared Spectroscopy) device.

[0076] Then, the brain wave data output by the brain wave sensor ES and the heart rate data output by the heart rate sensor HS are converted into emotion index values, ie, arousal level and activity level, in the index value calculation unit 132. Specifically, using a method similar to the above-mentioned method for calculating arousal level and activity level, arousal level is calculated from "β waves / α waves of brain waves" and activity level is calculated from "standard deviation of heart rate LF (Low Frequency) components (low frequency components of heart rate waveform signal)."

[0077] The emotion index value representing the arousal level calculated by the index value calculation unit 132 is input as a correct value to the first AI model 121ma. The emotion index value representing the activity level calculated by the index value calculation unit 132 is input as a correct value to the second AI model 121mb.

[0078] That is, the first AI model 121ma is trained using a supervised learning data set in which the input values ​​are behavior data (appearance information) of gaze, face direction, and facial expression, and the correct answer value is an emotion index value representing arousal level. Therefore, the trained first AI model 121mA is generated as an AI model in which the input values ​​are behavior data of gaze, face direction, and facial expression, and the output value is arousal level, as shown in FIG.

[0079] In addition, the second AI model 121mb is trained using a supervised learning data set in which the input values ​​are behavior data (appearance information) of gaze, face direction, and facial expression, and the correct answer value is an emotion index value representing activity. Therefore, the trained second AI model 121mB is generated as an AI model in which the input values ​​are behavior data of gaze, face direction, and facial expression, and the output value is activity, as shown in FIG.

[0080] Next, the flow of the learning method (AI learning process) of the first AI model 121ma and the second AI model 121mb will be described with reference to Fig. 5. Note that specific tasks such as mounting the sensor are performed by an operator in charge of learning the AI ​​model, a subject (driver U1), etc.

[0081] A first AI model 121ma before learning that estimates an alertness level using the gaze, face direction, and facial appearance information as inputs, and a second AI model 121mb before learning that estimates an activity level using the gaze, face direction, and facial appearance information as inputs are stored in a storage device (memory, etc.) of the learning device 10. The first AI model 121ma before learning and the second AI model 121mb before learning are separately created in advance by an AI designer or developer using a computer for AI design and according to the specifications of the AI ​​model (usage form, required performance, etc.).

[0082] The driver U1 gets into the vehicle equipped with the learning device 10 and adjusts the position, direction, sensitivity, etc. of the camera C mounted on the vehicle. The driver U1 also wears the brain wave sensor ES and the heart rate sensor HS. When the above-mentioned preparations for learning are complete, the driver U1 etc. starts up the learning device 10 and learning begins. If necessary (for example, when the vehicle is traveling, a learning condition), the driver U1 starts driving (operating) the vehicle.

[0083] The camera C captures an image that is appearance information of the user U1, and outputs the image information to the learning device 10.

[0084] The brain wave sensor ES and the heart rate sensor HS detect the brain waves and heart rate, which are bio-signals of the user U1, and output the brain waves and heart rate to the learning device 10.

[0085] Then, the learning device 10 performs subsequent processing by treating the image information, brain waves, and heart rate of user U1 at the same timing as a pair of data (one data set).

[0086] The learning device 10 converts the brain waves (β waves / α waves of brain waves) of the inputted biological signal into an emotion index value representing the degree of arousal by emotion index value conversion processing in the index value calculation section 132. Also, the learning device 10 converts the heart rate (standard deviation of the heart rate LF component) of the inputted biological signal into an emotion index value representing the degree of activity by emotion index value conversion processing in the index value calculation section 132.

[0087] The learning device 10 converts the input image information of the user U1 into behavior data in the behavior determination unit 133 to generate information on the gaze, face direction, and facial expression. At this time, since the image information is time-series data that is continuous over time, the image information is converted into behavior data in units of an appropriate period (appropriate timing and time length based on the emotion index value calculation timing) relative to the emotion estimation timing (emotion index value calculation timing for generating learning data). This period may be determined based on experiments, etc., so that the estimated emotion index value of the AI ​​model becomes an appropriate value. Then, the behavior determination unit 133 outputs the behavior data of this period length as one piece of learning data.

[0088] The learning device 10 then generates learning data for the first AI model 121ma, which uses the gaze, facial direction, and facial expression information generated by the behavior determination unit 133 as input values ​​and the awakening level as a correct value. The learning device 10 also generates learning data for the second AI model 121mb, which uses the gaze, facial direction, and facial expression information generated by the behavior determination unit 133 as input values ​​and the activity level as a correct value.

[0089] Then, the learning device 10 aggregates a large number of pieces of learning data generated at each timing to generate a learning data set for each model. After that, the learning device 10 trains the first AI model 121ma and the second AI model 121mb using each of the generated learning data sets. Note that the learning data may be input to the first AI model 121ma and the second AI model 121mb sequentially at the timing when the learning data is created, and learning may be performed.

[0090] Specifically, the learning device 10 sequentially inputs the learning data of each generated learning data set to the first AI model 121ma and the second AI model 121mb. Then, the learning device 10 performs learning such as adjusting parameters such as weights in the first AI model 121ma and the second AI model 121mb using a learning algorithm such as an error backpropagation learning method.

[0091] As a result, the first AI model 121ma and the second AI model 121mb are trained using a large amount of supervised training data generated based on each piece of data sequentially detected by each sensor. Then, the trained first AI model 121mA and the trained second AI model 121mB used in the emotion estimation device 20 are generated.

[0092] <4. Learning device for emotion estimation AI> Fig. 6 is a block diagram showing the configuration of the learning device 10. Fig. 6 shows components necessary for explaining the features of this embodiment, and omits the description of general components.

[0093] 6, learning device 10 includes communication unit 11, storage unit 12, and controller 13. Learning device 10 can be configured as a so-called computer device. Although not shown in the figure, learning device 10 includes an input device such as a keyboard and an output device such as a display.

[0094] The communication unit 11 is an interface for communicating data with other devices and various sensors via a communication network, and is configured, for example, by a network interface card (NIC).

[0095] The storage unit 12 includes a volatile memory and a non-volatile memory. The volatile memory includes, for example, a RAM (Random Access Memory). The non-volatile memory includes, for example, a ROM (Read Only Memory), a flash memory, and a hard disk drive. The non-volatile memory stores programs and data that can be read by the controller 13. At least a part of the programs and data stored in the non-volatile memory may be obtained from another computer device (server device) connected by wire or wirelessly, or from a portable recording medium.

[0096] The storage unit 12 is provided with a first AI model storage unit 121A and a second AI model storage unit 121B. The first AI model storage unit 121A and the second AI model storage unit 121B store a first AI model 121ma and a second AI model 121mb before learning, which are to be learned, respectively. Furthermore, the storage unit 12 is provided with a task table 123 and a data type table 124 as data tables for various processes. The task table 123 and the data type table 124 will be described later.

[0097] The controller 13 realizes various functions of the learning device 10 and includes a processor that performs arithmetic processing and the like. The processor includes, for example, a CPU (Central Processing Unit). The controller 13 may be configured with one processor or multiple processors. When configured with multiple processors, the processors are connected to each other so that they can communicate with each other and cooperate to execute processing. Note that the learning device 10 can also be configured with a cloud server, in which case the CPU that configures the processor may be a virtual CPU.

[0098] The controller 13 includes, as its functions, an acquisition unit 131, an index value calculation unit 132, a behavior determination unit 133, a generation unit 134, and a provision unit 135. In this embodiment, the functions of the controller 13 are realized by a processor executing arithmetic processing in accordance with a program stored in the storage unit 12.

[0099] The acquisition unit 131 acquires various pieces of information (image (video) information, brain wave information, heart rate information) detected by the camera C, the brain wave sensor ES, and the heart rate sensor HS via the communication unit 11. The acquisition unit 131 stores the acquired various pieces of information in a data table formed in the storage unit 12 as necessary for subsequent processing. The acquisition unit 131 acquires these pieces of information at substantially the same time, and stores them as one data set in one data record in the data table (data type table 124). These pieces of data are then used to create one piece of supervised learning data.

[0100] FIG. 7 is a schematic diagram showing changes over time in a subject's biosignal. In the two graphs in FIG. 7, the horizontal axis is time and the vertical axis is the level of the biosignal. The upper graph in FIG. 7 shows the time series data waveform of β / α brain waves as a biosignal. The lower graph in FIG. 7 shows the time series data waveform of the standard deviation of the LF component of the heart rate as a biosignal. Below the two graphs in FIG. 7, video data of the subject's facial image (still images (frame images) at each timing) acquired at the same timing as the biosignal in the graph is depicted typically.

[0101] The graph in Figure 7 shows the transition of biosignals when the subject performs various tasks. The tasks performed are those that put the subject in a special emotional state, such as tasks that make the electroencephalogram biosignals (arousal level of an emotional indicator) swing to a higher level, or tasks that make the heart rate biosignals (activity level of an emotional indicator) swing to a higher level, etc. There are several types of tasks that are known to put the subject in a specific mental and physical state, and evidence has been obtained from research in the fields of medicine, psychology, etc., and such tasks can be selected and used as appropriate.

[0102] Fig. 8 is a diagram showing an example of the task table 123. As shown in Fig. 8, the items of the task table 123 include "task ID", "biological signal type", "corresponding index type", "task content", and "index value transition state".

[0103] "Task ID" is task ID data that is identification information for identifying task information. Task ID data is also a primary key of a data record in task table 123. That is, in task table 123, a data record is configured for each task ID data, and data of each item associated with the task ID data is stored in the data record.

[0104] The "biological signal type" stores the type of measurement value based on the biological signal detected by the sensor. This biological signal type data is data correlated with the corresponding index type data.

[0105] The "corresponding index type" is the emotional index type to be learned by the first AI model 121ma and the second AI model 121mb. As the corresponding index type, for example, arousal level and activity level are stored.

[0106] "Task content" is the specific content of the task that the subject executes for the learning process of the first AI model 121ma and the second AI model 121mb.

[0107] The "index value transition state" is the state to which the corresponding emotional index value is estimated to transition when the subject performs the corresponding task content (stored in the same record). For example, data such as "high arousal: arousal level becomes on the arousal side (high arousal level) (high index value state)" is stored.

[0108] According to this task table 123, for example, in the task data with task ID ST01, the biological signal is electroencephalogram data, the corresponding index type is alertness, the state transition of alertness is high alertness, and the specific task content is "mentally add displayed numbers." This task table 123 is used when selecting a task to be performed by the subject when bringing the subject into a desired biological state (emotional state).

[0109] Returning to FIG. 7, FIG. 7 shows the fluctuation periods La, Lb, Lc, and Ld of the biosignals due to the execution of various tasks Ta, Tb, Tc, and Td. When the subject executes the task Ta, the subject shows fluctuations in the biosignals corresponding to the task Ta in the fluctuation period La, and expresses an emotion in the second quadrant of the psychological plane of FIG. 2 (according to the subject's emotional declaration, "sadness"). When the subject executes the task Tb, the subject shows fluctuations in the biosignals corresponding to the task Tb in the fluctuation period Lb, and expresses an emotion in the second quadrant of the psychological plane of FIG. 2 (according to the subject's emotional declaration, "melancholy"). When the subject executes the task Tc, the subject shows fluctuations in the biosignals corresponding to the task Tc in the fluctuation period Lc, and expresses an emotion in the third quadrant of the psychological plane of FIG. 2 (according to the subject's emotional declaration, "calmness"). When the subject executes the task Td, the subject shows fluctuations in the biosignals corresponding to the task Td in the fluctuation period Ld, and expresses an emotion in the first quadrant of the psychological plane of FIG. 2 (according to the subject's emotional declaration, "joy"). It should be noted that the period Ln is a period of a neutral emotional state in which the emotion is intermediate and ambiguous.

[0110] Fig. 9 is an enlarged schematic diagram showing the change over time of the subject's biosignal. Fig. 9 is a further enlarged schematic diagram of the transition of the biosignal similar to that of Fig. 7, and the configuration and format within the figure are the same as those of Fig. 7. Fig. 9 shows the transition of the biosignal of the subject when a task Te different from that of Fig. 7 is executed. The fluctuation of the biosignal due to the execution of the task Te by the subject during the fluctuation period Le is shown.

[0111] When the subject performed task Te, the beta and alpha waves of the brain wave increased and swung toward the awakening side, as shown by the arrow Ve in Figure 9. This means that the rate of change of the beta and alpha waves of the brain wave was fast (the amount of change per unit time was large), and the intensity of the emotion was strong.

[0112] Furthermore, when the subject performed task Te, the standard deviation of the LF component of the heart rate of the subject decreased and swung toward the parasympathetic nerve activity (weak emotion) side, as shown by the arrow Vh in Figure 9. The rate of change in the standard deviation of the LF component of the heart rate was fast (the amount of change per unit time was large), which means that the intensity of the emotion was strong.

[0113] From these, when the subject performs the task Te, the subject expresses the emotion in the second quadrant of the psychological plane in Figure 2. Then, the subject's questionnaire after the task execution shows that the subject expressed the emotion of "melancholy" according to the subject's emotion declaration. Then, the frame images P1, P2, and P3 are recognized as the facial images of the subject when expressing the emotion of "melancholy."

[0114] In this way, the acquisition unit 131 acquires, as emotion expression bioinformation of the subject, data related to the biosignal of the β wave / α wave of the brainwave, which is the bioinformation of the subject, and data related to the biosignal of the standard deviation of the LF component of the heartbeat, which is the bioinformation, when the subject expresses emotion.The pair of the emotion expression bioinformation, the brainwave data and the heartbeat data, is called the data type of the bioinformation.

[0115] The acquiring unit 131 also acquires image data of a face image, which is appearance information, when the subject expresses emotion. The acquiring unit 131 then stores the electroencephalogram data, the heartbeat data, and the image data at substantially the same timing in one data record in the data type table 124 as one data set.

[0116] As described above, when the subject expresses emotion, emotion expression biometric information is generated by acquiring appearance information and biometric information at the same timing for the subject. The emotion expression biometric information includes the subject's biometric information and the fluctuation speed of the biometric signal (emotion intensity). In other words, the brain wave data and heart rate data (emotion expression biometric information) when the subject expresses emotion are recorded as biometric information when the subject's emotion is well expressed from the subject's face image acquired at the same timing. This makes it possible to clearly define the emotion type and emotion intensity for the brain wave data and heart rate data (emotion expression biometric information) when the subject expresses emotion.

[0117] Furthermore, the biological information acquired from the subject includes the ratio of beta waves to alpha waves (beta waves / alpha waves) in the electroencephalogram, which is a biological signal, and the standard deviation of the low frequency component (LF component) of the heart rate waveform signal in the heart rate, which is a biological signal. Since the electroencephalogram correlates with the level of arousal, it is possible to obtain an AI model that can accurately estimate the level of arousal as an emotional index value through learning. Since the heart rate correlates with the level of activity, it is possible to obtain an AI model that can accurately estimate the level of activity as an emotional index value through learning. As a result, emotions can be accurately estimated from the level of arousal and the level of activity.

[0118] Fig. 10 is a diagram showing an example of the data type table 124. Note that the dashed arrow in Fig. 10 means that the data records on both sides of the arrow are continuously connected. As shown in Fig. 10, the items of the data set in the data type table 124 include "data ID", "emotion quadrant", "emotion type", "initial value", "after emotion expression", "fluctuation speed", and "intensity" of electroencephalogram data, "initial value", "after emotion expression", "fluctuation speed", and "intensity", "image data", and "measurement time" of heart rate data.

[0119] Among the above data sets, the pair of electroencephalogram data and heart rate data is the data type of biological information. How to handle the data type of biological information will be described later.

[0120] "Data type ID" is ID data that is identification information for identifying a data type (data set). The data type ID data is also the primary key of a data record in the data type table 124. That is, in the data type table 124, a data record is configured for each data type ID data, and data of each item linked to the data type ID data is stored in the data record.

[0121] The "emotion quadrant" is data for one of the four quadrants from the first to the fourth on the psychological plane in Fig. 2. The data type table 124 includes data corresponding to all four quadrants. The "emotion type" is data indicating which emotion in the "emotion quadrant" it is, and is determined by the subject's emotional declaration as a result of a questionnaire given to the subject after the task is performed.

[0122] The "initial value," "after emotion expression," "fluctuation rate," and "intensity" of the electroencephalogram data are data related to the biosignals of the electroencephalogram's beta and alpha waves when the subject expresses emotion. The "initial value" is electroencephalogram data from the early stage of emotion expression, and "after emotion expression" is electroencephalogram data from the final stage of emotion expression. The "fluctuation rate" is the fluctuation rate (amount of fluctuation per unit time) of the electroencephalogram data from the early stage to the final stage of emotion expression. "Intensity" indicates the intensity of emotion expressed in the electroencephalogram data, and is "strong" when the "fluctuation rate" is equal to or greater than a specified value and is "weak" when the "fluctuation rate" is less than the specified value.

[0123] The "initial value", "after emotion expression", "fluctuation rate", and "intensity" of the heart rate data are data related to the biosignal of the standard deviation of the LF component of the heart rate when the subject is expressing emotion. The "initial value" is the heart rate data at the beginning of emotion expression, and "after emotion expression" is the heart rate data at the end of emotion expression. The "fluctuation rate" is the rate of fluctuation of the heart rate data (amount of fluctuation per unit time) from the beginning to the end of emotion expression. "Intensity" indicates the intensity of emotion expressed in the heart rate data, and is "strong" when the "fluctuation rate" is equal to or greater than a predetermined value and is "weak" when the "fluctuation rate" is less than the predetermined value.

[0124] In addition, the electroencephalogram data and heart rate data may be data representing waveform data when emotions other than those described above are expressed (data for reproducing the characteristics of the electroencephalogram waveform and heart rate waveform), or may be data that reproduces the waveform itself, for example.

[0125] "Image data" is an image data file of a facial image when the subject expresses emotion at the same timing as the electroencephalogram data and the heart rate data. "Measurement time" is the measurement time of the electroencephalogram data and the heart rate data.

[0126] Note that the signal levels (absolute values) of electroencephalograms and heartbeats vary due to individual differences, surrounding environment, etc. For this reason, it is preferable that a large number of data sets, which have the same "emotion quadrant," "emotion type," and "intensity" of data, but have different "initial values," "after emotion expression," and "fluctuation rates" of electroencephalogram data and different "initial values," "after emotion expression," and "fluctuation rates" of heartbeat data, are stored in the data type table 124.

[0127] Furthermore, the learning device 10 may receive and use a data set of learning data created by the learning data creation device 70, which will be described later, from the learning data creation device 70. In this case, the acquisition unit 131 receives the data set provided by the learning data creation device 70 via the network and stores it in the storage unit 12.

[0128] Returning to Fig. 6, the index value calculation unit 132 calculates emotion index values ​​representing arousal and activity based on the brainwave and heartbeat data of biological information. As described above, the emotion index value representing arousal can be calculated from "β waves / α waves of brainwaves". Moreover, the emotion index value representing activity can be calculated from "standard deviation of heartbeat LF component". Note that data such as calculation formulas required for calculating these emotion index values ​​are stored in the storage unit 12.

[0129] The behavior determination unit 133 performs an analysis process on image information including the face of the user U1 (see FIG. 5) to determine predetermined types of behavior of the user U1, such as the line of sight, face direction, facial expression, etc. Note that the process performed by the behavior determination unit 133 is equivalent to the process performed by the behavior determination unit 232 in FIG. 3, and each unit outputs data of the line of sight, face direction, and facial expression in the same format.

[0130] For example, regarding the line of sight of the user U1, the behavior determination unit 133 performs recognition processing such as feature calculation and shape discrimination using the left and right eyeballs of the user U1 as detection objects from an image including the face of the user U1. Based on the result of the recognition processing, the behavior determination unit 133 determines the behavior of the line of sight and the gaze point of the user U1 by a predetermined line of sight detection processing using, for example, the position of the inner corner of the eye, the center position of the iris and pupil of the eye, the center position of the corneal reflection image (Purkinje image) by near-infrared illumination, the center position of the eyeball, etc. The line of sight of the user U1 can be expressed, for example, as a two-dimensional coordinate of the position where the line of sight vector of the user U1 penetrates the virtual plane, which is set in front of the user U1 and directly facing the user U1.

[0131] Furthermore, for example, with regard to the direction of the face of user U1, the behavior determination unit 133 performs recognition processing such as feature calculation and shape determination using the face of user U1 as a detection target from an image including the face of user U1. Based on the result of the recognition processing, the behavior determination unit 133 determines the behavior of the direction of the face of user U1 by a predetermined face direction detection processing using, for example, the positions of the eyes, nose, mouth, etc., the position of the tip of the nose, the facial contour, the center position in the width direction of the facial contour, etc.

[0132] Furthermore, for example, with respect to the facial expression of user U1, the behavior determination unit 133 performs recognition processing such as feature calculation and shape discrimination with the face of user U1 as a detection target from an image including the face of user U1. Based on the result of the recognition processing, the behavior determination unit 133 determines the behavior of the facial expression of user U1 by a predetermined facial expression detection processing using, for example, the angle of the corners of the mouth, the angle of the eyebrows, the degree of opening of the eyes, and the like.

[0133] The generation unit 134 uses the gaze, facial direction, and facial expression appearance information of the user U1 determined by the behavior determination unit 133 in the same learning data as input values, and inputs the arousal level, which is the emotion index value calculated by the index value calculation unit 132, as a correct value into the first AI model 121ma, and learns the first AI model 121ma.

[0134] In addition, the generation unit 134 uses the gaze, facial direction, and facial expression appearance information of the user U1 determined by the behavior determination unit 133 in the same learning data as input values, and inputs the activity level, which is the emotion index value calculated by the index value calculation unit 132, as a correct answer value to the second AI model 121mb, and learns the second AI model 121mb.

[0135] In other words, the generation unit 134 generates a supervised learning dataset in which appearance information of the user U1 based on information detected by a remote sensor (non-contact sensor: camera) is used as input value data, and emotional index values ​​(arousal and activity) calculated based on the biosignals (brain waves and heart rate) of the user U1 detected by a contact-type biosignal sensor (contact sensor: brain wave sensor, heart rate sensor) are used as correct value data.

[0136] Then, the generation unit 134 uses the generated supervised learning data set to train the first AI model 121ma and the second AI model 121mb. Through this training, the trained first AI model 121mA and the trained second AI model 121mB become AI models that input appearance information of a user (driver) and output the alertness and activity of the user.

[0137] The generation unit 134 uses a learning algorithm such as an error backpropagation learning method to adjust parameters such as weights in the first AI model 121ma and the second AI model 121mb, thereby learning the first AI model 121ma and the second AI model 121mb.

[0138] The providing unit 135 provides the trained first AI model 121mA (data) and the trained second AI model 121mB (data) generated by the generating unit 134 to the emotion estimation device 20 (see FIGS. 1 and 3) used in the vehicle V2 described later via a network. As a result, the emotion estimation device 20 takes in the first AI model 121mA (data) and the second AI model 121mB (data) provided by the providing unit 135 and stores them in the first AI model storage unit 221A and the first AI model storage unit 221A. Therefore, the emotion estimation device 20 can estimate emotions based on image information acquired by the camera C by the emotion estimation function, and can use the estimated emotion information in various functions of the vehicle V2.

[0139] The providing unit 135 provides the trained first AI model 121mA (data) and the trained second AI model 121mB (data) generated by the generating unit 134 to the emotion estimation device 20 via a network, but it is also possible to provide them by another method. For example, it is also possible to provide the first AI model 121mA (data) and the second AI model 121mB (data) by writing the data of the first AI model 121mA and the data of the second AI model 121mB to an LSI forming an AI platform to generate an AI execution LSI and incorporating the AI ​​execution LSI into the emotion estimation device 20. It is also possible to provide the first AI model 121mA (data) and the second AI model 121mB (data) to the emotion estimation device 20 using a data transmission medium other than communication, such as a memory card or an optical disk recording medium.

[0140] <5. Example of operation of emotion estimation AI learning device 1> Fig. 11 is a flowchart showing the learning process of the emotion estimation AI executed by the controller 13 of the learning device 10 in Fig. 6. This flowchart shows the technical contents of a computer program that causes a computer device to realize the learning process of the emotion estimation AI model. In addition, the computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0141] The process shown in FIG. 11 is executed when the designer or the like of the learning device 10 executes the learning process of the emotion estimation AI model, for example, when a start operation of the learning process is performed by an operation unit such as a keyboard. If learning data sets during vehicle driving are to be collected and generated, the process is executed based on a collection start operation of the learning data set by an operator, a subject, or the like for the in-vehicle learning device.

[0142] In step S201, the controller 13 (acquisition unit 131) acquires the appearance information and biological signals of the user U1, specifically, the data of the camera captured image from the camera C, and also acquires biological signals (brain wave data and heart rate data) from the brain wave sensor ES and the heart rate sensor HS, and stores this information at substantially the same time as one data set in the storage unit 12, and then proceeds to step S202.

[0143] In the case of performing learning with the learning device 10 installed in a research and development room or the like using each data collected by the in-vehicle device, steps S201 and subsequent processes will be performed using each data collected and accumulated by the in-vehicle device provided to the learning device 10 via wireless communication or a recording medium.

[0144] In step S202, the controller 13 (behavior determination unit 133) determines the behavior (gaze, face orientation, expression) based on each data of the camera image acquired by the acquisition unit 131 in step S201, and then proceeds to step S203.

[0145] In step S203, the controller 13 (index value calculation unit 132) calculates the arousal level and activity level, which are emotion index values, based on the brain wave data and heart rate data of the biological signals acquired by the acquisition unit 131 in step S201, and then proceeds to step S204.

[0146] In step S204, the controller 13 (generation unit 134) creates supervised learning data for alertness learning, using the gaze, facial direction, and facial expression data of the behavior determined in step S202 as input values ​​and the alertness calculated in step S203 as the correct answer value, and then proceeds to step S205.

[0147] In step S205, the controller 13 (generation unit 134) creates supervised learning data for activity learning in which the gaze, facial direction, and facial expression data of the behavior determined in step S202 are used as input values ​​and the activity calculated in step S203 is used as the correct answer value, and then the process proceeds to step S206.

[0148] In step S206, the controller 13 (generation unit 134) provides the supervised learning data for alertness learning created in step S204 to the first AI model 121ma before completion of learning, to train the first AI model 121ma, and then proceeds to step S207.

[0149] In step S207, the controller 13 (generation unit 134) provides the supervised learning data for activity learning created in step S205 to the second AI model 121mb before completion of learning, to train the second AI model 121mb, and then proceeds to step S208.

[0150] In step S208, it is determined whether the learning of each model is completed, in this case whether the learning run of vehicle V1 is completed, for example, based on the ignition switch state or the operation of the vehicle driver (learning operator, test subject), and if completed, the processing ends; if not, the processing returns to step S201 and learning continues.

[0151] After that, the trained first AI model 121mA and the trained second AI model 121mB that have completed training are provided to an AI model utilization device such as the emotion estimation device 20 used in the vehicle V2 shown in Figure 1, based on instructions from an operator operating the learning device 10.

[0152] 11, each time data is collected from each sensor, supervised learning data is generated to train each AI model. However, each time data is collected from each sensor, supervised learning data may be created and accumulated, and the first AI model 121ma before learning is completed and the second AI model 121mb before learning is completed may be trained using a learning data set consisting of the accumulated learning data.

[0153] In this case, step S206 accumulates the supervised learning data for alertness learning generated in step S204. Step S207 accumulates the supervised learning data for activity learning generated in step S205. Then, step S208 judges whether the creation of the learning data is completed (for example, whether the learning run of the vehicle V1 is completed is judged based on, for example, the state of the ignition switch or the operation of the vehicle driver (learning operator, subject)), and if completed, the process ends, and if not, the process returns to step S201 and continues the generation process of the learning data.

[0154] After that, in a development / design room or the like, the first AI model 121ma and the second AI model 121mb will be trained using the accumulated training data set.

[0155] In addition, in the learning process of the above AI model, learning data is generated for each measurement data unit from each sensor. However, it is also possible to generate learning data using data obtained by statistically processing a plurality of data sets over an appropriate period (sensor output data, behavior data, or emotion index values) and use the data as learning data.

[0156] In addition, while the vehicle is running, biosignal data such as brain waves and heart rate, and external appearance information data such as images captured by a camera can be recorded, and the recorded data can be used as input data to generate learning data and train the first AI model 121ma and the second AI model 121mb using the learning device 10 in a development / design room, etc.

[0157] <6. Learning Data Creation System> Next, a learning data creation system that creates learning data for an AI model will be described with reference to FIG. 12. FIG. 12 is a block diagram showing the configuration of a learning data creation system 50. FIG. 12 shows components necessary for explaining the features of this embodiment, and general components are omitted. In this regard, the learning device 10 shown in FIG. 12 shows only components necessary for the explanation here compared to the learning device 10 shown in FIG. 6, and detailed explanations are omitted. Also, explanations of components with the same names as those already mentioned may be omitted.

[0158] 12, the learning data creation system 50 includes a server device 60 and a learning data creation device 70. The learning data creation system 50 may also include a learning device 10. The learning device 10, the server device 60, and the learning data creation device 70 are connected to each other via a communication network (not shown) so as to be able to communicate bidirectionally.

[0159] 6 and 10, the learning device 10 stores a data type table 124 in the storage unit 12. The data set of the data type table 124 includes a data type of bioinformation (emotion expression bioinformation) which is a pair of electroencephalogram data and heart rate data. In response to a request from the learning data creation device 70, the learning device 10 transmits the data set of the data type table 124 to the learning data creation device 70.

[0160] The server device 60 stores appearance information 621 of a plurality of people in the storage unit 62. The appearance information 621 is appearance information of a person when expressing an emotion (emotion expression appearance information) including image data including a person's face image and text data indicating an emotion type corresponding to the face image and its intensity (emotion intensity). That is, the appearance information 621 includes image data of known emotions that well express emotion types on the psychological plane shown in FIG. 2, such as joy, sadness, melancholy, relaxation, and unpleasantness. Appearance information 621 with known emotions can be easily associated with a data set in the data type table 124 that includes data of the same emotion type.

[0161] The appearance information 621 may be collected in advance from data such as still images and videos included in a publicly available AI data set or open source. The server device 60 transmits the appearance information 621 to the training data creation device 70 in response to a request from the training data creation device 70.

[0162] The training data creation device 70 includes a communication unit 71, a storage unit 72, and a controller 73. The training data creation device 70 can be configured as a so-called computer device. Although not shown in the figure, the training data creation device 70 includes input devices such as a keyboard and a touch panel, and an output device such as a display.

[0163] The storage unit 72 stores a data type table 721, appearance information 722, and a learning data set 723.

[0164] Data type table 721 is data type table 124 received from learning device 10. The data set of data type table 721 includes the data type of bioinformation, which is emotion expression bioinformation (electroencephalogram data, heart rate data) acquired by learning device 10 when a subject expresses emotion for creating learning data.

[0165] The appearance information 722 is the appearance information 621 of a plurality of people received from the server device 60. The appearance information 722 is emotion expression appearance information in which emotions are known at a timing different from the emotion expression biological information (electroencephalogram data, heart rate data) in the data type table 721.

[0166] The training data set 723 is a training data set including training data used for training the AI ​​model. The training data set 723 includes data sets of electroencephalogram data, heart rate data, and image data associated with each other by the controller 73, which are added as training data.

[0167] The controller 73 realizes various functions of the learning data creation device 70, and includes a processor that performs arithmetic processing, etc. The processor includes, for example, a CPU.

[0168] The controller 73 includes, as its functions, an acquisition unit 731, a behavior determination unit 732, a data extraction unit 733, a creation unit 734, and a provision unit 735. In this embodiment, the functions of the controller 73 are realized by a processor executing arithmetic processing in accordance with a program stored in the storage unit 72.

[0169] Acquiring unit 731 acquires (receives) a data set of data type table 124 from learning device 10 via communication unit 71. Acquiring unit 731 stores the acquired data set of the data type in data type table 721 formed in storage unit 72 as necessary for subsequent processing.

[0170] Furthermore, the acquiring unit 731 acquires (receives) appearance information 621 from the server device 60 via the communication unit 71. The acquiring unit 731 stores the acquired appearance information 621 in the storage unit 72 as appearance information 722 for subsequent processing as necessary. Note that the acquiring unit 731 may acquire (receive) appearance information including a person's face image and text data of emotion types from a server device or the like external to the learning data creation system 50 via a communication network.

[0171] The behavior determination unit 732 performs an analysis process on image data including a person's face image of the appearance information 722. As a result, the behavior determination unit 732 determines a predetermined type of behavior such as the line of sight, face direction, and expression of the person in the image data. Furthermore, the behavior determination unit 732 determines the intensity of the emotion expressed by the face image with respect to the determined behavior.

[0172] Data extraction unit 733 extracts data to be associated from data type table 721 and appearance information 722 acquired by acquisition unit 731 and stored in storage unit 72. Specifically, data extraction unit 733 extracts a data type of biometric information, which is a pair of electroencephalogram data and heart rate data, having the same emotion type and emotion intensity, and image data of a face image. Note that, when there are multiple data types of biometric information having the same emotion type and emotion intensity in data type table 721, they are extracted randomly so that no regularity (regularity) occurs in the series of work processes for creating learning data.

[0173] The creation unit 734 associates the electroencephalogram data, heartbeat data, and image data extracted by the data extraction unit 733 as one data set, and creates learning data. The creation unit 734 stores the created data set of the learning data in the storage unit 72 as the learning data set 723.

[0174] The providing unit 735 provides the data of the learning dataset 723 created by the creating unit 734 to the learning device 10 via the network. As a result, the learning device 10 receives the data of the learning dataset 723 provided by the providing unit 735 and stores it in the storage unit 12. Therefore, the learning device 10 can use the learning data created by the learning data creating device 70 for the AI ​​model installed therein, and the AI ​​model can be trained using the learning data.

[0175] <7. Example of operation of the learning data creation device> Fig. 13 is a flowchart showing the learning data creation process executed by the controller 73 of the learning data creation device 70 in Fig. 12. This flowchart shows the technical contents of a computer program that causes a computer device to realize the process of creating learning data for an AI model. The computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0176] The process shown in FIG. 13 is executed when a designer or the like of the learning device 10 executes a process of creating learning data for an AI model, and an operation to start the creation process is performed via an operation unit such as a keyboard.

[0177] In step S301, the controller 73 (acquisition unit 731) acquires a data set of the data type of biological information (electroencephalogram data, heart rate data) from the learning device 10, stores it in the storage unit 72, and proceeds to step S302.

[0178] In step S302, the controller 73 (acquisition unit 731) acquires appearance information including a person's face image and emotion type data from the server device 60, stores it in the storage unit 72, and then proceeds to step S303.

[0179] In step S303, the controller 73 (behavior determination unit 732) determines the behavior of the person based on the image data including the face image of the person acquired by the acquisition unit 731 in step S302, and further determines the emotion intensity represented by the face image, and then proceeds to step S304.

[0180] In step S304, the controller 73 (data extraction unit 733) extracts the data type of biometric information, which is a pair of electroencephalogram data and heart rate data, and the image data of a facial image, which have the same emotion type and emotion intensity, from the data type table 721 and the appearance information 722 stored in the storage unit 72, and proceeds to step S305.

[0181] If at least one of the intensities of the electroencephalogram data and the heart rate data in data type table 721 is "strong", the emotion intensity is set to "strong". If both the intensities of the electroencephalogram data and the heart rate data are "weak", the emotion intensity is set to "weak".

[0182] In step S305, the controller 73 (creation unit 734) associates the electroencephalogram data, heartbeat data, and image data extracted by the data extraction unit 733 in step S304 as one data set, creates learning data, and proceeds to step S306.

[0183] In step S306, it is determined whether the creation of the learning data is complete, and here whether there is any remaining biological information (electroencephalogram data, heart rate data) and appearance information (image data) that can be used as learning data, based on the operation of the operator operating the learning data creation device 70, and if it is complete, the processing is terminated. If it is not complete, the process returns to step S301 and the creation of the learning data is continued.

[0184] Thereafter, the created learning data set 723 is provided to the learning device 10 shown in FIG. 6 and the like, based on instructions from an operator who operates the learning data creation device 70 and the like.

[0185] As described above, a large amount of learning data is required for learning an AI model in which a person's facial expression (appearance information) is associated with biological signals (biological information) such as brain waves and heartbeats. In response to this, the learning data creation device 70 acquires emotion expression biological information (data type table 721 (brain wave data, heartbeat data)), which is biological information when a subject expresses emotion, for creating learning data, and further acquires emotion expression appearance information (appearance information 722 (image data)), which is appearance information with a known emotion, at a timing different from the emotion expression biological information. Then, the learning data creation device 70 associates emotion expression biological information (brain wave data, heartbeat data) and emotion expression appearance information (image data) that are the same emotion to create learning data (learning data set 723).

[0186] The data type table 721 (brain wave data, heart rate data) of the learning data creation device 70 is the data type table 124 of the learning device 10, and is composed of brain wave data and heart rate data generated in the process of learning an AI model in the learning device 10. On the other hand, the appearance information 722 (image data) of the learning data creation device 70 is the appearance information 621 of the server device 60, and is composed of data such as still images and videos in a generally public AI data set, etc. In other words, the brain wave data and heart rate data, and the image data are acquired (measured, photographed) at different times.

[0187] The learning data creation device 70 associates biosignals (biological information) of brain waves and heartbeats that are acquired (measured, photographed) for the same emotion but at different times with image data (appearance information) that records a person's facial expression to create learning data. This allows the biosignals (biological information) of brain waves and heartbeats, which require the subject to wear contact sensors such as brain wave sensors and heartbeat sensors during measurement, to be associated with a large amount of image data (appearance information). As a result, it becomes possible to create a large amount of learning data required for training the AI ​​model in a short period of time. By learning using a large amount of learning data, it is expected that the accuracy of emotion estimation by the AI ​​model will improve.

[0188] In addition, if an AI model is trained using a large number of training data in which the same electroencephalogram and heart rate biosignals are associated with a large number of image data, over-training may occur. For this reason, it is preferable that a large number of data sets are stored in the data type table 124 shown in Fig. 10, which have the same "emotion quadrant", "emotion type", and "strength" of data, but have different "initial value", "after emotion expression", and "fluctuation speed" of electroencephalogram data, and different "initial value", "after emotion expression", and "fluctuation speed" of heart rate data.

[0189] As described above, the brainwave data and heartbeat data (emotion expression bioinformation) when the subject expresses emotion include a fluctuation speed (signal fluctuation information) that indicates the fluctuation state of the biosignal. The fluctuation speed (signal fluctuation information) is information indicating the intensity of emotion. The fluctuation speed of the biosignal is related to the strength of the emotion intensity. Since the signal level (absolute value) of the brainwave and heartbeat varies due to individual differences, surrounding environment, etc., by focusing on the fluctuation speed of the biosignal, it becomes possible to commonly use data that has not been calibrated and has individual differences, etc. In other words, it becomes easier to associate the brainwave and heartbeat biosignals (biological information) with image data (appearance information) that records a person's facial expression, and it is possible to associate a larger number of data.

[0190] <8. Example of operation of emotion estimation AI learning device 2> Fig. 14 is a flowchart showing the learning process of the emotion estimation AI executed by the controller 13 of the learning device 10 in Fig. 12. This flowchart shows the technical contents of a computer program that causes a computer device to realize the learning process of the emotion estimation AI model. In addition, the computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0191] The process shown in FIG. 14 is executed when a designer or the like of the learning device 10 in FIG. 12 executes a learning process of the emotion estimation AI model, and an operation to start the learning process is performed via an operation unit such as a keyboard.

[0192] In step S401, the controller 13 (acquisition unit 131) acquires the data set of learning data (emotion expression biological information (electroencephalogram data, heart rate data) and emotion expression appearance information (image data)) created and provided by the learning data creation device 70 from the storage unit 12, and proceeds to step S402.

[0193] In step S402, the controller 13 (behavior determination unit 133) determines the behavior (gaze, facial direction, and facial expression) based on the appearance information (image data) acquired by the acquisition unit 131 in step S401, and proceeds to step S403.

[0194] In step S403, the controller 13 (index value calculation section 132) calculates the arousal level and activity level, which are emotion index values, based on the biological information (electroencephalogram data, heart rate data) acquired by the acquisition section 131 in step S401, and proceeds to step S404.

[0195] In step S404, the controller 13 (generation unit 134) creates supervised learning data for alertness learning, using the gaze, facial direction, and facial expression data of the behavior determined in step S402 as input values ​​and the alertness calculated in step S403 as the correct answer value, and then proceeds to step S405.

[0196] In step S405, the controller 13 (generation unit 134) creates supervised learning data for activity learning in which the gaze, facial direction, and facial expression data of the behavior determined in step S402 are used as input values ​​and the activity calculated in step S403 is used as the correct answer value, and then the process proceeds to step S406.

[0197] In step S406, the controller 13 (generation unit 134) provides the supervised learning data for alertness learning created in step S404 to the first AI model 121ma before completion of learning, to train the first AI model 121ma, and then proceeds to step S407.

[0198] In step S407, the controller 13 (generation unit 134) provides the supervised learning data for activity learning created in step S405 to the second AI model 121mb before completion of learning, to train the second AI model 121mb, and then proceeds to step S408.

[0199] In step S408, it is determined whether learning of each model is complete, and here whether a data set (emotion expression biological information (electroencephalogram data, heart rate data) and emotion expression appearance information (image data)) that can be used as learning data remains, based on the operation of the operator operating the learning device 10, and if it is complete, the processing ends; if it is not complete, the process returns to step S301 and continues creating the learning data.

[0200] After that, the trained first AI model 121mA and the trained second AI model 121mB that have completed training are provided to an AI model utilization device such as the emotion estimation device 20 used in the vehicle V2 shown in Figure 1, based on instructions from an operator operating the learning device 10.

[0201] As described above, the learning device 10 uses learning data created from emotion-expression biometric information (electroencephalogram data, heart rate data) and emotion-expression appearance information (image data) that are acquired at different times and associated with each other to generate supervised learning data in which emotion-expression appearance information is used as an input value and an emotion index value calculated from the emotion-expression biometric information is used as a correct answer value, and learns an AI model. Since the time required for creating the learning data is shortened, the time required for learning the AI ​​model can be shortened. This allows the learning of the AI ​​model to be completed quickly. In addition, since a large amount of learning data is created, the accuracy of the AI ​​model can be improved by learning using the large amount of learning data, making it possible to estimate emotions well.

[0202] Moreover, the AI ​​model trained using the training data created as described above is an arousal level estimation AI model (first AI model 121ma) that estimates a central nervous system arousal level, which is an emotional index value calculated based on an electroencephalogram biosignal. Since the time required for creating training data related to electroencephalogram data (arousal level) is shortened, the time required for training the arousal level estimation AI model can be shortened. This allows the training of the arousal level estimation AI model to be completed quickly. Furthermore, since a large amount of training data related to electroencephalogram data (arousal level) is created, the accuracy of the arousal level estimation AI model can be improved by training using the large amount of training data, and it becomes possible to properly estimate emotions from arousal levels.

[0203] Moreover, the AI ​​model trained using the training data created as described above is an activity estimation AI model (first AI model 121mb) that estimates the autonomic nervous system activity, which is an emotion index value calculated based on the heart rate biosignal. Since the time required for creating training data related to heart rate data (activity) is shortened, the time required for training the activity estimation AI model can be shortened. This allows the training of the activity estimation AI model to be completed quickly. Furthermore, since a large amount of training data related to heart rate data (activity) is created, the accuracy of the activity estimation AI model can be improved by training using the large amount of training data, and it becomes possible to properly estimate emotions from activity.

[0204] Moreover, according to the above configuration, the first AI model 121mA and the second AI model 121mB of the emotion estimation device 20 (see FIG. 1 ) are trained AI models that have completed training using a large amount of training data created in a short period of time by the training data creation device 70. Since the training period for the AI ​​model is shortened, the emotion estimation device 20 can be quickly provided to vehicles and the like. Furthermore, since the accuracy of the AI ​​model is improved by training using a large amount of training data, the emotion estimation device 20 can estimate emotions well.

[0205] <9. Points to note> Various technical features disclosed as embodiments in this specification can be modified in various ways without departing from the spirit of the technical creation. In other words, the above embodiments are illustrative in all respects and are not restrictive. The technical scope of the present invention is indicated by the claims, not the description of the above embodiments, and includes all modifications that fall within the meaning and scope of the claims. In addition, the multiple embodiments shown in this specification may be combined as appropriate to the extent possible.

[0206] In the above embodiment, various functions are realized by software through the arithmetic processing of the CPU according to the program, but at least some of these functions may be realized by electrical hardware resources. Examples of hardware resources include ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays). Conversely, at least some of the functions realized by hardware resources may be realized by software.

[0207] Also, it may include a computer program that causes a processor (computer) to realize at least a part of the functions of the learning device 10, the emotion estimation device 20, and the learning data creation device 70. Such a computer program can be stored in a computer-readable nonvolatile recording medium (for example, in addition to the above-mentioned nonvolatile memory, an optical recording medium (for example, an optical disk), a magneto-optical recording medium (for example, a magneto-optical disk), a USB memory, or an SD card, etc.) and provided (sold, etc.), and can also be provided by a method of providing the program from a server device via a communication line such as the Internet, that is, by downloading. [Explanation of symbols]

[0208] 10 Learning Device 12 Storage section 13 Controller 20 Emotion estimation device 22 Memory section 23 Controller 30 Vehicle control device 40 Vehicle Control System 50 Learning Data Creation System 60 Server equipment 62 Storage section 70 Learning data creation device 72 Memory section 73 Controller 121mA, 121ma 1st AI model 121mB, 121mb 2nd AI model 124 Data Type Table 131 Acquisition Department 132 Index value calculation unit 133 Behavior Judgment Unit 134 Generation part 135 Provision Department 222m Emotion estimation model 231 Acquisition Department 232 Behavior Judgment Unit 233 Emotion estimation part 234 Provision Department 621 Appearance Information 721 Data Type Table 722 Appearance Information 723 Training Data Set 731 Acquisition Department 732 Behavior Judgment Unit 733 Data Extraction Department 734 Creation Department 735 Provision Department C Camera ES Brainwave Sensor HS Heart Rate Sensor U1, U2 users

Claims

1. A learning data creation method for creating learning data for an AI model that estimates biological information related to an emotion of a subject based on appearance information of the subject, comprising: acquiring emotion expression biometric information, which is the biometric information, when the subject expresses the emotion for creating the learning data; Acquire emotion-expressing appearance information, the emotion being the known appearance information; creating the learning data by associating the emotion-expression biometric information and the emotion-expression appearance information, which are the same emotion; How to create training data.

2. The emotion expression biological information is The biometric information; Emotional intensity and Including, The emotional expression appearance information is Image data including a human face image; text data representing an emotion type and an emotion intensity corresponding to the facial image; Including, The training data generating method according to claim 1 .

3. the emotion expression biometric information is generated by acquiring the appearance information and the biometric information of the subject at the same time; The training data generating method according to claim 1 .

4. The emotion expression biological information includes signal variation information representing a variation state of a biological signal, and the signal variation information is information indicating emotion intensity. The training data generating method according to claim 3 .

5. The biological information is The ratio of beta waves to alpha waves in the brainwaves, which are biological signals, The standard deviation of the low frequency components of the heart rate waveform signal of the heart rate, which is a biological signal, Including, The training data generating method according to claim 3 .

6. The AI ​​model is an arousal level estimation AI model that estimates a central nervous system arousal level, which is an emotion index value calculated based on a biosignal of an electroencephalogram, as the bioinformation to be estimated, and calculates the central nervous system arousal level based on a ratio of beta waves to alpha waves in the electroencephalogram. The training data generating method according to claim 1 .

7. The AI ​​model is an activity estimation AI model that estimates an autonomic nervous system activity level, which is an emotion index value calculated based on a heartbeat biosignal, as the bioinformation to be estimated, and calculates the autonomic nervous system activity level based on a standard deviation of a low-frequency component of a heartbeat waveform signal at the heartbeat. The training data generating method according to claim 1 .

8. A learning data creation program for creating learning data for an AI model that estimates biological information related to an emotion of a subject based on appearance information of the subject, the learning data creation program comprising: acquiring emotion expression biometric information, which is the biometric information, when the subject expresses the emotion for creating the learning data; Acquire emotion-expressing appearance information, the emotion being the known appearance information; causing a computer to execute a method of creating the learning data by associating the emotion-expression biometric information and the emotion-expression appearance information, which are the same emotion; A program for creating learning data.

9. A learning data creation device that creates learning data for an AI model that estimates biological information related to an emotion of a subject based on appearance information of the subject, the learning data creation device comprising: acquiring emotion expression biometric information, which is the biometric information, when the subject expresses the emotion for creating the learning data; Acquire emotion-expressing appearance information, the emotion being the known appearance information; creating the learning data by associating the emotion-expression biometric information and the emotion-expression appearance information, which are the same emotion; Training data creation device.

10. A learning data creation system for creating learning data for an AI model that estimates biological information related to an emotion of a subject based on appearance information of the subject, comprising: A server device and a learning data creation device, The server device includes: storing emotion-expressing appearance information including image data including a human face image and text data indicating an emotion type and emotion intensity corresponding to the face image, the emotion being the known appearance information; The learning data creation device includes: acquiring emotion expression biometric information, which is the biometric information, when the subject expresses the emotion for creating the learning data; acquiring the emotion expression appearance information from the server device; creating the learning data by associating the emotion-expression biometric information and the emotion-expression appearance information, which are the same emotion; Learning data creation system.

11. A learning method for an AI model that estimates biological information related to an emotion of a subject based on appearance information of the subject, comprising: Acquire learning data created by associating emotion expression biometric information, which is the biometric information, with emotion expression appearance information, which is the appearance information for the same emotion and for which the emotion is known, when the subject expresses the emotion for creating the learning data; Calculating an emotion index value based on a bio-signal for the emotion expression bio-information of the created learning data; generating supervised learning data in which the emotion expression appearance information is used as an input value and the emotion index value is used as a correct answer value; Training the AI ​​model with the supervised training data; How to learn.

12. A learning device for an AI model that estimates biological information related to an emotion of a subject based on appearance information of the subject, Acquire learning data created by associating emotion expression biometric information, which is the biometric information, with emotion expression appearance information, which is the appearance information for the same emotion and for which the emotion is known, when the subject expresses the emotion for creating the learning data; Calculating an emotion index value based on a bio-signal for the emotion expression bio-information of the created learning data; generating supervised learning data in which the emotion expression appearance information is used as an input value and the emotion index value is used as a correct answer value; Training the AI ​​model with the supervised training data; Learning device.

13. An emotion estimation device that estimates an emotion of a subject of emotion estimation, comprising: An AI model that estimates biological information related to the subject's emotions based on the subject's appearance information; an emotion estimation model that estimates the emotion based on the estimated biological information; Equipped with The AI ​​model is Acquire learning data created by associating emotion expression biometric information, which is the biometric information, with emotion expression appearance information, which is the appearance information for the same emotion and for which the emotion is known, when the subject expresses the emotion for creating the learning data; Calculating an emotion index value based on a bio-signal for the emotion expression bio-information of the created learning data; generating supervised learning data in which the emotion expression appearance information is used as an input value and the emotion index value is used as a correct answer value; The supervised learning data is generated by learning the Emotion estimation device.

Images