Learning data generation method, learning method, learning device, learning program, and emotion estimation device

Abstract

To form an AI model for which learning is performed by learning data in a state that emotions are expressed stably and continuously, and to improve accuracy in estimating emotions.SOLUTION: A learning data generation method is a method for generating learning data for an AI model for estimating biological information related to emotions of an emotion estimation target person on the basis of appearance information on the emotion estimation target person, acquires the appearance information and a biological signal on a subject for generating learning data at the same timing, estimates a variation state of the biological signal on the basis of the acquired biological signal, and extracts the appearance information and the biological signal used for generating the learning data according to the estimated variation state of the biological signal.SELECTED DRAWING: Figure 14A

Description

[Technical field]

[0001] The present invention relates to a learning data generation method, a learning method, a learning device, a learning program, and an emotion estimation device related to an emotion estimation AI. [Background technology]

[0002] 2. Description of the Related Art Conventionally, an emotion estimation system that estimates emotion based on biological signals (for example, heart rate and brain waves) is known (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-185138 A Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, as artificial intelligence (AI) becomes more sophisticated and widespread, it is conceivable that AI can also be applied to emotion estimation devices. In order to create and improve the accuracy of an AI model used in emotion estimation devices, it is necessary to sample various data related to emotion estimation, such as biosignals, from many subjects, create supervised learning data based on this sampling data, and train a pre-learning AI model using the data.

[0005] However, a person's mental and physical state may become unstable due to some kind of disturbance, even if the person has the same emotional state. If various data related to emotion estimation, such as biosignals, in such an unstable state is used to generate training data, the performance (reliability) of the generated supervised training data decreases, and the performance of the AI ​​model trained using the training data decreases, which is an issue.

[0006] In view of the above problems, the present invention aims to provide a technology that can form an AI model trained using highly reliable training data, thereby improving the accuracy of emotion estimation. [Means for solving the problem]

[0007] An exemplary training data generation method of the present invention is a method for creating training data for an AI model that estimates bioinformation related to the emotions of a subject of emotion estimation based on appearance information of the subject, the method comprising the steps of: acquiring the appearance information and biosignals of a subject for creating training data at the same time; estimating a fluctuation state of the biosignals based on the acquired biosignals; and extracting the appearance information and the biosignals to be used for creating the training data in accordance with the estimated fluctuation state of the biosignals. Effect of the Invention

[0008] According to the present invention, learning data for AI model training can be generated based on emotion-related biological information in a state where the biological information is appropriate for use as learning data, for example, a stable and continuous state. Therefore, the AI ​​model is trained using various data such as highly reliable biological signals. This generates a highly accurate trained AI model, and as a result, it is expected that the accuracy of emotion estimation using the AI ​​model will be improved. [Brief description of the drawings]

[0009] [Figure 1] A conceptual explanatory diagram showing emotion estimation processing by an emotion estimation device. [Diagram 2] A diagram showing an example of a multi-dimensional (2-dimensional) model (mental plane) for emotion estimation. [Diagram 3] A diagram showing an example of a psychological plane including a neutral region. [Figure 4] Block diagram showing the configuration of a vehicle control system [Diagram 5] A flowchart showing an emotion estimation process executed by a controller of the emotion estimation device in FIG. [Figure 6]Conceptual diagram showing the learning process of emotion estimation AI using a learning device [Figure 7] Block diagram showing the configuration of a learning device [Figure 8A] Schematic diagram showing the time change of the subject's emotion index value [Figure 8B] A schematic diagram showing the time change of emotion index values ​​including the subject's neutral region. [Figure 9] A diagram showing an example of a task table. [Figure 10] A schematic diagram showing the time change in arousal level during task execution to estimate the neutral region of arousal level. [Figure 11] Schematic diagram showing the change in emotion index value (arousal level) over time [Figure 12] A graph showing the convergence of the upper limit estimate of the neutral region over the data accumulation period. [Figure 13] FIG. 13 is a diagram showing an example of a neutral region table. [Figure 14A] Schematic diagram showing the time change of the subjects' quantized emotion index values [Figure 14B] A schematic diagram showing the time change of the quantized emotion index value including the subject's neutral region. [Figure 15] A flowchart showing the learning process of the emotion estimation AI executed by the controller of the learning device of FIG. [Figure 16] A conceptual explanatory diagram showing emotion estimation processing by an emotion estimation device. [Figure 17] Conceptual diagram showing the learning process of emotion estimation AI using a learning device DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the contents of the embodiments described below.

[0011] <1. Conceptual structure of emotion estimation device> First, the feeling estimation device 20 will be described.

[0012] The emotion estimation model 222m of the emotion estimation device 20 estimates emotions based on two emotion index values, which are indexes indicating mental and physical states related to emotions. One of the emotion index values ​​used in this embodiment is the central nervous system arousal level (hereinafter referred to as arousal level), and its index value can be calculated using "β waves / α waves of electroencephalograms." The other emotion index value is the autonomic nervous system activity level (hereinafter referred to as activity level), and its index value can be calculated using the "standard deviation of heartbeat LF (Low Frequency) components (low frequency components of heartbeat waveform signals)."

[0013] The emotion estimation model 222m is configured by a model (calculation formula or conversion data table) for emotion estimation based on arousal and activity. The emotion estimation model 222m is configured by a multidimensional model (here, a two-dimensional model with arousal and activity as two axes) that estimates emotion using arousal and activity as parameters. The two-dimensional model is created based on medical evidence (papers, etc.) showing the relationship between each of a plurality of indexes and emotion (relationship between arousal and activity and emotion). Alternatively, the two-dimensional model is created based on the results of a questionnaire by many subjects (data consisting of emotion declaration by subjects and arousal and activity at that time (based on electroencephalogram and heart rate measurement values)). The emotion estimation model 222m can be not only a two-dimensional model, but also a multidimensional model of three or more dimensions.

[0014] FIG. 2 is a diagram showing an example of a multi-dimensional (two-dimensional) model (psychological plane) for emotion estimation. According to various medical evidence related to psychology, psychology can be estimated based on two types of indicators (emotion index values) that indicate physical conditions. In the psychological plane shown in FIG. 2, the vertical axis is "arousal level (arousal-unaroused)" and the horizontal axis is "activity level of the autonomic nervous system (sympathetic nervous activity (strong emotion)-parasympathetic nervous activity (weak emotion)").

[0015] In this psychological plane, each of the four quadrants separated by the vertical and horizontal axes is assigned a corresponding emotion (type). The distance from each axis indicates the intensity of the corresponding emotion. The first quadrant is assigned to the emotions of "fun, joy, anger, and sadness." The second quadrant is assigned to the emotion of "melancholy." The third quadrant is assigned to the emotions of "relaxation and calm." And the fourth quadrant is assigned to the emotions of "anxiety, fear, and discomfort."

[0016] The position of the axis is appropriately set based on an experiment, for example, by measuring the degree of arousal and activity of the subject and performing statistical processing. In addition, in the method of determining a neutral region (see Figs. 10 and 11) described later, a method of using the center of the determined neutral region as the axis, and in the quantization method (see Figs. 8A and 8B) described later, a method of measuring the range of variation of the emotion index value after quantization for many subjects and using the average of the median values ​​of the measured range of variation as the axis are possible.

[0017] Then, two types of emotion index values ​​(arousal and activity) obtained based on the biosignal are plotted on a psychological plane to obtain coordinates, from which emotion can be estimated. Specifically, emotion and its intensity can be estimated based on which quadrant of the psychological plane the plotted coordinates are in, where they are located within the quadrant, and how far they are from the origin. Note that the emotion estimation model shown in Figure 2 is a two-dimensional plane, but it can become a multidimensional space of three or more dimensions depending on the number of indices used.

[0018] Since estimating emotion intensity is difficult and involves relatively large errors, and its applications are limited, a realistic and useful method is to determine the emotion based on which quadrant of the psychological plane the emotion index value is in and to use that emotion.

[0019] For this reason, in the emotion estimation device 20 shown in Fig. 1, the emotion estimation model 222m is configured with a binary value (two values ​​with the axis as a boundary) for the vertical axis "arousal level" and a binary value for the horizontal axis "activity level". The inputs of the emotion estimation model 222m are the quantized arousal level and the quantized activity level, each of which can take two values.

[0020] Furthermore, when the emotion intensity is strong, that is, when the emotion index value swings significantly toward the maximum or minimum value, the emotion estimation accuracy is high. However, when the emotion intensity is weak, that is, when the emotion index value is near the median, the emotion estimation accuracy is low. For this reason, a method can be considered in which the area near the median of the emotion index value is considered to be a neutral area, and then it is determined that there is no estimated emotion, or that emotion estimation is impossible.

[0021] 3 is a diagram showing an example of a psychological plane including a neutral region, in which hatched regions Rn1 and Rn2 are neutral regions.

[0022] The upper and lower limits of the neutral region for the emotion index of "arousal" are YP and YN, and the region between the upper limit YP and the lower limit YN is the arousal neutral region Rn1 for "arousal". The upper and lower limits of the neutral region for the emotion index of "activity" are XP and XN, and the region between the upper limit XP and the lower limit XN is the activity neutral region Rn2 for "activity".

[0023] The neutral regions (upper limit YP, lower limit YN, upper limit XP, and lower limit XN) can be appropriately set by experiment, etc. In addition, since appropriate values ​​for setting the neutral region are calculated when the first AI model 121mA and the second AI model 121mB are learned, the calculated values ​​can also be applied. Specific examples of methods for determining the neutral region will be described later (see FIG. 10 and FIG. 11).

[0024] In this case, in the emotion estimation device 20 shown in Fig. 1, the emotion estimation model 222m is configured with three values ​​(three values ​​bounded by the upper and lower boundaries of the neutral region) on the vertical axis "arousal level" and three values ​​on the horizontal axis "activity level". The inputs of the emotion estimation model 222m are the quantized arousal level and the quantized activity level, each of which can take three values.

[0025] The emotion estimation device 20 estimates emotions by fitting (plotting as coordinates) the input emotion indicators “arousal level” and “activity level” to the emotion estimation model 222m shown on such a psychological plane.

[0026] The emotion estimation model 222m of the emotion estimation device 20 uses the arousal and activity of the emotion index calculated based on the electroencephalogram data and the heartbeat data, and therefore receives the electroencephalogram data and the heartbeat data as input. Note that in this embodiment, the emotion estimation model 222m receives the arousal and activity of the emotion index estimated by an AI model from the gaze, face direction, and voice, rather than the electroencephalogram data and the heartbeat data detected by the electroencephalogram sensor and the heartbeat sensor.

[0027] This technology eliminates the need to equip the subject of emotion estimation with contact sensors such as brainwave and heart rate sensors. This technology uses non-contact sensors such as cameras and microphones to collect gaze, face direction, and voice data (appearance information), and inputs this data into an AI model to estimate brainwave and heart rate data. For example, when estimating the emotions of a vehicle driver and using them for vehicle driving control, it is practically difficult to equip the driver with contact sensors, making this technology particularly useful.

[0028] In other words, calculation of emotion indices usually requires biosignals, but most biosignals are obtained from contact-type sensors, which limits their application. For this reason, it is desirable to use sensors that detect information based on the appearance of the emotion estimation subject (subject) that can be detected by non-contact sensors. Note that appearance information here includes not only image data but also information that can be detected by non-contact sensors such as voice (remote detection sensors: cameras, microphones, etc.).

[0029] In the use example of the feeling estimation device 20 shown in FIG. 1, a user U2 (emotion estimation target person) is a driver of a vehicle V2. The first AI model 121mA and the second AI model 121mB used in the estimation method are used to estimate the emotion of the driver (user U2). The first AI model 121mA and the second AI model 121mB are separately trained, provided as trained models, and loaded into the feeling estimation device 20. Details of the feeling estimation device 20 will be described later.

[0030] The feeling estimation device 20 can be realized by a computer device mounted on the vehicle V2, or by a server connected to the vehicle V2 via a network. The server may be a physical server or a virtual server.

[0031] The vehicle V2 is equipped with a camera C and a microphone M as on-board sensors. Then, gaze data and facial direction data of a user U2 (driver) based on an image captured by the camera C, and voice data of the user U2 (driver) acquired by the microphone M are input to a first AI model 121mA and a second AI model 121mB.

[0032] In the illustrated example, the image captured by the camera C is processed by image recognition processing or the like to be processed into gaze data (image of the eyeball or data processed into text and numerical data indicating gaze (direction)) and facial direction data (image of the face or data processed into text and numerical data indicating the facial direction) and input to the first AI model 121mA and the second AI model 121mB, but the image captured by the camera C may be input. The first AI model 121mA and the second AI model 121mB (AI models designed and trained according to the input data type) corresponding to the input data type are mounted on the emotion estimation device 20.

[0033] Since gaze and facial movement are important in emotion estimation, gaze data and facial movement data are preferably video data for a period that matches the emotion estimation timing, for example, a video of a predetermined length just before the emotion estimation timing. Similarly, audio data is preferably a video of a predetermined length just before the emotion estimation timing. For example, when the emotion estimation time length is 10 seconds, at the emotion estimation timing Ta, gaze data, facial movement data, and audio data based on facial video data from time Ta-10 seconds (start point) to time Ta (end point) are input to the first AI model 121mA and the second AI model 121mB. The first AI model 121mA and the second AI model 121mB are trained using learning input data in this data format.

[0034] The first AI model 121mA is an AI model that inputs gaze data, face direction data, and voice data of the user U2, and outputs an emotion index value of arousal level, and is designed to have a structure suitable for the input and output. The first AI model 121mA is trained with a large number of learning data sets in which gaze data, face direction data, and voice data are input data, and the arousal level corresponding to the input data is used as correct answer data.

[0035] The correct answer data is quantized data of the arousal level (a value corresponding to the arousal level input in the emotion estimation model 222m (a binary or ternary value (if there is a neutral region))). The first AI model 121mA is configured to output the quantized data of the arousal level.

[0036] The second AI model 121mB is an AI model that inputs gaze data, face direction data, and voice data of the user U2, and outputs an emotion index value of activity, and is designed to have a structure suitable for the input and output. The second AI model 121mB is trained with a large number of learning data sets in which gaze data, face direction data, and voice data are input data, and activity corresponding to the input data is used as correct answer data.

[0037] The correct answer data is quantized data of the activity (a value corresponding to the activity input in the emotion estimation model 222m (binary or ternary (if there is a neutral region))). The second AI model 121mB outputs the quantized data of the activity.

[0038] The learning method of the first AI model 121mA and the second AI model 121mB will be described later in detail.

[0039] The first AI model 121mA and the second AI model 121mB output quantized data of two types of emotion indexes, arousal level and activity level, estimated (output) based on the gaze data, face direction data, and voice data of the user U2 to the emotion estimation model 222m. The emotion estimation model 222m estimates an emotion based on the input quantized arousal level and quantized activity level.

[0040] <2. Vehicle control system> Next, a vehicle control system that uses estimated emotion data to control a vehicle will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of a vehicle control system 40. Fig. 4 shows components necessary for explaining the features of this embodiment, and omits the description of general components.

[0041] 4, the vehicle control system 40 includes the feeling estimation device 20, the vehicle control device 30, an actuator unit 41, and a notification unit 42. Although not shown in the figure, the vehicle control system 40 includes input devices such as a keyboard and a touch panel, and an output device such as a display.

[0042] <2-1. Emotion estimation device> The feeling estimation device 20 includes a communication unit 21, a storage unit 22, and a controller 23. The feeling estimation device 20 is generally required to have quick response in control, and is therefore mounted on the vehicle V2 as in this example, but may also be configured as a server connected to the vehicle V2 via a network.

[0043] The storage unit 22 is provided with a first AI model storage unit 221A and a second AI model storage unit 221B that store the trained first AI model 121mA and the trained second AI model 121mB shown in FIG.

[0044] The first AI model 121mA and the second AI model 121mB are stored in the first AI model storage unit 221A and the second AI model storage unit 221B, for example, in the following manner.

[0045] The creator of the vehicle control system 40, when assembling the system, etc., installs memories in which the trained first AI model 121mA and the second AI model 121mB are written as the first AI model storage unit 221A and the second AI model storage unit 221B. Alternatively, the creator of the vehicle control system 40 connects an external storage device in which the trained first AI model 121mA and the second AI model 121mB are stored to the vehicle control system 40 (emotion estimation device 20), reads the first AI model 121mA and the second AI model 121mB, and writes them to the first AI model storage unit 221A and the second AI model storage unit 221B. Alternatively, the creator of the vehicle control system 40 receives the trained first AI model 121mA and the second AI model 121mB from the learning device 10 via a network, and writes them to the first AI model storage unit 221A and the second AI model storage unit 221B.

[0046] The storage unit 22 is also provided with an emotion estimation model storage unit 222 that stores the emotion estimation model 222m shown in Fig. 1. The emotion estimation model storage unit 222 stores various data necessary for emotion estimation, such as a psychological plane table 224 that is data for estimating emotions from emotion index values ​​shown in Fig. 2 or 3.

[0047] The psychological plane table 224 is data forming the emotion estimation model (psychological plane) shown in Fig. 2 or 3, and is a data group that associates two types of quantized emotion index values ​​(quantized arousal and quantized activity, which are quantized values ​​of arousal and activity) with the corresponding emotion information. In addition, such emotion estimation models (various data constituting the emotion estimation models) are also stored in the storage unit 22 (emotion estimation model storage unit 222) at the time of designing or assembling the emotion estimation device 20 in a manner similar to that of the first AI model 121mA and the second AI model 121mB.

[0048] In addition, the psychological plane table 224 can be replaced with data of an arithmetic (logical) processing formula for calculating emotions from two types of quantized emotion index values ​​(quantized arousal level and quantized activity level). In that case, the controller 23 described later will calculate emotions by arithmetic processing based on the arithmetic (logical) processing formula.

[0049] The controller 23 controls various operations of the feeling estimation device 20, includes a processor that performs arithmetic processing, and is configured by, for example, a CPU. The controller 23 has, as its functions, an acquisition unit 231, a behavior determination unit 232, a feeling estimation unit 233, and a providing unit 234. The functions of the controller 23 are realized by the processor executing arithmetic processing in accordance with a program stored in the storage unit 22.

[0050] The acquisition unit 231 acquires various appearance information (image information, audio information) of the user U2, who is a target of emotion estimation, captured by the camera C and the microphone M via the communication unit 21. The acquisition unit 231 stores the acquired various appearance information in a data table formed in the storage unit 22 as necessary for subsequent processing. The acquisition unit 231 acquires these pieces of appearance information at substantially the same time and stores them in the data table as one data set. These pieces of data are then used to estimate the behavior state and emotion at that time.

[0051] The behavior determination unit 232 determines predetermined types of behavior such as the gaze, facial direction, facial expression, and voice of the user U2 by performing an analysis process on the image information and voice information of the user U2 acquired by the acquisition unit 231. The predetermined types of behavior are behaviors corresponding to the types of data input to the first AI model 121mA and the second AI model 121mB, and in this embodiment, specifically, the gaze, facial direction, facial expression, and voice (contents).

[0052] Therefore, the behavior determination unit 232 performs various processes according to the input specifications (determined by design conditions, learning conditions, etc.) of the first AI model 121mA and the second AI model 121mB. The various processes include, for example, a process of cutting out images (a group of still images or a video) and audio data of a processing target time length, a process of cutting out the face and eyeballs of the user U2 from an image captured by the camera C and a process of detecting the direction, and a frequency analysis process of an output signal from the microphone M. In addition, in the process of cutting out audio data, if the time length of emotion estimation is 10 seconds, for example, data for 10 seconds immediately before the emotion estimation timing is cut out. The time length may be changed according to the data type.

[0053] The first AI model 121mA and the second AI model 121mB output appropriate estimated (correct) data in response to input of data of the same type (content and format) as the input data during learning. For this reason, the output data type (content type and format) of the behavior determination unit 232 needs to be the same as the input data during learning of the first AI model 121mA and the second AI model 121mB. The behavior determination unit 232 performs such data processing. Programs and data for realizing these operations of the behavior determination unit 232 are stored in the memory unit 22.

[0054] The emotion estimation unit 233 estimates the emotion of the user U2 based on the behavior data determined by the behavior determination unit 232. In detail, the emotion estimation unit 233 inputs the behavior data determined by the behavior determination unit 232 to the first AI model 121mA and the second AI model 121mB. As a result, the first AI model 121mA and the second AI model 121mB output the quantized emotion index value (quantized arousal level and quantized activity level) of the user U2 estimated based on the behavior information (appearance information) related to the gaze, face direction, facial expression, voice, etc. of the user U2. Then, the estimated quantized emotion index value of the user U2 output by the first AI model 121mA and the second AI model 121mB is input to the emotion estimation model 222m. Then, the emotion estimation model 222m estimates the emotion of the user U2 based on the estimated quantized emotion index value of the user U2 and outputs it.

[0055] According to the above configuration, the first AI model 121mA and the second AI model 121mB use the estimated output as a quantized emotion index value, so that the resolution of the data to be processed is lowered, and it is expected that the processing load will be lighter. Also, when the first AI model 121mA and the second AI model 121mB are trained, the resolution of the correct answer data of the training data is lowered, so that it is expected that the training processing load will be reduced and the work of generating training data will be reduced.

[0056] The estimated emotion index value used in the emotion estimation described above may be estimated using data obtained by performing statistical processing (average processing, low-pass filter processing, etc.) on a plurality of pieces of data for an appropriate time length. In this case, the statistical processing is performed by the emotion estimation unit 233.

[0057] The providing unit 234 provides the emotion of the user U2 estimated by the emotion estimation unit 233 to the vehicle control device 30. This enables the vehicle control device 30 to control the vehicle V2 based on the emotion of the user U2.

[0058] <2-2. Vehicle control device and other components> The vehicle control device 30 is, for example, an ECU (Electronic Control Unit) for vehicle control, and is mounted on the vehicle V2. The vehicle control device 30 includes a controller 31, a storage unit 32, and a communication unit 33.

[0059] The controller 31 performs various vehicle controls, such as control of the direction and speed of the vehicle V2, based on driving operations by the user U2 who is the driver of the vehicle V2 and information from various connected sensors (not shown), such as a vehicle speed sensor, an air-fuel ratio sensor, and a steering angle sensor. Furthermore, the controller 31 receives information related to the emotion of the user U2 (estimated emotion) from the emotion estimation device 20 via the communication unit 33. The controller 31 then performs vehicle control using the received estimated emotion information, such as control of the speed, accelerator sensitivity, and brake sensitivity. Specifically, when the driver is excited, the controller 31 performs driving control that is on the safe side, that is, that suppresses the influence of the driver's excitement, by lowering the accelerator sensitivity (to make it harder to accelerate), increasing the brake sensitivity (to make it easier to stop), and lowering the upper limit of the maximum speed control.

[0060] In addition, the controller 31 notifies the driver of the estimated emotion information itself and notifies the driver according to the estimated emotion information. Specifically, when the driver is excited, the controller 31 displays and provides voice guidance such as "calm down," and controls the quality of the voice guidance to be calm in speech and expression.

[0061] The storage unit 32 is configured with various memories like the storage unit 12, and stores data used by the controller 31 in processing, such as programs, processing coefficient data, temporary storage data during processing, etc. The storage unit 32 is provided with a plurality of data tables for various processing in which information related to emotions received from the emotion estimation device 20 is associated with control signals for the vehicle V2 and notification information for the user U2 in the vehicle cabin.

[0062] The communication unit 33 is an interface for communicating data among the feeling estimation device 20, the actuator unit 41, and the notification unit 42, and is, for example, a Controller Area Network (CAN) interface.

[0063] The actuator unit 41 is composed of various driving components such as a motor that realizes various operations of the vehicle, and the operations are driven and controlled by the vehicle control device 30. Specifically, the actuator unit 41 is, for example, an engine and a motor that generate a driving force in the vehicle V2, a steering actuator that drives the steering of the vehicle V2, a brake actuator that drives the brakes of the vehicle V2, and the like.

[0064] The notification unit 42 notifies the user U2 in the vehicle cabin of the notification information from the vehicle control device 30 by a visual, auditory, or other means. Specifically, for example, the notification unit 42 is composed of a liquid crystal display that conveys the notification information to the user U2 by characters, images, videos, or the like, and a speaker that conveys the notification information by voice, warning sound, or the like.

[0065] The vehicle control device 30 generates and outputs a control signal for the vehicle V2 based on information from various sensors and the emotion of the user U2 provided by the emotion estimation device 20. Specifically, for example, when the user U2 is excited or scared, the vehicle control device 30 displays and sounds a message such as "calm down" via a display unit or a speaker. In addition, the vehicle control device 30 controls the speed of the vehicle V2 based on information from a vehicle speed sensor, an obstacle sensor (radar, etc.), etc., and when the user U2 is excited, the vehicle control device 30 controls the speed to be slower than when the user U2 is in a normal emotion.

[0066] That is, the vehicle control system 40 estimates the emotion of the user U2 who is the driver of the vehicle V2, and performs the operation of the vehicle V2, for example, driving control, taking into consideration the estimated emotion. With this configuration, the vehicle V2 can be controlled according to the emotion of the user U2. Therefore, according to this embodiment, it is possible to control the vehicle V2 according to the emotion of the driver that affects safe driving, which can contribute to safe driving of the vehicle V2.

[0067] <2-3. Processing of emotion estimation device> Fig. 5 is a flowchart showing emotion estimation processing executed by the controller 23 of the emotion estimation device 20 in Fig. 4. This flowchart shows the technical contents of a computer program that causes a computer device to realize the emotion estimation processing. In addition, the computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0068] The process shown in FIG. 5 is initiated when the vehicle V2 starts and various controls based on the estimated emotion are initiated, and is then repeatedly executed at appropriate timing for the various controls, that is, at timings when emotion estimation is required, or over a time period when emotion estimation is required.

[0069] In step S101, the controller 23 (acquisition unit 231) acquires data (appearance information) indicating the behavior of the user U2, specifically, an image captured by a camera and a voice collected by a microphone of the user U2, and proceeds to step S102. The acquired various appearance information is stored in a data table of the storage unit 22 as necessary.

[0070] In step S102, the controller 23 (behavior determination unit 232) determines the behavior of the user U2, specifically, the gaze, facial direction, facial expression, and voice (content), based on the various data (images captured by the camera, audio collected by the microphone) acquired by the acquisition unit 231 in step S101, and proceeds to step S103.

[0071] In step S103, the controller 23 (emotion estimation unit 233) inputs each piece of data based on the behavior judgment result by the behavior judgment unit 232 judged in step S102 as input values ​​to the first AI model 121mA and the second AI model 121mB, and has the first AI model 121mA and the second AI model 121mB estimate the quantized emotion index values ​​of the quantized alertness and quantized activity, respectively, and proceeds to step S104.

[0072] In step S104, the controller 23 (emotion estimation unit 233) applies (inputs) the two types of estimated quantized emotion index values ​​(quantized arousal level and quantized activity level) to the emotion estimation model 222m (data of the psychological plane table 224) to estimate the emotion of the user U2, and proceeds to step S105.

[0073] In step S105, the controller 23 (providing unit 234) provides (outputs) the estimated emotion information of the user U2 estimated by the emotion estimation unit 233 to the vehicle control device 30, and ends the process.

[0074] Since each estimated emotion is estimated to be adversely affected (misjudgment) due to noise in the sensor output, it is preferable to adopt an estimated emotion that has been subjected to statistical processing, such as adopting an average or mode over a predetermined period. For this reason, it is preferable for the controller 23 (providing unit 234) to add a timestamp to the estimated emotion, store it, and provide the estimated emotion that has been subjected to statistical processing to the vehicle control device 30 (step S105). It is also possible for the vehicle control device 30 to store estimated emotion information, perform statistical processing, and use the results for control.

[0075] <3. Learning method of emotion estimation AI> Next, the learning method of the first AI model 121ma and the second AI model 121mb will be described. In order to easily distinguish between the models before and after learning, the models before learning are referred to as the first AI model 121ma and the second AI model 121mb, and the models after learning are referred to as the first AI model 121mA and the second AI model 121mB, and are written so that the models before and after learning can be distinguished by the last character of the code. Figure 6 is a conceptual explanatory diagram showing the learning process of the emotion estimation AI by the learning device 10.

[0076] In this embodiment, the user U1 is a subject for generating learning data (a subject for generating learning data), and the subject is a member of the development team of the emotion estimation device 20. In the case of a personal emotion estimation device 20 in which the user is limited, the subject is preferably the user, but since there is similarity even if the subject is a different person, the subject may be the user and a different person. In the case of a general-purpose emotion estimation device 20 in which the user is not limited, the subject is the user and a different person, but if learning data is generated by multiple subjects, learning data by subjects with various characteristics can be obtained. As a result, the emotion estimation device 20 is expected to be a highly versatile device. In this embodiment, since the emotion estimation device 20 is assumed to be mounted on a vehicle, learning is performed on a vehicle-mounted device in a similar environment, and the subject user U1 is the driver of the vehicle V1.

[0077] The appropriate learning data used to train an AI model is determined by the purpose of the AI ​​model. For example, in the case of an AI-applied device for a specific driver, the appropriate information is information on that driver, and in the case of an AI-applied device for various drivers, the appropriate information is information on a wide variety of drivers. In addition, in the case of an AI-applied device for a driver, the appropriate information is information on the driver, and in the case of an AI-applied device for each vehicle occupant other than the driver, the appropriate information is information on various occupants other than the driver. In order to increase the accuracy of emotion estimation by the AI ​​model, a large amount of information is required as learning data.

[0078] In this embodiment, the AI ​​application device is applicable to drivers in general (not specific individuals, but general drivers). In order to make the device control the vehicle based on the emotions of the drivers, it is preferable that the experimental test run for learning the AI ​​is an experimental test run with various patterns by multiple subjects of various types. Since the learning by each subject is similar, the learning by one subject (driver) will be described.

[0079] Furthermore, for example, if the AI-applied device is a medical treatment device in a medical institution, the user U1 suitable for the subject (information collection target) will be a patient or a doctor in the medical institution. If the AI-applied device is an educational instruction device in an educational institution, the user U1 suitable for the subject will be a student or a teacher. If the AI-applied device is an e-sports-related device or an entertainment content-related device, the user U1 suitable for the subject will be an e-sports player or a content viewer.

[0080] It is also possible to mount the learning device 10 on the vehicle V1 and have it perform learning, or to install the learning device 10 in a location other than the vehicle V1, such as a research and development laboratory, and have it perform learning. In the latter case, each biological data (measurement data such as brain waves, heart rate, images, and voice) of the subject user U1 who is riding in the vehicle V1 is input to the learning device 10 via a recording medium or a communication line. In this embodiment, an example will be described in which the learning device 10 is mounted on the vehicle V1 and has it perform learning.

[0081] A storage unit for the AI ​​models is provided in a storage device (memory, etc.) of the learning device 10 mounted on the vehicle V1, and a first AI model 121ma and a second AI model 121mb before learning are stored. Then, the learning device 10 executes a learning process described below, whereby learning of the AI ​​models before learning is performed, and a trained AI model, that is, a first AI model 121mA and a second AI model 121mB to be implemented in the AI ​​application device, is generated.

[0082] The trained first AI model 121mA and the trained second AI model 121mB are used in the emotion estimation device 20 shown in FIG. 1 and FIG. 4 described above. Therefore, the training data of the first AI model 121ma is input data of behavior information such as the gaze, face direction, facial expression, and voice of the user U1 (appearance information, data of the same type and format as the input data of the first AI model 121mA of the emotion estimation device 20), and the correct answer data is a quantized value (encoded value) of the emotion index value representing the arousal level. The training data of the second AI model 121mb is input data of behavior information such as the gaze, face direction, facial expression, and voice of the user U1 (appearance information, data of the same type and format as the input data of the second AI model 121mB of the emotion estimation device 20), and the correct answer data is a quantized value (encoded value) of the emotion index value representing the activity level.

[0083] In this embodiment, the quantized values ​​of the arousal level and the activity level (quantized arousal level and quantized activity level) are set to two values, positive and negative. In addition, when the emotion estimation model 222m having the neutral region described above is applied, the quantized values ​​are set to three values, positive, neutral, and negative.

[0084] For this reason, the vehicle V1 on which the learning device 10 is mounted is provided with various on-board sensors that output data used by (input to) the emotion estimation device 20. In this embodiment, specifically, the on-board sensors include a camera C and a microphone M. The camera C outputs captured image information showing the face of the driver U1 so that data on the line of sight and facial direction of the driver U1 can be obtained. The microphone M detects audio information such as sounds emitted by the driver U1. The camera C and the microphone M are installed, for example, near the windshield or dashboard of the vehicle V1 so that the direction of the driver U1 is the direction of imaging and the direction of sound collection.

[0085] The outputs from the camera C and microphone M are processed into behavior data by a behavior determination unit 135 having a configuration similar to that of the behavior determination unit 232 in Fig. 4, and are processed into behavior data on gaze, facial direction, facial expression, and voice. These behavior data are then input as input values ​​to the first AI model 121ma and the second AI model 121mb of the learning device 10.

[0086] A biosensor is attached to the driver U1. In this embodiment, the biosensors are an electroencephalogram sensor S11 that detects an electroencephalogram and a heartbeat sensor S12 that detects a heartbeat. For example, a headgear-type electroencephalogram sensor is used as the brainwave sensor S11. For example, a chest belt-type electrocardiogram-type heartbeat sensor is used as the heartbeat sensor S12. Note that the biosensor may be replaced or replaced with another sensor depending on the bioinformation to be acquired, wearability, and the like. The other biosensor may be, for example, an optical heartbeat (pulse) sensor, a blood pressure monitor, or a NIRS (Near Infrared Spectroscopy) device.

[0087] Then, the brain wave data output by the brain wave sensor S11 and the heart rate data output by the heart rate sensor S12 are converted into emotion index values, ie, arousal level and activity level, in the index value calculation unit 132. Specifically, using a method similar to the above-mentioned method for calculating arousal level and activity level, arousal level is calculated using "brain wave beta waves / alpha waves," and activity level is calculated using "standard deviation of heart rate LF (Low Frequency) components (low frequency components of heart rate waveform signal)."

[0088] Furthermore, in order to make each emotion index value into data that matches the emotion estimation resolution, that is, the resolution (binary) of the emotion estimation model 222m, the quantization unit 133 performs quantization processing at that resolution. Then, the emotion index value representing the quantized arousal level is input as a correct value to the first AI model 121ma. Moreover, the emotion index value representing the quantized activity level is input as a correct value to the second AI model 121mb.

[0089] In other words, the first AI model 121ma is trained using a supervised learning data set in which the input values ​​are behavior data (appearance information) of gaze, face direction, facial expression, and voice, and the correct answer value is an emotion index value representing a quantized arousal level. Therefore, the trained first AI model 121mA is generated as an AI model in which the input values ​​are behavior data of gaze, face direction, facial expression, and voice, and the output value is arousal level, as shown in FIG.

[0090] In addition, the second AI model 121mb is trained using a supervised learning data set in which the input values ​​are behavior data (appearance information) of gaze, face direction, facial expression, and voice, and the correct answer value is an emotion index value representing quantized activity. Therefore, the trained second AI model 121mB is generated as an AI model in which the input values ​​are behavior data of gaze, face direction, facial expression, and voice, and the output value is activity, as shown in FIG.

[0091] In addition, when the subject is in an unstable mental state or environmental state, there is a problem that inappropriate learning data is generated as learning data, and the AI ​​model learns using the inappropriate learning data. Therefore, the learning device 10 is provided with an extraction unit 134a and an extraction unit 134b that extract appropriate learning data, respectively, after the quantization unit 133 and before (or after) the behavior determination unit 135. This prevents inappropriate learning data from being input to the first AI model 121ma and the second AI model 121mb. Details of the extraction unit 134a and the extraction unit 134b will be described later.

[0092] Next, the flow of the learning method (AI learning process) of the first AI model 121ma and the second AI model 121mb will be described with reference to Fig. 6. Note that specific tasks such as mounting the sensor are performed by an operator in charge of learning the AI ​​model, a subject (driver U1), etc.

[0093] A first AI model 121ma before learning that estimates a quantized wakefulness level using information on gaze, face direction, facial expression, and voice as input, and a second AI model 121mb before learning that estimates a quantized activity level using information on gaze, face direction, facial expression, and voice as input are stored in a storage device (memory, etc.) of the learning device 10. The first AI model 121ma before learning and the second AI model 121mb before learning are separately created in advance by an AI designer or developer using a computer for AI design and according to the specifications of the AI ​​model (usage form, required performance, etc.).

[0094] The driver U1 gets into the vehicle equipped with the learning device 10 and adjusts the position, direction, sensitivity, etc. of the camera C and microphone M mounted on the vehicle. The driver U1 also wears the brain wave sensor S11 and the heart rate sensor S12. When the above-mentioned preparations for learning are complete, the driver U1 etc. starts up the learning device 10 and learning begins. If necessary (for example, when the vehicle is traveling, a learning condition), the driver U1 starts driving (operating) the vehicle.

[0095] The camera C and microphone M detect an image, which is appearance information of the user U1, and a voice emitted by the user U1, and output the image information and the voice information to the study device .

[0096] The brain wave sensor S11 and the heart rate sensor S12 detect the brain waves and the heart rate, which are bio-signals of the user U1, and output the brain waves and the heart rate to the learning device 10.

[0097] Then, the learning device 10 performs subsequent processing using the image information and voice information of the user U1 at the same timing, and the brainwaves and heartbeat as a pair of data (one piece of learning data).

[0098] In the index value calculation unit 132, the learning device 10 converts the brain waves and heart rate of the input biological signals into an emotion index value representing the degree of arousal (for example, beta waves / alpha waves of the brain waves) and an emotion index value representing the degree of activity (for example, the standard deviation of the heart rate LF component) by emotion index value processing.

[0099] Furthermore, the learning device 10 performs quantization processing on the converted arousal levels and activity levels in the quantization section 133 to generate quantized arousal levels and quantized activity levels. Note that a configuration may also be adopted in which statistical processing such as averaging is performed on the emotion index values ​​calculated in a predetermined statistical processing period, and the statistically processed statistically processed emotion index values ​​are output to the quantization section 133. With regard to the statistical processing period, a period during which an appropriate emotion index value is output (appropriate timing and duration based on the emotion index value calculation timing) may be determined based on experiments or the like.

[0100] Furthermore, the learning device 10 extracts, in the extraction unit 134a, quantized awakening levels and quantized activity levels suitable as learning data from the quantized awakening levels and quantized activity levels generated in the quantization unit 133, and sets these as data for generating learning data.

[0101] The learning device 10 converts the input image information and voice information of the user U1 into behavior data in the behavior determination unit 135 to generate information on the gaze, face direction, facial expression, and voice. At this time, since the image information and voice information are time-series data that are continuous in time, the image information and voice information are converted into behavior data in units of an appropriate period (appropriate timing and time length based on the emotion index value calculation timing) relative to the emotion estimation timing (emotion index value calculation timing for generating learning data). This period may be determined based on experiments, etc., so that the estimated quantized emotion index value of the AI ​​model becomes an appropriate value. Then, the behavior determination unit 135 outputs the behavior data of this period length as one piece of learning data.

[0102] Furthermore, the learning device 10 extracts behavior data suitable as learning data from the behavior data generated by the behavior determination unit 135 in the extraction unit 134b, and sets the extracted behavior data as data for generating learning data. Details of the extraction units 134a and 134b will be described later.

[0103] Then, the learning device 10 generates learning data for the first AI model 121ma, which uses the gaze, face direction, facial expression, and voice information extracted by the extraction unit 134b as input values ​​and the quantized awakening level extracted by the extraction unit 134a as a correct answer value. The learning device 10 also generates learning data for the second AI model 121mb, which uses the gaze, face direction, facial expression, and voice information extracted by the extraction unit 134b as input values ​​and the quantized activity level extracted by the extraction unit 134a as a correct answer value. Then, the learning device 10 aggregates a large number of learning data generated at each timing to form a learning data set for each model. After that, the learning device 10 performs learning of the first AI model 121ma and the second AI model 121mb using each of the generated learning data sets. Note that the learning data may be input to the first AI model 121ma and the second AI model 121mb sequentially at the timing when the learning data is created, and learning may be performed.

[0104] Specifically, the learning device 10 sequentially inputs the learning data of each generated learning data set to the first AI model 121ma and the second AI model 121mb. Then, the learning device 10 performs learning such as adjusting parameters such as weights in the first AI model 121ma and the second AI model 121mb using a learning algorithm such as an error backpropagation learning method.

[0105] As a result, the first AI model 121ma and the second AI model 121mb are trained using a large amount of supervised training data generated based on each piece of data sequentially detected by each sensor. Then, the trained first AI model 121mA and the trained second AI model 121mB used in the emotion estimation device 20 are generated.

[0106] <4. Configuration of AI learning device for emotion estimation> Fig. 7 is a block diagram showing the configuration of the learning device 10. Fig. 7 shows components necessary for explaining the features of this embodiment, and omits the description of general components.

[0107] 7, learning device 10 includes communication unit 11, storage unit 12, and controller 13. Learning device 10 can be configured as a so-called computer device. Although not shown in the figure, learning device 10 includes an input device such as a keyboard and an output device such as a display.

[0108] The communication unit 11 is an interface for communicating data with other devices and various sensors via a communication network, and is configured, for example, by a network interface card (NIC).

[0109] The storage unit 12 includes a volatile memory and a non-volatile memory. The volatile memory includes, for example, a RAM (Random Access Memory). The non-volatile memory includes, for example, a ROM (Read Only Memory), a flash memory, and a hard disk drive. The non-volatile memory stores programs and data that can be read by the controller 13. At least a part of the programs and data stored in the non-volatile memory may be obtained from another computer device (server device) connected by wire or wirelessly, or from a portable recording medium.

[0110] The storage unit 12 is provided with a first AI model storage unit 121A and a second AI model storage unit 121B. The first AI model storage unit 121A and the second AI model storage unit 121B store a first AI model 121ma and a second AI model 121mb before learning, which are to be learned, respectively. Furthermore, the storage unit 12 is provided with a task table 123 and a neutral region table 124 as data tables for various processes. The task table 123 and the neutral region table 124 will be described later.

[0111] The controller 13 realizes various functions of the learning device 10 and includes a processor that performs arithmetic processing and the like. The processor includes, for example, a CPU (Central Processing Unit). The controller 13 may be configured with one processor or multiple processors. When configured with multiple processors, the processors are connected to each other so that they can communicate with each other and cooperate to execute processing. Note that the learning device 10 can also be configured with a cloud server, in which case the CPU that configures the processor may be a virtual CPU.

[0112] The controller 13 includes, as its functions, an acquisition unit 131, an index value calculation unit 132, a quantization unit 133, an extraction unit 134, a behavior determination unit 135, a generation unit 136, and a provision unit 137. In this embodiment, the functions of the controller 13 are realized by a processor executing arithmetic processing in accordance with a program stored in the storage unit 12.

[0113] The acquisition unit 131 acquires various pieces of information (image (video) information, audio information, brain wave information, heart rate information) detected by the camera C, microphone M, brain wave sensor S11, and heart rate sensor S12 via the communication unit 11. The acquisition unit 131 stores the acquired various pieces of information in a data table formed in the storage unit 12 as necessary for subsequent processing. The acquisition unit 131 acquires these pieces of information at substantially the same time and stores them in one data record in the data table as one data set. These pieces of data are then used to generate one piece of supervised learning data.

[0114] The index value calculation unit 132 calculates emotion index values ​​representing arousal and activity based on the brainwave and heartbeat data of the biological information. As described above, the emotion index value representing arousal can be calculated using "brainwave beta waves / alpha waves." Moreover, the emotion index value representing activity can be calculated using "standard deviation of heartbeat LF component." Note that data such as calculation formulas required for calculating these emotion index values ​​are stored in the storage unit 12.

[0115] The quantization unit 133 performs a quantization process on the emotion index value calculated by the index value calculation unit 132. In the emotion estimation model (psychological plane) shown in Fig. 2, emotion is estimated based on whether the emotion index value (β wave / α wave of electroencephalogram) indicating the degree of arousal is higher or lower than a threshold value (X axis) and whether the emotion index value (standard deviation of heart rate LF component) indicating the degree of activity is higher or lower than a threshold value (Y axis). In the emotion estimation model (psychological plane) shown in Fig. 3, emotion is estimated based on whether the degree of arousal is higher than a neutral state, a neutral state (normal state, neutral state), or lower than a neutral state, and whether the degree of activity is higher than a neutral state, a neutral state (normal state, neutral state), or lower than a neutral state.

[0116] That is, the learning device 10 quantizes the emotion index value with a resolution that matches the emotion estimation resolution.

[0117] Next, a specific method of the quantization process (encoding process) will be described.

[0118] The first example is a method of quantizing emotion index values ​​when the stratification (quantization value) of emotion index values ​​does not include a neutral region.

[0119] FIG. 8A is a schematic diagram showing the change over time in the emotion index value of a subject. In the graph included in FIG. 8A, the horizontal axis is time, and the vertical axis is the emotion index value. The graph in FIG. 8A corresponds to the emotion estimation model (psychological plane) shown in FIG. 2, and "0" on the vertical axis corresponds to the emotion index value at the axis position (median, calibration reference value) in FIG. 2. Below the graph in FIG. 8A, video data of the subject's facial image (still images (frame images) at each timing) acquired at the same timing as the emotion index value of the graph is depicted typically.

[0120] In the graph of Fig. 8A, the dashed curve indicates the emotion index value before quantization, and the discrete values ​​(positive / negative (decision value)) shown by the solid line indicate the quantized emotion index value after quantization. To be more specific, in the graph of Fig. 8A, the dashed curve is the emotion index value before quantization calculated by the index value calculation unit 132, and the discrete values ​​of the solid line are the quantized emotion index value after quantization converted by the quantization unit 133.

[0121] The quantization unit 133 performs quantization processing (stratification processing) on ​​the emotion index value before quantization calculated by the index value calculation unit 132, using the median value of the emotion index value as a threshold. The quantization unit 133 calculates and outputs a positive (high index value state) quantized emotion index value in a period LH in which the emotion index value fluctuates on the higher level side than the threshold. The quantization unit 133 also calculates and outputs a negative (low index value state) quantized emotion index value in a period LL in which the emotion index value fluctuates on the lower level side than the threshold. In this way, it is possible to obtain positive and negative quantized emotion index values ​​in the high index value state and low index value state of the subject, respectively.

[0122] In the example of FIG. 8A, frame images P1 to P6 and P10 to P12 are recognized as images in a state where the quantized emotion index value is positive, and frame images P7 to P9 are recognized as images in a state where the quantized emotion index value is negative.

[0123] The second example is a method of quantizing emotion index values ​​when the stratification (quantization value) of emotion index values ​​includes a neutral region.

[0124] FIG. 8B is a schematic diagram showing the time change of the emotion index value including the neutral region Rn of the subject. In the graph included in FIG. 8B, the horizontal axis is time, and the vertical axis is the emotion index value. The graph in FIG. 8B corresponds to the emotion estimation model (mental plane) including the neutral region shown in FIG. 3, and "0" on the vertical axis corresponds to the emotion index value at the axis position (median, calibration reference value) in FIG. 3. The neutral region Rn, which represents a neutral state of emotion, exists near the median. Below the graph in FIG. 8B, video data of the subject's facial image acquired at the same timing as the emotion index value of the graph is drawn typically.

[0125] The quantization unit 133 performs quantization processing on the emotion index value before quantization calculated by the index value calculation unit 132, using as thresholds an upper limit value (first threshold value) RnU and a lower limit value (second threshold value) RnD of the neutral region Rn of the emotion index value. The quantization unit 133 calculates and outputs a positive (high index value state) quantized emotion index value in a period LH in which the emotion index value fluctuates on the higher level side than the upper limit value RnU. The quantization unit 133 calculates and outputs a negative (low index value state) quantized emotion index value in a period LL in which the emotion index value fluctuates on the lower level side than the lower limit value RnD. The quantization unit 133 calculates and outputs a neutral (neutral state) quantized emotion index value in a period LN in which the emotion index value fluctuates between the upper limit value RnU and the lower limit value RnD (neutral region Rn). This makes it possible to obtain positive, negative, and neutral quantized emotion index values ​​in the subject's high index value state, low index value state, and neutral state, respectively.

[0126] In the example of FIG. 8B, frame images P22 to P25, and P30 to P32 are recognized as images in a state where the quantized emotion index value is positive, frame images P28 to P29 are recognized as images in a state where the quantized emotion index value is negative, and frame images P21, and P26 to P27 are recognized as images in a state where the quantized emotion index value is neutral.

[0127] When quantizing a subject's emotion index value using the above method, it is necessary to determine in advance the median of the subject's emotion index value that serves as a threshold (first example above), or the upper limit value RnU and lower limit value RnD of the neutral region Rn (second example above). In order to determine these thresholds, the subject is made to execute a specific task that puts the subject into a specific emotion index value state, and the emotion index value during the execution of the task is measured. Then, based on the measurement result of the emotion index value, the median of the subject's emotion index value that serves as a threshold, or the upper limit value RnU and lower limit value RnD of the neutral region Rn, is determined.

[0128] Fig. 9 is a diagram showing an example of the task table 123. As shown in Fig. 9, the items of the task table 123 include "task ID", "biological signal type", "corresponding index type", "task content", and "index value transition state".

[0129] "Task ID" is task ID data that is identification information for identifying task information. Task ID data is also a primary key of a data record in task table 123. That is, in task table 123, a data record is configured for each task ID data, and data of each item associated with the task ID data is stored in the data record.

[0130] The "biological signal type" stores the type of measurement value based on the biological signal detected by the sensor. This biological signal type data is data correlated with the corresponding index type data.

[0131] The "corresponding index type" is the emotion index type to be learned by the first AI model 121ma and the second AI model 121mb, and is the emotion index type to be quantized. As the corresponding index type, for example, arousal level and activity level are stored.

[0132] The "task content" is the specific content of the task that the subject executes in order to acquire the emotion index type threshold value required for the emotion index type quantization process.

[0133] The "index value transition state" is the state to which the corresponding emotional index value is estimated to transition when the subject performs the corresponding task content (stored in the same record). For example, data such as "high arousal: arousal level becomes on the arousal side (high arousal level) (high index value state)" is stored.

[0134] According to this task table 123, for example, task data with a task ID of ST01 has a biological signal that is electroencephalogram data, a corresponding indicator type that is alertness, a state transition of alertness that is high alertness, and a specific task content that is “mentally add the displayed numbers.”

[0135] Then, the task content to be executed by the subject (subject for threshold determination) is selected according to the emotion index value type to be subjected to the quantization process and the threshold type to be determined. For example, when the subject's alertness is to be transitioned to the alert side (high alertness) in order to determine the threshold for the quantization process of alertness, the "task content" "mentally add displayed numbers" with a "corresponding index type" of "alertness" and an "index value transition state" of "high alertness" is selected as the task to be executed by the subject. Then, the "task content" of the task selected in this way is notified (displayed) to the subject, and the subject is made to execute the task.

[0136] 10 is a schematic diagram showing the change in arousal level over time during task execution in an example in which tasks ST01 and ST02 in the task table 123 are selected to estimate the neutral region Rn (upper limit value RnU and lower limit value RnD) of arousal level. For ease of understanding, the emotion index value will be described here as arousal level.

[0137] In the graph of Figure 10, the horizontal axis is time and the vertical axis is alertness. Signal x represents the change in alertness over time when the subject performed the task content of task ST01, "mentally add the displayed numbers." Signal y represents the change in alertness over time when the subject performed the task content of task ST02, "remain still and clench your fist."

[0138] As shown in Fig. 10, when the subject is performing task ST01, which puts the subject in a high arousal state, the subject's arousal is in a high state (high index value state), so the measured arousal level of the subject is expected to change (fluctue) within the high arousal range. Therefore, the minimum arousal level during the execution of task ST01 (taking into account the time lag of the effect, the period from the start of execution after the equivalent time lag has elapsed to the end of execution after the equivalent time lag has elapsed) can be estimated to be the upper limit RnU of the neutral arousal level range Rn.

[0139] Furthermore, when the subject is performing task ST02, which results in a low level of arousal, the subject's level of arousal is in a low state (non-arousal state, low index value state), so the measured level of arousal of the subject is expected to change (fluctue) within the region of low arousal (non-arousal state). Therefore, the maximum level of arousal during the execution of task ST02 (taking into account the time lag of the effect, the period from the start of execution after the equivalent time lag has elapsed to the end of execution after the equivalent time lag has elapsed) can be estimated to be the lower limit RnD of the neutral region Rn of arousal.

[0140] Based on this technical concept, the quantization unit 133 determines a neutral region Rn for emotion index values ​​such as arousal level.

[0141] Specifically, the quantization unit 133 performs statistical processing on the time-varying data of the emotion index value (arousal level: signal x) obtained during execution of the first task, and determines the upper limit value RnU of the neutral region Rn. For example, the quantization unit 133 performs smoothing processing on the time-varying data of the emotion index value (arousal level, etc.) based on the subject's biological signal obtained during execution of the first task, and sets the minimum value of the signal x data after the smoothing processing as the upper limit value RnU of the neutral region Rn.

[0142] Furthermore, the quantization unit 133 performs statistical processing on the time-varying data of the emotion index value (activity: signal y) obtained during execution of the second task, and determines the lower limit RnD of the neutral region Rn. For example, the quantization unit 133 performs smoothing processing on the time-varying data of the emotion index value (arousal level, etc.) based on the subject's biological signal obtained during execution of the second task, and sets the maximum value of the signal y data after the smoothing processing as the lower limit RnD of the neutral region Rn.

[0143] The smoothing process is performed for the purpose of removing noise components such as minute peak signals.

[0144] Furthermore, taking into consideration the intended use of the estimated emotion, the upper limit RnU and the lower limit RnD may be determined by adding an appropriate margin to the "minimum value of the signal x data" and the "maximum value of the signal y data".

[0145] Furthermore, when the stratification (quantization value) of the emotion index value does not include a neutral region, the threshold value in the quantization process of the emotion index value can be set to the intermediate value between the above-mentioned upper limit value RnU and lower limit value RnD, or the like.

[0146] The quantization unit 133 determines an upper limit value RnU and a lower limit value RnD of the neutral region Rn for each type of emotion index value. For this purpose, in this embodiment, the subject (user) is requested to execute a first task for determining the upper limit value of the neutral region Rn and a second task for determining the lower limit value of the neutral region Rn as described above for each data of the awakening level and the activity level. Then, for each of the awakening level and the activity level, a process for determining each neutral region Rn of the awakening level and the activity level is performed according to the measurement (calculation) results of the awakening level and the activity level when each task is executed.

[0147] In the above, it is assumed that the first fluctuation range, which is the fluctuation range of the emotion index value (arousal: signal x) obtained when the first task is performed, and the second fluctuation range, which is the fluctuation range of the emotion index value (activity: signal y) obtained when the second task is performed, are separated from each other, and a neutral region Rn is defined as the area between the two fluctuation ranges (see Figure 10).

[0148] However, depending on the type of emotion index value and the contents of the task, the first and second variation ranges may overlap. In such a case, the neutral area may be determined based on the overlapping range of the first and second variation ranges. For example, a method is possible in which the overlapping range of the first and second variation ranges is set as the neutral area. Also, for example, a method is possible in which the overlapping range of the first and second variation ranges is expanded by adding an appropriate offset (a suitable value (e.g., 10% range expansion) is set from the viewpoint of preventing erroneous determination of emotion) to the overlapping range of the first and second variation ranges, or a method is possible in which the overlapping range is reduced by subtracting an offset (a suitable value (e.g., 10% range reduction) is set from the viewpoint of preventing inability to determine emotion) to set as the neutral area.

[0149] Next, a second example of the process of determining the neutral area will be described. In the second example, the subject (user U1) is not requested to perform specific tasks such as the first task and the second task. In the second example, the neutral area is determined by focusing on the fact that as the accumulation of time-series data of the measurement (calculation) of the emotion index value progresses, the upper limit value and the lower limit value of the neutral area estimated by the statistical processing converge to a certain value.

[0150] The method of determining the neutral region in this second example will be described with reference to Fig. 11. Fig. 11 is a schematic diagram showing the change over time of the emotion index value (arousal level). In the graph of Fig. 11, the horizontal axis represents time, and the vertical axis represents arousal level.

[0151] As shown in Fig. 11, a person's mental and physical state changes into various states based on changes in the environment, etc., that change over time, even without giving a specific task. Therefore, a considerable amount of time-series data of emotion index values ​​is statistically accumulated, and the neutral region (upper and lower limits) can be estimated by performing cluster analysis that divides the accumulated data into three states (high arousal, neutral, low arousal (sleep-like state)) based on the magnitude of the values. For the cluster analysis, known methods such as the k-means method, Otsu's multi-value method, and Gaussian mixture model can be used.

[0152] Fig. 12 is a diagram showing the convergence status of the upper limit estimate of the neutral region with respect to the data accumulation period (proportional to the amount of accumulated data) when the neutral region is determined by the statistical method as shown in Fig. 11. The convergence status of the lower limit estimate of the neutral region is also similar.

[0153] The amount of data used to estimate the upper limit of the neutral region using statistical methods increases over time. As the amount of data used in statistical processing increases over time, the estimated upper limit converges to a certain value over time. This tendency is similar for the lower limit. A neutral region identified by converged upper and lower limit values ​​is considered to be highly reliable. For this reason, the neutral region is determined when it is determined that the upper and lower limit values ​​calculated by the statistical method have converged. Convergence can be determined by a method in which the fluctuation value of the calculated value of the upper limit (lower limit) of the neutral region (for example, the average change value over several calculations) becomes equal to or less than an appropriately set convergence determination threshold.

[0154] The quantization unit 133 determines the upper and lower limits of the neutral region for each emotion index value type (arousal level and activity level).

[0155] The quantization unit 133 stores the determined upper and lower limit values ​​of the neutral region in the neutral region table 124 of the storage unit 12. In particular, the quantization unit 133 creates a data table of the determined upper and lower limit values ​​of the neutral region and stores the data table in the storage unit 12. It is effective to store neutral region information for each subject (user) in the neutral region table 124. In this case, it is also possible to use the neutral region information stored in the neutral region table 124 when recreating (adding) a learning dataset for the same subject.

[0156] Fig. 13 is a diagram showing an example of the neutral region table 124. As shown in Fig. 13, the items in the neutral region table 124 include "neutral information ID", "user ID", "emotion index type", "upper limit value", "lower limit value", and "acquisition date and time", and corresponding data is stored in each item column (storage unit).

[0157] The "neutral information ID" is identification information for identifying neutral information. The neutral information ID is also the primary key of the data record in the neutral area table 124. That is, in the neutral area table 124, a data record is configured for each neutral information ID, and data for each item linked to the neutral information ID is stored in the data record.

[0158] The "user ID" is identification information for identifying user information, that is, identification data of the user to which the value of the neutral region stored in the data record applies.

[0159] The “upper limit value” and the “lower limit value” are information on the upper limit value and the lower limit value of the neutral region determined by the quantization unit 133 .

[0160] The "acquisition date and time" stores the date and time when the information on the upper and lower limits of the neutral area was acquired (decided and stored). Note that the "upper limit" and "lower limit" and "acquisition date and time" for the same user ID and the same emotion index type are updated each time the latest information is acquired.

[0161] When the stratification (quantization value) of the emotion index value does not include a neutral region, the "upper limit value" and the "lower limit value" become a single "threshold" data item that stratifies the emotion index value into positive and negative (2-quantization), and, for example, the intermediate value between the "upper limit value" and the "lower limit value" is stored.

[0162] Alternatively, a method of estimating a neutral region based on an emotion declaration from a subject (user U1) is also possible.

[0163] Specifically, the quantization unit 133 (controller 13) asks the subject (user) a question about the subject's self-aware emotion index state (for example, a question about whether the subject's arousal state is one of three stages (awake, normal, not-awake (sleepy))), and accumulates the answer and the emotion index value (arousal level) measured (calculated) at that time. The quantization unit 133 then performs statistical processing on the accumulated data to estimate the neutral region.

[0164] For example, the average value of the highest 10% and the average value of the lowest 10% of the arousal level calculated when the subject's answer is "normal" are set as the upper and lower limits of the neutral region. Alternatively, for example, a method can be applied in which a value obtained by subtracting an appropriate offset value from the average value of the lowest 10% of the arousal level calculated when the subject's answer is "awakened" is set as the upper limit of the neutral region, and a value obtained by adding an appropriate offset value to the average value of the highest 10% of the arousal level calculated when the subject's answer is "not awakened" is set as the lower limit of the neutral region.

[0165] In addition, the subject's mental and physical state may not be uniform over time, and the emotional index value (arousal and activity) may not stably and continuously show the characteristics of either a high index value state, a low index value state, or a neutral state. For example, the mental and physical state (emotion index value) may not be fixed at either a high index value state or a low index value state, and may fluctuate between the two. In addition, even if an emotion is generated, there may be a delay before the emotion is expressed in the facial expression, and the emotion may not be expressed in the facial expression. If the appearance information and emotional index value of a subject in such an unstable state are adopted as learning data, the performance of the supervised learning data may deteriorate, and the performance of the AI ​​model may deteriorate.

[0166] For this reason, the extraction unit 134 (see FIG. 7) extracts data of a time period in which the fluctuation state is considered appropriate as learning data according to the fluctuation state of the quantized emotion index value as supervised learning data. In detail, the extraction unit 134 extracts data of a time period in which the emotion index value (arousal level and activity level) stably and continuously exhibits the characteristics of either a high index value state, a low index value state, or a neutral state as learning data. Next, a method for extracting supervised learning data will be described.

[0167] The extraction unit 134 includes the extraction unit 134a and extraction unit 134b shown in Fig. 6. Their operations are similar to those of the extraction unit 134 (the target data types are different). The extraction unit 134a extracts quantized emotion index values ​​(quantized arousal level and quantized liveliness). The extraction unit 134b extracts behavior data (appearance information) of gaze, face direction, facial expression, and voice.

[0168] FIG. 14A is a schematic diagram showing the time change of the quantized emotion index value of the subject. FIG. 14B is a schematic diagram showing the time change of the quantized emotion index value including the neutral region Rn of the subject. In the graphs included in FIG. 14A and FIG. 14B, the horizontal axis is time, and the vertical axis is the quantized emotion index value (positive / negative). Note that the emotion index value that is the original data of the quantized emotion index value is omitted from the illustration. Under the graphs in FIG. 14A and FIG. 14B, video data of the subject's face image acquired at the same timing as the emotion index value of the graph is drawn typically. Other than that, the configurations of FIG. 14A and FIG. 14B are the same as the configurations of FIG. 8A and FIG. 8B, and the description thereof will be omitted. In FIG. 14A and FIG. 14B, the discrete values ​​(positive / negative (judgment value)) shown by the solid lines are the quantized emotion index values ​​converted by the quantization unit 133.

[0169] When the quantized emotion index value remains stable at the same value for a period longer than a predetermined continuation determination threshold LR, the extraction unit 134 extracts data for that continuation period as learning data.

[0170] For example, in the emotion index value situation shown in Fig. 14A, the periods L1, L7, and L8 during which the quantized emotion index value maintains the same value are longer than the continuation judgment threshold LR, so each learning data is extracted and created using the quantized emotion index values ​​for those periods as correct answer data (positive, positive, negative) and face images (P41 to P45, P49, P51 to P52) as input data. Then, the extraction unit 134 excludes data for periods during which the same quantized emotion index value continued for a period shorter than the continuation judgment threshold LR. Therefore, data for periods L2 to L6, which have short durations, is excluded from the learning data.

[0171] 14B, the periods L11, L12, L18, and L19 during which the quantized emotion index value maintains the same value are longer than the continuation judgment threshold LR, so learning data is extracted and created using the quantized emotion index values ​​for those periods as correct answer data (positive, neutral, positive, negative) and face images (P61-P63, P65-66, P70, P72) as input data. Then, the data for periods L13-L17 is excluded from the learning data.

[0172] In this way, the extraction unit 134 extracts and employs emotion index values ​​and facial images in a state in which features are stably and continuously expressed as supervised learning data.

[0173] According to the above configuration, the learning device 10 creates learning data based on data in a state in which emotion index values ​​that are considered to be highly reliable are stably and continuously displayed, and uses the learning data to learn an AI model. Therefore, the AI ​​model is expected to estimate quantized emotion index values ​​with high accuracy. As a result, it is expected that the accuracy of emotion estimation using the AI ​​model will be improved.

[0174] Incidentally, since the quantized emotion index value is a value in a state in which the emotion index value exists in each quantized value region, it may be judged based on the length of time that the emotion index value exists in each quantized value region, instead of the duration of the quantized emotion index value.

[0175] In addition, it is considered that there is a time lag between the change in emotion and the change in facial expression. For this reason, in addition to the above-mentioned extraction process, it is also effective to remove from the learning data one or several images in the vicinity of the change in the quantized emotion index value, for example, before and after the timing of the change. For example, in the case of the situation shown in FIG. 14A, the method is to remove from the learning data images P1 and P5 in the vicinity of the timing of the change in the quantized emotion index value in period L1. By such a process, it becomes possible to remove from the learning data data that is unsuitable as learning data due to the influence of the time lag between the change in emotion and the change in facial expression. Therefore, it is expected that the AI ​​model will estimate the quantized emotion index value with high accuracy. As a result, it is expected that the accuracy of emotion estimation using the AI ​​model will be improved.

[0176] The behavior determination unit 135 performs analysis processing on image information including the face of the user U1 (see FIG. 6) and generated voice, thereby determining predetermined types of behavior of the user U1, such as the line of sight, face direction, facial expression, voice, etc. Note that the processing performed by the behavior determination unit 135 is equivalent to the processing performed by the behavior determination unit 232 in FIG. 4, and each unit outputs data of the line of sight, face direction, facial expression, and voice in the same format.

[0177] For example, regarding the line of sight of the user U1, the behavior determination unit 135 performs recognition processing such as feature calculation and shape discrimination using the left and right eyeballs of the user U1 as detection objects from an image including the face of the user U1. Based on the result of the recognition processing, the behavior determination unit 135 determines the behavior of the line of sight and the gaze point of the user U1 by a predetermined line of sight detection processing using, for example, the position of the inner corner of the eye, the center position of the iris and pupil of the eye, the center position of the corneal reflection image (Purkinje image) by near-infrared illumination, the center position of the eyeball, etc. The line of sight of the user U1 can be expressed by, for example, a two-dimensional coordinate system of a position where the line of sight vector of the user U1 penetrates the virtual plane, which is set in front of the user U1 and directly facing the user U1.

[0178] Furthermore, for example, with regard to the direction of the face of user U1, the behavior determination unit 135 performs recognition processing such as feature calculation and shape determination with the face of user U1 as a detection target from an image including the face of user U1. Based on the result of the recognition processing, the behavior determination unit 135 determines the behavior of the direction of the face of user U1 by a predetermined face direction detection processing using, for example, the positions of the eyes, nose, mouth, etc., the position of the tip of the nose, the facial contour, the center position in the width direction of the facial contour, etc.

[0179] Furthermore, for example, with respect to the facial expression of user U1, the behavior determination unit 135 performs recognition processing such as feature calculation and shape discrimination with the face of user U1 as a detection target from an image including the face of user U1. Based on the result of the recognition processing, the behavior determination unit 135 determines the behavior of the facial expression of user U1 by a predetermined facial expression detection processing using, for example, the angle of the corners of the mouth, the angle of the eyebrows, the degree of opening of the eyes, and the like.

[0180] Furthermore, for example, with regard to audio information including pronunciation of the user U1's voice, the behavior determination unit 135 determines the behavior of the user U1's voice based on the voice volume, speaking speed, speaking frequency, and language analysis results of the spoken content based on the voice recognition results (for example, the appearance of words and sentences related to joy, anger, anxiety, etc.).

[0181] The generation unit 136 takes as input values ​​the appearance information (extracted appearance information) of the user U1's gaze, facial direction, facial expression, and voice, which are determined by the behavior determination unit 135 and further extracted by the extraction unit 134 in the same learning data, and inputs the quantized awakening level, which is the quantized emotion index value quantized by the quantization unit 133 and further extracted by the extraction unit 134, into the first AI model 121ma as a correct value, and learns the first AI model 121ma.

[0182] In addition, the generation unit 136 takes as input values ​​the appearance information (extracted appearance information) of the user U1's gaze, facial direction, facial expression, and voice, which are determined by the behavior determination unit 135 and further extracted by the extraction unit 134 in the same learning data, and inputs the quantized activity, which is the quantized emotion index value quantized by the quantization unit 133 and further extracted by the extraction unit 134, into the second AI model 121mb as a correct answer value, and learns the second AI model 121mb.

[0183] In other words, the generation unit 136 generates a supervised learning dataset using appearance information of the user U1 based on information detected by remote sensors (non-contact sensors: camera, microphone) as input value data, and quantized emotion index values ​​(arousal and activity) calculated based on biosignals detected by contact biosignal sensors (contact sensors: brain wave sensor, heart rate sensor) and further quantized as correct value data.

[0184] Then, the generation unit 136 uses the generated supervised learning data set to train the first AI model 121ma and the second AI model 121mb. Through this training, the trained first AI model 121mA and the trained second AI model 121mB become AI models that estimate the quantized wakefulness and quantized activity of a user (driver) using the appearance information of the user as an input.

[0185] The generation unit 136 uses a learning algorithm such as an error backpropagation learning method to adjust parameters such as weights in the first AI model 121ma and the second AI model 121mb, thereby learning the first AI model 121ma and the second AI model 121mb.

[0186] According to the above configuration, the learning device 10 uses the emotion index value quantized according to the resolution of the emotion estimation model that performs emotion estimation using the emotion index value as the correct value of the AI ​​learning data, and learns the AI ​​model. Therefore, the AI ​​model learns with data of appropriate resolution (quantized emotion index) and outputs estimated data of appropriate resolution (quantized emotion index). This makes the learning of the AI ​​model more efficient, and also reduces the processing load when the AI ​​model is used (resulting in less wasteful processing).

[0187] The providing unit 137 provides the trained first AI model 121mA (data) and the trained second AI model 121mB (data) generated by the generating unit 136 to the feeling estimation device 20 (see FIGS. 1 and 4) used in the vehicle V2 described later via a network. As a result, the feeling estimation device 20 takes in the first AI model 121mA (data) and the second AI model 121mB (data) provided by the providing unit 137 and stores them in the first AI model storage unit 221A and the first AI model storage unit 221A. Therefore, the feeling estimation device 20 can estimate feelings based on image information and audio information acquired by the camera C and the microphone M by the feeling estimation function, and the estimated feeling information can be used in various functions of the vehicle V2.

[0188] The providing unit 137 provides the trained first AI model 121mA (data) and the trained second AI model 121mB (data) generated by the generating unit 136 to the emotion estimation device 20 via a network, but it is also possible to provide them by another method. For example, it is also possible to provide the first AI model 121mA (data) and the second AI model 121mB (data) by writing the data of the first AI model 121mA and the data of the second AI model 121mB to an LSI forming an AI platform to generate an AI execution LSI and incorporating the AI ​​execution LSI into the emotion estimation device 20. It is also possible to provide the first AI model 121mA (data) and the second AI model 121mB (data) to the emotion estimation device 20 using a data transmission medium other than communication, such as a memory card, an optical disk recording medium, etc.

[0189] <5. Example of operation of emotion estimation AI learning device> Fig. 15 is a flowchart showing the learning process of the emotion estimation AI executed by the controller 13 of the learning device 10 in Fig. 7. This flowchart shows the technical contents of a computer program that causes a computer device to realize the learning process of the emotion estimation AI model. In addition, the computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of multiple programs that work together.

[0190] 15 is executed when a designer of the learning device 10 performs a learning process of the emotion estimation AI model, for example, by operating a keyboard or other operating unit to start the learning process. If a learning dataset is to be collected and generated during vehicle driving, the process is executed based on an operation to start collection of the learning dataset by an operator or a subject on the in-vehicle learning device.

[0191] In step S201, the controller 13 (acquisition unit 131) acquires appearance information and biosignals of the user U1, specifically, data on images captured by the camera and audio collected by the microphone from the camera C and microphone M, and also acquires biosignals (brain wave data and heart rate data) from the brain wave sensor S11 and heart rate sensor S12, stores them in the memory unit 12, and proceeds to step S202.

[0192] In addition, when learning is performed by a learning device 10 installed in a research and development laboratory or the like using the various data collected by an in-vehicle device, step S201 and subsequent processing will be performed using the various data collected and accumulated by the in-vehicle device and provided to the learning device 10 via wireless communication or recording medium.

[0193] In step S202, the controller 13 (behavior determination unit 135) determines behavior (gaze, facial direction, and voice) based on the camera image and microphone voice data acquired by the acquisition unit 131 in step S201, and proceeds to step S203.

[0194] In step S203, the controller 13 (index value calculation section 132) calculates the arousal level and activity level, which are emotion index values, based on the electroencephalogram data and heart rate data of the biological signal acquired by the acquisition section 131 in step S201, and proceeds to step S204.

[0195] In step S204, the controller 13 (quantization unit 133) performs a quantization process on the awakening level and the activity level calculated by the index value calculation unit 132 in step S203, and then the process proceeds to step S205.

[0196] In step S205, the controller 13 (extraction unit 134) extracts data for a time period that is considered to be suitable as learning data, based on the fluctuation state of the quantized emotion index value, from the alertness (quantized alertness) and the quantized activity (quantized activity) quantized by the quantization unit 133 in step S204, and the behavior data determined by the behavior determination unit 135 in step S202, and then proceeds to step S206.

[0197] In step S206, the controller 13 (generation unit 136) generates supervised learning data for alertness learning using the gaze, facial direction, facial expression, and voice data of the behavior extracted in step S205 as input values ​​and the quantized alertness extracted in step S205 as the correct answer value, and then proceeds to step S207.

[0198] In step S207, the controller 13 (generation unit 136) generates supervised learning data for activity learning using the gaze, facial direction, facial expression, and voice data of the behavior extracted in step S205 as input values ​​and the quantized activity extracted in step S205 as a correct answer value, and then proceeds to step S208.

[0199] In step S208, the controller 13 (generation unit 136) provides the supervised learning data for alertness learning generated in step S206 to the first AI model 121ma before completion of learning, to train the first AI model 121ma, and then proceeds to step S209.

[0200] In step S209, the controller 13 (generation unit 136) provides the supervised learning data for activity learning generated in step S207 to the second AI model 121mb before completion of learning, to train the second AI model 121mb, and then proceeds to step S210.

[0201] In step S210, it is determined whether learning of each model is completed, in this case whether the learning run of vehicle V1 is completed, for example, based on the ignition switch state or the operation of the vehicle driver (learning operator, test subject), and if completed, the processing ends; if not, the processing returns to step S201 and learning continues.

[0202] After that, the trained first AI model 121mA and the trained second AI model 121mB that have completed training are provided to an AI model utilization device such as the emotion estimation device 20 used in the vehicle V2 shown in Figure 1, based on instructions from an operator operating the learning device 10.

[0203] 15, each time data is collected from each sensor, supervised learning data is generated to train each AI model. However, each time data is collected from each sensor, supervised learning data may be generated and accumulated, and the first AI model 121ma before learning is completed and the second AI model 121mb before learning is completed may be trained using a learning data set consisting of the accumulated learning data.

[0204] In this case, step S208 accumulates the supervised learning data for alertness learning generated in step S206. Step S209 accumulates the supervised learning data for activity learning generated in step S207. Then, step S210 judges whether the creation of the learning data is completed (for example, whether the learning run of vehicle V1 is completed is judged based on, for example, the state of the ignition switch or the operation of the vehicle driver (learning operator, subject)), and if completed, ends the process, and if not, returns to step S201 and continues the generation process of the learning data.

[0205] After that, in a development / design room or the like, the first AI model 121ma and the second AI model 121mb will be trained using the accumulated training data set.

[0206] In addition, in the learning process of the above AI model, learning data is generated for each measurement data unit from each sensor. However, it is also possible to generate learning data using data obtained by statistically processing a plurality of data sets over an appropriate period (sensor output data, behavior data, or emotion index values) and use the data as learning data.

[0207] In addition, while the vehicle is running, biosignal data such as brain waves and heart rate, and appearance information data such as images captured by a camera and sounds picked up by a microphone can be recorded, and the recorded data can be used as input data to generate learning data and train the first AI model 121ma and the second AI model 121mb using the learning device 10 in a development / design room, etc.

[0208] <6. Variations> In the above embodiment, a learning device 10 has been described that generates an AI model that estimates emotion index values ​​(arousal and activity) using an image captured by a camera and an audio picked up by a microphone as input. However, it is also possible to realize a learning device that generates an AI model that estimates emotions using an image captured by a camera and an audio picked up by a microphone as input.

[0209] Fig. 16 is a conceptual explanatory diagram showing emotion estimation processing by the emotion estimation device 50. Fig. 17 is a conceptual explanatory diagram showing learning processing of the emotion estimation AI by the learning device 60. Note that in this modified example, the same components as those shown in Figs. 1, 6, etc. are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0210] The emotion estimation device 50 acquires, from a camera C and a microphone M, appearance information of a user U2 who is a subject. The behavior determination unit 232 generates data on gaze, face direction, facial expression, and voice based on the acquired camera image and microphone audio, and outputs each of the data as input value data to a trained third AI model 121mC. The third AI model 121mC is a trained AI model that estimates emotions based on input values ​​of gaze, face direction, facial expression, and voice data, and outputs emotion information of the user U2.

[0211] The learning device 60 acquires electroencephalogram data and heartbeat data of biological signals detected by the electroencephalogram sensor S11 and the heartbeat sensor S12. The index value calculation unit 132 calculates emotion index values ​​representing arousal and activity levels, respectively, based on the acquired electroencephalogram data and heartbeat data, and outputs the emotion index values ​​to the quantization unit 133. The quantization unit 133 performs quantization processing on each piece of emotion index value data representing arousal and activity levels, respectively, to calculate quantized arousal levels and quantized activity levels, and outputs the quantized arousal levels and quantized activity levels to the extraction unit 134. The extraction unit 134a extracts each piece of data for a time period that is considered to be suitable data as learning data according to the fluctuation state of the quantized arousal levels and quantized activity levels, and outputs the data to the emotion estimation model 622m (psychological plane model portion).

[0212] The emotion estimation model 622m estimates emotions by applying the extracted quantized arousal level and quantized activity level to the psychological plane model, and outputs the estimated emotion information to the third AI model 121mc (before learning and during learning) as a correct answer value of the learning data. The emotion estimation model 622m has the same configuration as the emotion estimation model 222m shown in Fig. 1, and estimates emotions using a psychological plane table 624 having the same configuration as the psychological plane table 224 shown in Fig. 1.

[0213] Furthermore, the learning device 60 acquires, from the camera C and the microphone M, the appearance information, that is, the images captured by the camera and the audio collected by the microphone. The extraction unit 134b extracts each piece of data of the images captured by the camera and the audio collected by the microphone for a time period considered to be suitable data as learning data according to the fluctuation state of the quantized wakefulness and the quantized activity, and outputs the data to the behavior determination unit 135. Note that the data extracted by the extraction unit 134a and the extraction unit 134b corresponds to the same time period.

[0214] The behavior determination unit 135 generates data on gaze, face direction, facial expression, and voice based on the extracted camera captured image and microphone collected voice, and outputs each data as an input value of learning data to the third AI model 121mc before learning (during learning). Depending on the content of the data generated by the behavior determination unit 135, it is also possible for the extraction unit 134b to extract data (extract data for the relevant time period) from the data generated by the behavior determination unit 135.

[0215] The learning device 60 then adjusts parameters such as weights in the third AI model 121mc using a learning algorithm such as a backpropagation learning method based on the input learning data. The learning device 60 then executes the above learning process on the third AI model 121mc using the above data sequentially input (measured) during the learning period, and generates a trained third AI model 121mC.

[0216] As described above, according to the learning device 60 of the modified example, emotion estimation is performed by inputting an emotion index value quantized according to the resolution of the emotion estimation model that performs emotion estimation using the emotion index value to the emotion estimation model. Then, the estimated emotion is used as a correct answer value of supervised learning data for AI learning. Therefore, for generating learning data, data processing using data with appropriate resolution (quantized emotion index) is performed. This makes learning of the AI ​​model efficient. In addition, since the emotion estimation device using the AI ​​model can be configured only with the third AI model 121mC as the estimation model, a model (such as a mental plane table) that estimates emotions from emotion index values ​​is not required, and the processing load on the emotion estimation device can be reduced.

[0217] <7. Points to note> Various technical features disclosed as embodiments in this specification can be modified in various ways without departing from the spirit of the technical creation. In other words, the above embodiments are illustrative in all respects and are not restrictive. The technical scope of the present invention is indicated by the claims, not the description of the above embodiments, and includes all modifications that fall within the meaning and scope of the claims. In addition, the multiple embodiments shown in this specification may be combined as appropriate to the extent possible.

[0218] In the above embodiment, various functions are realized by software through the arithmetic processing of the CPU according to the program, but at least some of these functions may be realized by electrical hardware resources. Examples of hardware resources include ASICs (Application Specific Integrated Circuits) and FPGAs (Field Programmable Gate Arrays). Conversely, at least some of the functions realized by hardware resources may be realized by software.

[0219] The computer program may also include a computer program that causes a processor (computer) to realize at least a part of the functions of the learning device 10 and the emotion estimation device 20. Such a computer program can be stored in a computer-readable nonvolatile recording medium (for example, in addition to the above-mentioned nonvolatile memory, an optical recording medium (for example, an optical disk), a magneto-optical recording medium (for example, a magneto-optical disk), a USB memory, or an SD card, etc.) and provided (sold, etc.), and can also be provided by a method of providing the program from a server device via a communication line such as the Internet, that is, by downloading the program. [Explanation of symbols]

[0220] 10, 60 Learning Device 12 Storage section 13 Controller 20, 50 Emotion estimation device 22 Memory section 23 Controller 30 Vehicle control device 31 Controller 40 Vehicle Control System 41 Actuator section 42 Notification Department 121mA, 121ma 1st AI model 121mB, 121mb 2nd AI model 121mC, 121mc 3rd AI model 131 Acquisition Department 132 Index value calculation unit 133 Quantization section 134, 134a, 134b extraction part 135 Behavior Judgment Unit 136 Generation part 137 Provision Department 222m, 622m Emotion estimation model 231 Acquisition Department 232 Behavior Judgment Unit 233 Emotion estimation part 234 Provision Department C Camera M Microphone S11 Brainwave Sensor S12 Heart Rate Sensor Rn Neutral Region U1, U2 users

Claims

1. A method for creating learning data for an AI model that estimates biological information related to an emotion of a target of emotion estimation based on appearance information of the target of emotion estimation, comprising: Acquiring the appearance information and the biological signal of the subject for creating learning data at the same time; Estimating a fluctuation state of the biological signal based on the acquired biological signal; extracting the appearance information and the biological signal to be used for creating the learning data according to the estimated fluctuation state of the biological signal; Training data generation method.

2. The fluctuation state of the biological signal is a fluctuation state of an emotion index value calculated based on the biological signal. The training data generating method according to claim 1 .

3. the fluctuation state of the biological signal is a fluctuation state of a quantized emotion index value obtained by quantizing an emotion index value calculated based on the biological signal, The training data generating method according to claim 1 .

4. The fluctuation state of the quantized emotion index value is a duration during which the quantized emotion index value has the same value. The training data generating method according to claim 3 .

5. The emotion index value is a central nervous system arousal level or an autonomic nervous system activity level, The central nervous system arousal level is calculated based on a ratio of beta waves to alpha waves in the electroencephalogram, which is the biological signal; The autonomic nervous system activity is calculated based on a standard deviation of a low frequency component of a heart rate waveform signal of a heart rate, which is the biological signal. The training data generating method according to any one of claims 2 to 4.

6. A learning method for an AI model that estimates biological information related to an emotion of a target of emotion estimation based on appearance information of the target of emotion estimation, comprising: Acquiring the appearance information and the biological signal of the subject for creating learning data at the same time; Estimating a fluctuation state of the biological signal based on the acquired biological signal; extracting the appearance information and the biological signal to be used for creating learning data according to the estimated fluctuation state of the biological signal; generating supervised learning data in which appearance learning information based on the extracted appearance information is used as an input value and biological learning information based on the extracted biological signal is used as a correct answer value; Training the AI ​​model with the generated supervised training data; How to learn.

7. The biological learning information is an emotion index value calculated based on the biological signal, The fluctuation state of the biological signal is a fluctuation state of the emotion index value. The learning method according to claim 6.

8. the biological learning information is a quantized emotion index value obtained by quantizing an emotion index value calculated based on the biological signal, The fluctuation state of the biological signal is a duration during which the quantized emotion index value is the same value. The learning method according to claim 6.

9. The emotion index value is a central nervous system arousal level or an autonomic nervous system activity level, The central nervous system arousal level is calculated based on a ratio of beta waves to alpha waves in the electroencephalogram, which is the biological signal; The autonomic nervous system activity is calculated based on a standard deviation of a low frequency component of a heart rate waveform signal of a heart rate, which is the biological signal. The learning method according to claim 7 or 8.

10. A learning device that learns an AI model that estimates biological information related to an emotion of an emotion estimation target based on appearance information of the emotion estimation target, Acquiring the appearance information and the biological signal of the subject for creating learning data at the same time; Estimating a fluctuation state of the biological signal based on the acquired biological signal; extracting the appearance information and the biological signal to be used for creating learning data according to the estimated fluctuation state of the biological signal; generating supervised learning data in which appearance learning information based on the extracted appearance information is used as an input value and biological learning information based on the extracted biological signal is used as a correct answer value; Training the AI ​​model with the generated supervised training data; Learning device.

11. A learning program for learning an AI model that estimates biological information related to an emotion of a target of emotion estimation based on appearance information of the target of emotion estimation, Acquiring the appearance information and the biological signal of the subject for creating learning data at the same time; Estimating a fluctuation state of the biological signal based on the acquired biological signal; extracting the appearance information and the biological signal to be used for creating learning data according to the estimated fluctuation state of the biological signal; generating supervised learning data in which appearance learning information based on the extracted appearance information is used as an input value and biological learning information based on the extracted biological signal is used as a correct answer value; Training the AI ​​model with the generated supervised training data; Study program.

12. A learning method for an AI model that estimates biological information related to an emotion of a target of emotion estimation based on appearance information of the target of emotion estimation, comprising: The appearance information, the electroencephalogram, and the heart rate of the subject for creating learning data are acquired at the same timing; calculating a degree of arousal based on a ratio of beta waves to alpha waves in the acquired electroencephalogram; quantizing the calculated wakefulness level to calculate a quantized wakefulness level; Calculating activity based on a standard deviation of a low-frequency component of the acquired heartbeat waveform signal; quantizing the calculated activity to calculate a quantized activity; Estimating a fluctuation state of the calculated quantized wakefulness or the calculated quantized activity; extracting the appearance information, the quantized wakefulness level, and the quantized activity level to be used for creating learning data according to the estimated fluctuation state; Generate first supervised learning data in which appearance learning information based on the extracted appearance information is used as an input value and the extracted quantized awakening level is used as a correct answer value; generating second supervised learning data in which appearance learning information based on the extracted appearance information is used as an input value and the extracted quantized activity is used as a correct answer value; Using the generated first supervised learning data, an alertness AI model is trained to estimate the quantized alertness based on the appearance information; Using the generated second supervised learning data, an activity AI model is trained to estimate the quantized activity based on the appearance information. How to learn.

13. An emotion estimation device that estimates an emotion of a subject of emotion estimation, comprising: acquiring appearance information of the subject; An emotion index value is estimated by an AI model based on the acquired appearance information; Estimating an emotion based on the estimated emotion index value using an emotion estimation model; The AI ​​model is Acquiring the appearance information and the biological signal of the subject for creating learning data at the same time; Estimating a fluctuation state of the biological signal based on the acquired biological signal; Calculating an emotion index value based on the acquired biological signal; extracting the appearance information and the emotion index value to be used for creating learning data according to the estimated fluctuation state of the biological signal; generating supervised learning data in which appearance learning information based on the extracted appearance information is used as an input value and the extracted emotion index value is used as a correct answer value; A pre-learning AI model is generated by learning the pre-learning AI model using the generated supervised learning data. Emotion estimation device.

14. An emotion estimation device that estimates an emotion of a subject of emotion estimation, comprising: acquiring appearance information of the subject; Estimating an arousal level of the emotion index value based on the acquired appearance information using an arousal level AI model; Estimating the activity of the emotion index value based on the acquired appearance information using an activity AI model; Estimating an emotion based on the estimated arousal level and the estimated activity level; The wakefulness AI model and the activity AI model are The appearance information, the electroencephalogram, and the heart rate of the subject for creating learning data are acquired at the same timing; Calculating a level of arousal based on a ratio of beta waves to alpha waves in the acquired electroencephalogram; Calculating activity based on a standard deviation of a low-frequency component of the acquired heartbeat waveform signal; Estimating a fluctuation state of the calculated wakefulness level or the calculated activity level; extracting the appearance information, the arousal level, and the activity level to be used for creating learning data according to the estimated fluctuation state; generating first supervised learning data in which appearance learning information based on the extracted appearance information is used as an input value and the extracted awakening level is used as a correct answer value; generating second supervised learning data in which appearance learning information based on the extracted appearance information is used as an input value and the extracted activity is used as a correct answer value; A wakefulness level AI model that estimates the wakefulness level based on the appearance information is trained using the generated first supervised learning data; An activity level AI model that estimates the activity level based on the appearance information is generated by learning the generated second supervised learning data. Emotion estimation device.

Images