Learning device, estimation device, on-board emotion estimation system, method of learning, and learning program

Abstract

To provide a technology for forming an AI model that can derive an emotion index value without the need of a sensor device for detecting a biological signal.SOLUTION: A learning device causes an AI model (121 m, 122 m) to learn using a learning data set including learning input data that includes at least one of the pieces of information of input operation to a vehicle (V1) by a subject (U1) and information output from the vehicle (V1) in accordance with input operation to the vehicle (V1) and including an emotion index value in accordance with a biological signal of the subject as correct answer data.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a learning device, an estimation device, an in-vehicle emotion estimation system, a learning method, and a learning program. [Background technology]

[0002] 2. Description of the Related Art There is known an apparatus that estimates a person's emotion based on a biosignal measured by a biosensor (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-63324 A Summary of the Invention [Problem to be solved by the invention]

[0004] A method is being considered in which an emotion index value is derived as an index showing a person's emotion, and emotion estimation is performed based on the emotion index value. If a person's biosignal is detected, the emotion index value can be derived from the detected biosignal. However, it is difficult to have each person who is the subject of emotion estimation wear a sensor for detecting the biosignal, and wearing a sensor is often bothersome for each person who is the subject of emotion estimation.

[0005] The present invention aims to provide a technique for forming an AI model that can derive an emotion index value without requiring the wearing of a sensor for detecting biological signals. [Means for solving the problem]

[0006] The learning device of the present invention has learning input data including at least one of information on an input operation by a subject into a learning device and information output from the learning device in response to the input operation into the learning device, and trains an AI model using a learning dataset having an emotion index value corresponding to the biological signal of the subject as correct answer data. Effect of the Invention

[0007] According to the present invention, an AI model can be generated that can derive an emotion index value corresponding to correct answer data based on information of an input operation by a device user to a target device (the same type of device as the learning device) that corresponds to learning input data or information output from the target device in response to an input operation to the target device. Therefore, it is possible to provide a technology for forming an AI model that can derive an emotion index value without requiring the wearing of a sensor for detecting biological signals. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic configuration diagram of a feeling estimation device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of a two-dimensional model related to emotion estimation in the first embodiment of the present invention. [Diagram 3] FIG. 11 is a diagram showing another example of the two-dimensional model related to emotion estimation in the first embodiment of the present invention. [Figure 4] 1 is a configuration block diagram of a vehicle control system according to a first embodiment of the present invention. [Diagram 5] 1 is a block diagram of a sensor unit mounted on a vehicle according to a first embodiment of the present invention; [Figure 6] 4 is a flowchart of a feeling estimation process executed by the feeling estimation device in the first embodiment of the present invention. [Figure 7] FIG. 10 is a schematic configuration diagram of a learning device according to an example EX1_1 belonging to the first embodiment of the present invention. [Figure 8] FIG. 10 is a configuration block diagram of a learning device according to an example EX1_1 belonging to the first embodiment of the present invention. [Figure 9] 11 is a flowchart of a learning process executed by a learning device according to an example EX1_1 belonging to the first embodiment of the present invention. [Figure 10] FIG. 11 is a schematic configuration diagram of a learning device according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 11] FIG. 11 is a configuration block diagram of a learning device according to an example EX1_2 belonging to the first embodiment of the present invention. [Figure 12] FIG. 11 is a diagram showing the alertness levels of two subjects before and after normalization according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 13] FIG. 11 is a diagram showing a task table for normalization according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 14] FIG. 11 is a schematic diagram showing a change in alertness over time during task execution according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 15] FIG. 11 is a schematic diagram showing a change in wakefulness over time according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 16] FIG. 11 is a diagram showing the convergence state of an upper limit estimate value in a neutral region according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 17] FIG. 11 is a diagram showing a neutral region table according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 18] FIG. 11 is a diagram showing the alertness levels of two subjects before and after normalization according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 19] FIG. 11 is a schematic configuration diagram of a learning device according to Example EX1_3 belonging to the first embodiment of the present invention. [Figure 20] FIG. 11 is a configuration block diagram of a learning device according to an example EX1_3 belonging to the first embodiment of the present invention. [Figure 21] FIG. 11 is a schematic configuration diagram of a learning device according to Example EX1_4 belonging to the first embodiment of the present invention. [Figure 22] FIG. 11 is a schematic configuration diagram of a feeling estimation device according to Example EX1_4 belonging to the first embodiment of the present invention. [Diagram 23] FIG. 11 is a diagram showing the relationship between test subjects and learning devices, and the relationship between subjects and target devices according to the second embodiment of the present invention. [Figure 24]FIG. 11 is a schematic configuration diagram of a learning device according to a second embodiment of the present invention, when the learning device is a computer device. [Diagram 25] FIG. 11 is a schematic configuration diagram of a feeling estimation device according to a second embodiment of the present invention, when a target device is a computer device. [Figure 26] FIG. 11 is a diagram illustrating input operation information and device output information of a computer device when the learning device is a computer device according to a second embodiment of the present invention. [Figure 27] FIG. 11 is a diagram showing input operation information and device output information of a computer device when the target device is a computer device according to the second embodiment of the present invention. [Figure 28] FIG. 11 is a schematic configuration diagram of a learning device according to a second embodiment of the present invention, when the learning device is a game device. [Figure 29] FIG. 11 is a schematic configuration diagram of a feeling estimation device according to a second embodiment of the present invention, when a target device is a game device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an example of an embodiment of the present invention will be specifically described with reference to the drawings. In each of the drawings, the same parts are given the same reference numerals, and duplicated descriptions of the same parts are omitted as a rule. In this specification, for the sake of simplifying the description, a symbol or code referring to information, a signal, a physical quantity, a functional part, a circuit, an element, or a part, etc. may be written, and the name of the information, signal, physical quantity, functional part, circuit, element, or part, etc. corresponding to the symbol or code may be omitted or abbreviated. For example, the first AI model referred to by "221m" described later (see FIG. 1) may be written as the first AI model 221m, or may be abbreviated as the AI ​​model 221m or model 221m, but they all refer to the same thing. In addition, AI is an abbreviation for artificial intelligence.

[0010] <<First embodiment>> A first embodiment of the present invention will be described. FIG. 1 shows a schematic configuration of a feeling estimation device 20 according to the first embodiment. The feeling estimation device 20 includes an input data generation unit 232, a first AI model 221m, a second AI model 222m, and a feeling estimation model 223m. The feeling estimation device 20 has a function of estimating the feeling of a person U2. The person U2 is a target of feeling estimation, and is hereinafter referred to as a target person U2. The target person U2 is a driver of a vehicle V2 as a target vehicle. Here, it is assumed that the vehicle V2 is an automobile or the like that runs on a road surface. The vehicle V2 may be any type of vehicle.

[0011] The emotion estimation model 223m estimates emotions based on two emotion indices, which are indices indicating the mental and physical state of the subject U2. The mental and physical state of any person includes a psychological state, and therefore each emotion index can also be said to be an index indicating a psychological state. The value of an emotion index is called an emotion index value. One of the emotion indices used in this embodiment is the arousal level of the central nervous system (hereinafter referred to as the arousal level), and the other emotion index used in this embodiment is the activity level of the autonomic nervous system (hereinafter referred to as the activity level). That is, one emotion index value is expressed by the arousal level, and the other emotion index value is expressed by the activity level. The arousal level can be derived based on, for example, electroencephalogram data. The activity level can be derived based on, for example, heart rate data.

[0012] The emotion estimation model 223m is a model for emotion estimation based on arousal and activity, and may be configured with a calculation formula or a conversion data table that specifies emotion from arousal and activity. The emotion estimation model 223m is a multidimensional model that estimates emotion using arousal and activity as parameters, and is a two-dimensional model with arousal and activity as two axes here. The two-dimensional model is created based on medical evidence (papers, etc.) showing the relationship between arousal and activity and emotion. Alternatively, the two-dimensional model is created based on the results of a questionnaire by many subjects. The questionnaire results of the subjects include the arousal and activity of the subjects specified from the measured values ​​of the subject's brain waves and heart rate, and the subject's declared emotion at the time of measuring the brain waves and heart rate. Note that a multidimensional model of three or more dimensions may be used as the emotion estimation model 223m.

[0013] FIG. 2 shows an example of a two-dimensional model related to emotion estimation. FIG. 2 shows a two-dimensional psychological plane PP defined in the two-dimensional model. According to various medical evidence related to psychology, psychology can be estimated based on two types of indices that indicate mental and physical states. The psychological plane PP shown in FIG. 2 has a vertical axis corresponding to the arousal level and a horizontal axis corresponding to the activity level.

[0014] Points indicating the awakening and activity of the subject U2 can be plotted (placed) on the psychological plane PP. The higher the awakening of the subject U2, the more the coordinates of the plotted (placed) points move in the positive direction of the vertical axis, and the lower the awakening of the subject U2, the more the coordinates of the plotted (placed) points move in the negative direction of the vertical axis. The higher the activity of the subject U2, the more the coordinates of the plotted (placed) points move in the positive direction of the horizontal axis, and the lower the activity of the subject U2, the more the coordinates of the plotted (placed) points move in the negative direction of the horizontal axis. From the origin of the psychological plane PP, the positive side of the vertical axis corresponds to an awakened state, and the negative side of the vertical axis corresponds to an unawakened state. From the origin of the psychological plane PP, the positive side of the horizontal axis corresponds to a state in which the sympathetic nerves are activated (a state in which a relatively strong emotion is present), and the negative side of the horizontal axis corresponds to a state in which the parasympathetic nerves are activated (a state in which a relatively weak emotion is present).

[0015] The vertical and horizontal axes intersect at the origin of the psychological plane PP, and the psychological plane PP has first to fourth quadrants separated by the vertical and horizontal axes. The first quadrant is an area where the vertical axis component has a positive value and the horizontal axis component also has a positive value when viewed from the origin of the psychological plane PP. The second quadrant is an area where the vertical axis component has a positive value and the horizontal axis component also has a negative value when viewed from the origin of the psychological plane PP. The third quadrant is an area where the vertical axis component has a negative value and the horizontal axis component also has a negative value when viewed from the origin of the psychological plane PP. The fourth quadrant is an area where the vertical axis component has a negative value and the horizontal axis component has a positive value when viewed from the origin of the psychological plane PP.

[0016] In the psychological plane PP, each quadrant is assigned a corresponding psychological state (in other words, an emotion). The first quadrant is assigned the psychological states of "fun, joy, anger, sadness." The second quadrant is assigned the psychological state of "melancholy." The third quadrant is assigned the psychological state of "relaxation, calm." The fourth quadrant is assigned the psychological states of "anxiety, fear, unpleasantness." The distance from the origin indicates the intensity of the corresponding psychological state. For example, when a point indicating the arousal and activity of subject U2 is plotted in the second quadrant, the greater the distance between the plotted point and the origin, the stronger the psychological state of "melancholy" for subject U2.

[0017] In this way, the emotion of any person depends on the arousal level and activity level of the person. The position of each axis on the psychological plane PP may be appropriately set based on an experiment or the like. Each axis may be set based on the position of a neutral area described later.

[0018] The emotion estimation model 223m can estimate the psychological state of the subject U2 from coordinates obtained by plotting (arranging) points indicating the arousal and activity of the subject U2 on the psychological plane PP. The psychological state and its intensity of the subject U2 can be estimated based on which quadrant of the psychological plane PP the coordinates of the plotted point are in, where in the quadrant they are located, and the distance between the plotted point and the origin. The first to fourth quadrants on the psychological plane PP can be considered as first to fourth classes, respectively, and in this case, the emotion estimation model 223m can be said to classify the psychological state of the subject U2 into one of the first to fourth classes. In addition, when the emotion estimation model 223m is a multidimensional model of three or more dimensions, a multidimensional psychological space of three or more dimensions is used depending on the number of types of emotion index values ​​to be used.

[0019] Incidentally, when the emotional intensity is strong, that is, when the emotional index value fluctuates greatly towards the maximum value side or the minimum value side, the emotional estimation accuracy is likely to increase. On the other hand, when the emotional intensity is weak, that is, when the emotional index value is near the median value, the emotional estimation accuracy is likely to decrease. Therefore, it is also possible to adopt a method of setting the region near the median value of the emotional index value as a neutral region and making a determination such as non-estimated emotion or non-estimable emotion for the emotional index value corresponding to the neutral region.

[0020] FIG. 3 is a diagram showing an example of a psychological plane PP including a neutral region. In FIG. 3, the hatched region NR1 extending along the horizontal axis and the hatched region NR2 extending along the vertical axis are both neutral regions. Among these, the region NR1 is an arousal neutral region, and the region NR2 is an activity neutral region.

[0021] The arousal neutral region NR1 is a region associated with an arousal level that is equal to or higher than a lower limit value YN and equal to or lower than an upper limit value YP. "YN < YP". The horizontal axis passing through the origin of the psychological plane PP is located within the arousal neutral region NR1. The arousal neutral region NR1 is a region extending along the direction of the horizontal axis of the psychological plane PP, and the horizontal axis of the psychological plane PP may be arranged at the center of the arousal neutral region NR1. The activity neutral region NR2 is a region associated with an activity level that is equal to or higher than a lower limit value XN and equal to or lower than an upper limit value XP. "XN < XP". The vertical axis passing through the origin of the psychological plane PP is located within the activity neutral region NR2. The activity neutral region NR2 is a region extending along the direction of the vertical axis of the psychological plane PP, and the vertical axis of the psychological plane PP may be arranged at the center of the activity neutral region NR2. A part of the arousal neutral region NR1 and a part of the activity neutral region NR2 overlap each other in the region including the origin of the psychological plane PP. The origin of the psychological plane PP may be arranged at the center of the region where the regions NR1 and NR2 overlap.

[0022] The setting of these neutral regions (i.e., the setting of the upper limit YP, the lower limit YN, the upper limit XP, and the lower limit XN) may be appropriately performed through experiments, etc. An example of a method for setting the neutral region will be described later. Each position in the neutral region may be understood as belonging to none of the first to fourth quadrants of the psychological plane PP.

[0023] Calculating emotion index values ​​usually requires a biosignal, and in most cases, a contact-type biosensor is required to acquire the biosignal. However, wearing a biosensor is troublesome for emotion estimation subjects, especially for vehicle drivers (subject U2 in this case) who tend to frequently wear and remove the biosensor. For this reason, an emotion estimation method that does not require wearing a biosensor is required.

[0024] Here, when an arbitrary person operates an arbitrary device, the emotion of the person is expressed in the input operation to the device and also in the information output from the device in response to the input operation. In other words, when focusing on the subject U2 and the vehicle V2, the emotion of the subject U2 is expressed in the input operation to the vehicle V2 and also in the information output from the vehicle V2 in response to the input operation to the vehicle V2. The input operation to the vehicle V2 is the driving operation of the vehicle V2 by the subject U2, and corresponds to the accelerator pedal operation, the brake pedal operation, the steering operation, and the like. The information output from the vehicle V2 includes the running state information of the vehicle V2, and the running state information of the vehicle V2 indicates the speed, the steering angle, the acceleration, and the like of the vehicle V2. The information output from the vehicle V2 may include the degree of sway of the vehicle V2 in response to the driving operation. On the other hand, emotions are correlated with biological information such as brain waves and heart rate. Therefore, the correlation between the information of the input operation to the vehicle V2 and the information output from the vehicle V2 in response to the input operation to the vehicle V2, and biological information such as brain waves and heart rate can be modeled.

[0025] In the feeling estimation device 20, as shown in Fig. 1, the input data generation unit 232 generates estimation input data based on the reference data D2. The reference data D2 includes output information (sensor output information Sout2) from a sensor unit provided in the vehicle V2. Details of these data and information will be described later.

[0026] The AI ​​models 221m and 222m are trained models obtained through prior training. In this specification, training refers to machine learning. The training method will be described later. The first AI model 221m is a model that outputs (calculates) the alertness of the subject U2 based on the input data for estimation, and has a structure suitable for input and output. The second AI model 222m is a model that outputs (calculates) the activity of the subject U2 based on the input data for estimation, and has a structure suitable for input and output. Any AI model may be configured by a neural network. The alertness and activity of the subject U2 are each a type of biological information of the subject U2. In any person, the alertness correlates with brain waves, and the activity correlates with heart rate.

[0027] The emotion estimation model 223m estimates the emotion of the subject U2 based on the arousal level and activity level output from the AI ​​models 221m and 222m.

[0028] 4 is a configuration block diagram of the vehicle control system SYS according to the first embodiment. The vehicle control system SYS includes a feeling estimation device 20, a vehicle control device 30, an actuator unit 40, a notification unit 50, and a sensor unit 60. The feeling estimation device 20, the vehicle control device 30, the actuator unit 40, the notification unit 50, and the sensor unit 60 are all mounted on a vehicle V2.

[0029] The feeling estimation device 20 includes a communication unit 21, a storage unit 22, and a controller 23. In the vehicle control system SYS, vehicle control can be performed using the feeling estimation result by the feeling estimation device 20. When performing vehicle control requiring quick response using the feeling estimation result, it is desirable to mount the feeling estimation device 20 on the vehicle V2 as in this example. However, the feeling estimation device 20 may be configured using a server device provided outside the vehicle V2. In this case, an in-vehicle block having a part of the feeling estimation device 20 is installed on the vehicle V2, while a server block having the remaining part of the feeling estimation device 20 is installed on the server device, and the feeling estimation device 20 is configured by the cooperation of the in-vehicle block and the server block. The in-vehicle block and the server block share any information by communication via a predetermined communication network. For example, among the function blocks 231 to 234 described later, the function blocks 232 and 233 are realized by the server block, and the other function blocks 231 and 234 are realized by the in-vehicle block. In addition, the allocation of the functional blocks to the server block and the in-vehicle block is arbitrary. The server device is composed of one or more computer devices connected to a communication network. When the emotion estimation device 20 is configured by the cooperation of the in-vehicle block and the server block, the part of the memory contents of the memory unit 22 necessary for realizing the operation of the server block is held in a memory unit (not shown) in the server device. That is, for example, when the functional block 233 is provided in the server block, the memory units 221 to 223 shown in FIG. 4 are held in a memory unit (not shown) in the server device.

[0030] The communication unit 21 transmits and receives any signal between the emotion estimation device 20 and a counterpart device different from the communication unit 21. The counterpart device for the communication unit 21 includes a vehicle control device 30. The communication unit 21 can communicate with the counterpart device via an in-vehicle network formed in the vehicle V2. The in-vehicle network may include, for example, a Controller Area Network (CAN) and may further include an Audio Visual Communication Local Area Network (AVCLAN). The counterpart device for the communication unit 21 may further include a sensor unit 60, and in this case, the communication unit 21 may be able to directly receive output information from the sensor unit 60. The counterpart device for the communication unit 21 may further include an external device (such as a server device connected to the above-mentioned communication network) provided outside the vehicle V2.

[0031] The storage unit 22 is configured to have a non-volatile memory such as a ROM (Read only memory) or a flash memory, and a volatile memory such as a RAM (Random access memory). The storage unit 22 includes a first AI model storage unit 221 that stores a first AI model 221m, and a second AI model storage unit 222 that stores a second AI model 222m. The storage unit 22 further includes an emotion estimation model storage unit 223 that stores an emotion estimation model 223m. In addition to the emotion estimation model 223m, the emotion estimation model storage unit 223 also stores various data required for emotion estimation, such as a psychological plane table 223t. The psychological plane table 223t is data that forms the psychological plane PP shown in FIG. 2 or FIG. 3, and has a data group that associates two types of emotion index values ​​(arousal and activity) with the corresponding emotion information. The storage unit 22 further includes a program storage unit 224 that stores various programs. The programs stored in the program storage unit 224 include a program 224p.

[0032] The controller 23 comprehensively controls the operation of each part in the feeling estimation device 20. The controller 23 includes a calculation processing unit including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) as hardware resources. The controller 23 includes functional blocks 231 to 234.

[0033] The controller 23 is a program execution device (computer) capable of executing any program. The controller 23 executes the program 224p to realize each function of the controller 23 (including each function of the functional blocks 231 to 234). All of the operations of the controller 23 described in this embodiment may be operations that are realized by the controller 23 executing the program 224p. The program 224p may be composed of a plurality of programs. The controller 23 can transmit and receive any information to and from a counterpart device (a counterpart device for the communication unit 21) using the communication unit 21, but in the following, a description of the communication unit 21 may be omitted.

[0034] The functional blocks of the controller 23 will be described. Functional blocks 231, 232, 233, and 234 are an acquisition unit, an input data generation unit, a feeling estimation unit, and an estimation result output unit, respectively. The acquisition unit 231 acquires information used to estimate the feeling of the subject U2. The information acquired by the acquisition unit 231 includes information generated by the sensor unit 60. The input data generation unit 232 generates input data for estimation based on the information acquired by the acquisition unit 231 (see also FIG. 1).

[0035] The feeling estimation unit 233 estimates the feeling of the subject U2 using the AI ​​models 221m and 222m and the feeling estimation model 223m based on the input data for estimation. Specifically, the input data for estimation generated by the input data generation unit 232 is input to the AI ​​models 221m and 222m. As a result, the AI ​​models 221m and 222m output the arousal level and activity level, which are the emotion index values ​​of the subject U2. Next, the feeling estimation unit 233 inputs the arousal level and activity level output from the AI ​​models 221m and 222m to the feeling estimation model 223m. As a result, the feeling of the subject U2 is estimated by the feeling estimation model 223m, and the estimation result is output from the feeling estimation model 223m.

[0036] Coordinates obtained by plotting (arranging) points indicating the arousal and activity of the subject U2 on the psychological plane PP are referred to as plot coordinates. The plot coordinates represent the positions of the points indicating the arousal and activity of the subject U2 (positions on the psychological plane PP). The emotion estimation unit 233 estimates the emotion of the subject U2 by identifying in which quadrant of the psychological plane PP the plot coordinates are located. It may be considered that the estimation of the emotion of the subject U2 is performed by the emotion estimation model 223m.

[0037] When the plot coordinates are located in the i-th quadrant, the emotion estimation unit 233 (emotion estimation model 223m) estimates that the emotion of the subject U2 corresponds to the psychological state assigned to the i-th quadrant. Here, i represents any integer. Therefore, for example, when the plot coordinates are located in the first quadrant, the emotion estimation unit 233 (emotion estimation model 223m) estimates that the subject U2 has an emotion corresponding to "joy, joy, anger, or sadness". Similarly, for example, when the plot coordinates are located in the second quadrant, the emotion estimation unit 233 (emotion estimation model 223m) estimates that the subject U2 has an emotion corresponding to "melancholy". The same applies when the plot coordinates are located in the third or fourth quadrant.

[0038] When the psychological plane PP including the neutral region is used, the plot coordinates may be located within the neutral region. In this case, the emotion estimation unit 233 (emotion estimation model 223m) estimates that the subject U2 does not have any particular emotion or determines that emotion estimation of the subject U2 is impossible.

[0039] The estimation result output unit 234 outputs the estimation result of the emotion of the subject U2 by the emotion estimation unit 233 to the vehicle control device 30. This makes it possible for the vehicle control device 30 to control the vehicle V2 based on the emotion of the subject U2.

[0040] The vehicle control device 30 is configured by, for example, an ECU (Electronic Control Unit) for vehicle control. The vehicle control device 30 includes a communication unit 31, a storage unit 32, and a controller 33.

[0041] The communication unit 31 transmits and receives any signal between the vehicle control device 30 and a counterpart device different from the vehicle control device 30. The counterpart device for the communication unit 31 includes the emotion estimation device 20, the actuator unit 40, the notification unit 50, and the sensor unit 60, and the communication unit 31 can communicate with the counterpart device via a vehicle network formed in the vehicle V2. The counterpart device for the communication unit 31 may further include an external device (such as a server device connected to the above-mentioned communication network) provided outside the vehicle V2. The controller 33 can transmit and receive any information to and from the counterpart device (the counterpart device for the communication unit 31) using the communication unit 31, but hereinafter, a description of the communication unit 31 may be omitted.

[0042] The storage unit 32 is configured to include a non-volatile memory such as a ROM or a flash memory, and a volatile memory such as a RAM. The storage unit 32 stores programs to be executed by the controller 33, as well as any data to be referenced by the controller 33.

[0043] The controller 33 performs driving control of the vehicle V2 based on the driving operation of the subject U2 of the vehicle V2 and the output information of the sensor unit 60. The controller 33 also receives estimated emotion information of the subject U2 from the emotion estimation device 20 via the communication unit 31. The estimated emotion information of the subject U2 represents the emotion of the subject U2 estimated by the emotion estimation device 20 (emotion estimation unit 233). The controller 33 can perform vehicle control using the estimated emotion information. In vehicle control using the estimated emotion information, for example, the speed, accelerator sensitivity, and brake sensitivity of the vehicle V2 are controlled. More specifically, for example, when the driver is excited, the controller 33 can perform driving control that is closer to the safe side than when the driver is not excited. Examples of driving control that is closer to the safe side include control that lowers the accelerator sensitivity (to make it harder to accelerate (to make acceleration gentle)), increases the brake sensitivity (to make it easier to stop), and lowers the upper limit of the maximum speed control.

[0044] In addition, the controller 33 performs notification control for notifying various types of notification information to the occupants of the vehicle V2. The notification information is transmitted to the notification unit 50 via the communication unit 31. The occupants of the vehicle V2 include the target person U2 who is the driver of the vehicle V2, and may also include other passengers.

[0045] The actuator unit 40 has various driving components constituted by a motor and the like that realize the running of the vehicle V2. The operation of the actuator unit 40 is driven and controlled by the vehicle control device 30. The actuator unit 40 includes an engine and a motor that generate a driving force for the vehicle V2, a steering actuator that drives the steering of the vehicle V2, a brake actuator that drives the brakes of the vehicle V2, and the like.

[0046] The notification unit 50 visually or audibly notifies the occupant of the vehicle V2 of the notification information from the vehicle control device 30. The notification unit 50 has a display device and a speaker. The notification unit 50 can notify the notification information from the vehicle control device 30 by displaying an image on the display device. The notification unit 50 can notify the notification information from the vehicle control device 30 by outputting an audio signal from the speaker.

[0047] The sensor unit 60 has a sensor that detects the content of the driving operation of the vehicle V2 by the subject U2 and sensors that detect various states of the vehicle V2.

[0048] The vehicle control device 30 (controller 33) generates and outputs a control signal for the vehicle V2 based on the estimated emotion information of the subject U2 provided by the emotion estimation device 20, in addition to information from various sensors. For example, the vehicle control device 30 controls the speed of the vehicle V2 based on information from a vehicle speed sensor, a distance measurement sensor, etc., and when it is estimated that the subject U2 is emotionally excited, the vehicle control device 30 performs speed control that reduces the speed of the vehicle V2 (control to limit (reduce) the maximum speed), control to limit (reduce) the maximum acceleration, etc., compared to when it is not.

[0049] The vehicle control device 30 (controller 33) can also cause the notification unit 50 to perform a notification based on the estimated emotion information of the subject U2. At this time, the vehicle control device 30 (controller 33) can use the notification unit 50 to notify the estimated emotion information itself, or can perform a notification according to the estimated emotion information. For example, when it is estimated that the subject U2 is excited or scared, the vehicle control device 30 (controller 33) uses the notification unit 50 to display a text or output a voice such as "Let's calm down." At this time, control is performed to make the quality of the voice in the voice output calm. To achieve this, the storage unit 32 is provided with a data table for various processes in which the estimated emotion information received from the emotion estimation device 20 is associated with the control signal and notification information of the vehicle V2. Note that the control to cause the notification unit 50 to perform a notification based on the estimated emotion information of the subject U2 may be performed by the controller 23 of the emotion estimation device 20.

[0050] In this way, the vehicle control system SYS estimates the emotion of the subject U2 who is the driver of the vehicle V2, and controls the driving of the vehicle V2 taking the estimated emotion into consideration. This enables the driving control of the vehicle V2 according to the driver's emotion that affects safe driving, and is expected to contribute to safe driving of the vehicle V2.

[0051] 5 shows an internal block diagram of the sensor unit 60. The sensor unit 60 includes sensors 61 to 67. The sensors 61, 62, and 63 are an accelerator pedal sensor, a brake pedal sensor, and a steering wheel sensor, respectively. The sensors 64, 65, 66, and 67 are a vehicle speed sensor, a steering angle sensor, a G sensor, and a distance measurement sensor, respectively. Each vehicle described in this embodiment is provided with operational parts that receive driving operations from the driver, and the operational parts include an accelerator pedal, a brake pedal, and a steering wheel.

[0052] The accelerator pedal sensor 61 detects the operation of the accelerator pedal of the vehicle V2 by the subject U2, and generates and outputs accelerator pedal operation information indicating the operation of the accelerator pedal. The brake pedal sensor 62 detects the operation of the brake pedal of the vehicle V2 by the subject U2, and generates and outputs brake pedal operation information indicating the operation of the brake pedal. The steering wheel sensor 63 detects the operation of the steering wheel of the vehicle V2 by the subject U2, and generates and outputs steering wheel operation information indicating the operation of the steering wheel.

[0053] The vehicle speed sensor 64 detects the speed of the vehicle V2 and outputs vehicle speed information (vehicle speed pulse) representing the detected speed. The steering angle sensor 65 detects the steering angle (steering angle) of the vehicle V2 and outputs steering angle information representing the detected steering angle. The G sensor 66 detects acceleration in a predetermined axial direction applied to the vehicle V2 and generates and outputs the acceleration detection result as acceleration information. The G sensor 66 may detect acceleration in two mutually orthogonal axial directions or may detect acceleration in three mutually orthogonal axial directions.

[0054] The distance measurement sensor 67 generates and outputs distance measurement information by performing distance measurement. In the distance measurement by the distance measurement sensor 67, the distance between the vehicle V2 and a solid object (three-dimensional object) located within the distance measurement area around the vehicle V2 is detected, and the orientation of the solid object as viewed from the vehicle V2 is also detected. These detection results are included in the distance measurement information. The distance measurement sensor 67 may be configured with a LIDAR (Light Detection and Ranging) that performs distance measurement using light, or may be configured with a radar that performs distance measurement using radio waves. The distance measurement sensor 67 may be configured with a combination of LIDAR and radar.

[0055] Hereinafter, information obtained by the sensor unit 60 of the vehicle V2 will be referred to as sensor output information Sout2. The sensor output information Sout2 includes accelerator pedal operation information, brake pedal operation information, steering wheel operation information, vehicle speed information, steering angle information, acceleration information, and distance measurement information obtained by the sensor unit 60 of the vehicle V2. The sensor output information Sout2 may further include information indicating the temperature and humidity in the passenger compartment of the vehicle V2. The controller 33 performs driving control of the vehicle V2 (e.g., control of the speed and traveling direction of the vehicle V2) through drive control of the actuator unit 40 based on the sensor output information Sout2.

[0056] In driving the vehicle V2, the subject U2 operates the accelerator pedal, brake pedal, and steering wheel provided on the vehicle V2. That is, the driving operation of the vehicle V2 by the subject U2 includes the operation of the accelerator pedal, brake pedal, and steering wheel of the vehicle V2. Therefore, the sensor output information Sout2 includes information on the driving operation of the vehicle V2, and the accelerator pedal operation information, brake pedal operation information, and steering wheel operation information of the vehicle V2 belong to the information on the driving operation of the vehicle V2.

[0057] The controller 33 controls the speed of the vehicle V2 (controls the acceleration or deceleration of the vehicle V2) based on the operation of the accelerator pedal and the brake pedal of the vehicle V2. The controller 33 controls the steering angle of the vehicle V2 based on the operation of the steering wheel of the vehicle V2.

[0058] The sensor output information Sout2 includes information that depends on the driving operation of the vehicle V2. The information that depends on the driving operation of the vehicle V2 corresponds to the vehicle speed information, steering angle information, acceleration information, and distance measurement information of the vehicle V2. This is because the speed, steering angle, and acceleration of the vehicle V2 change according to the driving operation of the vehicle V2, and the distance between the vehicle V2 and an object other than the vehicle V2 (for example, the distance ahead) changes according to the driving operation of the vehicle V2.

[0059] Fig. 6 is a flowchart showing the emotion estimation process executed by the controller 23. The emotion estimation process may be a process realized by the controller 23 executing the above-mentioned program 224p. The emotion estimation process shown in Fig. 6 is activated at the timing when various controls based on the estimated emotion are started after the start of the vehicle V2, and is thereafter executed once or repeatedly executed multiple times at any timing when emotion estimation is required. The emotion estimation process has the processes of steps S21 to S25.

[0060] In the emotion estimation process, first, data acquisition process is performed in step S21. In the data acquisition process, the acquisition unit 231 acquires reference data D2 for an estimation evaluation period. The estimation evaluation period is an arbitrary period having a predetermined time length (for example, several seconds to several tens of seconds), and in the emotion estimation process, the emotion of the subject U2 in the estimation evaluation period is estimated. The reference data D2 includes sensor output information Sout2 (see FIG. 1 and FIG. 5) obtained by the sensor unit 60 of the vehicle V2. Therefore, the reference data D2 acquired in step S21 includes the sensor output information Sout2 in the estimation evaluation period. Information other than the sensor output information Sout2 (for example, photographed image data of the subject U2, photographed image data around the vehicle V2) may be further included in the reference data D2. After step S21, the process proceeds to step S22.

[0061] In step S22, the input data generation unit 232 generates AI input data by performing data processing on the reference data D2 in the estimation evaluation period. In the feeling estimation device 20, the AI ​​input data generated in step S22 functions as estimation input data. In the feeling estimation device 20, the AI ​​input data generated here is input to the AI ​​models 221m and 222m as estimation input data. After step S22, the process proceeds to step S23.

[0062] In the data processing, the reference data D2 is processed into a data format to be input to the AI ​​models 221m and 222m, and the processed data is obtained as AI input data. The data processing is set according to the input specifications of the AI ​​models 221m and 222m. The data processing may include, for example, various signal processes (averaging, peak detection, frequency analysis, high-frequency component extraction, low-frequency component extraction, etc.) for each piece of information in the reference data D2. In addition, the AI ​​input data may be time-series data consisting of data (instantaneous values) acquired at appropriate predetermined time intervals during the estimation evaluation period.

[0063] More specifically, for example, a statistical value of the amount of depression of the accelerator pedal during the evaluation period for estimation is included in the AI ​​input data. Similarly, for example, a statistical value of the amount of depression of the brake pedal during the evaluation period for estimation is included in the AI ​​input data. Furthermore, for example, time series data of the amount of steering wheel operation during the evaluation period for estimation is included in the AI ​​input data. Furthermore, for example, a statistical value of the speed of vehicle V2 during the evaluation period for estimation is included in the AI ​​input data. Furthermore, for example, a statistical value of the steering angle of vehicle V2 during the evaluation period for estimation is included in the AI ​​input data. Furthermore, for example, a statistical value of the acceleration of vehicle V2 during the evaluation period for estimation is included in the AI ​​input data. The statistical values ​​include, for example, an average value, a maximum value, a minimum value, a standard deviation, and a variance.

[0064] Also, for example, in data processing, the front inter-vehicle distance and the rear inter-vehicle distance are derived from the distance measurement information during the estimation evaluation period, and the front inter-vehicle distance and the rear inter-vehicle distance are included in the AI ​​input data. For vehicle V2, the front inter-vehicle distance refers to the distance between vehicle V2 and another vehicle traveling immediately ahead of vehicle V2. For vehicle V2, the rear inter-vehicle distance refers to the distance between vehicle V2 and another vehicle traveling immediately behind vehicle V2.

[0065] In step S23, emotion index values ​​are derived in the AI ​​models 221m and 222m based on the input data for estimation. That is, the emotion estimation unit 233 inputs the input data for estimation to the AI ​​model 221m, and thus the arousal level is derived in the AI ​​model 221m. The emotion estimation unit 233 inputs the input data for estimation to the AI ​​model 222m, and thus the activity level is derived (estimated) in the AI ​​model 222m. The arousal level and activity level derived here are the arousal level and activity level of the subject U2 in the estimation evaluation period. After step S23, the process proceeds to step S24.

[0066] In step S24, the feeling estimation unit 233 estimates the feeling of the subject U2 by inputting the arousal level and activity level derived in step S23 to the feeling estimation model 223m, and generates estimated feeling information representing the estimation result. After step S24, the process proceeds to step S25.

[0067] In step S25, the estimation result output unit 234 outputs the estimated emotion information of the subject U2 obtained in step S24 to the vehicle control device 30. With the processing of step S25, one cycle of the emotion estimation processing ends.

[0068] Incidentally, the emotion estimated in one emotion estimation process may be affected by noise, etc. For this reason, the estimation result output unit 234 may perform statistical processing on multiple pieces of estimated emotion information obtained by executing the emotion estimation process multiple times, and output the information obtained thereby to the vehicle control device 30.

[0069] The first embodiment includes the following Examples EX1_1 to EX1_5. The matters described above in the first embodiment are applied to the following Examples EX1_1 to EX1_5 unless otherwise specified and unless there is a contradiction. However, in each example, for matters that contradict the matters described above in the first embodiment, the description in each example may take precedence. Furthermore, unless there is a contradiction, matters described in any of Examples EX1_1 to EX1_5 can also be applied to any other example (i.e., any two or more of the multiple examples can be combined).

[0070] [Example EX1_1] Example EX1_1 will be described. In the example EX1_1, a learning device for obtaining the AI ​​models 221m and 222m mounted on the feeling estimation device 20 will be described.

[0071] FIG. 7 shows a schematic configuration of the learning device 10 according to the embodiment EX1_1. The learning device 10 includes an index value derivation unit 133, an input data generation unit 132, a first AI model 121m, and a second AI model 122m. The learning device 10 learns the AI ​​models 121m and 122m. The trained AI models 121m and 122m are mounted on the emotion estimation device 20 as the AI ​​models 221m and 222m, respectively (see FIGS. 1 and 4). The learning shown in the embodiment EX1_1 and any embodiment described later is supervised machine learning, and therefore, a learning data set including teacher data (correct answer data) is required for learning.

[0072] In FIG. 7, U1 represents a subject involved in the generation of the learning dataset. The subject U1 gets in and drives the vehicle V1. In other words, the subject U1 is the driver of the vehicle V1. The learning dataset is generated by the subject U1 driving the vehicle V1. The learning dataset is generated by acquiring biometric information of the subject U1. The vehicle V1 is assumed to be an automobile or the like that runs on a road surface, similar to the vehicle V2.

[0073] When forming an individual-specific emotion estimation device 20 in which the emotion estimator is limited, the subject U1 is the same person as the target person U2. However, since there is similarity between different people, the subject U1 may be a person different from the target person U2. When forming a general-purpose emotion estimation device 20 in which the person whose emotion is estimated is not limited, the subject U1 and the target person U2 may be the same person. If multiple subjects U1 are prepared, a learning dataset from people with various characteristics can be obtained, and it is expected that a highly versatile emotion estimation device 20 can be obtained.

[0074] A plurality of subjects U1 may be prepared, and each subject U1 may be the driver of a vehicle V1 and perform various patterns of experimental driving, and at this time, a vehicle V1 may be assigned to each subject U1. This makes it easier to obtain a large amount of learning data set in a short time. The plurality of subjects U1 may or may not include the subject U2. However, in the following, for the sake of concreteness and clarification of the explanation, attention will be focused on one subject U1 and one vehicle V1 unless otherwise necessary.

[0075] A biosensor is attached to the subject U1 to acquire bioinformation of the subject U1. Here, it is assumed that a brainwave sensor ES and a heartbeat sensor HS are attached to the subject U1 as the biosensors. The brainwave sensor ES detects the brainwaves of the subject U1, and outputs brainwave data indicating the results of the brainwave detection. The brainwave data includes information on the detected brainwaves. The heartbeat sensor HS detects the heartbeat of the subject U1, and outputs heartbeat data indicating the results of the heartbeat detection. The heartbeat data includes information on the detected heartbeat. For example, a headgear-type brainwave sensor is used as the brainwave sensor ES. For example, a chest belt-type electrocardiogram-type heartbeat sensor is used as the heartbeat sensor HS.

[0076] Data obtained from the biosensor attached to the subject U1 is called biodata. Therefore, the biodata includes brainwave data and heart rate data, and represents bioinformation of the subject U1. Depending on the type of bioinformation to be acquired or the wearing comfort, other sensors may be included in the biosensor and attached to the subject U1. Examples of other sensors include a blood pressure monitor, a NIRS (Near Infrared Spectroscopy) device, or an optical heart rate sensor.

[0077] The biological data of the subject U1 is input to the index value derivation section 133. The index value derivation section 133 derives and outputs the arousal level and activity level of the subject U1 as emotion index values ​​of the subject U1 based on the biological data of the subject U1.

[0078] The index value derivation unit 133 derives the level of awakening of the subject U1 based on the electroencephalogram data of the subject U1. The level of awakening is derived based on the beta / alpha waves of the electroencephalogram. That is, a value according to the beta / alpha waves of the electroencephalogram (for example, a value proportional to the beta / alpha waves of the electroencephalogram) is derived as the level of awakening. The beta / alpha waves of the electroencephalogram are the ratio of beta waves of the electroencephalogram to alpha waves of the electroencephalogram. More specifically, for example, the beta / alpha waves of the electroencephalogram are "INT_β / INT_α" or "PW_β / PW_α". INT_β and INT_α represent the intensities of the beta and alpha waves of the electroencephalogram, respectively. PW_β and PW_α represent the powers of the beta and alpha waves of the electroencephalogram, respectively.

[0079] The index value derivation unit 133 derives the activity of the subject U1 based on the heartbeat data of the subject U1. More specifically, the activity is calculated based on the LF component of the time interval between heartbeats (hereinafter referred to as the heartbeat interval). The heartbeat sensor HS may detect an action potential of the heart of the subject U1, which is an electric potential generated on the skin surface of the subject U1, and generate a cardiac potential signal indicating the detected action potential. The heartbeat sensor HS detects a heartbeat interval, which is a time interval between the heartbeats of the subject U1, based on the cardiac potential signal. The heartbeat interval corresponds to the RR interval of the cardiac potential signal (however, it may be the interval between two adjacent Q waves in the cardiac potential signal, etc.). The heartbeat data from the heartbeat sensor HS includes a signal indicating the heartbeat interval (may further include a cardiac potential signal). It is not essential that the cardiac potential signal is generated in the heartbeat sensor HS. The index value derivation unit 133 derives the LF component of the heartbeat interval based on the heartbeat data. The LF components of the heartbeat interval are frequency components in a predetermined low frequency band (for example, a frequency band from 0.04 to 0.15 Hz) among the frequency components included in the heartbeat interval. For the subject U1, the index value derivation unit 133 derives the activity level of the subject U1 by, for example, calculating the standard deviation in the amplitude of the LF components of the heartbeat interval. At this time, a value according to the standard deviation (for example, a value proportional to the standard deviation) is derived as the activity level. In the amplitude spectrum of the heartbeat interval, the frequency components in the low frequency band of the heartbeat interval are made up of 1st to kth frequency components (k is an integer equal to or greater than 2), and the standard deviation corresponds to the standard deviation in the amplitude of the 1st to kth frequency components.

[0080] The vehicle V1 is equipped with a sensor unit equivalent to the sensor unit 60 equipped in the vehicle V2. The sensor unit of the vehicle V1 generates sensor output information equivalent to the sensor output information Sout2 obtained by the sensor unit 60 of the vehicle V2. The sensor output information generated by the sensor unit of the vehicle V1 is called sensor output information Sout1. The sensor output information Sout1 includes accelerator pedal operation information, brake pedal operation information, steering wheel operation information, vehicle speed information, steering angle information, acceleration information, and distance measurement information of the vehicle V1. The above-mentioned matters regarding those information also apply to the sensor unit and sensor output information Sout1 of the vehicle V1. However, in this application, the vehicle V2 and the subject U2 are read as the vehicle V1 and the subject U1. That is, for example, in the sensor unit of the vehicle V1, the accelerator pedal sensor 61 detects the operation contents of the accelerator pedal of the vehicle V1 by the subject U1, and generates and outputs accelerator pedal operation information indicating the operation contents of the accelerator pedal. Similarly, for example, in the sensor section of the vehicle V1, the vehicle speed sensor 64 detects the speed of the vehicle V1, and outputs vehicle speed information (vehicle speed pulse) representing the detected speed.

[0081] Reference data D1 is input to the input data generating unit 132 of the learning device 10. The reference data D1 includes sensor output information Sout1. The reference data D1 may further include information other than the sensor output information Sout1 (for example, photographed image data of the subject U1, photographed image data of the periphery of the vehicle V1). In any case, the same type of data as the reference data D2 is included in the reference data D1.

[0082] The input data generating unit 132 converts the reference data D1 into learning input data suitable for the structure of the AI ​​models 121m and 122m. In the learning device 10, the AI ​​model 121m is trained so that the awakening level of the subject U1 can be estimated (inferred) based on the learning input data. That is, the learning device 10 supplies the awakening level output from the index value derivation unit 133 to the AI ​​model 121m as correct answer data, and trains the AI ​​model 121m using a learning algorithm such as backpropagation based on the learning input data. Similarly, in the learning device 10, the AI ​​model 122m is trained so that the activity level of the subject U1 can be estimated (inferred) based on the learning input data. That is, the learning device 10 supplies the activity level output from the index value derivation unit 133 to the AI ​​model 122m as correct answer data, and trains the AI ​​model 122m using a learning algorithm such as backpropagation based on the learning input data.

[0083] Fig. 8 is a configuration block diagram of a learning device 10 according to an embodiment EX1_1. The learning device 10 may be mounted on a vehicle V1, but here, it is considered that the learning device 10 is provided outside the vehicle V1. The learning device 10 may be formed by any one or more computer devices. The learning device 10 may be formed by using cloud computing.

[0084] A database DB is connected to the learning device 10 by wire or wirelessly. The database DB may be composed of multiple physically separated recording devices. The database DB stores a large number of pairs of biometric data of a subject U1 and reference data D1 of a vehicle V1. The biometric data and reference data D1 in one pair are data generated simultaneously.

[0085] Consider that there are M learning evaluation periods, each of which is different from the others. In this case, the first to Mth sets of data in the first to Mth learning evaluation periods are stored in the database DB. The i-th set of data includes biological data of the subject U1 in the i-th learning evaluation period and reference data D1 in the i-th learning evaluation period. The reference data D1 in the i-th learning evaluation period includes sensor output information Sout1 output from the sensor unit of the vehicle V1 in the i-th learning evaluation period. Here, M represents an integer value sufficiently larger than 1, and ranges from tens of thousands to hundreds of thousands, for example. i represents an arbitrary integer. The subject U1 becomes the driver of the vehicle V1 and performs various patterns of experimental driving to obtain the first to Mth sets of data. The first to Mth sets of data are supplied to the learning device 10.

[0086] In addition, since the emotion estimation process (see FIG. 6) is performed using the trained AI models 121m and 122m as the AI ​​models 221m and 222m of the emotion estimation device 20, it is preferable that the length of each learning evaluation period is set to be the same as the length of each estimation evaluation period.

[0087] The learning device 10 includes a communication unit 11, a storage unit 12, and a controller 13. The communication unit 11 transmits and receives any signal between the learning device 10 and a counterpart device different from the learning device 10. The counterpart device for the communication unit 11 includes a database DB. The storage unit 12 includes a non-volatile memory such as a ROM or a flash memory, and a volatile memory such as a RAM. The storage unit 12 includes a first AI model storage unit 121 that stores a first AI model 121m, and a second AI model storage unit 122 that stores a second AI model 122m. The storage unit 12 further includes a program storage unit 123 that stores various programs. The programs stored in the program storage unit 123 include a program 123p.

[0088] The controller 13 comprehensively controls the operation of each part in the learning device 10. The controller 13 includes a calculation processing unit including a CPU, a GPU, etc. as hardware resources. The controller 13 has function blocks 131 to 134. The function blocks 131, 132, 133, and 134 are an acquisition unit, an input data generation unit, an index value derivation unit, and a learning processing unit, respectively.

[0089] The controller 13 is a program execution device (computer) capable of executing any program. The controller 13 executes the program 123p to realize each function of the controller 13 (including each function of the functional blocks 131 to 134). All of the operations of the controller 13 described in this embodiment may be operations that are realized by the controller 13 executing the program 123p. The program 123p may be composed of a plurality of programs. The controller 13 can transmit and receive any information to and from a counterpart device (a counterpart device for the communication unit 11) using the communication unit 11, but in the following, the description of the communication unit 11 may be omitted.

[0090] Each functional block of the controller 13 will be described. The acquisition unit 131 sequentially acquires group data from the database DB. The input data generation unit 132 generates learning input data for each group data based on the reference data D1 in the group data acquired by the acquisition unit 231 (see also FIG. 7). The index value derivation unit 133 derives and outputs the emotion index value (here, arousal level and activity level) of the subject U1 based on the biological data of the subject U1 for each group data.

[0091] A learning data set is formed by the emotion index values ​​(here, arousal and activity) of the subject U1 and the learning input data. The learning processing unit 134 uses the learning data set to execute learning of the AI ​​models 121m and 122m.

[0092] The learning data set for the AI ​​model 121m has arousal level as the correct answer data. In learning, the AI ​​model 121m estimates (infers) the arousal level of the subject U1 based on the learning input data. The learning processing unit 134 updates the parameters (weights of the neural network, etc.) of the AI ​​model 121m based on the error between the estimated arousal level and the arousal level as the correct answer data. When a termination condition, such as the error being sufficiently low, is satisfied, the learning of the AI ​​model 121m ends. The AI ​​model 121m at the time of completing the learning is applied to the emotion estimation device 20 (see FIG. 1 and FIG. 4) as the A1 model 221m.

[0093] In contrast, the learning data set for the AI ​​model 122m has the activity level as the correct answer data. In learning, the AI ​​model 122m estimates (infers) the activity level of the subject U1 based on the learning input data. The learning processing unit 134 updates the parameters (weights of the neural network, etc.) of the AI ​​model 122m based on the error between the estimated activity level and the activity level as the correct answer data. When a termination condition, such as the error being sufficiently low, is satisfied, the learning of the AI ​​model 122m ends. The AI ​​model 122m at the time when learning ends is applied to the emotion estimation device 20 (see Figs. 1 and 4) as the A1 model 222m.

[0094] The input data generation unit 132 generates AI input data by performing data processing on the reference data D1 in the pair data. The AI-generated data generated by the input data generation unit 132 functions as learning input data. In the data processing in the input data generation unit 132, the reference data D1 is processed into a format of data to be input to the AI ​​models 121m and 122m, and the processed data is obtained as AI input data. The data processing in the input data generation unit 132 is set according to the input specifications of the AI ​​models 121m and 122m. The AI ​​models 121m and 122m trained using the learning input data are used as the AI ​​models 221m and 222m that estimate emotion index values ​​from the estimation input data in the emotion estimation device 20 (see FIG. 4). Therefore, the data processing in the input data generation unit 132 is the same as the data processing in the input data generation unit 232 (see FIG. 4 and FIG. 6). Therefore, the AI ​​input data (input data for learning) generated by the input data generation unit 132 and the AI ​​input data (input data for estimation) generated by the input data generation unit 232 are the same type of data.

[0095] Fig. 9 shows an operation flowchart of the learning process executed by the learning device 10. The learning process may be a process realized by the controller 13 executing the above-mentioned program 123p. The learning process shown in Fig. 9 is started when the program 123p is started in the learning device 10 after the necessary amount of group data is obtained. The accumulation of group data in the database DB and the learning process shown in Fig. 9 may be performed in parallel. The learning process includes steps S11 to S15.

[0096] First, in step S11, the acquisition unit 131 acquires group data from the database DB. As described above, the group data includes biometric data and reference data D1 (see FIG. 8). Learning is performed in mini-batch units, and group data having a data amount equivalent to the mini-batch is acquired in step S11. After step S11, the process proceeds to step S12.

[0097] In step S12, the index value derivation unit 133 derives emotion index values ​​(arousal and activity) of the subject U1 for each group data based on the biological data of the subject U1. Also in step S12, the input data generation unit 132 generates AI input data for each group data through data processing based on the reference data D1. The AI-generated data generated by the input data generation unit 132 functions as learning input data. After step S12, the processes of steps S13 and S14 are executed. The order of execution of the processes of steps S13 and S14 is arbitrary. The processes of steps S13 and S14 may be executed at the same time.

[0098] In step S13, the learning processing unit 134 trains the first AI model 121m using a learning data set having learning input data and alertness as correct answer data. In training the AI ​​model 121m, the AI ​​model 121m estimates (infers) the alertness of the subject U1 based on the learning input data. The learning processing unit 134 updates the parameters (such as neural network weights) of the AI ​​model 121m based on the error between the estimated alertness and the alertness as correct answer data.

[0099] In step S14, the learning processing unit 134 trains the second AI model 122m using a learning data set having learning input data and activity levels as correct answer data. In training the AI ​​model 122m, the AI ​​model 122m estimates (infers) the activity level of the subject U1 based on the learning input data. The learning processing unit 134 updates the parameters (such as neural network weights) of the AI ​​model 122m based on the error between the estimated activity level and the activity level as correct answer data.

[0100] After steps S13 and S14, in step S15, the learning processing unit 134 determines whether a determined termination condition is satisfied. For example, the termination condition is satisfied when the error in each learning in steps S13 and S14 becomes equal to or less than a predetermined value. Alternatively, the termination condition is satisfied when the number of mini-batch unit learnings reaches a certain number. If the termination condition is satisfied (Y in step S15), the learning process in FIG. 9 ends. If the termination condition is not satisfied (N in step S15), the process returns to step S11. If the process returns to step S11, new group data is acquired from the database DB, and the processes from step S12 onwards are repeated.

[0101] As described above, when an arbitrary person operates an arbitrary device, the emotion of the person is expressed in the input operation to the device and also in the information output from the device in response to the input operation. In consideration of this, information on the input operation to the learning device by the subject U1 and information output from the learning device in response to the input operation to the learning device are included in the learning input data. Then, the AI ​​model (121m, 122m) is trained using a learning data set having the learning input data and having an emotion index value corresponding to the biological signal of the subject U1 as correct answer data. By using the AI ​​model (trained AI model) obtained through this training, it becomes possible to obtain the emotion index value of an arbitrary person (subject U2) using information on the input operation to the device and information output from the device in response to the input operation without using a contact type sensor or the like for obtaining a biological signal. In addition, a method of estimating a person's biological information (for example, arousal level, activity level) by using a photograph of the person's face, etc. is also considered. However, in this method, depending on the shooting conditions, such as backlight or low illuminance, it may be difficult to obtain a high-quality face image. In the method according to this embodiment, such concerns do not arise.

[0102] In this embodiment, the learning device is a vehicle V1 as a learning vehicle, and the subject U1 is the driver of the vehicle V1. The input operation to the learning device is a driving operation to the vehicle V1 performed by the subject U1, and corresponds to an accelerator pedal operation, a brake pedal operation, and a steering wheel operation to the vehicle V1. In this embodiment, the information output from the learning device in response to the input operation to the learning device is information output from the vehicle V1 in response to the driving operation to the vehicle V1. The information output from the vehicle V1 in response to the driving operation to the vehicle V1 includes driving state information of the vehicle V1, and the driving state information of the vehicle V1 represents the speed, steering angle, acceleration, etc. of the vehicle V1. The information output from the vehicle V1 may include the degree of sway of the vehicle V1 in response to the driving operation, the forward vehicle distance, and the rear vehicle distance, etc. By applying the trained model obtained through this learning method to a vehicle, it becomes possible to obtain the emotion index value of the driver (subject U2) using the information of the input operation to the vehicle V2 and the information output from the vehicle V2 in response to the input operation, without attaching a contact type sensor or the like for obtaining a biological signal to the driver (subject U2). Furthermore, in the feeling estimation device 20, feeling estimation is performed based on the driving operation and running state information of the vehicle V2, rather than on biological signals that are easily affected by individual differences and the surrounding environment, etc., making it easier to obtain stable estimation accuracy.

[0103] As described above, it is preferable that the biosignal of subject U1 includes information on brain waves. Since brain waves are correlated with arousal level, it is possible to obtain an AI model that can properly estimate arousal level as an emotional index value through learning. As described above, it is preferable that the biosignal of subject U1 includes information on heart rate. Since heart rate is correlated with activity level, it is possible to obtain an AI model that can properly estimate activity level as an emotional index value through learning. Emotions can be properly estimated from arousal level and activity level.

[0104] A trained model obtained using the above-mentioned training data set can be installed in an estimation device. An emotion estimation device 20 (see Figs. 1 and 4), which is one type of the estimation device, includes AI models 221m and 222m as trained models. Input data for estimation, which includes information on an input operation by the subject U2 to the target device and information output from the target device in response to the input operation to the target device, is input to the trained models (221m, 222m). This allows the emotion index value of the subject U2 to be estimated. That is, it is possible to obtain the emotion index value of an arbitrary person (subject U2) without using a contact sensor or the like for obtaining a biological signal.

[0105] In this embodiment, the target device is a vehicle V2 as a target vehicle, and the target person U2 is the driver of the vehicle V2. The input operation to the target device is a driving operation of the vehicle V2 performed by the target person U2, which corresponds to an accelerator pedal operation, a brake pedal operation, and a steering wheel operation to the vehicle V2. In this embodiment, the information output from the target device in response to the input operation to the target device is information output from the vehicle V2 in response to a driving operation to the vehicle V2. The information output from the vehicle V2 in response to the driving operation to the vehicle V2 includes running state information of the vehicle V2, and the running state information of the vehicle V2 indicates the speed, steering angle, acceleration, etc. of the vehicle V2. The information output from the vehicle V2 may include the degree of sway of the vehicle V2 in response to the driving operation, the front vehicle distance, and the rear vehicle distance, etc.

[0106] Specifically, the feeling estimation device 20 has AI models 221m and 222m as a first trained model trained using a first training data set and a second trained model trained using a second training data set. The first training data set is supervised learning data having the above-mentioned training input data and having an emotion index value (here, arousal level) according to the brain waves of the subject U1 as correct answer data. The second training data set is supervised learning data having the above-mentioned training input data and having an emotion index value (here, activity level) according to the heart rate of the subject U1 as correct answer data. In the feeling estimation device 20, the above-mentioned estimation input data is input to each trained model (221m, 222m), thereby obtaining first and second emotion index values ​​(here, arousal level and activity level of the subject U2) from the first and second trained models. The emotion estimation model 223 provided in the emotion estimation device 20 estimates the emotion of the subject U2 based on the first and second emotion index values. This makes it possible to estimate the emotions of the subject U2 without having to attach a contact sensor or the like for obtaining a biosignal to the subject U2.

[0107] In the learning device 10 according to the embodiment EX1_1, the arousal level of the subject U1 is used as the emotion index value corresponding to the brain waves of the subject U1, and the activity level of the subject U1 is used as the emotion index value corresponding to the heart rate of the subject U1. Therefore, the first emotion index value and the second emotion index value in the emotion estimation device 20 are the arousal level and activity level of the target person U2. Since the brain waves are correlated with the arousal level and the heart rate is correlated with the activity level, while the emotion of a person is correlated with the arousal level and the activity level, the emotion estimation device 20 enables good emotion estimation.

[0108] The vehicle control system SYS in FIG. 4 includes an in-vehicle emotion estimation system. The in-vehicle emotion estimation system includes an emotion estimation device 20 and a sensor unit 60. As described above, the sensor unit 60 is installed in the vehicle V2 and generates sensor output information Sout2 used for driving control of the vehicle V2 (see FIG. 1 and FIG. 5). The emotion estimation device 20 generates input data for estimation based on the sensor output information Sout2 (more specifically, based on reference data D2 including the sensor output information Sout2). The in-vehicle emotion estimation system makes it possible to estimate the emotion of the driver (subject U2) without attaching a contact-type sensor or the like for obtaining a biological signal to the driver (subject U2). In addition, since emotion estimation is performed using the sensor output information Sout2 originally used in the driving control of the vehicle V2, there is no need to add a dedicated part for emotion estimation to the vehicle V2.

[0109] Of the information that can be included in the learning input data, information on the input operations by subject U1 into the learning device is referred to as learning stage device input information, and information output from the learning device in response to the input operations into the learning device is referred to as learning stage device output information. Of the information that can be included in the input data for estimation, information on the input operation on the target device by the subject U2 is referred to as device input information at the estimation stage, and information output from the target device in response to the input operation on the target device is referred to as device output information at the estimation stage. In the method described above in Example EX1_1, both the device input information at the learning stage and the device output information at the learning stage are included in the learning input data, and both the device input information at the estimation stage and the device output information at the estimation stage are included in the estimation input data. However, any one of the device input information in the learning stage and the device output information in the learning stage may be included in the learning input data (this also applies to other examples and embodiments described later).Similarly, any one of the device input information in the estimation stage and the device output information in the estimation stage may be included in the estimation input data (this also applies to other examples and embodiments described later).

[0110] That is, for example, the following first modification technique may be adopted in the example EX1_1 (similarly in other examples and other embodiments described later). In the first modification technique, among the device input information in the learning stage and the device output information in the learning stage, only the device input information in the learning stage is included in the learning input data, and among the device input information in the estimation stage and the device output information in the estimation stage, only the device input information in the estimation stage is included in the estimation input data. When an arbitrary person operates an arbitrary device, the emotion of the person is expressed in the input operation to the device. Therefore, the first modification technique can also obtain the same action and effect as when both the device input information and the device output information are used. However, by using both the device input information and the device output information, it is expected that the accuracy of emotion estimation will be improved compared to the first modification technique.

[0111] Also, for example, the following second modification technique may be adopted in the example EX1_1 (similarly in other examples and other embodiments described later). In the second modification technique, among the device input information in the learning stage and the device output information in the learning stage, only the device output information in the learning stage is included in the learning input data, and among the device input information in the estimation stage and the device output information in the estimation stage, only the device output information in the estimation stage is included in the estimation input data. When an arbitrary person operates an arbitrary device, the emotion of the person is expressed in the information output from the device in response to the input operation. Therefore, the second modification technique can also obtain the same action and effect as when both the device input information and the device output information are used. However, by using both the device input information and the device output information, it is expected that the accuracy of emotion estimation will be improved compared to the second modification technique.

[0112] [Example EX1_2] An example EX1_2 will be described. There are individual differences in biosignals such as brainwave information and heart rate information, and the biosignals are also affected by the surrounding environment. As a kind of calibration to reduce these, a normalization process is executed in the example EX1_2.

[0113] FIG. 10 shows a schematic configuration of the learning device 10 according to the embodiment EX1_2. FIG. 11 shows a configuration block diagram of the learning device 10 according to the embodiment EX1_2. The learning device 10 according to the embodiment EX1_2 is added with a normalization unit 135 based on the learning device 10 according to the embodiment EX1_1 (see FIG. 7 and FIG. 8). The normalization unit 135 performs normalization processing on the emotion index value derived by the index value derivation unit 133. The normalization unit 135 is one of the functional blocks provided in the controller 13 of the learning device 10. In addition, the learning device 10 according to the embodiment EX1_2 is added with a task table 124 and a neutral region table 125 to the storage unit 12 based on the learning device 10 according to the embodiment EX1_1. Except for the addition of the normalization unit 135 and the tables 124 and 125, the learning device 10 according to the embodiment EX1_2 has the same configuration as the learning device 10 according to the embodiment EX1_1 (see FIG. 7 and FIG. 8).

[0114] The emotion index value derived by the index value derivation unit 133 is normalized in the normalization process. The concept of normalization will be explained by focusing on the arousal level as the emotion index value. FIG. 12 is a schematic diagram showing the change over time in the emotion index values ​​(arousal levels) of two subjects. In each of the four graphs a1, a2, b1, and b2 shown in FIG. 12, the horizontal axis corresponds to time, and the vertical axis corresponds to the arousal level based on the biological signal.

[0115] In Fig. 12, graphs a1 and a2 on the left side show data for the first subject, and graphs b1 and b2 on the right side show data for the second subject. In Fig. 12, graphs a1 and b1 on the top show data before normalization, and graphs a2 and b2 on the bottom show data after normalization. For the sake of concrete explanation, assume an example of a state in which the wakefulness level fluctuates between an upper limit and a lower limit during the measurement period for each subject.

[0116] As shown in graphs a1 and b1 in FIG. 12, the fluctuation range of the awakening degree of the first and second subjects is significantly different, and the fluctuation range of the second subject is larger than that of the first subject. That is, in this example, it can be said that the first subject has a narrower range of the measured awakening degree (in other words, a lower sensitivity to the awakening degree) compared to the second subject. Conversely, it can be said that the second subject has a wider range of the measured awakening degree (in other words, a higher sensitivity to the awakening degree) compared to the first subject. Note that the sensitivity to the awakening degree is influenced not only by the subject's innate individual differences, but also by various conditions such as the surrounding environment (ambient temperature and humidity, brightness, etc.) and the subject's condition (whether the subject is hungry, etc.). Therefore, even if it is data of a single subject, multiple data with different surrounding environments, subject conditions, or data collection periods can be interpreted as being equivalent to data of multiple subjects.

[0117] In the emotion index value, a neutral state exists near the center of the range of the fluctuation. The area corresponding to the neutral state is called the neutral area NR. The shaded area in each graph of FIG. 12 is the neutral area NR. In the case of wakefulness, the neutral state exists between the wakefulness state and the non-wakefulness state (sleep-like state). Such a neutral state area tends to be relatively easy to distinguish (estimate) by performing a specific task.

[0118] As shown in graphs a1 and b1 in Fig. 12, the neutral region NR differs between the first and second subjects. That is, the upper limit value Max1 and the lower limit value Min1 of the neutral region NR in the first subject differ from the upper limit value Max2 and the lower limit value Min2 of the neutral region NR in the second subject. The width W1 (=Max1-Min1) of the neutral region NR in the first subject differs from the width W2 (=Max2-Min2) of the neutral region NR in the second subject.

[0119] In the normalization process, the emotion index value derived by the index value derivation unit 133 is corrected (i.e., normalized) so that the neutral region NR of each subject is located in the same position as a predetermined standard neutral region NR0 and has the same width as the standard neutral region NR0. The standard upper limit value and standard lower limit value, which are the upper and lower limits of the standard neutral region NR0, are represented by the symbols "Max0" and "Min0", respectively. The standard width W0, which is the width of the standard neutral region NR0, is represented by "W0=Max0-Min0". The standard upper limit value Max0, the standard lower limit value Min0, and the width W0 are stored in the storage unit 12 as preset values.

[0120] Specifically, the emotion index value (arousal level) is corrected using the following calculation formulas (a1) to (a6). The correction here is a correction by normalization processing, and can be called normalization. AW1B and AW2B respectively represent the arousal levels of the first and second subjects before normalization (before correction). AW1A and AW2A respectively represent the arousal levels of the first and second subjects after normalization (after correction). AW1A = AW1B × A1 + B1 (a1) AW2A = AW2B × A2 + B2 (a2) A1=(Max0-Min0) / (Max1-Min1) ···(a3) A2=(Max0-Min0) / (Max2-Min2) ···(a4) B1=((Max0+Min0)-(Max1+Min1)×A1) / 2 (a5) B2=((Max0+Min0)-(Max2+Min2)×A2) / 2 (a6)

[0121] In general, the subject U1 can be regarded as the nth subject (n is any natural number). The upper and lower limits of the neutral region NR for the nth subject are represented by "Maxn" and "Minn", respectively. Then, the emotion index value (arousal level) can be corrected using the following calculation formulas (b1) to (b3). AWnB represents the arousal level of the nth subject before normalization (before correction), and AWnA represents the arousal level of the nth subject after normalization (after correction). The arousal level AWnB is the arousal level of the nth subject derived by the index value derivation unit 133 based on the electroencephalogram data of the nth subject as the subject U1. The normalization unit 135 performs normalization processing on the arousal level AWnB to obtain the arousal level AWnA, which is the arousal level after normalization. AWnA = AWnB × An + Bn (b1) An=(Max0-Min0) / (Maxn-Minn) ···(b2) Bn=((Max0+Min0)-(Maxn+Minn)×An) / 2 (b3)

[0122] When normalizing the arousal level of the nth subject by such a normalization method (corresponding to the calculation formulas (b1) to (b3)), it is necessary to estimate the upper limit value Maxn and the lower limit value Minn of the neutral region NR for the nth subject. For this estimation, the nth subject is made to perform a specific task, and the arousal level at that time is measured. Then, the upper limit value Maxn and the lower limit value Minn are estimated based on the measured value of the arousal level. The task to be performed is a task that puts the subject who performed the task into a specific mental and physical state (specific state). The task to be performed is, for example, a task in which the arousal level fluctuates above the upper limit value of the neutral region NR or a task in which the arousal level fluctuates below the lower limit value of the neutral region NR. As such tasks, there are several types of tasks for which evidence has been obtained in research in the fields of medicine, psychology, and the like. By having the nth subject perform these tasks, the nth subject is put into a specific mental and physical state, and normalization processing can be performed using the biological signal of the nth subject in the specific mental and physical state. Incidentally, measurement of the degree of awakening refers to deriving the degree of awakening of any subject by the index value derivation unit 133 based on the electroencephalogram data of the subject.

[0123] The activity level can also be normalized (corrected) in a similar manner. When normalizing (correcting) the activity level, the alertness and brain wave data in the above description of the embodiment EX1_2 can be read as activity level and heart rate data. In other words, each parameter for alertness can be replaced with a parameter for the activity level to be corrected. The same applies when an emotion index value different from alertness and activity level is used. The neutral area NR for alertness corresponds to the alertness neutral area NR1 shown in FIG. 3, and the neutral area NR for activity corresponds to the activity neutral area NR2 shown in FIG. 3.

[0124] 13 is a diagram showing an example of the task table 124. Items in the task table 124 include "task ID", "corresponding index type", "task content", and "index value transition state".

[0125] "Task ID" is task ID data, which is identification information for identifying task information. Task ID data is also the primary key of a data record in task table 124. That is, in task table 124, a data record is configured for each task ID data, and task data linked to the task ID data is stored in that data record. The task data includes data for the items "Corresponding index type," "Task content," and "Index value transition state."

[0126] "Corresponding index type" is the emotional index type to be normalized, and a "corresponding index type" corresponding to each task ID is stored in the task table 124. In this case, arousal level or activity level is stored as the "corresponding index type." "Task content" indicates the specific content of the task performed by the subject for normalization processing.

[0127] The "index value transition state" indicates how the corresponding emotion index value is expected to transition when the subject performs the corresponding task. For example, the "index value transition state" indicates high arousal or low arousal. High arousal indicates that when the nth subject performs the corresponding task, the arousal level of the nth subject increases beyond the upper limit value Maxn of the neutral region NR. Low arousal indicates that when the nth subject performs the corresponding task, the arousal level of the nth subject falls below the lower limit value Minn of the neutral region NR.

[0128] In the example of task table 124 in FIG. 13, task data for a task ID of "ST01" and a task ID of "ST02" are illustrated. A task to which a task ID of "ST01" is assigned may hereinafter be referred to as task ST01. A task to which a task ID of "ST02" is assigned may hereinafter be referred to as task ST02. In the task data for task ST01, the target emotion index is arousal level, the index value transition state (i.e., the transition state of arousal level) is high arousal level, and the task content is "mentally add displayed numbers." In the task data for task ST02, the target emotion index is arousal level, the index value transition state (i.e., the transition state of arousal level) is low arousal level, and the task content is "remain calm and clench your fist."

[0129] The task content to be performed by the subject is selected depending on the target of the normalization process and whether the boundary of the neutral region NR to be estimated is the upper limit or the lower limit. Using task ST01, the upper limit of the neutral region NR of the alertness level can be estimated, and using task ST02, the lower limit of the neutral region NR of the alertness level can be estimated.

[0130] An explanation will be added with reference to FIG. 14. For a given subject, the upper limit and lower limit of the neutral region NR of the level of alertness are represented by the symbols "NRU" and "NRD", respectively. In this example, tasks ST01 and ST02 are selected to estimate the upper limit NRU and the lower limit NRD. FIG. 14 is a schematic diagram showing the change in alertness over time when the subject of interest executes tasks ST01 and ST02. In the graph shown in FIG. 14, the horizontal axis corresponds to time, and the vertical axis corresponds to alertness. Signal α represents the change in alertness over time of the subject of interest when the subject of interest executes the task contents of task ST01. Signal β represents the change in alertness over time of the subject of interest when the subject of interest executes the task contents of task ST02.

[0131] During the first task execution period in which the subject of interest is performing the task ST01, the degree of arousal of the subject of interest is in a relatively high state (high index value state), so the measured degree of arousal of the subject of interest is expected to change (fluctue) within the high arousal range. On the other hand, it is considered that there is a time lag from the start of the execution of the task ST01 until the degree of arousal fluctuates due to the influence of the task ST01, and from the end of the task ST01 until the influence of the task ST01 disappears from the degree of arousal. For this reason, the time when a first predetermined time has elapsed from the start time of the first task execution period is set as the first measurement start time, and the time when a second predetermined time has elapsed from the end time of the first task execution period is set as the first measurement end time. The period from the first measurement start time to the first measurement end time is set as the first task evaluation period. Then, the minimum value of the degree of arousal of the subject of interest during the first task evaluation period can be estimated as the upper limit value NRU.

[0132] During the second task execution period in which the subject of interest is performing the task ST02, the arousal level of the subject of interest is in a relatively low state (low index value state, non-awakened state), so the measured arousal level of the subject of interest is expected to change (fluctue) within the low arousal level range. On the other hand, it is considered that there is a time lag from the start of the execution of the task ST02 until the arousal level fluctuates due to the influence of the task ST02, and from the end of the task ST02 until the influence of the task ST02 disappears from the arousal level. For this reason, the time when the first predetermined time has elapsed from the start time of the second task execution period is set as the second measurement start time, and the time when the second predetermined time has elapsed from the end time of the second task execution period is set as the second measurement end time. The period from the second measurement start time to the second measurement end time is set as the second task evaluation period. Then, the maximum value of the arousal level of the subject of interest during the second task evaluation period can be estimated as the lower limit NRD.

[0133] Based on this method, the neutral region for the emotion index value of arousal can be estimated. That is, if the subject of interest is the nth subject, the normalization unit 135 regards the upper limit value NRU and the lower limit value NRD as the upper limit value Maxn and the lower limit value Minn, respectively, of the neutral region NR in the arousal of the nth subject. After the upper limit value Maxn and the lower limit value Minn are specified, the arousal can be normalized according to the above formulas (b1) to (b3) (that is, the arousal level after normalization AWnA can be obtained from the arousal level before normalization AWnB). The same applies to the other emotion index values.

[0134] The normalization unit 135 determines the neutral region NR by the above method, specifically, performs statistical processing on the time-varying data of the emotion index value (arousal level: signal α) obtained during the execution of task ST01 to determine the upper limit value NRU of the neutral region NR. For example, the normalization unit 135 performs smoothing processing on the signal α in the first task evaluation period, and sets the minimum value of the signal α after the smoothing processing as the upper limit value NRU of the neutral region NR. Similarly, the normalization unit 135 performs smoothing processing on the signal β in the second task evaluation period, and sets the maximum value of the signal β after the smoothing processing as the lower limit value NRD of the neutral region NR. The smoothing processing is performed for the purpose of removing noise components, etc.

[0135] The method of determining the upper limit NRU and lower limit NRD of the neutral region NR has been described with a focus on the level of arousal, but the normalization unit 135 determines the upper limit and lower limit of the neutral region for each type of emotion index value. For this reason, a task ST03 for determining the upper limit of the neutral region of activity and a task ST04 for determining the lower limit of the neutral region of activity are prepared, and the subject is requested to perform tasks ST03 and ST04 (tasks ST03 and ST04 are not shown). As with the level of arousal, the neutral region of activity is determined according to the measurement results of activity when each task is performed.

[0136] In the above, it is assumed that the first variation range, which is the variation range of the emotion index value (signal α) obtained when task ST01 is executed, and the second variation range, which is the variation range of the emotion index value (signal β) obtained when task ST02 is executed, are separated from each other (see FIG. 14). Under this assumption, the space between the first and second variation ranges can be set as a neutral region NR.

[0137] However, depending on the type of emotion index value or the content of the task, the first and second variation ranges may overlap. In such a case, the neutral region may be determined based on the overlapping range of the first and second variation ranges. For example, the overlapping range of the first and second variation ranges may be set as the neutral region. Alternatively, for example, the overlapping range of the first and second variation ranges may be expanded by adding an appropriate offset to the overlapping range of the first and second variation ranges and set as the neutral region. In this case, the range may be expanded by an appropriate value (for example, 10%) from the viewpoint of preventing erroneous determination of emotion. Alternatively, for example, the overlapping range of the first and second variation ranges may be reduced by an appropriate offset and set as the neutral region. In this case, the range may be reduced by an appropriate value (for example, 10%) from the viewpoint of preventing inability to determine emotion.

[0138] Next, another method for determining the neutral region will be described. In this method, the subject is not required to execute specific tasks such as the above tasks ST01 and ST02. In this method, the neutral region is determined by focusing on the fact that as the accumulation of time-series data of emotion index values ​​progresses, the upper and lower limit values ​​of the neutral region estimated by statistical processing converge to certain values.

[0139] This will be described with reference to Fig. 15. Focus will be given on arousal level as an emotion index value. Fig. 15 is a schematic diagram showing the change over time in the emotion index value (arousal level), and in the graph of Fig. 15, the horizontal axis corresponds to time, and the vertical axis corresponds to arousal level.

[0140] As shown in FIG. 15, a person's mental and physical state changes into various states based on changes in the environment, etc., which change over time, even without giving a specific task. Therefore, a considerable amount of time-series data of emotion index values ​​can be statistically processed and a cluster analysis can be performed to divide the accumulated data into three states (high arousal, neutral, low arousal) based on the magnitude of the values. The cluster analysis can estimate the neutral region (upper and lower limits of the neutral region). For the cluster analysis, known methods such as the k-means method, Otsu's multi-value method, and Gaussian mixture model can be used.

[0141] Fig. 16 is a diagram showing the convergence status of the upper limit estimate of the neutral region when the neutral region is determined by the statistical method as shown in Fig. 15. The convergence status of the lower limit estimate of the neutral region is similar.

[0142] The amount of data used to estimate the upper limit of the neutral region using a statistical method increases over time. As the amount of data used in statistical processing increases over time, the estimated upper limit (estimated upper limit) converges to a certain value over time. This tendency is the same for the lower limit. A neutral region specified by the converged upper limit and lower limit values ​​is considered to be highly reliable. For this reason, the neutral region is determined at the stage where it is determined that the upper limit and lower limit values ​​calculated by the statistical method have converged. For example, a method is conceivable in which values ​​estimated to be the upper limit of the neutral region are sequentially specified, and the convergence is determined when the variance of the specified multiple values ​​becomes equal to or less than a predetermined convergence determination threshold. The same is true for the lower limit of the neutral region. In addition, even when a statistical method is used, the normalization unit 135 determines the upper limit and lower limit of the neutral region for each type of emotion index value.

[0143] The normalization unit 135 stores the determined upper and lower limit values ​​of the neutral region in the neutral region table 125 of the storage unit 12. In detail, the normalization unit 135 creates a data table of the determined upper and lower limit values ​​of the neutral region and stores the data table in the storage unit 12. When there are multiple subjects as the subject U1, it is effective to store neutral region information (information indicating the upper and lower limit values ​​of the neutral region) for each subject in the neutral region table 125. In this case, when recreating (adding) a learning dataset for the same subject, the neutral region information stored in the neutral region table 125 can be used.

[0144] 17 is a diagram showing an example of the neutral region table 125. The items in the neutral region table 125 include a "neutral information ID", a "user ID", an "emotion index type", an "upper limit value", a "lower limit value", and an "acquisition date and time".

[0145] In table 125, the "neutral information ID" is identification information for identifying neutral information. The neutral information ID is also the primary key of the data record in neutral area table 125. That is, in neutral area table 125, a data record is configured for each neutral information ID, and data of each item linked to the neutral information ID is stored in the data record.

[0146] In table 125, "user ID" is identification information for identifying a user. The above-mentioned n-th subject corresponds to a user. A different user ID is assigned to the first subject and the second subject. The same is true for the third subject, etc.

[0147] In the table 125 , the “upper limit value” and the “lower limit value” are information on the upper limit value and the lower limit value of the neutral region determined by the normalization unit 135 .

[0148] In table 125, the "acquisition date and time" stores date and time information on the upper and lower limit values ​​of the neutral region obtained (determined and stored). Even after the upper and lower limit values ​​of the neutral region for each user are stored in table 125, the data in table 125 can be updated based on the latest information.

[0149] Alternatively, a method of determining the neutral region based on an emotion declaration from the subject is also possible. For example, the normalization unit 135 asks the subject (user) a question regarding the emotion index value that the subject is aware of, and accumulates the answer and the arousal level measured (calculated) at that time. The question may, for example, inquire as to which of three stages (awake, normal, not awakened (sleepy)) the subject's arousal state is in. The normalization unit 135 then performs statistical processing on the accumulated data to estimate the neutral region. The arousal level measured when the subject's answer is "normal" is K. A Suppose there are individual awakening levels (K A is a sufficiently large integer). For example, A Among the arousal levels, the top 10% and bottom 10% are extracted, and the average of the top 10% and bottom 10% are set as the upper and lower limits of the neutral region, respectively. The arousal level measured when the subject answered "Arousal" is K B Suppose there are individual awakening levels (K B is a sufficiently large integer). For example, B Of the arousal levels, the lowest 10% of arousal levels can be extracted, and the average of the lowest 10% of arousal levels minus an appropriate offset value can be used as the upper limit of the neutral region. Similarly, for example, the arousal level measured when the subject's answer was "not aroused" can be K C Suppose there are individual awakening levels (K C is a sufficiently large integer). For example, C The top 10% of the arousal levels can be extracted from the individual arousal levels, and the average value of the top 10% arousal levels plus an appropriate offset value can be set as the lower limit of the neutral region. The same applies to the activity level.

[0150] The normalization method described above is referred to as the first normalization method. The first normalization method is a method in which normalization is performed based on the neutral region of the emotion index value, but normalization can also be performed by another method. The method described next is a method (hereinafter referred to as the second normalization method) in which normalization is performed based on the range of emotion index values ​​calculated within an appropriate measurement period.

[0151] Fig. 18 is a schematic diagram showing the change over time in the emotion index values ​​(arousal level) of two subjects. In each of the four graphs a1, a2, b1, and b2 shown in Fig. 18, the horizontal axis corresponds to time, and the vertical axis corresponds to the arousal level based on the biological signal.

[0152] In Fig. 18, graphs a1 and a2 on the left side show data for the first subject, and graphs b1 and b2 on the right side show data for the second subject. In Fig. 18, graphs a1 and b1 on the top show data before normalization, and graphs a2 and b2 on the bottom show data after normalization by the second normalization method. For the sake of concrete explanation, an example of a state in which the wakefulness level fluctuates between an upper limit and a lower limit during the measurement period for each subject is assumed.

[0153] As shown in graphs a1 and b1 in FIG. 18, the fluctuation range of the arousal level of the first and second subjects is significantly different, and the fluctuation range is larger for the second subject than for the first subject. That is, in this example, the first subject has a narrower range of measured arousal levels (in other words, a lower sensitivity to arousal levels) compared to the second subject. Conversely, the second subject has a wider range of measured arousal levels (in other words, a higher sensitivity to arousal levels) compared to the first subject. Note that the sensitivity to arousal levels is influenced not only by the subjects' innate individual differences, but also by various conditions such as the surrounding environment (ambient temperature and humidity, brightness, etc.) and the subject's condition (whether the subject is hungry, etc.).

[0154] The second normalization method is based on the idea that the range of variation of the emotion index value (arousal level) during an appropriate measurement period changes depending on the subject's sensitivity to the emotion index value and the surrounding environment. For this reason, the second normalization method corrects the emotion index value so that the size of the range of variation of the emotion index value (arousal level) is the same across multiple subjects.

[0155] As shown in graphs a1 and b1 of FIG. 18, the fluctuation range CH of the wakefulness differs between the first and second subjects due to individual differences, etc. In the description of the second normalization method, Max1 and Min1 respectively represent the upper limit value and the lower limit value in the fluctuation range CH of the wakefulness of the first subject. In the description of the second normalization method, Max2 and Min2 respectively represent the upper limit value and the lower limit value in the fluctuation range CH of the wakefulness of the second subject. In the example shown in FIG. 18, at least the upper limit value Max1 and the upper limit value Max2 are different from each other. In the description of the second normalization method, W1 represents the width of the fluctuation range CH of the wakefulness of the first subject, and is "W1=Max1-Min1". In the description of the second normalization method, W2 represents the width of the fluctuation range CH of the wakefulness of the second subject, and is "W2=Max2-Min2".

[0156] In the normalization process according to the second normalization method, the emotion index value derived by the index value derivation unit 133 is corrected (i.e., normalized) so that the variation range CH of each subject is located in the same position as the predetermined standard variation range CH0 and has the same width as the standard variation range CH0. In the explanation of the second normalization method, "Max0" and "Min0" respectively represent the standard upper limit value and the standard lower limit value which are the upper limit value and the lower limit value of the standard variation range CH0. In the explanation of the second normalization method, "W0" represents the standard width which is the width of the standard variation range CH0, and "W0=Max0-Min0". The standard upper limit value Max0, the standard lower limit value Min0, and the width W0 are stored in the storage unit 12 as preset values.

[0157] Specifically, the emotion index value (arousal level) is corrected using the following calculation formulas (c1) to (c6). The correction here is a correction by normalization processing, and can be called normalization. AW1B and AW2B respectively represent the arousal levels of the first and second subjects before normalization (before correction). AW1A and AW2A respectively represent the arousal levels of the first and second subjects after normalization (after correction). AW1A = AW1B × A1 + B1 (c1) AW2A = AW2B × A2 + B2 (c2) A1=(Max0-Min0) / (Max1-Min1) ···(c3) A2=(Max0-Min0) / (Max2-Min2) ···(c4) B1=((Max0+Min0)-(Max1+Min1)×A1) / 2 (c5) B2=((Max0+Min0)-(Max2+Min2)×A2) / 2 (c6)

[0158] In general, the subject U1 can be regarded as the nth subject (n is any natural number). In the explanation of the second normalization method, "Maxn" and "Minn" respectively represent the upper and lower limits of the fluctuation range CH for the nth subject. Then, the emotion index value (arousal level) can be corrected using the following calculation formulas (d1) to (d3). AWnB represents the arousal level of the nth subject before normalization (after correction), and AWnA represents the arousal level of the nth subject after normalization (after correction). The arousal level AWnB is the arousal level of the nth subject derived by the index value derivation unit 133 based on the electroencephalogram data of the nth subject as the subject U1. The normalization unit 135 performs normalization processing on the arousal level AWnB to obtain the arousal level AWnA, which is the arousal level after normalization. AWnA = AWnB × An + Bn (d1) An=(Max0-Min0) / (Maxn-Minn) ···(d2) Bn=((Max0+Min0)-(Maxn+Minn)×An) / 2 (d3)

[0159] When the second normalization method is used, a measurement period having a necessary length is set, and the controller 13 is caused to perform the following measurement process during the measurement period. In the measurement process, the controller 13 uses the index value derivation unit 133 to measure the wakefulness of the nth subject from the electroencephalogram data of the nth subject during the measurement period. The controller 13 then specifies the upper limit Maxn and lower limit Minn for the nth subject by specifying the fluctuation range CH of the measured wakefulness during the measurement period. After the upper limit Maxn and lower limit Minn are specified, the wakefulness can be normalized according to the above formulas (d1) to (d3) (i.e., the wakefulness after normalization AWnA can be obtained from the wakefulness before normalization AWnB). Note that the measurement of the wakefulness refers to the index value derivation unit 133 deriving the wakefulness of the nth subject based on the electroencephalogram data of the nth subject as the subject U1.

[0160] The activity level can also be normalized (corrected) in a similar manner. When normalizing (correcting) the activity level using the second normalization method, the arousal level and brainwave data in the above description of the second normalization method can be read as activity level and heart rate data. In other words, each parameter for the arousal level can be replaced with a parameter for the activity level to be corrected. The same applies when an emotion index value different from the arousal level and activity level is used. The neutral region NR for the arousal level corresponds to the arousal level neutral region NR1 shown in FIG. 3, and the neutral region NR for the activity level corresponds to the activity level neutral region NR2 shown in FIG. 3.

[0161] According to the second normalization method, there is no need for the subject to perform a special task, to collect data when performing the task, and to perform calculations on the collected data, etc. This reduces the burden on the subject, and also reduces the processing load of the learning device and the learning time.

[0162] In the embodiment EX1_2, the controller 13 creates normalization data for normalizing (correcting) the alertness and activity of the subject U1, and stores the normalization data in the storage unit 12. The data stored in the neutral region table 125 is an example of the normalization data. For the nth subject, the upper limit value Maxn and lower limit value Minn for the alertness and the upper limit value Maxn and lower limit value Minn for the activity, which are specified (determined) by any of the above-mentioned methods, are included in the normalization data.

[0163] Thereafter, when the learning process is performed, the awakening level and activity level derived by the index value derivation unit 133 are normalized by the normalization unit 135 using the normalization data. Then, in the learning process, the awakening level after normalization is input to the AI ​​model 121m as the correct answer data, and the activity level after normalization is input to the AI ​​model 122m as the correct answer data. The learning method of the AI ​​models 121m and 122m is the same as that shown in the embodiment EX1_1, except that the correct answer data input to the AI ​​models 121m and 122m are the awakening level and activity level after normalization.

[0164] When there are multiple subjects as the subject U1, normalization data is created for each subject and stored in the storage unit 12. In the learning process, in order to perform appropriate normalization, the group data provided to the learning device 10 is provided to the learning device 10 with the group data of which subject is specified. It is sufficient to include subject identification information in the group data. The subject identification information in certain group data indicates which subject the biometric data and reference data D1 in the group data belong to.

[0165] The trained AI models 121m and 122m are incorporated into the emotion estimation device 20 as AI models 221m and 222m (see FIGS. 1 and 4). In the embodiment EX1_2, the arousal level derived by the AI ​​model 221m corresponds to the normalized arousal level, and the activity level derived by the AI ​​model 222m corresponds to the normalized activity level. This enables highly accurate emotion estimation in which the influence of individual differences and the like is suppressed.

[0166] [Example EX1_3] An embodiment EX1_3 will be described. The learning device 10 can also quantize emotion index values. This will be described.

[0167] Fig. 19 shows a schematic configuration of the learning device 10 according to the embodiment EX1_3. Fig. 20 shows a configuration block diagram of the learning device 10 according to the embodiment EX1_3. A quantization unit 136 is added to the learning device 10 according to the embodiment EX1_3 based on the learning device 10 according to the embodiment EX1_2 (see Figs. 10 and 11). The quantization unit 136 is one of the functional blocks provided in the controller 13 of the learning device 10. Except for the addition of the quantization unit 136, the learning device 10 according to the embodiment EX1_3 has the same configuration as the learning device 10 according to the embodiment EX1_2 (see Figs. 10 and 11).

[0168] The quantization unit 136 performs quantization processing on the normalized emotion index value output from the normalization unit 135. In the quantization processing, the normalized emotion index value is quantized. The quantization processing is performed for each type of emotion index value. The alertness after the quantization processing is referred to as the quantized alertness, and the activity after the quantization processing is referred to as the quantized activity. The quantized alertness and quantized activity are output from the quantization unit 136.

[0169] The wakefulness level derived by the index value derivation unit 133 and the normalized wakefulness level output by the normalization unit 135 are continuous values ​​or have a high enough resolution to be considered as continuous values. Similarly, the activity level derived by the index value derivation unit 133 and the normalized activity level output by the normalization unit 135 are continuous values ​​or have a high enough resolution to be considered as continuous values.

[0170] In the quantization process, the emotion index value is quantized using the boundary set in the psychological plane PP as a threshold value.

[0171] First, consider the first pattern in which a neutral region is not set on the psychological plane PP. In the first pattern, the normalized awakening level output from the normalization unit 135 is quantized in the quantization unit 136 to either an awakening level having a first value or an awakening level having a second value. That is, the quantized awakening level has a first value or a second value. In the first pattern, when the normalized awakening level is located on the positive side of the vertical axis direction as viewed from the origin of the psychological plane PP, the quantized awakening level has a first value. In the first pattern, when the normalized awakening level is located on the negative side of the vertical axis direction as viewed from the origin of the psychological plane PP, the quantized awakening level has a second value. The quantized awakening level having a first value represents an awakened state, and the quantized awakening level having a second value represents a non-awakened state.

[0172] Similarly, in the first pattern, the normalized activity output from the normalization unit 135 is quantized in the quantization unit 136 to either an activity having a first value or an activity having a second value. That is, the quantized activity has a first value or a second value. In the first pattern, when the normalized activity is located on the positive side of the horizontal axis direction as viewed from the origin of the psychological plane PP, the quantized activity has a first value. In the first pattern, when the normalized activity is located on the negative side of the horizontal axis direction as viewed from the origin of the psychological plane PP, the quantized activity has a second value. The quantized activity having a first value represents an active state, and the quantized activity having a second value represents an inactive state.

[0173] Next, consider the second pattern in which a neutral region is set in the psychological plane PP. In the second pattern, the normalized awakening level is quantized using the upper and lower limits of the awakening level neutral region NR1 as two thresholds. Therefore, in the second pattern, the normalized awakening level output from the normalization unit 135 is quantized in the quantization unit 136 to an awakening level having a first value, an awakening level having a second value, or an awakening level having a third value. That is, the quantized awakening level has a first value, a second value, or a third value. The quantized awakening level having a first value represents an awakened state, and the quantized awakening level having a second value represents a non-awakened state. The quantized awakening level having a third value represents a neutral state with respect to the awakening level.

[0174] Regarding the second pattern, considering the above-mentioned subject U1, it is as follows. When the normalized alertness derived for subject U1 is greater than the upper limit of the alertness neutral region NR1, the quantized alertness has a first value. When the normalized alertness derived for subject U1 is less than the lower limit of the alertness neutral region NR1, the quantized alertness has a second value. When the normalized alertness derived for subject U1 is within the alertness neutral region NR1, the quantized alertness has a third value.

[0175] Similarly, in the second pattern, the normalized activity is quantized using the upper and lower limits of the activity neutral region NR2 as two thresholds. Therefore, in the second pattern, the normalized activity output from the normalization unit 135 is quantized in the quantization unit 136 to an activity having a first value, an activity having a second value, or an activity having a third value. That is, the quantized activity has a first value, a second value, or a third value. The quantized activity having a first value represents an active state, and the quantized activity having a second value represents an inactive state. The quantized activity having a third value represents a neutral state with respect to activity.

[0176] Regarding the second pattern, consider the above-mentioned subject U1 as follows: When the normalized activity derived for subject U1 is greater than the upper limit of the activity-neutral region NR2, the quantized activity has a first value. When the normalized activity derived for subject U1 is less than the lower limit of the activity-neutral region NR2, the quantized activity has a second value. When the normalized activity derived for subject U1 is within the activity-neutral region NR2, the quantized activity has a third value.

[0177] In the learning process according to the embodiment EX1_3, the awakening level and the activity level derived by the index value derivation unit 133 are normalized by the normalization unit 135. The normalization method is as shown in the embodiment EX1_2. The normalized awakening level and the normalized activity level are output from the normalization unit 135. The quantization unit 136 derives the quantized awakening level by performing a quantization process on the normalized awakening level, and derives the quantized activity level by performing a quantization process on the normalized activity level. In the learning process, the quantized awakening level is input to the AI ​​model 121m as correct answer data, and the quantized activity level is input to the AI ​​model 122m as correct answer data. The learning method of the AI ​​models 121m and 122m is the same as that shown in the embodiment EX1_1, except that the correct answer data input to the AI ​​models 121m and 122m are the quantized awakening level and the quantized activity level.

[0178] The trained AI models 121m and 122m are incorporated into the emotion estimation device 20 as AI models 221m and 222m (see Figs. 1 and 4). In the embodiment EX1_3, the arousal level derived by the AI ​​model 221m corresponds to the quantized arousal level, and the activity level derived by the AI ​​model 222m corresponds to the quantized activity level. Since normalization is performed during learning, it is possible to perform highly accurate emotion estimation with the influence of individual differences and the like suppressed. In addition, the quantization reduces the computation load during learning. The reduction in computation load leads to a reduction in learning time. The emotion estimation model 223m is a model corresponding to the quantized arousal level and the quantized activity level, that is, a model in which the arousal level axis and the activity level axis have a resolution corresponding to the quantized value. The arousal level axis and the activity level axis correspond to the vertical axis and the horizontal axis of the psychological plane PP, respectively (see Figs. 2 and 3). In a model in which the alertness axis and activity axis have resolution corresponding to quantized values, the values ​​on the alertness axis and the activity axis each have a two-value resolution, or, if a neutral region NR is set, a three-value resolution.

[0179] [Example EX1_4] Example EX1_4 will be described. FIG. 21 shows a schematic configuration of the learning device 10 according to Example EX1_4. Based on the learning device 10 according to Example EX1_1 (see FIGS. 7 and 8), the AI ​​models 121m and 122m are replaced with the AI ​​model 127m to obtain the learning device 10 according to Example EX1_4. Except for this replacement, the learning device 10 according to Example EX1_4 has the same configuration as the learning device 10 according to Example EX1_1 (see FIGS. 7 and 8).

[0180] In the embodiment EX1_4, the input data generating unit 132 converts the reference data D1 into learning input data suitable for the structure of the AI ​​model 127m. In the learning device 10 according to the embodiment EX1_4, the AI ​​model 127m is trained so that both the awakening level and the activity level of the subject U1 can be estimated (inferred) based on the learning input data. That is, the learning device 10 supplies the awakening level and the activity level output from the index value derivation unit 133 to the AI ​​model 127m as correct answer data, and trains the AI ​​model 127m using a learning algorithm such as the backpropagation method based on the learning input data.

[0181] The learning processing unit 134 executes learning of the AI ​​model 127m using a learning data set. The learning data set has the arousal level and activity level derived by the index value derivation unit 133 and the learning input data generated by the input data generation unit 132. The correct answer data in the learning data set for the AI ​​model 127m are the arousal level and activity level. In learning, the AI ​​model 127m estimates (infers) the arousal level and activity level of the subject U1 based on the learning input data. The learning processing unit 134 updates the parameters of the AI ​​model 127m (weights of a neural network, etc.) based on the error between the estimated arousal level and activity level and the arousal level and activity level as correct answer data. When a termination condition, such as the error being sufficiently low, is satisfied, the learning of the AI ​​model 127m is terminated. The AI ​​model 127m at the time of the completion of learning is applied to the emotion estimation device 20.

[0182] FIG. 22 shows a schematic configuration of the feeling estimation device 20 according to the embodiment EX1_4. In the feeling estimation device 20 according to the embodiment EX1_4, an AI model 227m is provided instead of the AI ​​models 221m and 222m. That is, the feeling estimation device 20 according to the embodiment EX1_4 is obtained by replacing the AI ​​models 221m and 222m with the AI ​​model 227m based on the feeling estimation device 20 shown in FIG. 1 and FIG. 4. The AI ​​model 127m at the time when learning is completed, that is, the learned AI model 127m is incorporated into the feeling estimation device 20 as the A1 model 227m. The AI ​​model 227m outputs (calculates) the arousal level and activity level of the subject U2 based on the input data for estimation. The feeling estimation model 223m according to the embodiment EX1_4 estimates the feeling of the subject U2 based on the arousal level and activity level output from the AI ​​model 227m.

[0183] The embodiment EX1_2 and the embodiment EX1_4 may be combined. That is, the AI ​​models 121m and 122m may be replaced with the AI ​​model 127m based on the learning device 10 (see Figs. 10 and 11) according to the embodiment EX1_2. In this case, the AI ​​model 127m is learned by using the normalized awakening level and activity level output from the normalization unit 135 as correct answer data.

[0184] The embodiment EX1_3 and the embodiment EX1_4 may be combined. That is, the AI ​​models 121m and 122m may be replaced with the AI ​​model 127m based on the learning device 10 (see FIG. 19 and FIG. 20) according to the embodiment EX1_3. In this case, the quantized awakening level and the quantized activity level output from the quantization unit 136 are used as the correct answer data to learn the AI ​​model 127m.

[0185] [Example EX1_5] An example EX1_5 will be described. Although arousal and activity are given as examples of emotion index values, the emotion index values ​​handled by the learning device 10 and the emotion estimation device 20 are not limited to arousal and activity. When the biometric data of the subject U1 includes brainwave data, one emotion index value derived by the index value derivation unit 133 of the learning device 10 may be any value based on the brainwave data of the subject U1 (which may include the brainwave data itself). When the biometric data of the subject U1 includes heartbeat data, another emotion index value derived by the index value derivation unit 133 of the learning device 10 may be any value based on the heartbeat data of the subject U1 (which may include the heartbeat data itself). The biometric data provided to the learning device 10 may be other than brainwave data and heartbeat data. The biometric data provided to the learning device 10 may include other data related to biometric information in addition to brainwave data and heartbeat data.

[0186] <<Second embodiment>> A second embodiment of the present invention will be described. As described above, when a person operates an arbitrary device, the person's emotions are expressed in the input operation to the device and also in the information output from the device in response to the input operation. In consideration of this, information on the input operation to the learning device by the subject U1 and information output from the learning device in response to the input operation to the learning device are included in the learning input data. Then, the learning device 10 trains an AI model using a learning data set that has the learning input data and has an emotion index value corresponding to the biological signal of the subject U1 as correct answer data.

[0187] An estimation device including a trained model trained using a training data set can be configured. One type of such estimation device is the emotion estimation device 20. For example, the emotion estimation device 20 in FIG. 1 includes AI models 221 and 222 as trained models. The emotion index value of the subject U2 is estimated by inputting estimation input data having information of an input operation by the subject U2 to the target device and information output from the target device in response to the input operation to the target device into the trained model. In the emotion estimation device 20, the emotion of the subject U2 is estimated based on the emotion index value (e.g., arousal level and activity level) estimated for the subject U2.

[0188] See Figure 23. For the sake of simplicity, Information on the input operation by the subject U1 to the learning device is referred to as input operation information IN1. Information output from the learning device in response to an input operation to the learning device by the subject U1 is referred to as device output information OUT1. Information on the input operation on the target device by the target person U2 is referred to as input operation information IN2. Information output from the target device in response to an input operation by the target person U2 to the target device is referred to as device output information OUT2.

[0189] In the first embodiment, the learning device and the target device are vehicles V1 and V2, respectively. In the first embodiment, the input operation information IN1 and the device output information OUT1 are represented by the sensor output information Sout1 obtained from the sensor unit of the vehicle V1 (see FIG. 7). In the first embodiment, the input operation information IN2 and the device output information OUT2 are represented by the sensor output information Sout2 obtained from the sensor unit of the vehicle V2 (see FIG. 1).

[0190] The learning device and the target device are not limited to a vehicle. For example, the learning device and the target device may be a computer device. A case where the learning device and the target device are a computer device (hereinafter, referred to as a first case) will be described.

[0191] Fig. 24 shows a schematic configuration of the learning device 10 in the first case. Fig. 25 shows a schematic configuration of the feeling estimation device 20 in the first case.

[0192] The learning device in the first case is a computer CP1, and the subject U1 is an operator who operates the computer CP1. The computer CP1 has a keyboard KB1, a mouse MS1, and a display unit DSP1 as a man-machine interface. The subject U1 inputs a desired instruction to the computer CP1 by operating the keyboard KB1 and the mouse MS1. The computer CP1 performs arithmetic processing according to the input instruction and displays an image according to the input instruction on the display unit DSP1. The target device in the first case is a computer CP2, and the subject U2 is an operator who operates the computer CP2. The computer CP2 has a keyboard KB2, a mouse MS2, and a display unit DSP2 as a man-machine interface. The subject U2 inputs a desired instruction to the computer CP2 by operating the keyboard KB2 and the mouse MS2. The computer CP2 performs arithmetic processing according to the input instruction and displays an image according to the input instruction on the display unit DSP2.

[0193] FIG. 26 shows an overview of the input operation information IN1 and the device output information OUT1 in the first case.

[0194] In the first case, the input operation information IN1 includes operation information of the keyboard KB1 by the subject U1. The subject U1 performs a pressing operation of pressing a key on the keyboard KB1 to give a desired instruction to the computer device CP1. The operation information of the keyboard KB1 includes the input pressure (pressure for pressing a key) in the pressing operation on the keyboard KB1, and the operation speed of the keyboard KB1. The operation speed of the keyboard KB1 is the average value of the reciprocal of the repetition interval when the pressing operation on the keyboard KB1 is performed repeatedly.

[0195] In the first case, the input operation information IN1 includes operation information of the mouse MS1 by the subject U1. The mouse MS1 is provided with a button that receives an operation from the subject U1. When the subject U1 performs a button pressing operation to press the button of the mouse MS1, the computer device CP1 executes a process according to the button pressing operation. The operation information of the mouse MS1 includes an input pressure (pressure to press the button) in the button pressing operation to the mouse MS1. The operation information of the mouse MS1 also includes movement information of the mouse MS1. The computer device CP1 displays a pointer on the display unit DSP1. When the subject U1 performs a movement operation to move the mouse MS1 on the desk, the computer device CP1 moves the pointer on the display unit DSP1 according to the movement operation of the mouse MS1. The movement information of the mouse MS1 includes the movement amount, movement speed, and movement trajectory of the mouse MS1 in the movement operation of the mouse MS1. In addition, if the mouse MS1 is provided with a wheel, the operation content of the wheel may also be included in the operation information of the mouse MS1.

[0196] In the first case, the device output information OUT1 includes operation response information of the keyboard KB1. The operation response information of the keyboard KB1 indicates the processing contents performed by the computer device CP1 in response to an input operation on the keyboard KB1. For example, the additional character display interval when characters are additionally displayed on the display unit DSP1 in response to a pressing operation on the keyboard KB1 is included in the operation response information of the keyboard KB1.

[0197] In the first case, the device output information OUT1 includes operation correspondence information of the mouse MS1. The operation correspondence information of the mouse MS1 indicates the processing content performed by the computer device CP1 in response to an input operation to the mouse MS1. For example, the movement speed of the pointer on the display unit DSP1 in response to the movement operation of the mouse MS1 is included in the operation correspondence information of the mouse MS1.

[0198] The computer device CP1 has a function of outputting the input operation information IN1 and the device output information OUT1 as data. In the first case, as shown in FIG. 24, the input operation information IN1 and the device output information OUT1 related to the computer device CP1 are included in the reference data D1. Except for the content of the reference data D1 being different from the first embodiment, the configuration and operation of the learning device 10 are as shown in the first embodiment. The learning device 10 in the first case may be provided with a normalization unit 135, or may be provided with both the normalization unit 135 and the quantization unit 136.

[0199] 27 shows an overview of the input operation information IN2 and the device output information OUT2 in the first case. The input operation information IN1 and the device output information OUT1 are information related to the computer device CP1, while the input operation information IN2 and the device output information OUT2 are information related to the computer device CP2. Except for this, the input operation information IN2 is the same type of information as the input operation information IN1, and the device output information OUT2 is the same type of information as the device output information OUT1.

[0200] In the first case, the input operation information IN2 includes operation information of the keyboard KB2 by the subject U2. The subject U2 performs a pressing operation of pressing a key on the keyboard KB2 to give a desired instruction to the computer device CP2. The operation information of the keyboard KB2 includes the input pressure (pressure for pressing a key) in the pressing operation on the keyboard KB2, and the operation speed of the keyboard KB2. The operation speed of the keyboard KB2 is the average value of the reciprocal of the repetition interval when the pressing operation on the keyboard KB2 is performed repeatedly.

[0201] In the first case, the input operation information IN2 includes operation information of the mouse MS2 by the subject U2. The mouse MS2 is provided with a button that receives an operation from the subject U2. When the subject U2 performs a button pressing operation to press the button of the mouse MS2, the computer device CP2 executes a process according to the button pressing operation. The operation information of the mouse MS2 includes the input pressure (pressure to press the button) in the button pressing operation on the mouse MS2. The operation information of the mouse MS2 also includes movement information of the mouse MS2. The computer device CP2 displays a pointer on the display unit DSP2. When the subject U2 performs a movement operation to move the mouse MS2 on the desk, the computer device CP2 moves the pointer on the display unit DSP2 according to the movement operation of the mouse MS2. The movement information of the mouse MS2 includes the movement amount, movement speed, and movement trajectory of the mouse MS2 in the movement operation of the mouse MS2. In addition, if a wheel is provided on the mouse MS2, the operation content of the wheel may also be included in the operation information of the mouse MS2.

[0202] In the first case, the device output information OUT2 includes operation correspondence information of the keyboard KB2. The operation correspondence information of the keyboard KB2 indicates the processing contents performed by the computer device CP2 in response to an input operation on the keyboard KB2. For example, the additional character display interval when characters are additionally displayed on the display unit DSP2 in response to a pressing operation on the keyboard KB2 is included in the operation correspondence information of the keyboard KB2.

[0203] In the first case, the device output information OUT2 includes operation correspondence information of the mouse MS2. The operation correspondence information of the mouse MS2 indicates the processing content performed by the computer device CP2 in response to an input operation to the mouse MS2. For example, the movement speed of the pointer on the display unit DSP2 in response to the movement operation of the mouse MS2 is included in the operation correspondence information of the mouse MS2.

[0204] The computer device CP2 has a function of outputting the input operation information IN2 and the device output information OUT2 as data. In the first case, as shown in Fig. 25, the input operation information IN2 and the device output information OUT2 related to the computer device CP2 are included in the reference data D2. Except for the content of the reference data D2 being different from the first embodiment, the configuration and operation of the feeling estimation device 20 are as shown in the first embodiment.

[0205] In the first case, the feeling estimation device 20 may be mounted on the computer device CP2, or may be mounted on an electronic device other than the computer device CP2. In the first case, the system having the feeling estimation device 20 can perform emotion response processing according to the emotion estimated by the emotion estimation device 20. For example, when it is estimated that the subject U2 has a depressed emotion, a message such as "Let's take a short break" is displayed on the display unit DSP2 in the emotion response processing. This can improve the efficiency of work using the computer device CP2.

[0206] Also, for example, the learning device and the target device may be game devices. A case in which the learning device and the target device are game devices (hereinafter, referred to as a second case) will be described.

[0207] Fig. 28 shows a schematic configuration of the learning device 10 in the second case. Fig. 29 shows a schematic configuration of the feeling estimation device 20 in the second case.

[0208] The learning device in the second case is a game device GM1, and the subject U1 is an operator (player) who operates the game device GM1. The game device GM1 is equipped with a controller and a display unit as a man-machine interface. The subject U1 progresses the game on the game device GM1 by operating the controller of the game device GM1. The progress of the game is displayed on the display unit of the game device GM1. The target device in the second case is a game device GM2, and the subject U2 is an operator (player) who operates the game device GM2. The game device GM2 is equipped with a controller and a display unit as a man-machine interface. The subject U2 progresses the game on the game device GM2 by operating the controller of the game device GM2. The progress of the game is displayed on the display unit of the game device GM2.

[0209] In the second case, the input operation information IN1 includes operation information (information representing the content of the operation) of the controller of the game device GM1 by the subject U1, etc. In the second case, the device output information OUT1 includes the results of the game proceeding on the game device GM1, action information of the game, etc.

[0210] The game device GM1 has a function of outputting the input operation information IN1 and the device output information OUT1 as data. In the second case, as shown in FIG. 28, the input operation information IN1 and the device output information OUT1 related to the game device GM1 are included in the reference data D1. Except for the content of the reference data D1 being different from that of the first embodiment, the configuration and operation of the learning device 10 are as shown in the first embodiment. The learning device 10 in the second case may be provided with a normalization unit 135, or may be provided with both the normalization unit 135 and the quantization unit 136.

[0211] The input operation information IN1 and the device output information OUT1 are information related to the game device GM1, whereas the input operation information IN2 and the device output information OUT2 are information related to the game device GM2. Except for this, the input operation information IN2 is the same type of information as the input operation information IN1, and the device output information OUT2 is the same type of information as the device output information OUT1.

[0212] In the second case, the input operation information IN2 includes operation information (information representing the content of the operation) of the controller of the game device GM2 by the subject U2, etc. In the second case, the device output information OUT2 includes the results of the game proceeding on the game device GM2, action information of the game, etc.

[0213] The game device GM2 has a function of outputting the input operation information IN2 and the device output information OUT2 as data. In the second case, as shown in Fig. 29, the input operation information IN2 and the device output information OUT2 related to the game device GM2 are included in the reference data D2. Except for the content of the reference data D2 being different from the first embodiment, the configuration and operation of the feeling estimation device 20 are as shown in the first embodiment.

[0214] In the second case, the emotion estimation device 20 may be mounted on the game device GM2, or may be mounted on an electronic device other than the game device GM2. In the second case, the system having the emotion estimation device 20 can perform emotion response processing according to the emotion estimated by the emotion estimation device 20. For example, when it is estimated that the subject U2 has an unpleasant emotion, the emotion response processing notifies the subject U2 that "it is estimated that you are not feeling well. Please concentrate."

[0215] For example, the emotion estimation result can be used for mental training of the subject U2 in e-sports. When the subject U2 feels an unfavorable emotion (anxiety, anger, etc.) during game play, it is determined that intensive training corresponding to the emotional state is necessary. The emotion information of the subject U2 estimated by the emotion estimation device 20 is used in the training.

[0216] In addition, there are types of e-sports, such as a type in which multiple players play together and a type in which multiple players play against each other. In such types of e-sports, if the emotional state of each player is displayed, it becomes possible to perform advanced game play, such as changing game tactics according to the emotional state. It is also possible to configure the e-sports spectators to be able to understand the emotional state of each player while watching the game, which can increase the fun factor of watching the game.

[0217] Although this overlaps with what has already been mentioned, the first or second modified technique described above may be adopted in the second embodiment.

[0218] <<Additional Information>> A program for causing a computer device to execute any of the methods described in the embodiments of the present invention, and a non-volatile recording medium having the program recorded thereon, are included in the scope of the embodiments of the present invention. Any of the processes in the embodiments of the present invention may be realized by hardware such as a semiconductor integrated circuit, software equivalent to the program, or a combination of hardware and software.

[0219] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above embodiments are merely examples of the present invention, and the meanings of the terms of the present invention and each component are not limited to those described in the above embodiments. The specific numerical values ​​shown in the above description are merely examples, and can be changed to various numerical values ​​as a matter of course. [Explanation of symbols]

[0220] SYS Vehicle Control System 10 Learning Device 11 Communications Department 12 Storage section 121m 1st AI model 122m 2nd AI model 13 Controller 131 Acquisition Department 132 Input Data Generation Unit 133 Index value derivation part 134 Learning processing unit 135 Normalization section 136 Quantization section 20 Emotion estimation device 21 Communications Department 22 Memory section 221m 1st AI model 221m 2nd AI model 223m Emotion estimation model 23 Controller 231 Acquisition Department 232 Input Data Generation Unit 233 Emotion estimation part 234 Estimation result output section 30 Vehicle control device 40 Actuator section 50 Notification Department 60 Sensor section DB Database U1 Subject U2 Target Audience V1 and V2 vehicles

Claims

1. The AI ​​model is trained using a learning data set having learning input data including at least one of information on an input operation by a subject to a learning device and information output from the learning device in response to the input operation to the learning device, and having an emotion index value corresponding to a biological signal of the subject as correct answer data. ,Learning device.

2. The biological signal includes electroencephalogram information, The emotion index value includes a degree of arousal of the central nervous system based on the electroencephalogram. The learning device according to claim 1 .

3. The biological signal includes heart rate information, The emotion index value includes an activity level of the autonomic nervous system based on the heart rate. The learning device according to claim 1 .

4. the learning device is a learning vehicle, and the subject is a driver of the learning vehicle; the input operation to the learning device is a driving operation of the learning vehicle performed by the subject, The information output from the learning device includes driving state information of the learning vehicle corresponding to the driving operation of the learning vehicle. A learning device according to any one of claims 1 to 3.

5. The learning device includes learning input data including at least one of information on an input operation by a subject to a learning device and information output from the learning device in response to the input operation to the learning device, and a trained model trained using a training dataset having an emotion index value corresponding to a biological signal of the subject as correct answer data; At least one of information on an input operation by a subject to a target device and information output from the target device in response to the input operation to the target device is input to the trained model, thereby estimating an emotion index value of the subject. , Estimation device.

6. a first trained model having learning input data including at least one of information of an input operation by a subject to a learning device and information output from the learning device in response to the input operation to the learning device, and trained using a first training data set having an emotion index value according to the electroencephalogram of the subject as correct answer data; a second trained model trained using a second training data set having the learning input data and having an emotion index value according to the heart rate of the subject as correct answer data; An emotion estimation model for estimating an emotion of a subject, Input data for estimation including at least one of information on an input operation on a target device by the subject and information output from the target device in response to the input operation on the target device is input to each trained model, thereby estimating the emotion of the subject by the emotion estimation model based on a first emotion index value and a second emotion index value output from the first trained model and the second trained model. , estimation device.

7. the emotion index value according to the electroencephalogram of the subject is a degree of arousal of the central nervous system based on the electroencephalogram of the subject, the emotion index value according to the heart rate of the subject is an activity level of the autonomic nervous system based on the heart rate of the subject, the first emotion index value is a degree of arousal of the central nervous system of the subject; The second emotion index value is an activity of the subject's autonomic nervous system. R The estimation device according to claim 6 .

8. the learning device is a learning vehicle, and the subject is a driver of the learning vehicle; the input operation to the learning device is a driving operation of the learning vehicle performed by the subject, the information output from the learning device includes driving state information of the learning vehicle corresponding to the driving operation of the learning vehicle; the target device is a target vehicle, and the target person is a driver of the target vehicle; The input operation to the target device is a driving operation of the target vehicle performed by the target person, The information output from the target device includes driving state information of the target vehicle corresponding to the driving operation of the target vehicle.

8. An estimation device according to claim 6 or 7.

9. The estimation device according to claim 8 ; a sensor unit that is installed in the target vehicle and generates sensor output information that is used for driving control of the target vehicle; The estimation device generates the estimation input data based on the sensor output information. ,In-vehicle emotion estimation system.

10. The AI ​​model is trained using a learning data set having learning input data including at least one of information on an input operation by a subject to a learning device and information output from the learning device in response to the input operation to the learning device, and having an emotion index value corresponding to a biological signal of the subject as correct answer data. , learning methods.

11. A method for training an AI model using a learning data set having learning input data including at least one of information on an input operation by a subject to a learning device and information output from the learning device in response to the input operation to the learning device, and having an emotion index value corresponding to a biological signal of the subject as correct answer data, is performed by a computer. , learning programs.

Images