Emotion estimation device, on-board emotion estimation system, method for estimating emotion, and emotion estimation program

Abstract

To estimate emotions of drivers of peripheral vehicles in a practical mode.SOLUTION: An emotion estimation device generates travel state information of an other vehicle (V2) on the basis of travel state information of own vehicle (V1) and a relative physical relationship between own vehicle and the other vehicle and estimates emotion of the driver of the other vehicle on the basis of the travel state information of the other vehicle.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an emotion estimation device, an in-vehicle emotion estimation system, an emotion estimation method, and an emotion estimation program. [Background technology]

[0002] 2. Description of the Related Art There is known an apparatus that estimates a person's emotion based on a biosignal measured by a biosensor (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-63324 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a vehicle such as an automobile, if it is possible to estimate and understand the emotions of the drivers of surrounding vehicles, driving that takes these emotions into consideration becomes possible, which is thought to lead to safe driving. In this case, a reference method is considered in which the drivers of the surrounding vehicles are fitted with biosensors and the output signal of the biosensors or information based on the output signal of the biosensors is transmitted to the vehicle by wireless communication. However, the requirement to wear a biosensor is troublesome for the drivers of each vehicle. In addition, transmitting the output signal of the biosensor or information based on the output signal of the biosensor to the vehicle by wireless communication is difficult to realize (not practical) from the viewpoint of communication costs and personal information protection, etc.

[0005] An object of the present invention is to provide a technique capable of estimating the emotions of drivers of nearby vehicles in a practical manner. [Means for solving the problem]

[0006] The emotion estimation device according to the present invention generates driving state information of another vehicle based on driving state information of the host vehicle and a relative positional relationship between the host vehicle and the other vehicle, and estimates the emotion of a driver of the other vehicle based on the driving state information of the other vehicle. Effect of the Invention

[0007] According to the present invention, driving condition information of a vehicle driven by a driver of a surrounding vehicle, which is correlated with the emotions of the driver, is generated, and the emotions of the driver are estimated based on the driving condition information, thereby making it possible to provide a technology that can estimate the emotions of the drivers of surrounding vehicles in a practical manner. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic configuration diagram of a feeling estimation device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of a two-dimensional model related to emotion estimation in the first embodiment of the present invention. [Diagram 3] FIG. 11 is a diagram showing another example of the two-dimensional model related to emotion estimation in the first embodiment of the present invention. [Figure 4] 1 is a configuration block diagram of a vehicle control system according to a first embodiment of the present invention. [Diagram 5] 1 is a block diagram of a sensor unit mounted on a vehicle according to a first embodiment of the present invention; [Figure 6] 4 is a flowchart of a feeling estimation process executed by the feeling estimation device in the first embodiment of the present invention. [Figure 7] FIG. 10 is a schematic configuration diagram of a learning device according to an example EX1_1 belonging to the first embodiment of the present invention. [Figure 8] FIG. 10 is a configuration block diagram of a learning device according to an example EX1_1 belonging to the first embodiment of the present invention. [Figure 9] 11 is a flowchart of a learning process executed by a learning device according to an example EX1_1 belonging to the first embodiment of the present invention. [Figure 10]FIG. 11 is a schematic configuration diagram of a learning device according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 11] FIG. 11 is a configuration block diagram of a learning device according to an example EX1_2 belonging to the first embodiment of the present invention. [Figure 12] FIG. 11 is a diagram showing the alertness levels of two subjects before and after normalization according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 13] FIG. 11 is a diagram showing a task table for normalization according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 14] FIG. 11 is a schematic diagram showing a change in alertness over time during task execution according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 15] FIG. 11 is a schematic diagram showing a change in wakefulness over time according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 16] FIG. 11 is a diagram showing the convergence state of an upper limit estimate value in a neutral region according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 17] FIG. 11 is a diagram showing a neutral region table according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 18] FIG. 11 is a diagram showing the alertness levels of two subjects before and after normalization according to Example EX1_2 belonging to the first embodiment of the present invention. [Figure 19] FIG. 11 is a schematic configuration diagram of a learning device according to Example EX1_3 belonging to the first embodiment of the present invention. [Figure 20] FIG. 11 is a configuration block diagram of a learning device according to an example EX1_3 belonging to the first embodiment of the present invention. [Figure 21] FIG. 11 is a schematic configuration diagram of a learning device according to Example EX1_4 belonging to the first embodiment of the present invention. [Figure 22] FIG. 11 is a schematic configuration diagram of a feeling estimation device according to Example EX1_4 belonging to the first embodiment of the present invention. [Figure 23] FIG. 11 is a schematic configuration diagram of a learning device according to Example EX1_5 belonging to the first embodiment of the present invention. [Figure 24]FIG. 11 is a schematic configuration diagram of a feeling estimation device according to Example EX1_5 belonging to the first embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an example of an embodiment of the present invention will be specifically described with reference to the drawings. In each of the drawings, the same parts are given the same reference numerals, and duplicated descriptions of the same parts are omitted as a rule. In this specification, for the sake of simplifying the description, a symbol or code referring to information, a signal, a physical quantity, a functional part, a circuit, an element, or a part, etc. may be written, and the name of the information, signal, physical quantity, functional part, circuit, element, or part, etc. corresponding to the symbol or code may be omitted or abbreviated. For example, the first AI model referred to by "221m" described later (see FIG. 1) may be written as the first AI model 221m, or may be abbreviated as the AI ​​model 221m or model 221m, but they all refer to the same thing. In addition, AI is an abbreviation for artificial intelligence.

[0010] <<First embodiment>> A first embodiment of the present invention will be described. FIG. 1 shows a schematic configuration of a feeling estimation device 20 according to the first embodiment. The feeling estimation device 20 includes a reference data generation unit 232, an input data generation unit 233, a first AI model 221m, a second AI model 222m, and a feeling estimation model 223m. The feeling estimation device 20 has a function of estimating the feeling of a person U2. The person U2 is a subject of feeling estimation, and is hereinafter referred to as a subject U2. The subject U2 is a driver of a vehicle V2. A vehicle V1 is a vehicle that runs around the vehicle V2. In this embodiment, the vehicle is an automobile or the like that runs on a road surface. In this embodiment, the vehicle may be any type of vehicle. The vehicle V1 is referred to as a subject vehicle, and the vehicle V2 is referred to as a different vehicle.

[0011] The host vehicle V1 and the other vehicle V2 travel in the same direction. The traveling direction of the host vehicle V1 and the other vehicle V2 corresponds to the forward direction. The other vehicle V2 is located in front of the host vehicle V1. In other words, the other vehicle V2 is a vehicle in front of the host vehicle V1. The other vehicle V2 is a vehicle located immediately in front of the host vehicle V1 (immediately ahead vehicle), and no other vehicles exist between the other vehicle V2 and the host vehicle V1. Distance d12 is the inter-vehicle distance between the host vehicle V1 and the other vehicle V2. From the host vehicle V1, the distance d12 corresponds to the forward inter-vehicle distance, and from the other vehicle V2, the distance d12 corresponds to the rear inter-vehicle distance. The inter-vehicle distance d12 corresponds to the distance between the front end of the host vehicle V1's body and the rear end of the other vehicle V2's body.

[0012] In any vehicle, the driver's emotion is expressed in the vehicle's driving state information (speed, acceleration, steering angle, driving trajectory, etc.), so the driver's emotion can be estimated from the vehicle's driving state information. The emotion estimation device 20 according to this embodiment is applied to a host vehicle V1 and estimates the emotion of a driver (subject U2) of another vehicle V2.

[0013] In summary, the feeling estimation device 20 acquires driving state information of the host vehicle V1 based on sensor output information Sout1 output from a sensor unit installed in the host vehicle V1. Next, the feeling estimation device 20 converts the driving state information of the host vehicle V1 into driving state information of the other vehicle V2 based on the relative positional relationship between the host vehicle V1 and the other vehicle V2. The relative positional relationship between the host vehicle V1 and the other vehicle V2 is identified using an image sensor (camera) and a distance measuring sensor installed in the host vehicle V1. Then, the feeling estimation device 20 generates input data for estimation that matches the input specifications of the AI ​​model (221m, 222m) from reference data D2 including the driving state information of the other vehicle V2, and estimates the emotion of the subject U2 based on the input data for estimation.

[0014] In the feeling estimation device 20, an emotion index value of the subject U2 is derived using the AI ​​models 221m and 222m, and an emotion of the subject U2 is estimated from the derived emotion index value using an emotion estimation model 223m. This will be described in detail below.

[0015] The emotion estimation model 223m estimates emotions based on two emotion indices, which are indices indicating the mental and physical state of the subject U2. The mental and physical state of any person includes a psychological state, and therefore each emotion index can also be said to be an index indicating a psychological state. The value of an emotion index is called an emotion index value. One of the emotion indices used in this embodiment is the arousal level of the central nervous system (hereinafter referred to as the arousal level), and the other emotion index used in this embodiment is the activity level of the autonomic nervous system (hereinafter referred to as the activity level). That is, one emotion index value is expressed by the arousal level, and the other emotion index value is expressed by the activity level. The arousal level can be derived based on, for example, electroencephalogram data. The activity level can be derived based on, for example, heart rate data.

[0016] The emotion estimation model 223m is a model for emotion estimation based on arousal and activity, and may be configured with a calculation formula or a conversion data table that specifies emotion from arousal and activity. The emotion estimation model 223m is a multidimensional model that estimates emotion using arousal and activity as parameters, and is a two-dimensional model with arousal and activity as two axes here. The two-dimensional model is created based on medical evidence (papers, etc.) showing the relationship between arousal and activity and emotion. Alternatively, the two-dimensional model is created based on the results of a questionnaire by many subjects. The questionnaire results of the subjects include the arousal and activity of the subjects specified from the measured values ​​of the subject's brain waves and heart rate, and the subject's declared emotion at the time of measuring the brain waves and heart rate. Note that a multidimensional model of three or more dimensions may be used as the emotion estimation model 223m.

[0017] FIG. 2 shows an example of a two-dimensional model related to emotion estimation. FIG. 2 shows a two-dimensional psychological plane PP defined in the two-dimensional model. According to various medical evidence related to psychology, psychology can be estimated based on two types of indices that indicate mental and physical states. The psychological plane PP shown in FIG. 2 has a vertical axis corresponding to the arousal level and a horizontal axis corresponding to the activity level.

[0018] Points indicating the awakening and activity of the subject U2 can be plotted (placed) on the psychological plane PP. The higher the awakening of the subject U2, the more the coordinates of the plotted (placed) points move in the positive direction of the vertical axis, and the lower the awakening of the subject U2, the more the coordinates of the plotted (placed) points move in the negative direction of the vertical axis. The higher the activity of the subject U2, the more the coordinates of the plotted (placed) points move in the positive direction of the horizontal axis, and the lower the activity of the subject U2, the more the coordinates of the plotted (placed) points move in the negative direction of the horizontal axis. From the origin of the psychological plane PP, the positive side of the vertical axis corresponds to an awakened state, and the negative side of the vertical axis corresponds to an unawakened state. From the origin of the psychological plane PP, the positive side of the horizontal axis corresponds to a state in which the sympathetic nerves are activated (a state in which a relatively strong emotion is present), and the negative side of the horizontal axis corresponds to a state in which the parasympathetic nerves are activated (a state in which a relatively weak emotion is present).

[0019] The vertical and horizontal axes intersect at the origin of the psychological plane PP, and the psychological plane PP has first to fourth quadrants separated by the vertical and horizontal axes. The first quadrant is an area where the vertical axis component has a positive value and the horizontal axis component also has a positive value when viewed from the origin of the psychological plane PP. The second quadrant is an area where the vertical axis component has a positive value and the horizontal axis component also has a negative value when viewed from the origin of the psychological plane PP. The third quadrant is an area where the vertical axis component has a negative value and the horizontal axis component also has a negative value when viewed from the origin of the psychological plane PP. The fourth quadrant is an area where the vertical axis component has a negative value and the horizontal axis component has a positive value when viewed from the origin of the psychological plane PP.

[0020] In the psychological plane PP, each quadrant is assigned a corresponding psychological state (in other words, an emotion). The first quadrant is assigned the psychological states of "fun, joy, anger, sadness." The second quadrant is assigned the psychological state of "melancholy." The third quadrant is assigned the psychological state of "relaxation, calm." The fourth quadrant is assigned the psychological states of "anxiety, fear, unpleasantness." The distance from the origin indicates the intensity of the corresponding psychological state. For example, when a point indicating the arousal and activity of subject U2 is plotted in the second quadrant, the greater the distance between the plotted point and the origin, the stronger the psychological state of "melancholy" for subject U2.

[0021] In this way, the emotion of any person depends on the arousal level and activity level of the person. The position of each axis on the psychological plane PP may be appropriately set based on an experiment or the like. Each axis may be set based on the position of a neutral area described later.

[0022] The emotion estimation model 223m can estimate the psychological state of the subject U2 from coordinates obtained by plotting (arranging) points indicating the arousal and activity of the subject U2 on the psychological plane PP. The psychological state and its intensity of the subject U2 can be estimated based on which quadrant of the psychological plane PP the coordinates of the plotted point are in, where in the quadrant they are located, and the distance between the plotted point and the origin. The first to fourth quadrants on the psychological plane PP can be considered as first to fourth classes, respectively, and in this case, the emotion estimation model 223m can be said to classify the psychological state of the subject U2 into one of the first to fourth classes. In addition, when the emotion estimation model 223m is a multidimensional model of three or more dimensions, a multidimensional psychological space of three or more dimensions is used depending on the number of types of emotion index values ​​to be used.

[0023] Incidentally, when emotion intensity is strong, that is, when the emotion index value swings significantly toward the maximum or minimum value, emotion estimation accuracy tends to increase. On the other hand, when emotion intensity is weak, that is, when the emotion index value is near the median, emotion estimation accuracy tends to decrease. For this reason, a method can be adopted in which the region near the median of the emotion index value is set as a neutral region, and emotion index values ​​that fall into the neutral region are judged to have no estimated emotion or to be unable to be estimated.

[0024] Fig. 3 is a diagram showing an example of a psychological plane PP including a neutral region. In Fig. 3, a shaded region NR1 extending along the horizontal axis and a shaded region NR2 extending along the vertical axis are both neutral regions. Of these, region NR1 is an arousal neutral region, and region NR2 is an activity neutral region.

[0025] The arousal neutral region NR1 is a region associated with an arousal level equal to or higher than the lower limit value YN and equal to or lower than the upper limit value YP. "YN < YP". The horizontal axis passing through the origin of the psychological plane PP is located within the arousal neutral region NR1. The arousal neutral region NR1 is a region extending along the direction of the horizontal axis of the psychological plane PP, and the horizontal axis of the psychological plane PP may be arranged at the center of the arousal neutral region NR1. The activity neutral region NR2 is a region associated with an activity level equal to or higher than the lower limit value XN and equal to or lower than the upper limit value XP. "XN < XP". The vertical axis passing through the origin of the psychological plane PP is located within the activity neutral region NR2. The activity neutral region NR2 is a region extending along the direction of the vertical axis of the psychological plane PP, and the vertical axis of the psychological plane PP may be arranged at the center of the activity neutral region NR2. A part of the arousal neutral region NR1 and a part of the activity neutral region NR2 overlap with each other in a region including the origin of the psychological plane PP. The origin of the psychological plane PP may be arranged at the center of the region where the regions NR1 and NR2 overlap.

[0026] The setting of these neutral regions (i.e., the setting of the upper limit value YP, the lower limit value YN, the upper limit value XP, and the lower limit value XN) may be appropriately performed through experiments or the like. Examples of the setting method regarding the neutral region will be described later. Each position within the neutral region may be understood not to belong to any of the first to fourth quadrants of the psychological plane PP.

[0027] A biosignal is usually required to calculate the emotion index value, and a contact-type biosensor is usually required to acquire the biosignal. However, wearing a biosensor is troublesome for the emotion estimation subject, especially for the vehicle driver (the subject U2 in this case) who tends to frequently wear and remove the biosensor. For this reason, an emotion estimation method that does not require wearing a biosensor is required. Even if the subject U2 wears a biosensor, it is difficult to transmit information based on the output of the biosensor from the other vehicle V2 to the subject vehicle V1 so that the subject U2's emotion can be recognized by the subject vehicle V1, from the viewpoint of communication costs and personal information protection. In consideration of these, the emotion estimation device 20 according to this embodiment converts the driving state information of the subject vehicle V1 into driving state information of the other vehicle V2 to estimate the emotion of the driver of the other vehicle V2 (the subject U2).

[0028] As described above, in any vehicle, the driver's emotions are reflected in the vehicle's driving state information (speed, acceleration, steering angle, driving trajectory, etc.), so the driver's emotions can be estimated from the vehicle's driving state information. Meanwhile, emotions are correlated with biological information such as brain waves and heart rate. Therefore, it is possible to model the correlation between the driving state information of the other vehicle V2 and the biological information such as brain waves and heart rate of the driver of the other vehicle V2 (subject U2). In this case, by using a process of converting the driving state information of the host vehicle V1 into the driving state information of the other vehicle V2, it becomes possible to estimate the biological information of the driver of the other vehicle V2 (subject U2) from the driving state information of the host vehicle V1.

[0029] In the feeling estimation device 20, as shown in Fig. 1, the input data generation unit 233 generates estimation input data based on the reference data D2. The reference data D2 includes traveling state information of the other vehicle V2 based on the sensor output information Sout1 from the sensor unit provided in the host vehicle V1. The details of these data and information will be described later.

[0030] The AI ​​models 221m and 222m are trained models obtained through prior training. In this specification, training refers to machine learning. The training method will be described later. The first AI model 221m is a model that outputs (calculates) the alertness of the subject U2 based on the input data for estimation, and has a structure suitable for input and output. The second AI model 222m is a model that outputs (calculates) the activity of the subject U2 based on the input data for estimation, and has a structure suitable for input and output. Any AI model may be configured by a neural network. The alertness and activity of the subject U2 are each a type of biological information of the subject U2. In any person, the alertness correlates with brain waves, and the activity correlates with heart rate.

[0031] The emotion estimation model 223m estimates the emotion of the subject U2 based on the arousal level and activity level output from the AI ​​models 221m and 222m.

[0032] 4 is a configuration block diagram of the vehicle control system SYS according to the first embodiment. The vehicle control system SYS includes a feeling estimation device 20, a vehicle control device 30, an actuator unit 40, a notification unit 50, and a sensor unit 60. The feeling estimation device 20, the vehicle control device 30, the actuator unit 40, the notification unit 50, and the sensor unit 60 are all mounted on the host vehicle V1.

[0033] The feeling estimation device 20 includes a communication unit 21, a storage unit 22, and a controller 23. In the vehicle control system SYS, various controls can be performed using the feeling estimation result by the feeling estimation device 20. When performing vehicle control that requires quick response using the feeling estimation result, it is desirable to mount the feeling estimation device 20 on the vehicle V1 as in this example. However, the feeling estimation device 20 may be configured using a server device provided outside the vehicle V1. In this case, an in-vehicle block having a part of the feeling estimation device 20 is installed on the vehicle V1, while a server block having the remaining part of the feeling estimation device 20 is installed on the server device, and the feeling estimation device 20 is configured by the cooperation of the in-vehicle block and the server block. The in-vehicle block and the server block share any information by communication via a predetermined communication network. For example, among the function blocks 231 to 235 described later, the function blocks 233 and 234 are realized by the server block, and the other function blocks 231, 232, and 235 are realized by the in-vehicle block. In addition, the allocation of the functional blocks to the server block and the in-vehicle block may be arbitrary. The server device is composed of one or more computer devices connected to a communication network. When the emotion estimation device 20 is configured by the in-vehicle block and the server block working together, the part of the memory contents of the memory unit 22 necessary for realizing the operation of the server block is held in a memory unit (not shown) in the server device. That is, for example, when the functional block 234 is provided in the server block, the memory units 221 to 223 shown in FIG. 4 are held in a memory unit (not shown) in the server device.

[0034] The communication unit 21 transmits and receives any signal between the emotion estimation device 20 and a counterpart device different from the communication unit 21. The counterpart device for the communication unit 21 includes a vehicle control device 30. The communication unit 21 can communicate with the counterpart device via an in-vehicle network formed in the vehicle V1. The in-vehicle network may include, for example, a Controller Area Network (CAN) and may further include an Audio Visual Communication Local Area Network (AVCLAN). The counterpart device for the communication unit 21 may further include a sensor unit 60, in which case the communication unit 21 may be able to directly receive output information from the sensor unit 60. The counterpart device for the communication unit 21 may further include an external device (such as a server device connected to the above communication network) provided outside the vehicle V1.

[0035] The storage unit 22 is configured to have a non-volatile memory such as a ROM (Read only memory) or a flash memory, and a volatile memory such as a RAM (Random access memory). The storage unit 22 includes a first AI model storage unit 221 that stores a first AI model 221m, and a second AI model storage unit 222 that stores a second AI model 222m. The storage unit 22 further includes an emotion estimation model storage unit 223 that stores an emotion estimation model 223m. In addition to the emotion estimation model 223m, the emotion estimation model storage unit 223 also stores various data required for emotion estimation, such as a psychological plane table 223t. The psychological plane table 223t is data that forms the psychological plane PP shown in FIG. 2 or FIG. 3, and has a data group that associates two types of emotion index values ​​(arousal and activity) with the corresponding emotion information. The storage unit 22 further includes a program storage unit 224 that stores various programs. The programs stored in the program storage unit 224 include a program 224p.

[0036] The controller 23 comprehensively controls the operation of each part in the feeling estimation device 20. The controller 23 includes a calculation processing unit including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) as hardware resources. The controller 23 includes functional blocks 231 to 235.

[0037] The controller 23 is a program execution device (computer) capable of executing any program. The controller 23 executes the program 224p to realize each function of the controller 23 (including each function of the functional blocks 231 to 235). All of the operations of the controller 23 described in this embodiment may be operations that are realized by the controller 23 executing the program 224p. The program 224p may be composed of a plurality of programs. The controller 23 can transmit and receive any information to and from a counterpart device (a counterpart device for the communication unit 21) using the communication unit 21, but in the following, the description of the communication unit 21 may be omitted.

[0038] The functional blocks of the controller 23 will be described. The functional blocks 231, 232, 233, 234, and 235 are an acquisition unit, a reference data generation unit, an input data generation unit, a feeling estimation unit, and an estimation result output unit, respectively. The acquisition unit 231 acquires information used to estimate the feeling of the subject U2. The information acquired by the acquisition unit 231 includes information generated by the sensor unit 60.

[0039] The reference data generating unit 232 generates driving state information of the other vehicle V2 based on the information acquired by the acquiring unit 231. At this time, the driving state information of the host vehicle V1 is converted into driving state information of the other vehicle V2 based on the relative positional relationship between the host vehicle V1 and the other vehicle V2. The reference data generating unit 232 generates reference data D2 including the driving state information of the other vehicle V2 and supplies it to the input data generating unit 233. The input data generating unit 233 generates estimation input data that matches the input specifications of the AI ​​models 221m and 222m based on the reference data D2 (see also FIG. 1).

[0040] The feeling estimation unit 234 estimates the feeling of the subject U2 using the AI ​​models 221m and 222m and the feeling estimation model 223m based on the input data for estimation. Specifically, the input data for estimation generated by the input data generation unit 233 is input to the AI ​​models 221m and 222m. As a result, the AI ​​models 221m and 222m output the arousal level and activity level, which are the emotion index values ​​of the subject U2. Next, the feeling estimation unit 234 inputs the arousal level and activity level output from the AI ​​models 221m and 222m to the feeling estimation model 223m. As a result, the feeling of the subject U2 is estimated by the emotion estimation model 223m, and the estimation result is output from the emotion estimation model 223m.

[0041] Coordinates obtained by plotting (arranging) points indicating the arousal and activity of the subject U2 on the psychological plane PP are referred to as plot coordinates. The plot coordinates represent the positions of the points indicating the arousal and activity of the subject U2 (positions on the psychological plane PP). The emotion estimation unit 234 estimates the emotion of the subject U2 by identifying in which quadrant of the psychological plane PP the plot coordinates are located. It may be considered that the estimation of the emotion of the subject U2 is performed by the emotion estimation model 223m.

[0042] When the plot coordinates are located in the i-th quadrant, the emotion estimation unit 234 (emotion estimation model 223m) estimates that the emotion of the subject U2 corresponds to the psychological state assigned to the i-th quadrant. Here, i represents any integer. Therefore, for example, when the plot coordinates are located in the first quadrant, the emotion estimation unit 234 (emotion estimation model 223m) estimates that the subject U2 has an emotion corresponding to "joy, joy, anger, or sadness." Similarly, for example, when the plot coordinates are located in the second quadrant, the emotion estimation unit 234 (emotion estimation model 223m) estimates that the subject U2 has an emotion corresponding to "melancholy." The same applies when the plot coordinates are located in the third or fourth quadrant.

[0043] When the psychological plane PP including the neutral region is used, the plot coordinates may be located within the neutral region. In this case, the emotion estimation unit 234 (emotion estimation model 223m) estimates that the subject U2 does not have any particular emotion or determines that emotion estimation of the subject U2 is impossible.

[0044] The estimation result output unit 235 outputs the estimation result of the emotion of the subject U2 by the emotion estimation unit 234 to the vehicle control device 30. This makes it possible for the vehicle control device 30 to perform various controls based on the emotion of the subject U2.

[0045] The vehicle control device 30 is configured by, for example, an ECU (Electronic Control Unit) for vehicle control. The vehicle control device 30 includes a communication unit 31, a storage unit 32, and a controller 33.

[0046] The communication unit 31 transmits and receives any signal between the vehicle control device 30 and a counterpart device different from the vehicle control device 30. The counterpart device for the communication unit 31 includes the emotion estimation device 20, the actuator unit 40, the notification unit 50, and the sensor unit 60, and the communication unit 31 can communicate with the counterpart device via a vehicle network formed in the vehicle V1. The counterpart device for the communication unit 31 may further include an external device (such as a server device connected to the above-mentioned communication network) provided outside the vehicle V1. The controller 33 can transmit and receive any information to and from the counterpart device (the counterpart device for the communication unit 31) using the communication unit 31, but in the following, a description of the communication unit 31 may be omitted.

[0047] The storage unit 32 is configured to include a non-volatile memory such as a ROM or a flash memory, and a volatile memory such as a RAM. The storage unit 32 stores programs to be executed by the controller 33, as well as any data to be referenced by the controller 33.

[0048] The controller 33 performs driving control of the host vehicle V1 based on the driving operation of the host vehicle V1 by the driver of the host vehicle V1 and the output information of the sensor unit 60. The controller 33 also receives estimated emotion information of the subject U2 from the emotion estimation device 20 via the communication unit 31. The estimated emotion information of the subject U2 represents the emotion of the subject U2 estimated by the emotion estimation device 20 (emotion estimation unit 234). The controller 33 can perform processing according to the estimated emotion information. As the processing according to the estimated emotion information, the controller 33 can perform notification control for notifying the occupant of the host vehicle V1 of notification information. The notification information is transmitted to the notification unit 50 via the communication unit 31. The occupant of the host vehicle V1 includes the driver of the host vehicle V1 and may also include other passengers.

[0049] The actuator unit 40 has various drive components constituted by a motor and the like that realize the running of the host vehicle V1. The operation of the actuator unit 40 is driven and controlled by the vehicle control device 30. The actuator unit 40 includes an engine and a motor that generate a drive force for the host vehicle V1, a steering actuator that drives the steering of the host vehicle V1, and a brake actuator that drives the brakes of the host vehicle V1.

[0050] The notification unit 50 visually or audibly notifies the occupants of the vehicle V1 of the notification information from the vehicle control device 30. The notification unit 50 has a display device and a speaker. The notification unit 50 can notify the notification information from the vehicle control device 30 by displaying an image on the display device. The notification unit 50 can notify the notification information from the vehicle control device 30 by outputting an audio signal from the speaker.

[0051] The sensor unit 60 includes a sensor that detects the content of the driving operation of the host vehicle V1 by the driver of the host vehicle V1, and sensors that detect various states of the host vehicle V1.

[0052] The following control is possible as the notification control according to the estimated emotion information. That is, the vehicle control device 30 (controller 33) can notify the estimated emotion information itself using the notification unit 50, or can notify according to the estimated emotion information. For example, if it is estimated that the subject U2 feels angry, the subject U2 may drive violently or may drive aggressively. In such a case, the vehicle control device 30 (controller 33) uses the notification unit 50 to display a text or output a voice such as "The driver of the vehicle ahead seems to feel angry. We recommend that you change lanes to avoid being involved in an unexpected situation." Or, for example, if it is estimated that the subject U2 feels fear, the driving operation of the vehicle V1 may cause the subject U2 to feel fear. Therefore, in this case, the vehicle control device 30 (controller 33) uses the notification unit 50 to display a text or output a voice such as "The driver of the vehicle ahead seems to feel fear. Please drive safely with a sufficient distance between the vehicles." In order to realize these, a data table for various processes in which the estimated emotion information received from the feeling estimation device 20 is associated with the notification information can be provided in the storage unit 32. Note that the controller 23 of the feeling estimation device 20 may control the notification unit 50 to perform notification based on the estimated emotion information of the subject U2.

[0053] In this way, the vehicle control system SYS can estimate the emotion of the subject U2 who is the driver of the other vehicle V2, and can provide a notification or the like that takes the estimated emotion into consideration. This can prevent undesirable situations that may be caused by negative emotions, and is expected to contribute to safe driving of the host vehicle V1, and therefore safe driving of the other vehicle V2.

[0054] 5 shows an internal block diagram of the sensor unit 60. The sensor unit 60 includes sensors 61 to 66. The sensors 61, 62, 63, 64, 65, and 66 are a vehicle speed sensor, a steering angle sensor, a G sensor, a distance measurement sensor, an image sensor, and a GPS sensor, respectively. Each vehicle described in this embodiment is provided with operation parts that receive driving operations from the driver, and the operation parts include an accelerator pedal, a brake pedal, and a steering wheel. The sensor unit 60 is also provided with a group of sensors (not shown) that detects the operation contents of the accelerator pedal, brake pedal, and steering wheel of the host vehicle V1 by the driver of the host vehicle V1.

[0055] The functions of the sensors provided in the sensor unit 60 of the host vehicle V1 will be described. The vehicle speed sensor 61 detects the speed of the host vehicle V1 and outputs vehicle speed information (vehicle speed pulse) representing the detected speed. The steering angle sensor 62 detects the steering angle (steering angle) of the host vehicle V1 and outputs steering angle information representing the detected steering angle. The G sensor 63 detects acceleration in a predetermined axial direction applied to the host vehicle V1 and generates and outputs the acceleration detection result as acceleration information. The G sensor 63 may detect acceleration in two mutually orthogonal axial directions, or may detect acceleration in three mutually orthogonal axial directions.

[0056] The distance measurement sensor 64 generates and outputs distance measurement information by performing distance measurement. In the distance measurement by the distance measurement sensor 64, the distance between the vehicle V1 and a solid object (three-dimensional object) located within a distance measurement area around the vehicle V1 is detected, and the orientation of the solid object as viewed from the vehicle V1 is also detected. These detection results are included in the distance measurement information. The distance measurement area is an area set based on the position of the vehicle V1. It is assumed that another vehicle V2 exists within the distance measurement area. Therefore, the distance measurement by the distance measurement sensor 64 detects the inter-vehicle distance d12 and also detects the orientation of the other vehicle V2 as viewed from the vehicle V1. The distance measurement sensor 64 may be configured with a LIDAR (Light Detection and Ranging) that performs distance measurement using light, or may be configured with a radar that performs distance measurement using radio waves. The distance measurement sensor 64 may be configured with a combination of LIDAR and radar. The distance measurement sensor 64 may be configured with one or more distance measurement sensors.

[0057] The image sensor 65 is a camera installed on the vehicle V1. The image sensor 65 generates camera information by photographing a predetermined photographing area. The image sensor 65 repeatedly photographs at a predetermined frame rate. The camera image is a photographed image of the photographing area. Image information indicating the camera image is called camera image information. The image sensor 65 may include a front camera that photographs the area in front of the vehicle V1, a rear camera that photographs the area behind the vehicle V1, a right camera that photographs the area to the right of the vehicle V1, and a left camera that photographs the area to the left of the vehicle V1. Only a part of the front camera, rear camera, right camera, and left camera may be included in the image sensor 65. Alternatively, the image sensor 65 may include a wide-angle camera that photographs the areas in front, to the side, and behind the vehicle V1. In addition to this, the method of configuring the cameras in the image sensor 65 is arbitrary. In any case, it is assumed that another vehicle V2 exists in the above-mentioned photographing area of ​​the image sensor 65. For this reason, the camera image includes an image of the other vehicle V2.

[0058] The GPS sensor 66 receives signals from a plurality of GPS satellites that form a GPS (Global Positioning System) and generates vehicle position information based on the reception results. The vehicle position information generated by the GPS sensor 66 represents the current location (current position) of the vehicle V1 by longitude and latitude, or represents the current location of the vehicle V1 by longitude, latitude and altitude.

[0059] Hereinafter, information obtained by the sensor unit 60 of the host vehicle V1 will be referred to as sensor output information Sout1. The sensor output information Sout1 includes vehicle speed information, steering angle information, acceleration information, distance measurement information, camera image information, and vehicle position information obtained by the sensor unit 60 of the host vehicle V1. The controller 33 can perform driving control of the host vehicle V1 (e.g., control of the speed and traveling direction of the host vehicle V1) through drive control of the actuator unit 40 based on the sensor output information Sout1.

[0060] The controller 23 can perform a tracking process of the other vehicle V2 by image recognition based on the camera image information. The tracking process will be described. It is assumed that an image of the other vehicle V2 is included in the camera image at any first and second time. The second time is any time after the first time. During the shooting period including the period from the first time to the second time, the other vehicle V2 continues to exist as a vehicle ahead of the host vehicle V1 within the shooting area of ​​the image sensor 65. The controller 23 detects the other vehicle V2 within the camera image based on the image information of the camera image at the first time. By repeatedly shooting at a predetermined frame rate between the first and second times, a camera image as a moving image is obtained. In the tracking process, the controller 23 tracks the other vehicle V2 within the moving image based on the camera image information between the first and second times (i.e., continues to detect the position of the other vehicle V2). The controller 23 detects the relative positional relationship between the host vehicle V1 and the other vehicle V2 based on the result of the tracking process and the distance measurement information. Although the relative positional relationship between the host vehicle V1 and the other vehicle V2 may change from moment to moment, the controller 23 can generate time series data of the relative positional relationship between the host vehicle V1 and the other vehicle V2 for any period of time. The process of detecting the relative positional relationship at each time and generating the time series data of the relative positional relationship is called the relative position detection process. The relative position detection process includes the above-mentioned tracking process. In this embodiment, the reference data generation unit 232 performs the relative position detection process.

[0061] Fig. 6 is a flowchart showing the emotion estimation process executed by the controller 23. The emotion estimation process may be a process realized by the controller 23 executing the above-mentioned program 224p. The emotion estimation process shown in Fig. 6 is activated at the timing when various controls based on the estimated emotion are started after the start of the host vehicle V1, and is thereafter executed once or repeatedly executed multiple times at any timing when emotion estimation is required. The emotion estimation process includes steps S21 to S28.

[0062] In the emotion estimation process, first, in step S21, a data acquisition process is performed. In the data acquisition process, the acquisition unit 231 acquires sensor output information Sout1 of the host vehicle V1 during an estimation evaluation period. The sensor output information Sout1 acquired by the acquisition unit 231 is supplied to the reference data generation unit 232. The estimation evaluation period is an arbitrary period having a predetermined time length (for example, several seconds to several tens of seconds), and in the emotion estimation process, the emotion of the subject U2 during the estimation evaluation period is estimated. After step S21, the process proceeds to step S22.

[0063] In step S22, the reference data generating unit 232 generates driving state information of the host vehicle V1 during the evaluation period for estimation based on the sensor output information Sout1. The driving state information of the host vehicle V1 includes speed information, steering angle information, and acceleration information of the host vehicle V1 at each time during the evaluation period for estimation. The speed information, steering angle information, and acceleration information of the host vehicle V1 respectively represent the speed, steering angle, and acceleration of the host vehicle V1. The driving state information of the host vehicle V1 includes information on the driving trajectory and the inter-vehicle distance d12 of the host vehicle V1 during the evaluation period for estimation. The driving trajectory information of the host vehicle V1 represents the driving trajectory of the host vehicle V1. The inter-vehicle distance d12 for the host vehicle V1 corresponds to the forward inter-vehicle distance. The reference data generating unit 232 can detect the driving trajectory of the host vehicle V1 based on the speed (vehicle speed pulse) and steering angle of the host vehicle V1 at each time during the evaluation period for estimation. Alternatively, the reference data generating unit 232 can detect the travel path of the host vehicle V1 during the estimation evaluation period based on the vehicle position information during the estimation evaluation period. The travel path of the host vehicle V1 identifies the degree of wobbling during the travel of the host vehicle V1. After step S22, the process proceeds to step S23.

[0064] In step S23, the reference data generating unit 232 generates time series data of the relative position relationship between the host vehicle V1 and the other vehicle V2 during the estimation evaluation period through the above-mentioned relative position detection process. Then, using the generated time series data, the reference data generating unit 232 converts the driving state information of the host vehicle V1 into driving state information of the other vehicle V2. The driving state information of the other vehicle V2 obtained by the conversion includes the speed information, steering angle information, and acceleration information of the other vehicle V2 at each time during the estimation evaluation period. The speed information, steering angle information, and acceleration information of the other vehicle V2 respectively represent the speed, steering angle, and acceleration of the other vehicle V2. The driving state information of the other vehicle V2 obtained by the conversion includes the driving trajectory information and the information of the inter-vehicle distance d12 of the other vehicle V2 during the estimation evaluation period. The driving trajectory information of the other vehicle V2 represents the driving trajectory of the other vehicle V2. The degree of wobbling during the driving of the other vehicle V2 is specified based on the driving trajectory of the other vehicle V2. The inter-vehicle distance d12 for the other vehicle V2 corresponds to the rear inter-vehicle distance.

[0065] If the relative positional relationship between the host vehicle V1 and the other vehicle V2 is unchanged during the estimation evaluation period, the speed, steering angle, acceleration, running path, and inter-vehicle distance d12 of the other vehicle V2 will be the same as the speed, steering angle, acceleration, running path, and inter-vehicle distance d12 of the host vehicle V1. If the inter-vehicle distance d12 increases by a constant amount during the estimation evaluation period, the speed of the other vehicle V2 will be greater than the speed of the host vehicle V1. If the relative position of the other vehicle V2 as seen from the host vehicle V1 during the estimation evaluation period fluctuates left and right when the host vehicle V1 is moving straight with a zero steering angle, the running path of the other vehicle V2 will also fluctuate left and right (i.e., the degree of wobble of the other vehicle V2 is large). In this way, the absolute behavior of the host vehicle V1 is converted into the absolute behavior of the other vehicle V2 based on the time series data of the relative positional relationship between the host vehicle V1 and the other vehicle V2, and the running state information of the other vehicle V2 is obtained.

[0066] The driving state information of the other vehicle V2 may include only an arbitrary part of the information of the other vehicle V2, including the speed information, steering angle information, acceleration information, driving trajectory information, and inter-vehicle distance d12. After step S23, the process proceeds to step S24.

[0067] In step S24, the reference data generating unit 232 supplies the reference data D2 including the driving state information of the other vehicle V2 to the input data generating unit 233. The reference data generating unit 232 may include other additional information in addition to the driving state information of the other vehicle V2 in the reference data D2. Examples of other additional information include the weather and temperature at the current location of the vehicle V1, the current time, the brightness around the vehicle V1, and the type of road on which the vehicle V1 is driving (whether it is an expressway or an ordinary road, etc.). After step S24, the process proceeds to step S25.

[0068] In step S25, the input data generation unit 233 generates AI input data by performing data processing on the reference data D2 in the estimation evaluation period. In the feeling estimation device 20, the AI ​​input data generated in step S25 functions as estimation input data. In the feeling estimation device 20, the AI ​​input data generated here is input to the AI ​​models 221m and 222m as estimation input data. After step S25, the process proceeds to step S26.

[0069] In the data processing, the reference data D2 is processed into a data format to be input to the AI ​​models 221m and 222m, and the processed data is obtained as AI input data. The data processing is set according to the input specifications of the AI ​​models 221m and 222m. The data processing may include, for example, various signal processes (averaging, peak detection, frequency analysis, high-frequency component extraction, low-frequency component extraction, etc.) for each piece of information in the reference data D2.

[0070] More specifically, for example, a statistical value of the speed of the other vehicle V2 during the evaluation period for estimation is included in the AI ​​input data. Also, for example, a statistical value of the steering angle of the other vehicle V2 during the evaluation period for estimation is included in the AI ​​input data. Also, for example, a statistical value of the acceleration of the other vehicle V2 during the evaluation period for estimation is included in the AI ​​input data. Also, for example, a statistical value of the inter-vehicle distance d12 during the evaluation period for estimation is included in the AI ​​input data. The statistical values ​​include, for example, an average value, a maximum value, a minimum value, a standard deviation, and a variance. Also, for example, in the data processing process, a fluctuation feature value indicating the fluctuation state of the other vehicle V2 is extracted from the driving trajectory of the other vehicle V2 during the evaluation period for estimation and is included in the AI ​​input data.

[0071] In step S26, emotion index values ​​are derived in the AI ​​models 221m and 222m based on the input data for estimation. That is, the emotion estimation unit 234 inputs the input data for estimation to the AI ​​model 221m, and thus an awakening level is derived in the AI ​​model 221m. The emotion estimation unit 234 inputs the input data for estimation to the AI ​​model 222m, and thus an activity level is derived (estimated) in the AI ​​model 222m. The awakening level and activity level derived here are the awakening level and activity level of the subject U2 in the estimation evaluation period. After step S26, the process proceeds to step S27.

[0072] In step S27, the feeling estimation unit 234 estimates the feeling of the subject U2 by inputting the arousal level and activity level derived in step S26 into the feeling estimation model 223m, and generates estimated feeling information representing the estimation result. After step S27, the process proceeds to step S28.

[0073] In step S28, the inference result output unit 235 outputs the inferred emotion information of the subject U2 obtained in step S27 to the vehicle control device 30. With the processing of step S28, one cycle of the emotion inference processing ends.

[0074] Incidentally, the emotion estimated in one emotion estimation process may be affected by noise, etc. For this reason, the estimation result output unit 235 may perform statistical processing on multiple pieces of estimated emotion information obtained by executing the emotion estimation process multiple times, and output the information obtained thereby to the vehicle control device 30.

[0075] The first embodiment includes the following Examples EX1_1 to EX1_6. The matters described above in the first embodiment are applied to the following Examples EX1_1 to EX1_6 unless otherwise specified and unless there is a contradiction. However, in each example, for matters that contradict the matters described above in the first embodiment, the description in each example may take precedence. Furthermore, unless there is a contradiction, matters described in any of Examples EX1_1 to EX1_6 can also be applied to any other example (i.e., any two or more of the multiple examples can be combined).

[0076] [Example EX1_1] Example EX1_1 will be described. In the example EX1_1, a learning device for obtaining the AI ​​models 221m and 222m mounted on the feeling estimation device 20 will be described.

[0077] FIG. 7 shows a schematic configuration of the learning device 10 according to the embodiment EX1_1. The learning device 10 includes an index value derivation unit 134, a reference data generation unit 132, an input data generation unit 133, a first AI model 121m, and a second AI model 122m. The learning device 10 learns the AI ​​models 121m and 122m. The trained AI models 121m and 122m are mounted on the emotion estimation device 20 as AI models 221m and 222m, respectively (see FIGS. 1 and 4). The learning shown in the embodiment EX1_1 and any embodiment described later is supervised machine learning, and therefore, a learning data set including teacher data (correct answer data) is required for learning.

[0078] In FIG. 7, person U3 represents a subject involved in the generation of the learning dataset. The subject U3 gets in and drives the vehicle V3. That is, the subject U3 is the driver of the vehicle V3. Hereinafter, the person U3 will be referred to as the subject U3, and the vehicle V3 will be specifically referred to as the test vehicle V3. The learning dataset is generated by the subject U3 driving the test vehicle V3. The learning dataset is generated by acquiring biometric information of the subject U3. The test vehicle V3 is assumed to be an automobile or the like that runs on a road surface, similar to the vehicles V1 and V2.

[0079] When forming an individual-specific emotion estimation device 20 in which the emotion estimator is limited, the subject U3 is the same person as the target person U2. In this case, the test vehicle V3 may be the same vehicle as the other vehicle V2 in FIG. 1. However, since there is similarity even between different people, the subject U3 may be a person different from the target person U2, and the test vehicle V3 may be a vehicle different from the other vehicle V2 in FIG. 1. When forming a general-purpose emotion estimation device 20 in which the person whose emotion is estimated is not limited, the subject U3 and the target person U2 may be the same person. If multiple subjects U3 are prepared, a learning data set of people with various characteristics can be obtained, and it is expected that a highly versatile emotion estimation device 20 can be obtained.

[0080] FIG. 7 shows vehicle V4 in addition to the test vehicle V3. The test vehicle V3 and vehicle V4 travel in the same direction. The travel direction of the test vehicle V3 and vehicle V4 corresponds to the forward direction. From the perspective of the test vehicle V3, the vehicle V4 is located behind. In other words, the vehicle V4 is the rear vehicle of the test vehicle V3. The vehicle V4 is a vehicle located immediately behind the test vehicle V3 (immediately rear vehicle), and no other vehicles exist between the test vehicle V3 and vehicle V4. The distance d34 is the inter-vehicle distance between the test vehicle V3 and vehicle V4. From the perspective of the test vehicle V3, the distance d34 corresponds to the rear inter-vehicle distance, and from the perspective of vehicle V4, the distance d34 corresponds to the front inter-vehicle distance. The inter-vehicle distance d34 corresponds to the distance between the rear end of the body of the test vehicle V3 and the front end of the body of the vehicle V4.

[0081] Since the subject U3 corresponds to a person simulating the target person U2, the subject vehicle V3 should be a vehicle in front of any vehicle. However, it is not essential that the subject vehicle V3 is always a vehicle in front of any vehicle. This is because the relationship between the behavior of a vehicle and the bio-information (brain waves and heart rate) of the driver of the vehicle is considered to be universal. Figure 7 illustrates the timing when the subject vehicle V3 becomes a vehicle in front of the vehicle V4.

[0082] A plurality of subjects U3 may be prepared, and each subject U3 may be a driver of a test vehicle V3 and perform various patterns of experimental driving, and at this time, a test vehicle V3 may be assigned to each subject U3. This makes it easier to obtain a large amount of learning data set in a short time. The plurality of subjects U3 may or may not include the target person U2. The plurality of test vehicles V3 assigned to the plurality of subjects U3 may or may not include the other vehicle V2 in FIG. 1. In the following, for the sake of concreteness and clarification of the explanation, unless otherwise necessary, attention will be paid to one subject U3 and one test vehicle V3, and the vehicle V4 will be the rear vehicle of the test vehicle V3 in question.

[0083] A biosensor is attached to the subject U3 to acquire bioinformation of the subject U3. Here, it is assumed that an electroencephalogram sensor ES and a heartbeat sensor HS are attached to the subject U3 as the biosensors. The electroencephalogram sensor ES detects the electroencephalograms of the subject U3, and outputs electroencephalogram data indicating the results of the electroencephalogram detection. The electroencephalogram data includes information on the detected electroencephalograms. The heartbeat sensor HS detects the heartbeat of the subject U3, and outputs heartbeat data indicating the results of the heartbeat detection. The heartbeat data includes information on the detected heartbeat. For example, a headgear-type electroencephalogram sensor is used as the electroencephalogram sensor ES. For example, a chest belt-type electrocardiogram-type heartbeat sensor is used as the heartbeat sensor HS.

[0084] Data obtained from the biosensor attached to the subject U3 is called biodata. Therefore, the biodata includes brainwave data and heart rate data, and represents bioinformation of the subject U3. Depending on the type of bioinformation to be acquired or the wearing comfort, other sensors may be included in the biosensor and attached to the subject U3. Examples of other sensors include a blood pressure monitor, a NIRS (Near Infrared Spectroscopy) device, or an optical heart rate sensor.

[0085] The biological data of the subject U3 is input to the index value derivation section 134. The index value derivation section 134 derives and outputs the arousal level and activity level of the subject U3 as emotion index values ​​of the subject U3 based on the biological data of the subject U3.

[0086] The index value derivation unit 134 derives the degree of awakening of the subject U3 based on the electroencephalogram data of the subject U3. The degree of awakening is derived based on the beta / alpha waves of the electroencephalogram. That is, a value according to the beta / alpha waves of the electroencephalogram (for example, a value proportional to the beta / alpha waves of the electroencephalogram) is derived as the degree of awakening. The beta / alpha waves of the electroencephalogram are the ratio of beta waves of the electroencephalogram to alpha waves of the electroencephalogram. More specifically, for example, the beta / alpha waves of the electroencephalogram are "INT_β / INT_α" or "PW_β / PW_α". INT_β and INT_α represent the intensities of the beta and alpha waves of the electroencephalogram, respectively. PW_β and PW_α represent the powers of the beta and alpha waves of the electroencephalogram, respectively.

[0087] The index value derivation unit 134 derives the activity of the subject U3 based on the heartbeat data of the subject U3. More specifically, the activity is calculated based on the LF component of the time interval between heartbeats (hereinafter referred to as the heartbeat interval). The heartbeat sensor HS may detect the action potential of the heart of the subject U3, which is an electric potential generated on the skin surface of the subject U3, and generate a cardiac potential signal indicating the detected action potential. The heartbeat sensor HS detects the heartbeat interval, which is the time interval between the heartbeats of the subject U3, based on the cardiac potential signal. The heartbeat interval corresponds to the RR interval of the cardiac potential signal (however, it may be the interval between two adjacent Q waves in the cardiac potential signal, etc.). The heartbeat data from the heartbeat sensor HS includes a signal indicating the heartbeat interval (may further include a cardiac potential signal). It is not essential that the cardiac potential signal is generated in the heartbeat sensor HS. The index value derivation unit 134 derives the LF component of the heartbeat interval based on the heartbeat data. The LF components of the heartbeat interval are frequency components in a predetermined low frequency band (for example, a frequency band from 0.04 to 0.15 Hz) among the frequency components included in the heartbeat interval. For the subject U3, the index value derivation unit 134 derives the activity level of the subject U3, for example, by calculating the standard deviation in the amplitude of the LF components of the heartbeat interval. At this time, a value according to the standard deviation (for example, a value proportional to the standard deviation) is derived as the activity level. In the amplitude spectrum of the heartbeat interval, the frequency components in the low frequency band of the heartbeat interval consist of 1st to kth frequency components (k is an integer equal to or greater than 2), and the standard deviation corresponds to the standard deviation in the amplitude of the 1st to kth frequency components.

[0088] The test vehicle V3 is equipped with a sensor unit equivalent to the sensor unit 60 equipped in the vehicle V1. Therefore, the sensor unit of the test vehicle V3 generates sensor output information Sout3 equivalent to the sensor output information Sout1 obtained by the sensor unit 60 of the vehicle V1. The sensor output information Sout3 includes the vehicle speed information, steering angle information, acceleration information, distance measurement information, camera image information, and vehicle position information of the test vehicle V3. The matters described above regarding the vehicle speed information, steering angle information, acceleration information, distance measurement information, camera image information, and vehicle position information also apply to the sensor unit and sensor output information Sout3 of the test vehicle V3. However, in this application, the host vehicle V1 is read as the test vehicle V3.

[0089] That is, for example, in the sensor section of the test vehicle V3, the vehicle speed sensor 61 detects the speed of the test vehicle V3 and outputs vehicle speed information (vehicle speed pulse) representing the detected speed. Similarly, for example, in the sensor section of the test vehicle V3, the distance measurement sensor 64 generates and outputs distance measurement information by performing distance measurement. In the distance measurement sensor 64 of the sensor section of the test vehicle V3, the distance between the test vehicle V3 and a three-dimensional object located in the distance measurement area around the test vehicle V3 is detected, and the direction of the three-dimensional object as viewed from the test vehicle V3 is also detected. These detection results are included in the distance measurement information. The distance measurement area related to the test vehicle V3 is an area that is set based on the position of the test vehicle V3. It is assumed that the vehicle V4 exists in the distance measurement area related to the test vehicle V3. Therefore, the distance measurement by the distance measurement sensor 64 of the sensor section of the test vehicle V3 detects the inter-vehicle distance d34 and also detects the direction of the vehicle V4 as viewed from the test vehicle V3. In the sensor section of the test vehicle V3, a vehicle V4 is present in the photographing area of ​​the image sensor 65, and the camera image obtained by the image sensor 65 includes an image of the vehicle V4.

[0090] In the learning device 10, the sensor output information Sout3 is input to the reference data generation unit 132. The reference data generation unit 132 generates reference data D3 including driving state information of the test vehicle V3 based on the sensor output information Sout3. The same type of data as the reference data D2 (see FIG. 1) is included in the reference data D3. Therefore, if the reference data D2 includes information on the speed information, steering angle information, acceleration information, driving trajectory information, and following distance d12 of the other vehicle V2, the reference data D3 includes information on the speed information, steering angle information, acceleration information, driving trajectory information, and following distance d34 of the test vehicle V3.

[0091] The input data generating unit 133 converts the reference data D3 into learning input data suitable for the structure of the AI ​​models 121m and 122m. In the learning device 10, the AI ​​model 121m is trained so that the awakening level of the subject U3 can be estimated (inferred) based on the learning input data. That is, the learning device 10 supplies the awakening level output from the index value derivation unit 134 to the AI ​​model 121m as correct answer data, and trains the AI ​​model 121m using a learning algorithm such as backpropagation based on the learning input data. Similarly, in the learning device 10, the AI ​​model 122m is trained so that the activity level of the subject U3 can be estimated (inferred) based on the learning input data. That is, the learning device 10 supplies the activity level output from the index value derivation unit 134 to the AI ​​model 122m as correct answer data, and trains the AI ​​model 122m using a learning algorithm such as backpropagation based on the learning input data.

[0092] Fig. 8 is a configuration block diagram of a learning device 10 according to an embodiment EX1_1. The learning device 10 may be mounted on a test vehicle V3, but here, it is considered that the learning device 10 is provided outside the test vehicle V3. The learning device 10 may be formed by any one or more computer devices. The learning device 10 may be formed by using cloud computing.

[0093] A database DB is connected to the learning device 10 by wire or wirelessly. The database DB may be composed of multiple physically separated recording devices. The database DB stores a large number of pairs of biometric data of a subject U3 and sensor output information Sout3 of a test vehicle V3. The biometric data and sensor output information Sout3 in one pair are data generated simultaneously.

[0094] Consider that there are 1st to Mth learning evaluation periods as M periods different from each other. In this case, the 1st to Mth sets of data in the 1st to Mth learning evaluation periods are stored in the database DB. The ith set of data includes biological data of the subject U3 in the ith learning evaluation period and sensor output information Sout3 in the ith learning evaluation period. The sensor output information Sout3 in the ith learning evaluation period includes sensor output information Sout3 output from the sensor unit of the test vehicle V3 in the ith learning evaluation period. Here, M represents an integer value sufficiently larger than 1, for example, ranging from tens of thousands to hundreds of thousands. i represents an arbitrary integer. The subject U3 becomes the driver of the test vehicle V3 and performs various patterns of experimental driving to obtain the 1st to Mth sets of data. The 1st to Mth sets of data are supplied to the learning device 10.

[0095] In addition, since the emotion estimation process (see FIG. 6) is performed using the trained AI models 121m and 122m as the AI ​​models 221m and 222m of the emotion estimation device 20, it is preferable that the length of each learning evaluation period is set to be the same as the length of each estimation evaluation period.

[0096] The learning device 10 includes a communication unit 11, a storage unit 12, and a controller 13. The communication unit 11 transmits and receives any signal between the learning device 10 and a counterpart device different from the learning device 10. The counterpart device for the communication unit 11 includes a database DB. The storage unit 12 includes a non-volatile memory such as a ROM or a flash memory, and a volatile memory such as a RAM. The storage unit 12 includes a first AI model storage unit 121 that stores a first AI model 121m, and a second AI model storage unit 122 that stores a second AI model 122m. The storage unit 12 further includes a program storage unit 123 that stores various programs. The programs stored in the program storage unit 123 include a program 123p.

[0097] The controller 13 comprehensively controls the operation of each part in the learning device 10. The controller 13 includes a calculation processing unit including a CPU, a GPU, etc. as hardware resources. The controller 13 has function blocks 131 to 135. The function blocks 131, 132, 133, 134, and 135 are an acquisition unit, a reference data generation unit, an input data generation unit, an index value derivation unit, and a learning processing unit, respectively.

[0098] The controller 13 is a program execution device (computer) capable of executing any program. The controller 13 executes the program 123p to realize each function of the controller 13 (including each function of the functional blocks 131 to 135). All of the operations of the controller 13 described in this embodiment may be operations that are realized by the controller 13 executing the program 123p. The program 123p may be composed of a plurality of programs. The controller 13 can transmit and receive any information to and from a counterpart device (a counterpart device for the communication unit 11) using the communication unit 11, but in the following, the description of the communication unit 11 may be omitted.

[0099] Each functional block of the controller 13 will be described. The acquisition unit 131 sequentially acquires group data from the database DB. The reference data generation unit 132 generates reference data D3 for each group data based on the sensor output information Sout3 acquired by the acquisition unit 131. The reference data D3 includes running state information of the test vehicle V3. The input data generation unit 133 generates learning input data for each group data based on the reference data D3 (see also FIG. 7). The index value derivation unit 134 derives and outputs the emotion index value (here, arousal level and activity level) of the subject U3 based on the biological data of the subject U3 for each group data.

[0100] A learning data set is formed by the emotion index values ​​(here, arousal level and activity level) of the subject U3 and the learning input data. The learning processing unit 135 uses the learning data set to execute learning of the AI ​​models 121m and 122m.

[0101] The learning data set for the AI ​​model 121m has arousal level as the correct answer data. In learning, the AI ​​model 121m estimates (infers) the arousal level of the subject U3 based on the learning input data. The learning processing unit 135 updates the parameters of the AI ​​model 121m (such as the weights of the neural network) based on the error between the estimated arousal level and the arousal level as the correct answer data. When a termination condition is satisfied, such as the error being sufficiently low, the learning of the AI ​​model 121m ends. The AI ​​model 121m at the time of completing the learning is applied to the emotion estimation device 20 (see FIG. 1 and FIG. 4) as the A1 model 221m.

[0102] In contrast, the learning data set for the AI ​​model 122m has the activity level as the correct answer data. In learning, the AI ​​model 122m estimates (infers) the activity level of the subject U3 based on the learning input data. The learning processing unit 135 updates the parameters (weights of the neural network, etc.) of the AI ​​model 122m based on the error between the estimated activity level and the activity level as the correct answer data. When a termination condition, such as the error being sufficiently low, is satisfied, the learning of the AI ​​model 122m ends. The AI ​​model 122m at the time when learning ends is applied to the emotion estimation device 20 (see Figs. 1 and 4) as the A1 model 222m.

[0103] The input data generation unit 133 generates AI input data by performing data processing on the reference data D3 in the pair data. The AI-generated data generated by the input data generation unit 133 functions as learning input data. In the data processing in the input data generation unit 133, the reference data D3 is processed into a format of data to be input to the AI ​​models 121m and 122m, and the processed data is obtained as AI input data. The data processing in the input data generation unit 133 is set according to the input specifications of the AI ​​models 121m and 122m. The AI ​​models 121m and 122m trained using the learning input data are used as the AI ​​models 221m and 222m that estimate emotion index values ​​from the estimation input data in the emotion estimation device 20 (see FIG. 4). Therefore, the data processing in the input data generation unit 133 is the same as the data processing in the input data generation unit 233 (see FIG. 4 and FIG. 6). Therefore, the AI ​​input data (input data for learning) generated by the input data generation unit 133 and the AI ​​input data (input data for estimation) generated by the input data generation unit 233 are the same type of data.

[0104] Fig. 9 shows an operation flowchart of the learning process executed by the learning device 10. The learning process may be a process realized by the controller 13 executing the above-mentioned program 123p. The learning process shown in Fig. 9 is started when the program 123p is started in the learning device 10 after the necessary amount of group data is obtained. The accumulation of group data in the database DB and the learning process shown in Fig. 9 may be performed in parallel. The learning process includes steps S11 to S16.

[0105] First, in step S11, the acquisition unit 131 acquires group data from the database DB. As described above, the group data has biological data and sensor output information Sout3 (see FIG. 8). Learning is performed in mini-batch units, and group data having a data amount equivalent to the mini-batch is acquired in step S11. After step S11, the process proceeds to step S12.

[0106] In step S12, the reference data generating unit 132 generates reference data D3 for each group data based on the sensor output information Sout3 in the group data acquired by the acquiring unit 231. The reference data D3 includes driving state information of the test vehicle V3. For the i-th group data, the driving state information of the test vehicle V3 includes the speed information, steering angle information, and acceleration information of the test vehicle V3 at each time in the i-th learning evaluation period. The speed information, steering angle information, and acceleration information of the test vehicle V3 respectively represent the speed, steering angle, and acceleration of the test vehicle V3. For the i-th group data, the driving state information of the test vehicle V3 includes the driving trajectory information and the vehicle distance d34 information of the test vehicle V3 in the i-th learning evaluation period. The driving trajectory information of the test vehicle V3 represents the driving trajectory of the test vehicle V3. The degree of wobbling during the driving of the test vehicle V3 is identified based on the driving trajectory of the test vehicle V3.

[0107] In step S13, the index value derivation unit 134 derives the emotion index values ​​(arousal and activity) of the subject U3 for each group data based on the biological data of the subject U3. Also in step S13, the input data generation unit 133 generates AI input data for each group data through data processing based on the reference data D3. The AI-generated data generated by the input data generation unit 133 functions as learning input data. After step S13, the processes of steps S14 and S15 are executed. The order of execution of the processes of steps S14 and S15 is arbitrary. The processes of steps S14 and S15 may be executed at the same time.

[0108] In step S14, the learning processing unit 135 trains the first AI model 121m using a learning data set having learning input data and alertness as correct answer data. In training the AI ​​model 121m, the AI ​​model 121m estimates (infers) the alertness of the subject U3 based on the learning input data. The learning processing unit 135 updates the parameters (such as neural network weights) of the AI ​​model 121m based on the error between the estimated alertness and the alertness as correct answer data.

[0109] In step S15, the learning processing unit 135 trains the second AI model 122m using a learning data set having learning input data and activity levels as correct answer data. In training the AI ​​model 122m, the AI ​​model 122m estimates (infers) the activity level of the subject U3 based on the learning input data. The learning processing unit 135 updates the parameters (such as neural network weights) of the AI ​​model 122m based on the error between the estimated activity level and the activity level as correct answer data.

[0110] After steps S14 and S15, in step S16, the learning processing unit 135 determines whether a determined termination condition is satisfied. For example, the termination condition is satisfied when the error in each learning of steps S14 and S15 becomes equal to or less than a predetermined value. Alternatively, the termination condition is satisfied when the number of mini-batch unit learnings reaches a certain number. If the termination condition is satisfied (Y in step S16), the learning process of FIG. 9 ends. If the termination condition is not satisfied (N in step S16), the process returns to step S11. If the process returns to step S11, new group data is acquired from the database DB, and the processes from step S12 onwards are repeated.

[0111] Note that, prior to storing the pair data in database DB, the sensor output information Sout3 may be supplied to an arbitrary computer device (not shown) having the function of reference data generation unit 132 to generate reference data D3. Then, the reference data D3 generated by the computer device may be accumulated in database DB instead of the sensor output information Sout3. In this case, the pair data stored in database DB will include reference data D3 instead of the sensor output information Sout3. In this case, reference data generation unit 132 is not necessary in learning device 10, and it is sufficient to supply reference data D3 in the pair data to input data generation unit 133.

[0112] As described above, the feeling estimation device 20 (controller 23) generates driving state information of the other vehicle V2 based on the driving state information of the host vehicle V1 and the relative positional relationship between the host vehicle V1 and the other vehicle V2 (steps S21 to S23 in FIG. 6). Then, the feeling estimation device 20 estimates the feeling of the subject U2, who is the driver of the other vehicle V2, based on the driving state information of the other vehicle V2 (based on the reference data D2 including the driving state information of the other vehicle V2) (steps S24 to S27 in FIG. 6). The feeling estimation device 20 makes it possible to grasp the feeling estimation result of the driver of the surrounding vehicle (here, the other vehicle V2) of the host vehicle V1 at the host vehicle V1. In addition, since the data used for feeling estimation is the driving state information of the host vehicle V1 and the data of the relative positional relationship between the host vehicle V1 and the other vehicle V2, it is not necessary for the driver of the surrounding vehicle to wear a biosensor, and therefore it is not necessary to transmit the detection result by the biosensor to the host vehicle V1. Therefore, the feeling estimation method by the feeling estimation device 20 is highly feasible. If the feelings of the drivers of the surrounding vehicles can be understood, driving can be performed taking the feelings into consideration, leading to safe driving. For example, it is possible to prevent possible trouble with surrounding vehicles by using the above-mentioned notification control, which is beneficial.

[0113] The feeling estimation device 20 (controller 23) generates time series data of the relative positional relationship between the host vehicle V1 and the other vehicle V2 using a sensor installed in the host vehicle V1 and observing the other vehicle V2. Then, the feeling estimation device 20 converts the driving state information of the host vehicle V1 into driving state information of the other vehicle V2 using the time series data (step S23 in FIG. 6). With this method, the driving state information of the other vehicle V2 can be generated using the above-mentioned sensor observing the other vehicle V2 and a sensor installed in the host vehicle V1 to obtain the driving state information of the host vehicle V1. In other words, the driving state information of the other vehicle V2 can be generated in the host vehicle V1 without obtaining information via communication from a sensor installed in the other vehicle V2, which is highly practical.

[0114] In this embodiment, the distance measurement sensor 64 and the image sensor 65 are used as sensors for observing the other vehicle V2. It is possible to accurately identify the relative positional relationship between the host vehicle V1 and the other vehicle V2 by using both the distance measurement sensor 64 and the image sensor 65. However, the relative positional relationship (including the inter-vehicle distance d12) between the host vehicle V1 and the other vehicle V2 may be identified by only one of the distance measurement sensor 64 and the image sensor 65.

[0115] In the distance measurement by the distance measuring sensor 64, the distance between the vehicle V1 and a three-dimensional object located within the distance measurement area around the vehicle V1 is detected, and the orientation of the three-dimensional object as viewed from the vehicle V1 is also detected. Since these detection results are included in the distance measurement information, it is possible to specify the relative positional relationship between the vehicle V1 and the other vehicle V2 corresponding to the above-mentioned three-dimensional object only from the distance measurement information. It is also possible to specify the relative positional relationship between the vehicle V1 and the other vehicle V2 from the position and size of the other vehicle V2 on the camera image obtained by the image sensor 65, without relying on the distance measurement information.

[0116] The emotion estimation device 20 has an estimator that estimates the emotion of a driver of an arbitrary vehicle based on driving state information of the vehicle. The emotion estimation device 20 estimates the emotion of the driver of the other vehicle V2 (subject U2) by inputting driving state information of the other vehicle V2 to the estimator. Since there is a correlation between driving state information and the driver's emotion in an arbitrary vehicle, this method can satisfactorily estimate the emotion of the driver of the other vehicle V2 (subject U2).

[0117] The above estimator is composed of models 221m, 222m, and 223m in the emotion estimation device 20 in Fig. 1. The input data generation unit 233 may also be considered to be included in the components of the estimator.

[0118] The estimator estimates the emotion of the driver of the other vehicle V2 (subject U2) by deriving an emotion index value related to the emotion of the driver of the other vehicle V2 based on the driving state information of the other vehicle V2 (steps S24 to S27 in FIG. 6). In this case, as described above, the emotion index value of the driver of the other vehicle V2 preferably includes the arousal level of the central nervous system and the activity level of the autonomic nervous system of the driver of the other vehicle V2. Since a person's emotion is correlated with the arousal level and the activity level, the estimator can perform good emotion estimation.

[0119] The vehicle control system SYS in FIG. 4 includes an in-vehicle emotion estimation system. The in-vehicle emotion estimation system includes an emotion estimation device 20 and a sensor unit 60. The emotion estimation device 20 generates the traveling state information of the other vehicle V2 based on the traveling state information of the host vehicle V1 based on the output information (Sout1) of the sensor unit 60 and the relative positional relationship between the host vehicle V1 and the other vehicle V2 (steps S21 to S23 in FIG. 6). Then, the emotion estimation device 20 estimates the emotion of the subject U2, who is the driver of the other vehicle V2, based on the traveling state information of the other vehicle V2 (based on the reference data D2 including the traveling state information of the other vehicle V2) (steps S24 to S27 in FIG. 6). The in-vehicle emotion estimation system makes it possible to grasp the emotion estimation result of the driver of the vehicle surrounding the host vehicle V1 (here, the other vehicle V2) at the host vehicle V1. The driver of the surrounding vehicle does not need to wear a biosensor, and therefore does not need to transmit the detection result by the biosensor to the host vehicle V1. Therefore, the emotion estimation method using the in-vehicle emotion estimation system is highly feasible. If the emotion of the driver of the surrounding vehicle can be grasped, driving that takes the emotion into consideration becomes possible, leading to safe driving. For example, it is possible to prevent possible trouble with surrounding vehicles by using the above-mentioned notification control, which is beneficial.

[0120] In this embodiment, the sensor unit 60 generates sensor output information Sout1 used for driving control of the host vehicle V1 (see Figs. 1 and 5). Since the emotion of the driver of the other vehicle V2 is estimated using the sensor output information Sout1 originally used for driving control of the host vehicle V1, there is no need to add a dedicated component for emotion estimation to the host vehicle V1.

[0121] [Example EX1_2] An example EX1_2 will be described. There are individual differences in biosignals such as brainwave information and heart rate information, and the biosignals are also affected by the surrounding environment. As a kind of calibration to reduce these, a normalization process is executed in the example EX1_2.

[0122] FIG. 10 shows a schematic configuration of the learning device 10 according to the embodiment EX1_2. FIG. 11 shows a configuration block diagram of the learning device 10 according to the embodiment EX1_2. The learning device 10 according to the embodiment EX1_2 is added with a normalization unit 136 based on the learning device 10 according to the embodiment EX1_1 (see FIG. 7 and FIG. 8). The normalization unit 136 performs normalization processing on the emotion index value derived by the index value derivation unit 134. The normalization unit 136 is one of the functional blocks provided in the controller 13 of the learning device 10. In addition, the learning device 10 according to the embodiment EX1_2 is added with a task table 124 and a neutral region table 125 to the storage unit 12 based on the learning device 10 according to the embodiment EX1_1. Except for the addition of the normalization unit 136 and the tables 124 and 125, the learning device 10 according to the embodiment EX1_2 has the same configuration as the learning device 10 according to the embodiment EX1_1 (see FIG. 7 and FIG. 8).

[0123] The emotion index value derived by the index value derivation unit 134 is normalized in a normalization process. The concept of normalization will be explained by focusing on the arousal level as an emotion index value. FIG. 12 is a schematic diagram showing the change over time in the emotion index values ​​(arousal levels) of two subjects. In each of the four graphs a1, a2, b1, and b2 shown in FIG. 12, the horizontal axis corresponds to time, and the vertical axis corresponds to the arousal level based on a biological signal.

[0124] In Fig. 12, graphs a1 and a2 on the left side show data for the first subject, and graphs b1 and b2 on the right side show data for the second subject. In Fig. 12, graphs a1 and b1 on the top show data before normalization, and graphs a2 and b2 on the bottom show data after normalization. For the sake of concrete explanation, assume an example of a state in which the wakefulness level fluctuates between an upper limit and a lower limit during the measurement period for each subject.

[0125] As shown in graphs a1 and b1 in FIG. 12, the fluctuation range of the awakening degree of the first and second subjects is significantly different, and the fluctuation range of the second subject is larger than that of the first subject. That is, in this example, it can be said that the first subject has a narrower range of the measured awakening degree (in other words, a lower sensitivity to the awakening degree) compared to the second subject. Conversely, it can be said that the second subject has a wider range of the measured awakening degree (in other words, a higher sensitivity to the awakening degree) compared to the first subject. Note that the sensitivity to the awakening degree is influenced not only by the subject's innate individual differences, but also by various conditions such as the surrounding environment (ambient temperature and humidity, brightness, etc.) and the subject's condition (whether the subject is hungry, etc.). Therefore, even if it is data of a single subject, multiple data with different surrounding environments, subject conditions, or data collection periods can be interpreted as being equivalent to data of multiple subjects.

[0126] In the emotion index value, a neutral state exists near the center of the range of the fluctuation. The area corresponding to the neutral state is called the neutral area NR. The shaded area in each graph of FIG. 12 is the neutral area NR. In the case of wakefulness, the neutral state exists between the wakefulness state and the non-wakefulness state (sleep-like state). Such a neutral state area tends to be relatively easy to distinguish (estimate) by performing a specific task.

[0127] As shown in graphs a1 and b1 in Fig. 12, the neutral region NR differs between the first and second subjects. That is, the upper limit value Max1 and the lower limit value Min1 of the neutral region NR in the first subject differ from the upper limit value Max2 and the lower limit value Min2 of the neutral region NR in the second subject. The width W1 (=Max1-Min1) of the neutral region NR in the first subject differs from the width W2 (=Max2-Min2) of the neutral region NR in the second subject.

[0128] In the normalization process, the emotion index value derived by the index value derivation unit 134 is corrected (i.e., normalized) so that the neutral region NR of each subject is located in the same position as a predetermined standard neutral region NR0 and has the same width as the standard neutral region NR0. The standard upper limit value and standard lower limit value, which are the upper and lower limits of the standard neutral region NR0, are represented by the symbols "Max0" and "Min0", respectively. The standard width W0, which is the width of the standard neutral region NR0, is represented by "W0=Max0-Min0". The standard upper limit value Max0, the standard lower limit value Min0, and the width W0 are stored in the storage unit 12 as preset values.

[0129] Specifically, the emotion index value (arousal level) is corrected using the following calculation formulas (a1) to (a6). The correction here is a correction by normalization processing, and can be called normalization. AW1B and AW2B respectively represent the arousal levels of the first and second subjects before normalization (before correction). AW1A and AW2A respectively represent the arousal levels of the first and second subjects after normalization (after correction). AW1A = AW1B × A1 + B1 (a1) AW2A = AW2B × A2 + B2 (a2) A1=(Max0-Min0) / (Max1-Min1) ···(a3) A2=(Max0-Min0) / (Max2-Min2) ···(a4) B1=((Max0+Min0)-(Max1+Min1)×A1) / 2 (a5) B2=((Max0+Min0)-(Max2+Min2)×A2) / 2 (a6)

[0130] In general, the subject U3 can be regarded as the nth subject (n is any natural number). The upper and lower limits of the neutral region NR for the nth subject are represented by "Maxn" and "Minn", respectively. Then, the emotion index value (arousal level) can be corrected using the following calculation formulas (b1) to (b3). AWnB represents the arousal level of the nth subject before normalization (before correction), and AWnA represents the arousal level of the nth subject after normalization (after correction). The arousal level AWnB is the arousal level of the nth subject derived by the index value derivation unit 134 based on the electroencephalogram data of the nth subject as the subject U3. The normalization unit 136 performs normalization processing on the arousal level AWnB to obtain the arousal level AWnA, which is the arousal level after normalization. AWnA = AWnB × An + Bn (b1) An=(Max0-Min0) / (Maxn-Minn) ···(b2) Bn=((Max0+Min0)-(Maxn+Minn)×An) / 2 (b3)

[0131] When normalizing the arousal level of the nth subject by such a normalization method (corresponding to the calculation formulas (b1) to (b3)), it is necessary to estimate the upper limit value Maxn and the lower limit value Minn of the neutral region NR for the nth subject. For this estimation, the nth subject is made to perform a specific task, and the arousal level at that time is measured. Then, the upper limit value Maxn and the lower limit value Minn are estimated based on the measured value of the arousal level. The task to be performed is a task that puts the subject who performed the task into a specific mental and physical state (specific state). The task to be performed is, for example, a task in which the arousal level fluctuates above the upper limit value of the neutral region NR or a task in which the arousal level fluctuates below the lower limit value of the neutral region NR. As such tasks, there are several types of tasks for which evidence has been obtained in research in the fields of medicine, psychology, and the like. By having the nth subject perform these tasks, the nth subject is put into a specific mental and physical state, and normalization processing can be performed using the biological signal of the nth subject in the specific mental and physical state. Incidentally, measurement of the degree of awakening refers to deriving the degree of awakening of any subject by the index value derivation unit 134 based on the electroencephalogram data of the subject.

[0132] The activity level can also be normalized (corrected) in a similar manner. When normalizing (correcting) the activity level, the alertness and brain wave data in the above description of the embodiment EX1_2 can be read as activity level and heart rate data. In other words, each parameter for alertness can be replaced with a parameter for the activity level to be corrected. The same applies when an emotion index value different from alertness and activity level is used. The neutral area NR for alertness corresponds to the alertness neutral area NR1 shown in FIG. 3, and the neutral area NR for activity corresponds to the activity neutral area NR2 shown in FIG. 3.

[0133] 13 is a diagram showing an example of the task table 124. Items in the task table 124 include "task ID", "corresponding index type", "task content", and "index value transition state".

[0134] "Task ID" is task ID data, which is identification information for identifying task information. Task ID data is also the primary key of a data record in task table 124. That is, in task table 124, a data record is configured for each task ID data, and task data linked to the task ID data is stored in that data record. The task data includes data for the items "Corresponding index type," "Task content," and "Index value transition state."

[0135] "Corresponding index type" is the emotional index type to be normalized, and a "corresponding index type" corresponding to each task ID is stored in the task table 124. In this case, arousal level or activity level is stored as the "corresponding index type." "Task content" indicates the specific content of the task performed by the subject for normalization processing.

[0136] The "index value transition state" indicates how the corresponding emotion index value is expected to transition when the subject performs the corresponding task. For example, the "index value transition state" indicates high arousal or low arousal. High arousal indicates that when the nth subject performs the corresponding task, the arousal level of the nth subject increases beyond the upper limit value Maxn of the neutral region NR. Low arousal indicates that when the nth subject performs the corresponding task, the arousal level of the nth subject falls below the lower limit value Minn of the neutral region NR.

[0137] In the example of task table 124 in FIG. 13, task data for a task ID of "ST01" and a task ID of "ST02" are illustrated. A task to which a task ID of "ST01" is assigned may hereinafter be referred to as task ST01. A task to which a task ID of "ST02" is assigned may hereinafter be referred to as task ST02. In the task data for task ST01, the target emotion index is arousal level, the index value transition state (i.e., the transition state of arousal level) is high arousal level, and the task content is "mentally add displayed numbers." In the task data for task ST02, the target emotion index is arousal level, the index value transition state (i.e., the transition state of arousal level) is low arousal level, and the task content is "remain calm and clench your fist."

[0138] The task content to be performed by the subject is selected depending on the target of the normalization process and whether the boundary of the neutral region NR to be estimated is the upper limit or the lower limit. Using task ST01, the upper limit of the neutral region NR of the alertness level can be estimated, and using task ST02, the lower limit of the neutral region NR of the alertness level can be estimated.

[0139] An explanation will be added with reference to FIG. 14. For a given subject, the upper limit and lower limit of the neutral region NR of the level of alertness are represented by the symbols "NRU" and "NRD", respectively. In this example, tasks ST01 and ST02 are selected to estimate the upper limit NRU and the lower limit NRD. FIG. 14 is a schematic diagram showing the change in alertness over time when the subject of interest executes tasks ST01 and ST02. In the graph shown in FIG. 14, the horizontal axis corresponds to time, and the vertical axis corresponds to alertness. Signal α represents the change in alertness over time of the subject of interest when the subject of interest executes the task contents of task ST01. Signal β represents the change in alertness over time of the subject of interest when the subject of interest executes the task contents of task ST02.

[0140] During the first task execution period in which the subject of interest is performing the task ST01, the degree of arousal of the subject of interest is in a relatively high state (high index value state), so the measured degree of arousal of the subject of interest is expected to change (fluctue) within the high arousal range. On the other hand, it is considered that there is a time lag from the start of the execution of the task ST01 until the degree of arousal fluctuates due to the influence of the task ST01, and from the end of the task ST01 until the influence of the task ST01 disappears from the degree of arousal. For this reason, the time when a first predetermined time has elapsed from the start time of the first task execution period is set as the first measurement start time, and the time when a second predetermined time has elapsed from the end time of the first task execution period is set as the first measurement end time. The period from the first measurement start time to the first measurement end time is set as the first task evaluation period. Then, the minimum value of the degree of arousal of the subject of interest during the first task evaluation period can be estimated as the upper limit value NRU.

[0141] During the second task execution period in which the subject of interest is performing the task ST02, the arousal level of the subject of interest is in a relatively low state (low index value state, non-awakened state), so the measured arousal level of the subject of interest is expected to change (fluctue) within the low arousal level range. On the other hand, it is considered that there is a time lag from the start of the execution of the task ST02 until the arousal level fluctuates due to the influence of the task ST02, and from the end of the task ST02 until the influence of the task ST02 disappears from the arousal level. For this reason, the time when the first predetermined time has elapsed from the start time of the second task execution period is set as the second measurement start time, and the time when the second predetermined time has elapsed from the end time of the second task execution period is set as the second measurement end time. The period from the second measurement start time to the second measurement end time is set as the second task evaluation period. Then, the maximum value of the arousal level of the subject of interest during the second task evaluation period can be estimated as the lower limit NRD.

[0142] Based on this method, the neutral region for the emotion index value of arousal can be estimated. That is, if the subject of interest is the nth subject, the normalization unit 136 regards the upper limit value NRU and the lower limit value NRD as the upper limit value Maxn and the lower limit value Minn, respectively, of the neutral region NR in the arousal level of the nth subject. After the upper limit value Maxn and the lower limit value Minn are specified, the arousal level can be normalized according to the above formulas (b1) to (b3) (that is, the arousal level AWnA after normalization can be obtained from the arousal level AWnB before normalization). The same applies to the other emotion index values.

[0143] The normalization unit 136 determines the neutral region NR by the above method, and specifically, performs statistical processing on the time-varying data of the emotion index value (arousal level: signal α) obtained during the execution of task ST01 to determine the upper limit value NRU of the neutral region NR. For example, the normalization unit 136 smoothes the signal α in the first task evaluation period, and sets the minimum value of the signal α after the smoothing process as the upper limit value NRU of the neutral region NR. Similarly, the normalization unit 136 smooths the signal β in the second task evaluation period, and sets the maximum value of the signal β after the smoothing process as the lower limit value NRD of the neutral region NR. The smoothing process is performed for the purpose of removing noise components, etc.

[0144] The method of determining the upper limit value NRU and the lower limit value NRD of the neutral region NR has been described with a focus on the level of arousal, but the normalization unit 136 determines the upper limit value and the lower limit value of the neutral region for each type of emotion index value. For this reason, a task ST03 for determining the upper limit value of the neutral region of activity and a task ST04 for determining the lower limit value of the neutral region of activity are prepared, and the subject is requested to perform tasks ST03 and ST04 (tasks ST03 and ST04 are not shown). As with the level of arousal, a process of determining the neutral region of activity is performed according to the measurement results of activity when each task is performed.

[0145] In the above, it is assumed that the first variation range, which is the variation range of the emotion index value (signal α) obtained when task ST01 is executed, and the second variation range, which is the variation range of the emotion index value (signal β) obtained when task ST02 is executed, are separated from each other (see FIG. 14). Under this assumption, the space between the first and second variation ranges can be set as a neutral region NR.

[0146] However, depending on the type of emotion index value or the content of the task, the first and second variation ranges may overlap. In such a case, the neutral region may be determined based on the overlapping range of the first and second variation ranges. For example, the overlapping range of the first and second variation ranges may be set as the neutral region. Alternatively, for example, the overlapping range of the first and second variation ranges may be expanded by adding an appropriate offset to the overlapping range of the first and second variation ranges and set as the neutral region. In this case, the range may be expanded by an appropriate value (for example, 10%) from the viewpoint of preventing erroneous determination of emotion. Alternatively, for example, the overlapping range of the first and second variation ranges may be reduced by an appropriate offset and set as the neutral region. In this case, the range may be reduced by an appropriate value (for example, 10%) from the viewpoint of preventing inability to determine emotion.

[0147] Next, another method for determining the neutral region will be described. In this method, the subject is not required to execute specific tasks such as the above tasks ST01 and ST02. In this method, the neutral region is determined by focusing on the fact that as the accumulation of time-series data of emotion index values ​​progresses, the upper and lower limit values ​​of the neutral region estimated by statistical processing converge to certain values.

[0148] This will be described with reference to Fig. 15. Focus will be given on arousal level as an emotion index value. Fig. 15 is a schematic diagram showing the change over time in the emotion index value (arousal level), and in the graph of Fig. 15, the horizontal axis corresponds to time, and the vertical axis corresponds to arousal level.

[0149] As shown in FIG. 15, a person's mental and physical state changes into various states based on changes in the environment, etc., which change over time, even without giving a specific task. Therefore, a considerable amount of time-series data of emotion index values ​​can be statistically processed and a cluster analysis can be performed to divide the accumulated data into three states (high arousal, neutral, low arousal) based on the magnitude of the values. The cluster analysis can estimate the neutral region (upper and lower limits of the neutral region). For the cluster analysis, known methods such as the k-means method, Otsu's multi-value method, and Gaussian mixture model can be used.

[0150] Fig. 16 is a diagram showing the convergence status of the upper limit estimate of the neutral region when the neutral region is determined by the statistical method as shown in Fig. 15. The convergence status of the lower limit estimate of the neutral region is similar.

[0151] The amount of data used to estimate the upper limit of the neutral region using a statistical method increases over time. As the amount of data used in statistical processing increases over time, the estimated upper limit (estimated upper limit) converges to a certain value over time. This tendency is the same for the lower limit. A neutral region specified by the converged upper limit and lower limit values ​​is considered to be highly reliable. For this reason, the neutral region is determined at the stage where it is determined that the upper limit and lower limit values ​​calculated by the statistical method have converged. For example, a method is conceivable in which values ​​estimated to be the upper limit of the neutral region are sequentially specified, and it is determined that the convergence has occurred when the variance of the specified multiple values ​​becomes equal to or less than a predetermined convergence determination threshold. The same is true for the lower limit of the neutral region. In addition, even when a statistical method is used, the normalization unit 136 determines the upper limit and lower limit of the neutral region for each type of emotion index value.

[0152] The normalization unit 136 stores the upper and lower limit values ​​of the determined neutral region in the neutral region table 125 of the storage unit 12. In detail, the normalization unit 136 creates a data table of the determined upper and lower limit values ​​of the neutral region and stores the data table in the storage unit 12. When there are multiple subjects as the subject U3, it is effective to store neutral region information (information indicating the upper and lower limit values ​​of the neutral region) for each subject in the neutral region table 125. In this case, when recreating (adding) a learning dataset for the same subject, the neutral region information stored in the neutral region table 125 can be used.

[0153] 17 is a diagram showing an example of the neutral region table 125. The items in the neutral region table 125 include a "neutral information ID", a "user ID", an "emotion index type", an "upper limit value", a "lower limit value", and an "acquisition date and time".

[0154] In table 125, the "neutral information ID" is identification information for identifying neutral information. The neutral information ID is also the primary key of the data record in neutral area table 125. That is, in neutral area table 125, a data record is configured for each neutral information ID, and data of each item linked to the neutral information ID is stored in the data record.

[0155] In table 125, "user ID" is identification information for identifying a user. The above-mentioned n-th subject corresponds to a user. A different user ID is assigned to the first subject and the second subject. The same is true for the third subject, etc.

[0156] In table 125, the “upper limit value” and the “lower limit value” are information on the upper limit value and the lower limit value of the neutral region determined by normalization unit 136.

[0157] In table 125, the "acquisition date and time" stores date and time information on the upper and lower limit values ​​of the neutral region obtained (determined and stored). Even after the upper and lower limit values ​​of the neutral region for each user are stored in table 125, the data in table 125 can be updated based on the latest information.

[0158] Alternatively, a method of determining the neutral region based on an emotion declaration from the subject is also possible. For example, the normalization unit 136 asks the subject (user) a question regarding the emotion index value that the subject is aware of, and accumulates the answer and the arousal level measured (calculated) at that time. The question may, for example, inquire as to which of three stages (awake, normal, not awakened (sleepy)) the subject's arousal state is in. The normalization unit 136 then performs statistical processing on the accumulated data to estimate the neutral region. The arousal level measured when the subject's answer is "normal" is K. A Suppose there are K A is a sufficiently large integer). For example, A Among the arousal levels, the top 10% and bottom 10% are extracted, and the average of the top 10% and bottom 10% are set as the upper and lower limits of the neutral region, respectively. The arousal level measured when the subject answered "Arousal" is K B Suppose there are K B is a sufficiently large integer). For example, B Of the arousal levels, the lowest 10% of arousal levels can be extracted, and the average of the lowest 10% of arousal levels minus an appropriate offset value can be used as the upper limit of the neutral region. Similarly, for example, the arousal level measured when the subject's answer was "not aroused" can be K C Suppose there are K C is a sufficiently large integer). For example, C The top 10% of the arousal levels can be extracted from the individual arousal levels, and the average value of the top 10% arousal levels plus an appropriate offset value can be set as the lower limit of the neutral region. The same applies to the activity level.

[0159] The normalization method described above is referred to as the first normalization method. The first normalization method is a method in which normalization is performed based on the neutral region of the emotion index value, but normalization can also be performed by another method. The method described next is a method (hereinafter referred to as the second normalization method) in which normalization is performed based on the range of emotion index values ​​calculated within an appropriate measurement period.

[0160] Fig. 18 is a schematic diagram showing the change over time in the emotion index values ​​(arousal level) of two subjects. In each of the four graphs a1, a2, b1, and b2 shown in Fig. 18, the horizontal axis corresponds to time, and the vertical axis corresponds to the arousal level based on the biological signal.

[0161] In Fig. 18, graphs a1 and a2 on the left side show data for the first subject, and graphs b1 and b2 on the right side show data for the second subject. In Fig. 18, graphs a1 and b1 on the top show data before normalization, and graphs a2 and b2 on the bottom show data after normalization by the second normalization method. For the sake of concrete explanation, an example of a state in which the wakefulness level fluctuates between an upper limit and a lower limit during the measurement period for each subject is assumed.

[0162] As shown in graphs a1 and b1 in FIG. 18, the fluctuation range of the arousal level of the first and second subjects is significantly different, and the fluctuation range is larger for the second subject than for the first subject. That is, in this example, the first subject has a narrower range of measured arousal levels (in other words, a lower sensitivity to arousal levels) compared to the second subject. Conversely, the second subject has a wider range of measured arousal levels (in other words, a higher sensitivity to arousal levels) compared to the first subject. Note that the sensitivity to arousal levels is influenced not only by the subjects' innate individual differences, but also by various conditions such as the surrounding environment (ambient temperature and humidity, brightness, etc.) and the subject's condition (whether the subject is hungry, etc.).

[0163] The second normalization method is based on the idea that the range of variation of the emotion index value (arousal level) during an appropriate measurement period changes depending on the subject's sensitivity to the emotion index value and the surrounding environment. For this reason, the second normalization method corrects the emotion index value so that the size of the range of variation of the emotion index value (arousal level) is the same across multiple subjects.

[0164] As shown in graphs a1 and b1 of FIG. 18, the fluctuation range CH of the wakefulness differs between the first and second subjects due to individual differences, etc. In the description of the second normalization method, Max1 and Min1 respectively represent the upper limit value and the lower limit value in the fluctuation range CH of the wakefulness of the first subject. In the description of the second normalization method, Max2 and Min2 respectively represent the upper limit value and the lower limit value in the fluctuation range CH of the wakefulness of the second subject. In the example shown in FIG. 18, at least the upper limit value Max1 and the upper limit value Max2 are different from each other. In the description of the second normalization method, W1 represents the width of the fluctuation range CH of the wakefulness of the first subject, and is "W1=Max1-Min1". In the description of the second normalization method, W2 represents the width of the fluctuation range CH of the wakefulness of the second subject, and is "W2=Max2-Min2".

[0165] In the normalization process according to the second normalization method, the emotion index value derived by the index value derivation unit 134 is corrected (i.e., normalized) so that the variation range CH of each subject is located in the same position as the predetermined standard variation range CH0 and has the same width as the standard variation range CH0. In the explanation of the second normalization method, "Max0" and "Min0" respectively represent the standard upper limit value and the standard lower limit value which are the upper limit value and the lower limit value of the standard variation range CH0. In the explanation of the second normalization method, "W0" represents the standard width which is the width of the standard variation range CH0, and "W0=Max0-Min0". The standard upper limit value Max0, the standard lower limit value Min0, and the width W0 are stored in the storage unit 12 as preset values.

[0166] Specifically, the emotion index value (arousal level) is corrected using the following calculation formulas (c1) to (c6). The correction here is a correction by normalization processing, and can be called normalization. AW1B and AW2B respectively represent the arousal levels of the first and second subjects before normalization (before correction). AW1A and AW2A respectively represent the arousal levels of the first and second subjects after normalization (after correction). AW1A = AW1B × A1 + B1 (c1) AW2A = AW2B × A2 + B2 (c2) A1=(Max0-Min0) / (Max1-Min1) ···(c3) A2=(Max0-Min0) / (Max2-Min2) ···(c4) B1=((Max0+Min0)-(Max1+Min1)×A1) / 2 (c5) B2=((Max0+Min0)-(Max2+Min2)×A2) / 2 (c6)

[0167] In general, the subject U3 can be regarded as the nth subject (n is any natural number). In the explanation of the second normalization method, "Maxn" and "Minn" respectively represent the upper and lower limits of the fluctuation range CH for the nth subject. Then, the emotion index value (arousal level) can be corrected using the following calculation formulas (d1) to (d3). AWnB represents the arousal level of the nth subject before normalization (after correction), and AWnA represents the arousal level of the nth subject after normalization (after correction). The arousal level AWnB is the arousal level of the nth subject derived by the index value derivation unit 134 based on the electroencephalogram data of the nth subject as the subject U3. The normalization unit 136 performs normalization processing on the arousal level AWnB to obtain the arousal level AWnA, which is the arousal level after normalization. AWnA = AWnB × An + Bn (d1) An=(Max0-Min0) / (Maxn-Minn) ···(d2) Bn=((Max0+Min0)-(Maxn+Minn)×An) / 2 (d3)

[0168] When the second normalization method is used, a measurement period having a necessary length is set, and the controller 13 is caused to perform the following measurement process during the measurement period. In the measurement process, the controller 13 uses the index value derivation unit 134 to measure the wakefulness of the nth subject from the electroencephalogram data of the nth subject during the measurement period. The controller 13 then specifies the upper limit Maxn and lower limit Minn for the nth subject by specifying the fluctuation range CH of the measured wakefulness during the measurement period. After the upper limit Maxn and lower limit Minn are specified, the wakefulness can be normalized according to the above formulas (d1) to (d3) (i.e., the wakefulness after normalization AWnA can be obtained from the wakefulness before normalization AWnB). Note that the measurement of the wakefulness refers to the index value derivation unit 134 deriving the wakefulness of the nth subject based on the electroencephalogram data of the nth subject as the subject U3.

[0169] The activity level can also be normalized (corrected) in a similar manner. When normalizing (correcting) the activity level using the second normalization method, the arousal level and brainwave data in the above description of the second normalization method can be read as activity level and heart rate data. In other words, each parameter for the arousal level can be replaced with a parameter for the activity level to be corrected. The same applies when an emotion index value different from the arousal level and activity level is used. The neutral region NR for the arousal level corresponds to the arousal level neutral region NR1 shown in FIG. 3, and the neutral region NR for the activity level corresponds to the activity level neutral region NR2 shown in FIG. 3.

[0170] According to the second normalization method, there is no need for the subject to perform a special task, to collect data when performing the task, and to perform calculations on the collected data, etc. This reduces the burden on the subject, and also reduces the processing load of the learning device and the learning time.

[0171] In the embodiment EX1_2, the controller 13 creates normalization data for normalizing (correcting) the alertness and activity of the subject U3, and stores the normalization data in the storage unit 12. The data stored in the neutral region table 125 is an example of the normalization data. For the nth subject, the upper limit value Maxn and the lower limit value Minn for the alertness and the upper limit value Maxn and the lower limit value Minn for the activity, which are specified (determined) by any of the above-mentioned methods, are included in the normalization data.

[0172] Thereafter, when the learning process is performed, the awakening level and activity level derived by the index value derivation unit 134 are normalized by the normalization unit 136 using the normalization data. Then, in the learning process, the awakening level after normalization is input to the AI ​​model 121m as the correct answer data, and the activity level after normalization is input to the AI ​​model 122m as the correct answer data. The learning method of the AI ​​models 121m and 122m is the same as that shown in the embodiment EX1_1, except that the correct answer data input to the AI ​​models 121m and 122m are the awakening level and activity level after normalization.

[0173] When there are multiple subjects as the subject U3, normalization data is created for each subject and stored in the storage unit 12. In the learning process, in order to perform appropriate normalization, the group data provided to the learning device 10 is provided to the learning device 10 with the group data of which subject is specified. It is sufficient to include subject identification information in the group data. The subject identification information in certain group data indicates which subject the biological data and sensor output information Sout3 in the group data belong to.

[0174] The trained AI models 121m and 122m are incorporated into the emotion estimation device 20 as AI models 221m and 222m (see FIGS. 1 and 4). In the embodiment EX1_2, the arousal level derived by the AI ​​model 221m corresponds to the normalized arousal level, and the activity level derived by the AI ​​model 222m corresponds to the normalized activity level. This enables highly accurate emotion estimation in which the influence of individual differences and the like is suppressed.

[0175] [Example EX1_3] An embodiment EX1_3 will be described. The learning device 10 can also quantize emotion index values. This will be described.

[0176] Fig. 19 shows a schematic configuration of the learning device 10 according to the embodiment EX1_3. Fig. 20 shows a configuration block diagram of the learning device 10 according to the embodiment EX1_3. A quantization unit 137 is added to the learning device 10 according to the embodiment EX1_3 based on the learning device 10 according to the embodiment EX1_2 (see Figs. 10 and 11). The quantization unit 137 is one of the functional blocks provided in the controller 13 of the learning device 10. Except for the addition of the quantization unit 137, the learning device 10 according to the embodiment EX1_3 has the same configuration as the learning device 10 according to the embodiment EX1_2 (see Figs. 10 and 11).

[0177] The quantization unit 137 performs quantization processing on the normalized emotion index value output from the normalization unit 136. In the quantization processing, the normalized emotion index value is quantized. The quantization processing is performed for each type of emotion index value. The alertness after the quantization processing is referred to as the quantized alertness, and the activity after the quantization processing is referred to as the quantized activity. The quantized alertness and quantized activity are output from the quantization unit 137.

[0178] The wakefulness level derived by the index value derivation unit 134 and the normalized wakefulness level output by the normalization unit 136 are continuous values ​​or have a high enough resolution to be considered as continuous values. Similarly, the activity level derived by the index value derivation unit 134 and the normalized activity level output by the normalization unit 136 are continuous values ​​or have a high enough resolution to be considered as continuous values.

[0179] In the quantization process, the emotion index value is quantized using the boundary set in the psychological plane PP as a threshold value.

[0180] First, consider the first pattern in which a neutral region is not set on the psychological plane PP. In the first pattern, the normalized awakening level output from the normalization unit 136 is quantized in the quantization unit 137 to either an awakening level having a first value or an awakening level having a second value. That is, the quantized awakening level has a first value or a second value. In the first pattern, when the normalized awakening level is located on the positive side of the vertical axis direction as viewed from the origin of the psychological plane PP, the quantized awakening level has a first value. In the first pattern, when the normalized awakening level is located on the negative side of the vertical axis direction as viewed from the origin of the psychological plane PP, the quantized awakening level has a second value. The quantized awakening level having a first value represents an awakened state, and the quantized awakening level having a second value represents a non-awakened state.

[0181] Similarly, in the first pattern, the normalized activity output from the normalization unit 136 is quantized in the quantization unit 137 to either an activity having a first value or an activity having a second value. That is, the quantized activity has a first value or a second value. In the first pattern, when the normalized activity is located on the positive side of the horizontal axis direction as viewed from the origin of the psychological plane PP, the quantized activity has a first value. In the first pattern, when the normalized activity is located on the negative side of the horizontal axis direction as viewed from the origin of the psychological plane PP, the quantized activity has a second value. The quantized activity having a first value represents an active state, and the quantized activity having a second value represents an inactive state.

[0182] Next, consider the second pattern in which a neutral region is set on the psychological plane PP. In the second pattern, the normalized awakening level is quantized using the upper and lower limits of the awakening level neutral region NR1 as two thresholds. Therefore, in the second pattern, the normalized awakening level output from the normalization unit 136 is quantized in the quantization unit 137 to an awakening level having a first value, an awakening level having a second value, or an awakening level having a third value. That is, the quantized awakening level has a first value, a second value, or a third value. The quantized awakening level having a first value represents an awakened state, and the quantized awakening level having a second value represents a non-awakened state. The quantized awakening level having a third value represents a neutral state with respect to the awakening level.

[0183] Regarding the second pattern, consider the above-mentioned subject U3 as follows. When the normalized alertness derived for subject U3 is greater than the upper limit of the alertness neutral region NR1, the quantized alertness has a first value. When the normalized alertness derived for subject U3 is less than the lower limit of the alertness neutral region NR1, the quantized alertness has a second value. When the normalized alertness derived for subject U3 is within the alertness neutral region NR1, the quantized alertness has a third value.

[0184] Similarly, in the second pattern, the normalized activity is quantized using the upper and lower limits of the activity neutral region NR2 as two thresholds. Therefore, in the second pattern, the normalized activity output from the normalization unit 136 is quantized in the quantization unit 137 to an activity having a first value, an activity having a second value, or an activity having a third value. That is, the quantized activity has a first value, a second value, or a third value. The quantized activity having a first value represents an active state, and the quantized activity having a second value represents an inactive state. The quantized activity having a third value represents a neutral state with respect to activity.

[0185] Regarding the second pattern, consider the above-mentioned subject U3 as follows: When the normalized activity derived for subject U3 is greater than the upper limit of the activity neutral region NR2, the quantized activity has a first value. When the normalized activity derived for subject U3 is less than the lower limit of the activity neutral region NR2, the quantized activity has a second value. When the normalized activity derived for subject U3 is within the activity neutral region NR2, the quantized activity has a third value.

[0186] In the learning process according to the embodiment EX1_3, the awakening level and the activity level derived by the index value derivation unit 134 are normalized by the normalization unit 136. The normalization method is as shown in the embodiment EX1_2. The normalized awakening level and the normalized activity level are output from the normalization unit 136. The quantization unit 137 derives the quantized awakening level by performing a quantization process on the normalized awakening level, and derives the quantized activity level by performing a quantization process on the normalized activity level. In the learning process, the quantized awakening level is input to the AI ​​model 121m as correct answer data, and the quantized activity level is input to the AI ​​model 122m as correct answer data. The learning method of the AI ​​models 121m and 122m is the same as that shown in the embodiment EX1_1, except that the correct answer data input to the AI ​​models 121m and 122m are the quantized awakening level and the quantized activity level.

[0187] The trained AI models 121m and 122m are incorporated into the emotion estimation device 20 as AI models 221m and 222m (see Figs. 1 and 4). In the embodiment EX1_3, the arousal level derived by the AI ​​model 221m corresponds to the quantized arousal level, and the activity level derived by the AI ​​model 222m corresponds to the quantized activity level. Since normalization is performed during learning, it is possible to perform highly accurate emotion estimation with the influence of individual differences and the like suppressed. In addition, the quantization reduces the computation load during learning. The reduction in computation load leads to a reduction in learning time. The emotion estimation model 223m is a model corresponding to the quantized arousal level and the quantized activity level, that is, a model in which the arousal level axis and the activity level axis have a resolution corresponding to the quantized value. The arousal level axis and the activity level axis correspond to the vertical axis and the horizontal axis of the psychological plane PP, respectively (see Figs. 2 and 3). In a model in which the alertness axis and activity axis have resolution corresponding to quantized values, the values ​​on the alertness axis and the activity axis each have a two-value resolution, or, if a neutral region NR is set, a three-value resolution.

[0188] [Example EX1_4] Example EX1_4 will be described. FIG. 21 shows a schematic configuration of the learning device 10 according to Example EX1_4. Based on the learning device 10 according to Example EX1_1 (see FIG. 7 and FIG. 8), the AI ​​models 121m and 122m are replaced with the AI ​​model 128m to obtain the learning device 10 according to Example EX1_4. Except for this replacement, the learning device 10 according to Example EX1_4 has the same configuration as the learning device 10 according to Example EX1_1 (see FIG. 7 and FIG. 8).

[0189] In the embodiment EX1_4, the input data generating unit 133 converts the reference data D3 into learning input data suitable for the structure of the AI ​​model 128m. In the learning device 10 according to the embodiment EX1_4, the AI ​​model 128m is trained so that both the awakening level and the activity level of the subject U3 can be estimated (inferred) based on the learning input data. That is, the learning device 10 supplies the awakening level and the activity level output from the index value derivation unit 134 to the AI ​​model 128m as correct answer data, and trains the AI ​​model 128m using a learning algorithm such as the backpropagation method based on the learning input data.

[0190] The learning processing unit 135 executes learning of the AI ​​model 128m using a learning data set. The learning data set has the arousal level and activity level derived by the index value derivation unit 134 and the learning input data generated by the input data generation unit 133. The correct answer data in the learning data set for the AI ​​model 128m are the arousal level and activity level. In learning, the AI ​​model 128m estimates (infers) the arousal level and activity level of the subject U3 based on the learning input data. The learning processing unit 135 updates the parameters of the AI ​​model 128m (weights of a neural network, etc.) based on the error between the estimated arousal level and activity level and the arousal level and activity level as correct answer data. When a termination condition, such as the error being sufficiently low, is satisfied, the learning of the AI ​​model 128m is terminated. The AI ​​model 128m at the time of the completion of learning is applied to the emotion estimation device 20.

[0191] FIG. 22 shows a schematic configuration of the feeling estimation device 20 according to the embodiment EX1_4. In the feeling estimation device 20 according to the embodiment EX1_4, an AI model 228m is provided instead of the AI ​​models 221m and 222m. That is, the feeling estimation device 20 according to the embodiment EX1_4 is obtained by replacing the AI ​​models 221m and 222m with the AI ​​model 228m based on the feeling estimation device 20 shown in FIG. 1 and FIG. 4. The AI ​​model 128m at the time when learning is completed, that is, the learned AI model 128m is incorporated into the feeling estimation device 20 as the A1 model 228m. The AI ​​model 228m outputs (calculates) the arousal level and activity level of the subject U2 based on the input data for estimation. The feeling estimation model 223m according to the embodiment EX1_4 estimates the feeling of the subject U2 based on the arousal level and activity level output from the AI ​​model 228m.

[0192] As described above, the feeling estimation device 20 has an estimator that estimates the feeling of a driver of an arbitrary vehicle based on the driving state information of the vehicle. In the feeling estimation device 20 in Fig. 22, the estimator is configured by the AI ​​model 228m and the feeling estimation model 223m. The input data generation unit 233 may also be considered to be included as a component of the estimator.

[0193] The embodiment EX1_2 and the embodiment EX1_4 may be combined. That is, the AI ​​models 121m and 122m may be replaced with the AI ​​model 128m based on the learning device 10 (see Figs. 10 and 11) according to the embodiment EX1_2. In this case, the AI ​​model 128m is learned by using the normalized awakening level and activity level output from the normalization unit 136 as correct answer data.

[0194] The embodiment EX1_3 and the embodiment EX1_4 may be combined. That is, the AI ​​models 121m and 122m may be replaced with the AI ​​model 128m based on the learning device 10 (see FIG. 19 and FIG. 20) according to the embodiment EX1_3. In this case, the quantized awakening level and the quantized activity level output from the quantization unit 137 are used as the correct answer data to learn the AI ​​model 128m.

[0195] [Example EX1_5] An example EX1_5 will be described. An AI model for estimating emotions itself, rather than emotion index values ​​such as arousal level and activity level, may be constructed in the learning device 10 and installed in the emotion estimation device 20. This will be described.

[0196] FIG. 23 shows a schematic configuration of the learning device 10 according to the embodiment EX1_5. Based on the learning device 10 according to the embodiment EX1_1 (see FIG. 7 and FIG. 8), the AI ​​models 121m and 122m are replaced with the AI ​​model 129m and the emotion estimation model 138 is added to the learning device 10, thereby obtaining the learning device 10 according to the embodiment EX1_5. Except for the replacement and addition, the learning device 10 according to the embodiment EX1_5 has the same configuration as the learning device 10 according to the embodiment EX1_1 (see FIG. 7 and FIG. 8). The controller 13 of the learning device 10 may realize each function of the emotion estimation model 138 and the AI ​​model 129m.

[0197] In the embodiment EX1_5, the arousal level and activity level of the subject U3 derived by the index value derivation unit 134 are supplied to the emotion estimation model 138. The emotion estimation model 138 estimates the emotion of the subject U3 based on the arousal level and activity level from the index value derivation unit 134. The emotion estimation model 138 is equivalent to the above-mentioned emotion estimation model 223m (see FIG. 1, etc.). The emotion estimation method of the subject U3 by the emotion estimation model 138 based on the arousal level and activity level of the subject U3 may be the same as the emotion estimation method of the subject U2 by the emotion estimation model 223m based on the arousal level and activity level of the subject U2.

[0198] In the embodiment EX1_5, the input data generating unit 133 converts the reference data D3 into learning input data suitable for the structure of the AI ​​model 129m. In the learning device 10 according to the embodiment EX1_5, the AI ​​model 129m is trained so that the emotion of the subject U3 can be estimated (inferred) based on the learning input data. That is, the learning device 10 supplies the emotion estimated by the emotion estimation model 138 (i.e., the estimated emotion of the subject U3) to the AI ​​model 129m as correct answer data, and trains the AI ​​model 129m using a learning algorithm such as the backpropagation method based on the learning input data.

[0199] The learning processing unit 135 executes learning of the AI ​​model 129m using a learning data set. The learning data set has the emotion estimated by the emotion estimation model 138 (i.e., the estimated emotion of the subject U3) and the learning input data generated by the input data generation unit 133. The correct answer data in the learning data set for the AI ​​model 129m is the emotion estimated by the emotion estimation model 138. In learning, the AI ​​model 129m estimates (infers) the emotion of the subject U3 based on the learning input data. The learning processing unit 135 updates the parameters of the AI ​​model 129m (weights of a neural network, etc.) based on the error between the emotion estimated by the AI ​​model 129m and the correct answer data (i.e., the emotion estimated by the emotion estimation model 138). When a termination condition, such as the error being sufficiently low, is satisfied, the learning of the AI ​​model 129m is terminated. The AI ​​model 129m at the time of completion of learning is applied to the emotion estimation device 20.

[0200] FIG. 24 shows a schematic configuration of the feeling estimation device 20 according to the embodiment EX1_5. In the feeling estimation device 20 according to the embodiment EX1_5, an AI model 229m is provided instead of the models 221m to 223m in FIG. 1 and the like. That is, the feeling estimation device 20 according to the embodiment EX1_5 is obtained by replacing the models 221m to 223m with the AI ​​model 229m based on the feeling estimation device 20 shown in FIG. 1 and FIG. 4. The AI ​​model 129m at the time when learning is completed, that is, the trained AI model 129m is incorporated into the feeling estimation device 20 as the A1 model 229m. The AI ​​model 229m estimates the feeling of the target U2 based on the input data for estimation.

[0201] As described above, the feeling estimation device 20 has an estimator that estimates the feeling of a driver of an arbitrary vehicle based on the driving state information of the vehicle. In the feeling estimation device 20 of FIG. 24, the estimator is configured by an AI model 229m. The input data generation unit 233 may also be considered to be included in the components of the estimator. The estimator (229m) according to the embodiment EX1_5 directly estimates the feeling of the subject U2 without deriving the feeling index value. In this configuration, since feeling estimation can be performed with a single AI model, it is possible that the configuration of the AI ​​model can be simplified or the calculation load can be reduced compared to the feeling estimation device 20 of FIG. 1.

[0202] The embodiment EX1_2 and the embodiment EX1_5 may be combined. In this case, a normalization unit 136 (see FIG. 10) is added to the learning device 10 according to the embodiment EX1_5. Then, the emotion of the subject U3 is estimated in the emotion estimation model 138 based on the normalized arousal level and activity level output from the normalization unit 136. The embodiment EX1_3 and the embodiment EX1_5 may be combined. In this case, a normalization unit 136 and a quantization unit 137 (see FIG. 19) are added to the learning device 10 according to the embodiment EX1_5. Then, the emotion of the subject U3 is estimated in the emotion estimation model 138 based on the quantized arousal level and quantized activity level output from the quantization unit 137.

[0203] [Example EX1_6] An example EX1_6 will be described. Although arousal and activity are given as examples of emotion index values, the emotion index values ​​handled by the learning device 10 and the emotion estimation device 20 are not limited to arousal and activity. When the biometric data of the subject U3 includes brainwave data, one emotion index value derived by the index value derivation unit 134 of the learning device 10 may be any value based on the brainwave data of the subject U3 (which may include the brainwave data itself). When the biometric data of the subject U3 includes heartbeat data, another emotion index value derived by the index value derivation unit 134 of the learning device 10 may be any value based on the heartbeat data of the subject U3 (which may include the heartbeat data itself). The biometric data provided to the learning device 10 may be other than brainwave data and heartbeat data. The biometric data provided to the learning device 10 may include other data related to biometric information in addition to brainwave data and heartbeat data.

[0204] <<Second embodiment>> A second embodiment of the present invention will be described. In the first embodiment, the other vehicle V2 is a vehicle located in front of the host vehicle V1, but the other vehicle V2 may be a vehicle located behind the host vehicle V1. That is, a modification in which the other vehicle V2 is a vehicle located behind the host vehicle V1 may be applied to the first embodiment, and in this case, the feeling estimation device 20 estimates the feeling of the driver of the vehicle (the other vehicle V2) behind the host vehicle V1.

[0205] When applying the transformation technique in which the other vehicle V2 is a vehicle located behind the vehicle V1 to the first embodiment, the longitudinal relationship between the vehicle V1 and the other vehicle V2 and the longitudinal relationship between the test vehicle V3 and the vehicle V4 may be reversed from that shown in the first embodiment. Therefore, in the transformation technique, the front vehicle and the rear vehicle in the first embodiment are replaced with the rear vehicle and the front vehicle, respectively, and the front inter-vehicle distance and the rear inter-vehicle distance are replaced with the rear inter-vehicle distance and the front inter-vehicle distance, respectively. In the second embodiment, the driving state information of the vehicle V1 includes the rear inter-vehicle distance based on the vehicle V1, while the driving state information of the other vehicle V2 includes the front inter-vehicle distance based on the other vehicle V2. The rear inter-vehicle distance based on the vehicle V1 refers to the distance between the vehicle V1 and the other vehicle V2 traveling behind the vehicle V1. The front inter-vehicle distance based on the other vehicle V2 refers to the distance between the other vehicle V2 and the vehicle V1 traveling ahead of the other vehicle V2.

[0206] As shown in FIG. 1, when the host vehicle V1 is located behind the other vehicle V2, the above-mentioned vehicle distance d12 is a forward vehicle distance as seen from the host vehicle V1 and a rear vehicle distance as seen from the other vehicle V2. Conversely, when the host vehicle V1 is located in front of the other vehicle V2, the above-mentioned vehicle distance d12 is a rear vehicle distance as seen from the host vehicle V1 and a forward vehicle distance as seen from the other vehicle V2. It is considered that when the vehicle distance d12 is relatively small, the subject U2 is likely to feel nervous, and when the vehicle distance d12 is relatively large, the subject U2 is likely to feel relaxed. In consideration of these, the vehicle distance d12 is included in the driving state information of the host vehicle V1 and the driving state information of the other vehicle V2. This is expected to improve the accuracy of estimating the emotion of the subject U2.

[0207] However, it is not essential that the vehicle distance d12 is included in the driving state information of the subject vehicle V1 and the driving state information of the other vehicle V2. Even if the vehicle distance d12 is excluded from each driving state information, it is possible to estimate the emotion of the subject U2 from other information in the driving state information. Similarly, it is not essential that the vehicle distance d34 (see FIG. 7) is included in the driving state information of the subject vehicle V3 and the driving state information of the vehicle V4.

[0208] <<Third embodiment>> A third embodiment of the present invention will be described. The first and second embodiments may be combined and implemented. The technology shown in the first embodiment estimates the emotion of a driver in a vehicle ahead of the host vehicle V1, and the technology shown in the second embodiment estimates the emotion of a driver in a vehicle behind the host vehicle V1.

[0209] Therefore, the feeling estimation device 20 according to the third embodiment can perform a first feeling estimation process and a second feeling estimation process. The first feeling estimation process is the feeling estimation process according to the first embodiment (see FIG. 6). In the first feeling estimation process, the feeling of the driver of the forward vehicle (V2) is estimated through generation of driving state information of the forward vehicle (V2) based on driving state information of the host vehicle V1 and the relative positional relationship between the host vehicle V1 and the forward vehicle (V2). The second feeling estimation process is the feeling estimation process according to the second embodiment. In the second feeling estimation process, the feeling of the driver of the backward vehicle (V2) is estimated through generation of driving state information of the backward vehicle (V2) based on driving state information of the host vehicle V1 and the relative positional relationship between the host vehicle V1 and the backward vehicle (V2).

[0210] However, as long as the relative positional relationship between the host vehicle V1 and the other vehicle V2 can be grasped by the host vehicle V1 (feeling estimation device 20), the positional relationship between the host vehicle V1 and the other vehicle V2 is arbitrary. Therefore, for example, the other vehicle V2 may be located diagonally forward or diagonally rearward of the host vehicle V1, or may be located to the side of the host vehicle V1.

[0211] <<Additional Information>> A program for causing a computer device to execute any of the methods described in the embodiments of the present invention, and a non-volatile recording medium having the program recorded thereon, are included in the scope of the embodiments of the present invention. Any of the processes in the embodiments of the present invention may be realized by hardware such as a semiconductor integrated circuit, software equivalent to the program, or a combination of hardware and software.

[0212] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above embodiments are merely examples of the present invention, and the meanings of the terms of the present invention and each component are not limited to those described in the above embodiments. The specific numerical values ​​shown in the above description are merely examples, and can be changed to various numerical values ​​as a matter of course. [Explanation of symbols]

[0213] SYS Vehicle Control System 10 Learning Device 11 Communications Department 12 Storage section 121m 1st AI model 122m 2nd AI model 128m, 129m AI model 13 Controller 131 Acquisition Department 132 Reference Data Generation Unit 133 Input Data Generation Unit 134 Index value derivation part 135 Learning processing unit 136 Normalization section 137 Quantization section 20 Emotion estimation device 21 Communications Department 22 Memory section 221m 1st AI model 221m 2nd AI model 223m Emotion estimation model 228m, 229m AI model 23 Controller 231 Acquisition Department 232 Reference Data Generation Unit 233 Input Data Generation Unit 234 Emotion estimation part 235 Estimation result output section 30 Vehicle control device 40 Actuator section 50 Notification Department 60 Sensor section DB Database V1 Vehicle V2 Other vehicles V3 test vehicle V4 Vehicle U2 Target Audience U3 Subjects

Claims

1. generating driving state information of the other vehicle based on driving state information of the host vehicle and a relative positional relationship between the host vehicle and the other vehicle, and estimating the emotion of the driver of the other vehicle based on the driving state information of the other vehicle; , emotion estimation device.

2. A sensor installed in the host vehicle and observing the other vehicle is used to generate time series data of the relative positional relationship, and information on the running state of the host vehicle is converted into information on the running state of the other vehicle using the time series data of the relative positional relationship. The emotion estimation device according to claim 1 .

3. The emotion of the driver of the other vehicle is estimated by inputting the driving state information of the other vehicle to an estimator that estimates the emotion of the driver of the other vehicle based on the driving state information of the vehicle. The emotion estimation device according to claim 1 .

4. In the estimator, an emotion of the driver of the other vehicle is estimated by deriving an emotion index value related to an emotion of the driver of the other vehicle based on the driving state information of the other vehicle; The emotion index value of the driver of the other vehicle includes an arousal level of the central nervous system and an activity level of the autonomic nervous system of the driver of the other vehicle. The emotion estimation device according to claim 3 .

5. In the estimator, the emotion of the driver of the other vehicle is directly estimated based on the driving state information of the other vehicle. The emotion estimation device according to claim 3 .

6. The driving state information of the host vehicle and the driving state information of the other vehicle include information on the distance between the host vehicle and the other vehicle. The emotion estimation device according to claim 1 .

7. A sensor unit installed in the vehicle; a feeling estimation device that generates driving state information of the other vehicle based on driving state information of the host vehicle based on output information of the sensor unit and a relative positional relationship between the host vehicle and the other vehicle, and estimates a feeling of a driver of the other vehicle based on the driving state information of the other vehicle. ,In-vehicle emotion estimation system.

8. The feeling estimation device generates time-series data of the relative positional relationship using a sensor in the sensor unit that observes the other vehicle, and converts running state information of the host vehicle into running state information of the other vehicle using the time-series data of the relative positional relationship. The in-vehicle emotion estimation system according to claim 7 .

9. The emotion estimation device estimates the emotion of the driver of the other vehicle by inputting the driving state information of the other vehicle to an estimator that estimates the emotion of the driver of the other vehicle based on the driving state information of the vehicle. The in-vehicle emotion estimation system according to claim 7 or 8.

10. the estimator estimates an emotion of the driver of the other vehicle by deriving an emotion index value related to an emotion of the driver of the other vehicle based on the traveling state information of the other vehicle; The emotion index value of the driver of the other vehicle includes an arousal level of the central nervous system and an activity level of the autonomic nervous system of the driver of the other vehicle. The in-vehicle emotion estimation system according to claim 9 .

11. In the estimator, the emotion of the driver of the other vehicle is directly estimated based on the driving state information of the other vehicle. The emotion estimation device according to claim 9 .

12. The driving state information of the host vehicle and the driving state information of the other vehicle include information on the distance between the host vehicle and the other vehicle. The emotion estimation device according to claim 7 or 8.

13. generating driving state information of the other vehicle based on driving state information of the host vehicle and a relative positional relationship between the host vehicle and the other vehicle, and estimating the emotion of the driver of the other vehicle based on the driving state information of the other vehicle; ,Emotion estimation method.

14. A method for generating driving state information of another vehicle based on driving state information of the own vehicle and a relative positional relationship between the own vehicle and another vehicle, and estimating the emotion of the driver of the other vehicle based on the driving state information of the other vehicle, is performed by a computer. , an emotion estimation program.

Images