Unpleasant emotion determination device and vehicle equipped with same

Abstract

To more properly determine whether or not an occupant feels uncomfortable.SOLUTION: It is determined whether or not an unpleasant feeling is generated in an occupant using an emotion estimation learned model for each occupant that outputs whether or not an unpleasant feeling is generated in the occupant with respect to a situation of a vehicle or an expression of the occupant by machine learning using the situation of the vehicle and a CG model created based on the expression of the occupant captured by an in-vehicle camera as input data.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an unpleasant emotion determination device and a vehicle equipped with the same, and more particularly to an unpleasant emotion determination device that determines whether an occupant is experiencing unpleasant emotions and a vehicle equipped with such an unpleasant emotion determination device. [Background technology]

[0002] Conventionally, one proposed technology of this type is to determine specific driving operations of the driver that make passengers feel uncomfortable based on the difference between the driver's emotions and the emotions of the passengers, and to generate advice to the driver based on the difference between the driving characteristics of the driver and the passengers regarding the specific driving operations (see, for example, Patent Document 1). This technology aims to achieve driving that does not make passengers uncomfortable. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-098779 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the above-mentioned technology, it is often difficult to estimate the emotions of passengers. Emotions vary from person to person, making it difficult to estimate the emotions of passengers. Furthermore, it is difficult to estimate the emotions of passengers when it is not possible to capture the facial expressions of passengers. For this reason, it is difficult to accurately determine whether a passenger is experiencing unpleasant emotions.

[0005] The unpleasant emotion determination device and the vehicle equipped with the unpleasant emotion determination device of the present disclosure mainly aim to more accurately determine whether or not an occupant is experiencing unpleasant emotions. [Means for solving the problem]

[0006] The unpleasant emotion determination device and the vehicle equipped with the same according to the present disclosure employ the following measures to achieve the above-mentioned main object.

[0007] The unpleasant emotion determination device of the present disclosure includes: an occupant detection device that detects the presence of an occupant in the vehicle for each occupant; a vehicle status detection device that detects the status of the vehicle; An in-car camera that captures the passengers inside the car, an unpleasant emotion determination device that determines whether the occupant detected by the occupant detection device is experiencing unpleasant emotions using the vehicle situation and / or the image captured by the in-vehicle camera; The unpleasant emotion determination device for a vehicle, The vehicle status and a CG model created based on the facial expressions of the occupants photographed by the in-vehicle camera are used as input data, and machine learning is used to output whether the occupants are experiencing unpleasant emotions in relation to the vehicle status and / or the facial expressions of the occupants. The emotion estimation trained model for each occupant is used to determine whether the occupants are experiencing unpleasant emotions. It is characterized by:

[0008] The unpleasant emotion determination device disclosed herein uses a trained emotion estimation model for each occupant, which uses machine learning to output whether the occupant is experiencing unpleasant emotions based on the vehicle situation and the occupant's facial expressions, using as input data a computer graphics model (CG model) created based on the vehicle situation and the occupant's facial expressions captured by an in-vehicle camera. This trained emotion estimation model for each occupant is used to determine whether the occupant is experiencing unpleasant emotions. Because the trained emotion estimation model for each occupant is used to determine whether the occupant is experiencing unpleasant emotions, it is possible to more accurately determine whether the occupant is experiencing unpleasant emotions. Here, the vehicle situation includes data related to vehicle behavior, such as vehicle speed V, longitudinal acceleration of the vehicle, gradient, and lateral acceleration of the vehicle, as well as environmental data, such as weather, air temperature, interior temperature, carbon dioxide concentration inside the vehicle, presence or absence of solar radiation, and time of day.

[0009] In the unpleasant emotion determination device disclosed herein, when an image of the facial expression of the target occupant is obtained by the in-vehicle camera, the vehicle situation and a CG model based on the occupant's facial expression may be applied to the emotion estimation trained model to determine whether the target occupant is feeling unpleasant, and when an image of the facial expression of the target occupant cannot be obtained by the in-vehicle camera, the vehicle situation may be applied to the emotion estimation trained model to determine whether the target occupant is feeling unpleasant. In this way, when the facial expression of the target occupant can be captured, it is possible to more accurately determine whether the target occupant is feeling unpleasant, and it is possible to determine whether the target occupant is feeling unpleasant even when the facial expression of the target occupant cannot be captured.

[0010] In the unpleasant emotion determination device of the present disclosure, when an emotion estimation trained model corresponding to a target occupant is not obtained, an emotion estimation trained model for the target occupant may be created using a CG model created based on the vehicle situation and the facial expression of the target occupant captured by the in-vehicle camera. This makes it possible to more accurately determine whether an unpleasant emotion is occurring in a new target occupant.

[0011] A vehicle disclosed herein is a vehicle equipped with an unpleasant emotion determination device according to any one of the aspects of the present disclosure, and is characterized in that it determines whether unspecified occupants are feeling unpleasant emotions using an emotion estimation learned model applied to unspecified occupants obtained by using a plurality of emotion estimation learned models created by unpleasant emotion determination devices of other vehicles. In such a vehicle disclosed herein, the emotion estimation learned model applied to unspecified occupants obtained by using a plurality of emotion estimation learned models created by unpleasant emotion determination devices of other vehicles is used, so it is possible to more appropriately determine whether unspecified occupants are feeling unpleasant emotions. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating an example of the configuration of an automobile 20 equipped with an unpleasant emotion determination device according to an embodiment of the present disclosure, with a main electronic control unit 30 as the central block. [Figure 2] 10 is a flowchart showing an example of unpleasant emotion determination processing executed by the main electronic control unit 30. DETAILED DESCRIPTION OF THE INVENTION

[0013] Next, a mode (embodiment) for carrying out the present disclosure will be described. Fig. 1 is a block diagram showing an example of the configuration of an automobile 20 equipped with an unpleasant emotion determination device as an embodiment of the present disclosure, with a main electronic control unit 30 as the central block. The unpleasant emotion determination device of the embodiment corresponds to the main electronic control unit 30. As shown in the figure, the automobile 20 of the embodiment includes a drive unit 62 that outputs drive force to drive wheels (not shown), and a drive electronic control unit (hereinafter referred to as a drive ECU) 60 that drives and controls the drive unit 62.

[0014] The drive device 62 may be, for example, a combination of an engine and an automatic transmission, a motor, etc. The drive ECU 60 is configured as a microcomputer centered around a CPU (not shown), and controls the drive of the drive device 62 based on a drive control signal from the main ECU 30.

[0015] In addition to the drive unit 62 and drive ECU 60, the automobile 20 of the embodiment is equipped with an ignition switch 32, a shift position sensor 34, an accelerator position sensor 36, a brake position sensor 38, a vehicle speed sensor 40, an acceleration sensor 42, a gradient sensor 44, a yaw rate sensor 46, an in-vehicle camera 48, a seat pressure sensor 50, a surrounding recognition electronic control unit (hereinafter referred to as the surrounding recognition ECU) 52, a surrounding recognition device 53, a battery electronic control unit (hereinafter referred to as the battery ECU) 54, a battery 55, an air conditioning device electronic control unit (hereinafter referred to as the air conditioning ECU) 56, an air conditioning device 58, a brake electronic control unit (hereinafter referred to as the brake ECU) 64, a brake device 66, a steering electronic control unit (hereinafter referred to as the steering ECU) 68, a steering device 70, a center display 72, a meter 76, a GPS (Global Positioning System, Global Positioning Satellite) 78, a navigation system 80, a communication device 86, and the like.

[0016] The shift position sensor 34 detects the position of the shift lever. The accelerator position sensor 36 detects the accelerator opening according to the amount of depression of the accelerator pedal by the driver. The brake position sensor 38 detects the brake position according to the amount of depression of the brake pedal by the driver.

[0017] The vehicle speed sensor 40 detects the vehicle speed based on the wheel speed, etc. The acceleration sensor 42 detects, for example, the acceleration in the longitudinal direction of the vehicle. The gradient sensor 44 detects the road gradient. The yaw rate sensor 46 detects the lateral acceleration (yaw rate) in the left-right direction due to turning motion. The in-vehicle camera 48 is arranged from the front to the rear of the passenger compartment and captures images of the passengers in the passenger compartment. The seat pressure sensor 50 detects whether or not a passenger is present in the seat.

[0018] The periphery recognition ECU 52 is configured as a microprocessor centered around a CPU (not shown). Information about the vehicle and its surroundings (e.g., inter-vehicle distances D1, D2 between the vehicle and other vehicles ahead and behind the vehicle, the vehicle speeds of other vehicles, and the vehicle's position in the lane on the road) is input to the periphery recognition ECU 52 from the periphery recognition device 53 via an input port. Examples of the periphery recognition device 54 include a front camera, a rear camera, millimeter-wave radar, quasi-millimeter-wave radar, infrared laser radar, sonar, etc.

[0019] The battery ECU 54 is configured as a microprocessor centered around a CPU (not shown). The battery ECU 54 receives, via an input port, a battery voltage Vb from a voltage sensor (not shown) attached to the output terminal of the battery 55, a battery current Ib from a current sensor (not shown), and a battery temperature Tb from a temperature sensor (not shown) attached to the battery 55. The battery ECU 54 calculates the power storage rate SOC, input / output limits Win, Wout, etc. based on the battery voltage Vb, battery current Ib, etc.

[0020] The air conditioner ECU 56 is configured as a microcomputer centered around a CPU (not shown) and is incorporated into an air conditioner 58 that conditions the passenger compartment, and controls the drive of the air conditioner compressor in the air conditioner so that the temperature in the passenger compartment becomes a set temperature.

[0021] The brake ECU 64 is configured as a microcomputer centered around a CPU (not shown), and controls the operation of a well-known hydraulically driven brake device 66. The brake device 66 is configured to be able to generate braking force resulting from the brake depression force exerted by depressing the brake pedal and braking force resulting from hydraulic pressure adjustment.

[0022] The steering ECU 68 is configured as a microcomputer centered around a CPU (not shown), and drives and controls the actuator of the steering device 28, which is mechanically connected to a steering wheel (not shown) and the drive wheels via a steering shaft. The steering device 28 steers the drive wheels based on the steering operation of the driver, and also steers the drive wheels by driving the actuator by the steering ECU 68 based on a steering control signal from the main electronic control unit 30.

[0023] The center display 72 is located in the center in front of the driver's seat and passenger seat, and also functions as a touch panel to execute various vehicle settings, audio and media applications, and also functions as the display unit 84 of the navigation system to perform map navigation. The center display 72 is equipped with speakers and other components.

[0024] The GPS 78 is a device that detects the position of a vehicle based on signals transmitted from multiple GPS satellites.

[0025] The navigation system 80 is a system that guides the vehicle to a set destination, and includes map information 82 and a display unit 84. When a destination is set, the navigation system 80 sets a route based on the destination information, information on the current location (current vehicle position) acquired by the GPS 78, and the map information 82.

[0026] The communication device 86 transmits information about the vehicle to the management center 100 and receives various information from the management center 100 .

[0027] The main electronic control unit 30 is configured as a microcomputer centered around a CPU (not shown). For example, the main electronic control unit 30 receives an ignition switch signal from an ignition switch 32, a shift position from a shift position sensor 34, an accelerator opening from an accelerator position sensor 36, a brake position from a brake position sensor 38, a vehicle speed V from a vehicle speed sensor 40, an acceleration from an acceleration sensor 42, a gradient from a gradient sensor 44, and a yaw rate from a yaw rate sensor 46. Other inputs include an image from an in-vehicle camera 48 and a seat pressure Ps of each seat from a seat pressure sensor 50. The main electronic control unit 30 outputs, for example, a display control signal to a center display 72 and a communication control signal to a communication device 86.

[0028] The main electronic control unit 30 communicates with the surroundings recognition ECU 52, air conditioner ECU 56, drive ECU 60, brake ECU 64, steering ECU 68, and navigation system 80, exchanging various types of information.

[0029] The main electronic control unit 30 sets the required driving force and required power based on the accelerator opening from the accelerator position sensor 36 and the vehicle speed from the vehicle speed sensor 40, and sends a drive control signal to the drive ECU 60 so that the required driving force and required power are output from the drive device 62 to the vehicle.

[0030] Next, the operation of the automobile 20 configured as described above, particularly the operation when determining whether or not an occupant of the automobile 20 is experiencing discomfort, will be described. The occupants include the driver and passengers in the front passenger seat and rear seat. The discomfort may be due to motion sickness or an abnormal physical condition. FIG. 2 is a flowchart showing an example of the discomfort judgment process executed by the main electronic control unit 30. This process is executed repeatedly after the system is started.

[0031] When the unpleasant emotion determination process is executed, the main electronic control unit 30 first confirms the passenger in the vehicle (step S100). The passenger can be confirmed by identifying the person based on the analysis results of the video captured by the in-vehicle camera 48. Alternatively, if the only determination is whether or not a passenger is present, it can also determine which seat the passenger is sitting in based on the detection value from the seat pressure sensor 50.

[0032] Next, it is determined whether an emotion estimation learned model is stored for each passenger (step S110). For passengers for whom an emotion estimation learned model is not stored, an emotion estimation learned model using a CG model is created and stored by the processes of steps S120 to S140.

[0033] To create an emotion estimation trained model, first, a CG model (computer graphics model) of the target vehicle's face is created from video of the target vehicle captured by the in-vehicle camera 48 (step S120). Then, vehicle situation data is input (step S130), and a facial image of the target vehicle is input (step S140). The degree of the target vehicle's unpleasant emotion is learned based on the vehicle situation data, the CG model corresponding to the target vehicle's facial image, and the target vehicle's unpleasant emotion (step S150). This learning also includes creating a trained model for selecting a CG model corresponding to the target vehicle's facial image for the vehicle situation data. Examples of vehicle situation data include the vehicle speed V detected by the vehicle speed sensor 40, the vehicle's longitudinal acceleration α detected by the acceleration sensor 42, the gradient θ detected by the gradient sensor 44, the vehicle's lateral acceleration (yaw rate) detected by the yaw rate sensor 46, the weather, the air temperature, the temperature inside the vehicle, the carbon dioxide concentration inside the vehicle, the presence or absence of solar radiation, and the time of day. Machine learning, such as deep learning, can be used for the learning. This learning is repeated until the emotion estimation trained model using the CG model is completed (step S160).

[0034] When the creation of an emotion estimation trained model using a CG model of the subject has been completed, the vehicle situation data and the subject's facial image are input (step S180), and it is determined whether the subject's facial image has been input (step S190). The subject's facial image may not be input when the subject is facing away from the camera, so this is determined.

[0035] If it is determined in step S190 that the facial image of the subject has been input, a CG model corresponding to the input facial image is selected (step S200), and the vehicle situation data and the selected CG model are applied to the emotion estimation trained model to estimate the subject's level of unpleasant emotion (step S210), and this process ends. This allows the subject's level of unpleasant emotion to be more accurately estimated.

[0036] On the other hand, if it is determined in step S190 that the facial image of the subject could not be input, a CG model corresponding to the input vehicle situation data is selected (step S220), and the vehicle situation data and the selected CG model are applied to the emotion estimation trained model to estimate the subject's level of unpleasant emotion (step S230), and this process ends. This makes it possible to more accurately estimate the subject's level of unpleasant emotion even when the facial image of the subject cannot be input.

[0037] In the unpleasant emotion determination device mounted on the automobile 20 according to the embodiment described above, a CG model selected based on vehicle situation data and a facial image of the subject captured by the in-vehicle camera 48 is applied to the emotion estimation trained model to estimate the subject's level of unpleasant emotion. This allows for more accurate determination of whether the occupant is experiencing unpleasant emotion. Furthermore, when the subject's facial image can be input, a CG model corresponding to the input facial image is selected, and the vehicle situation data and the selected CG model are applied to the emotion estimation trained model to estimate the subject's level of unpleasant emotion, thereby allowing for more accurate estimation of the subject's level of unpleasant emotion. Furthermore, when the subject's facial image cannot be input, a CG model corresponding to the input vehicle situation data is selected, and the vehicle situation data and the selected CG model are applied to the emotion estimation trained model to estimate the subject's level of unpleasant emotion. This allows for more accurate estimation of the subject's level of unpleasant emotion even when the subject's facial image cannot be input.

[0038] In the unpleasant emotion determination device mounted on the automobile 20 of the embodiment, the CG model of the subject and the emotion estimation trained model are created by the main electronic control unit 30 mounted on the automobile 20. However, the CG model of the subject and the emotion estimation trained model may also be created by the management center 100, a cloud server, or the like.

[0039] In the embodiment, the unpleasant emotion determination device mounted on the automobile 20 uses an emotion estimation trained model created by the main electronic control unit 30 mounted on the automobile 20, but it is also possible to acquire and use an emotion estimation trained model created by another vehicle, etc. In this case, it is preferable to use, as the emotion estimation trained model, a trained model for each age group, such as children, adults, and the elderly, based on the degree of likelihood of unpleasant emotions, such as people who are prone to motion sickness and people who are not prone to motion sickness.

[0040] The correspondence between the main elements of the embodiment and the main elements of the invention described in the "Means for Solving the Problems" section will be explained below. In the embodiment, the in-vehicle camera 48 and the seat pressure sensor 50 correspond to the "occupant detection device," the passenger speed sensor S40, the acceleration sensor 42, the gradient sensor 44, the yaw rate sensor 46, etc. correspond to the "vehicle situation detection device," the in-vehicle camera 48 corresponds to the "in-vehicle camera," and the main electronic control unit 30 corresponds to the "unpleasant emotion determination device."

[0041] The correspondence between the main elements of the embodiments and the main elements of the invention described in the "Means for Solving the Problem" section does not limit the elements of the invention described in the "Means for Solving the Problem" section, since the embodiments are examples for specifically explaining the mode for implementing the invention described in the "Means for Solving the Problem" section. In other words, the interpretation of the invention described in the "Means for Solving the Problem" section should be based on the description in that section, and the embodiments are merely specific examples of the invention described in the "Means for Solving the Problem" section.

[0042] The present disclosure has been described above using embodiments, but the present disclosure is not limited to these embodiments in any way, and it goes without saying that the present disclosure can be embodied in various forms within the scope that does not deviate from the gist of the present disclosure. [Industrial Applicability]

[0043] The present disclosure can be used in industries such as the manufacturing of in-vehicle unpleasant emotion determination devices. [Explanation of symbols]

[0044] 20 automobile, 30 main electronic control unit, 32 ignition switch, 34 shift position sensor, 36 accelerator position sensor, 38 brake position sensor, 40 vehicle speed sensor, 42 acceleration sensor, 44 gradient sensor, 46 yaw rate sensor, 48 in-car camera, 50 seat pressure sensor, 52 peripheral recognition electronic control unit (peripheral recognition ECU), 53 peripheral recognition device, 54 battery electronic control unit (battery ECU), 55 battery, 56 air conditioning system electronic control unit (air conditioning ECU), 58 air conditioning system, 60 drive system electronic control unit (drive ECU), 62 drive system, 64 brake electronic control unit (brake ECU), 66 brake system, 68 steering electronic control unit (steering ECU), 70 steering system, 72 center display, 78 GPS, 80 navigation system, 82 map information, 84 display unit, 86 communications equipment, 100 control centers;

Claims

1. an occupant detection device that detects the presence of an occupant in the vehicle for each occupant; a vehicle status detection device that detects the status of the vehicle; An in-car camera that captures the passengers inside the car, an unpleasant emotion determination device that determines whether the occupant detected by the occupant detection device is experiencing unpleasant emotions using the vehicle situation and / or the image captured by the in-vehicle camera; The unpleasant emotion determination device for a vehicle, The vehicle status and a CG model created based on the facial expressions of the occupants photographed by the in-vehicle camera are used as input data, and machine learning is used to output whether the occupants are experiencing unpleasant emotions in relation to the vehicle status and / or the facial expressions of the occupants, thereby determining whether the occupants are experiencing unpleasant emotions using an emotion estimation trained model for each occupant. An unpleasant emotion determination device comprising:

2. The unpleasant emotion determination device according to claim 1, When an image of the facial expression of the target occupant is obtained by the in-vehicle camera, a CG model based on the vehicle situation and the facial expression of the occupant is applied to the emotion estimation trained model to determine whether the target occupant is experiencing unpleasant emotions; When an image of the facial expression of the target occupant cannot be obtained by the in-vehicle camera, the vehicle situation is applied to the emotion estimation trained model to determine whether the target occupant is experiencing unpleasant emotions. Unpleasant emotion determination device.

3. The unpleasant emotion determination device according to claim 1, When an emotion estimation trained model corresponding to the target occupant is not obtained, an emotion estimation trained model of the target occupant is created using a CG model created based on the vehicle situation and the facial expression of the target occupant photographed by the in-vehicle camera. Unpleasant emotion determination device.

4. A vehicle equipped with the unpleasant emotion determination device according to any one of claims 1 to 3, determining whether or not an unspecified occupant is experiencing unpleasant emotions using an emotion estimation learned model applied to an unspecified occupant obtained by using a plurality of emotion estimation learned models created by unpleasant emotion determination devices of other vehicles; vehicle.

Images