Vehicle operation control method, program, on-vehicle control device, and on-vehicle system

Abstract

To optimize execution of an operation (for example, notification or information recording) based on an estimated emotion of a driver.SOLUTION: A vehicle operation control method includes steps of: deriving driving behavior information by evaluating a driving behavior of a vehicle by a driver based on vehicle sensor information corresponding to a traveling state of the vehicle (S11); estimating an emotion of the driver based on estimation information (S12); and, when the estimated emotion corresponds to a specific emotion (a negative emotion), controlling execution or non-execution of a specific operation based on a comparison result obtained by comparing the driving behavior information with reference information held in advance (S13 to S19).SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a vehicle operation control method, a program, an in-vehicle control device, and an in-vehicle system. [Background technology]

[0002] A system has been proposed that estimates a user's emotion and provides a notification or the like according to the estimated emotion (see Patent Documents 1 and 2 below). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-173763 [Patent Document 2] Japanese Patent Application Laid-Open No. 2009-134008 Summary of the Invention [Problem to be solved by the invention]

[0004] It is being considered to apply such a system to in-vehicle applications and provide notifications related to driving. For example, when a driver is feeling negative emotions, it is possible to notify the user that they should drive safely or maintain a calm mind. Generally, negative emotions tend to induce dangerous driving, but not all drivers who feel negative emotions will drive dangerously. Some drivers can maintain safe driving even when they feel negative emotions. Providing such a notification to such a driver solely because they are feeling negative emotions would be excessive notification. Excessive notification may cause discomfort to the driver.

[0005] Another possible method is a reference recording method that uses the estimated result of negative emotions as a trigger to determine that dangerous driving is likely to occur and records information about the scene (recording driving behavior, camera images, etc.). The reference recording method can be used to collect information about scenes involving dangerous driving, and can also be used to provide feedback to drivers (fleet drivers, etc.) for safe driving guidance. However, considering that some drivers can maintain safe driving even when harboring negative emotions, the reference recording method also records unnecessary information. This unnecessarily increases the required recording capacity. If the information to be recorded is transmitted to a recording medium via wireless communication, communication costs also unnecessarily increase. Furthermore, there are inconveniences, such as the additional labor required to select the truly necessary information.

[0006] The present invention aims to provide a technique that contributes to optimizing the execution of an action (for example, notification or information recording) based on an estimated emotion of a driver. [Means for solving the problem]

[0007] The vehicle operation control method of the present invention is a vehicle operation control method executed by an on-board control device, which derives driving behavior information by evaluating the driving behavior of the vehicle by the driver of the vehicle based on vehicle sensor information corresponding to the vehicle's driving state, estimates the driver's emotions based on estimation information, and when the estimated emotion corresponds to a specific emotion, compares the driving behavior information with pre-stored reference information and controls whether or not to perform a specific operation based on the comparison result. [Effects of the Invention]

[0008] By comparing driving behavior information based on vehicle sensor information with reference information, it is possible to determine whether the driver's driving behavior is unusual or different from normal. Therefore, with the above method, it is possible to perform a specific action only when the estimated emotion corresponds to a specific emotion and the driver's driving behavior is determined to be unusual or different from normal. This makes it possible to prevent excessive execution of a specific action (excessive notification or unnecessary information recording) and to perform a specific action only when it is truly necessary. In other words, the execution of a specific action is optimized. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram showing the relationship between a user U1 and other components according to an embodiment of the present invention. [Figure 2] 1 is a schematic block diagram of an in-vehicle system according to an embodiment of the present invention. [Figure 3] 1 is a diagram illustrating an internal configuration of an in-vehicle control device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram illustrating an internal configuration of a vehicle sensor unit according to the embodiment of the present invention. [Figure 5] 1 is a diagram illustrating an internal configuration of a biosensor according to an embodiment of the present invention. [Figure 6] 1 is a functional block diagram of an in-vehicle control device according to a first example of an embodiment of the present invention. [Figure 7] 4 is a flowchart showing the operation of a controller in an on-vehicle control device according to a first example of an embodiment of the present invention. [Figure 8] FIG. 1 is a diagram showing the relationship between a plurality of target periods according to a first example pertaining to an embodiment of the present invention. [Figure 9] 10 is a flowchart of a target process according to a first example belonging to an embodiment of the present invention. [Figure 10] 10A to 10C are diagrams illustrating examples of a notification operation and a recording operation according to a first example belonging to an embodiment of the present invention. [Figure 11] FIG. 10 is a functional block diagram of an in-vehicle control device according to a second example of an embodiment of the present invention. [Figure 12] 10 is a flowchart of a target process according to a second example belonging to an embodiment of the present invention. [Figure 13] FIG. 10 is a functional block diagram of an in-vehicle control device according to a third example of an embodiment of the present invention. [Figure 14] FIG. 10 is a diagram showing the relationship between an in-vehicle control device and a server device according to a fourth example of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, examples of embodiments of the present invention will be described in detail with reference to the drawings. In each of the drawings, the same parts are designated by the same reference numerals, and duplicate descriptions of the same parts will be omitted as a general rule. In this specification, for the sake of simplicity, symbols or signs referring to information, signals, physical quantities, functional units, circuits, elements, or components may be used, and the names of the information, signals, physical quantities, functional units, circuits, elements, or components corresponding to the symbols or signs may be omitted or abbreviated.

[0011] FIG. 1 shows the relationship between a user U1 and other components assumed in an embodiment of the present invention. The user U1 is a passenger in a vehicle V1. The user U1 is the driver of the vehicle V1. The vehicle V1 may be any type of vehicle. Here, the vehicle V1 is assumed to be an automobile or the like that runs on a road. An in-vehicle system SYS is mounted on the vehicle V1, and each component of the in-vehicle system SYS is installed in an appropriate position in the vehicle V1.

[0012] A seat ST1 is installed in the cabin of the vehicle V1. A user U1 sits in the seat ST1. Since it is assumed that the user U1 is the driver, the seat ST1 is the driver's seat. Hereinafter, when simply referring to the cabin, unless otherwise specified, this refers to the cabin of the vehicle V1. Furthermore, hereinafter, unless otherwise specified, the inside of the vehicle refers to the internal area of ​​the vehicle V1, and the outside of the vehicle refers to the external area of ​​the vehicle V1.

[0013] The direction from the driver's seat of vehicle V1 toward the steering wheel is defined as "forward," and the direction from the steering wheel of vehicle V1 toward the driver's seat is defined as "rearward." The direction perpendicular to the fore-and-aft direction and parallel to the road surface on which vehicle V1 is traveling is defined as the left-right direction. User U1 sits in seat ST1 facing forward. The fore-and-aft direction and left-and-right direction correspond to the fore-and-aft direction and left-and-right direction as seen from user U1's perspective. Unless otherwise specified below, vehicle V1 is assumed to be located on a horizontal road surface, and the traveling direction of vehicle V1 is assumed to be forward.

[0014] 2 shows a schematic block diagram of the in-vehicle system SYS. The in-vehicle system SYS includes an in-vehicle control device 10, a cruise control device 20, an actuator unit 30, a vehicle sensor unit 40, a biosensor 60, an exterior camera 71, an interior camera 72, a recording medium 73, and an HMI 80. The components of the in-vehicle system SYS can exchange signals and information with each other through an in-vehicle network formed within the vehicle V1. The in-vehicle network includes, for example, a CAN (Controller Area Network) and an AVCLAN (Audio Visual Communication Local Area Network).

[0015] The in-vehicle control device 10 evaluates the driving behavior of the user U1 and estimates the emotions of the user U1, and can perform various controls based on the evaluation results. These various controls will be described in detail later. The driving control device 20 controls the driving of the vehicle V1 using an actuator unit 30. The actuator unit 30 has various driving components such as a motor that realizes the driving of the vehicle V1. Specifically, the actuator unit 30 includes an engine and a motor that generate driving force for the vehicle V1, a steering actuator that drives the steering mechanism of the vehicle V1, and a brake actuator that drives the brakes of the vehicle V1.

[0016] The vehicle sensor unit 40 has sensors that detect the details of the driving operation of the vehicle V1 by the driver of the vehicle V1 and sensors that detect various states of the vehicle V1. The vehicle sensor unit 40 outputs vehicle sensor information containing these detection results. The driving control device 20 realizes driving control of the vehicle V1 by driving and controlling the actuator unit 30 in accordance with the vehicle sensor information. At this time, the driving control device 20 can also perform driving control in accordance with the emotion estimation result. The biometric sensor 60 detects biometric information of the user U1. The emotion of the user U1 may be estimated based on the detection result of the biometric information by the biometric sensor 60.

[0017] The exterior camera 71 consists of one or more cameras that capture images of the scene outside the vehicle V1. The exterior camera 71 has a capture area set outside the vehicle V1 and generates an exterior camera image by capturing images of the scene within the capture area. The exterior camera image is an image captured by the capture area of ​​the exterior camera 71. Image information showing the exterior camera image is called exterior image information. The exterior camera 71 captures images at a predetermined frame rate. The capture area of ​​the exterior camera 71 includes all or part of the front area, rear area, right side area, and left side area of ​​the vehicle V1. The front area, rear area, right side area, and left side area of ​​the vehicle V1 are areas located in front, rear, right side, and left side of the vehicle V1, respectively, within the external area of ​​the vehicle V1.

[0018] The in-vehicle camera 72 consists of one or more cameras that capture images of the interior of the vehicle V1. The in-vehicle camera 72 has a capture area set inside the vehicle V1 (i.e., the interior of the vehicle V1), and generates an in-vehicle camera image by capturing images of the capture area. The in-vehicle camera image is an image captured in the capture area by the in-vehicle camera 72. Image information showing the in-vehicle camera image is called in-vehicle image information. The in-vehicle camera 72 captures images at a predetermined frame rate. User U1 is located within the capture area of ​​the in-vehicle camera 72, and therefore the in-vehicle camera image includes an image of user U1. Here, it is assumed that the face of user U1 is included in the capture area of ​​the in-vehicle camera 72, and therefore the in-vehicle camera image includes a facial image of user U1 (a captured image of user U1's face).

[0019] The recording medium 73 is a nonvolatile recording medium such as a magnetic disk or flash memory, and stores any information in a nonvolatile manner. The recording medium 73 may be a recording medium in an external device (for example, a server device connected to the Internet) provided outside the vehicle V1. The recording medium 73 may also be built into the on-vehicle control device 10.

[0020] The HMI 80 is a human-machine interface. The HMI 80 is provided with a display device 81, a speaker 82, and a microphone 83. The display device 81 has a display screen such as a liquid crystal display panel, and displays any image under the control of the in-vehicle control device 10, the cruise control device 20, or a display control device (not shown). The display device 81 is installed at an appropriate location in the cabin of the vehicle V1 so that each occupant of the vehicle V1 can see the display content of the display device 81. Multiple display devices 81 may be installed in the cabin of the vehicle V1. The display device 81 may be a component of a car navigation system installed in the vehicle V1. The car navigation system may be included in the in-vehicle system SYS. The speaker 82 outputs any sound (message, warning sound, music, etc.) under the control of the in-vehicle control device 10, the cruise control device 20, or an audio device (not shown). The speaker 82 is installed at an appropriate location in the cabin of the vehicle V1 so that each occupant of the vehicle V1 can hear the output sound of the speaker 82. A plurality of speakers 82 may be installed in the cabin of vehicle V1. Microphone 83 picks up ambient sounds around its installation position and converts them into an electrical audio signal. The audio signal obtained by conversion by microphone 83 is called a microphone signal. Microphone 83 is installed in an appropriate location in the cabin of vehicle V1 so that the signal components of the speech sounds of each occupant of vehicle V1 are included in the microphone signal (i.e., so that the speech sounds are included in the content picked up by microphone 83). A plurality of microphones 83 may be installed in the cabin of vehicle V1. In addition, HMI 80 may be provided with a vibration device that applies vibrations to occupants (particularly the driver) of vehicle V1, an operation input device that accepts arbitrary operations (including voice operations) from occupants of vehicle V1, and the like.

[0021] 3 shows the internal configuration of the vehicle-mounted control device 10. The vehicle-mounted control device 10 includes a controller 11, a memory 12, and a communication unit 13.

[0022] The controller 11 includes, as hardware resources, a processing unit including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), etc. The controller 11 may implement any function, operation, or process that should be implemented by the controller 11 by executing a program recorded in the memory 12 or any other recording medium.

[0023] The memory 12 is configured to include a non-volatile memory such as a ROM (Read Only Memory) or a flash memory, and a volatile memory such as a RAM (Random Access Memory). The memory 12 stores various data referenced by the controller 11 as well as various programs to be executed by the controller 11.

[0024] The communication unit 13 is a communication circuit (communication module) that transmits and receives any signal between the in-vehicle control device 10 and a counterpart device different from the in-vehicle control device 10. The counterpart device for the communication unit 13 includes components other than the in-vehicle control device 10 among the components of the in-vehicle system SYS shown in FIG. 2. The communication unit 13 can communicate with the counterpart device via an in-vehicle network formed in the vehicle V1. The counterpart device for the communication unit 13 may further include an external device (for example, a server device connected to the Internet) provided outside the vehicle V1. Note that the controller 11 can transmit and receive any information to and from the counterpart device using the communication unit 13, but hereinafter, a description of the communication unit 13 may be omitted.

[0025] FIG. 4 shows an internal block diagram of the vehicle sensor unit 40. The vehicle sensor unit 40 includes sensors 41 to 47. The sensors 41, 42, and 43 are an accelerator pedal sensor, a brake pedal sensor, and a steering sensor (handle sensor), respectively. The vehicle V1 is provided with operational components that receive driving operations from the driver, and the operational components include an accelerator pedal, a brake pedal, and a steering wheel (handle). The sensors 44, 45, 46, and 47 are a vehicle speed sensor, a steering angle sensor, a G sensor, and a GPS sensor, respectively.

[0026] The accelerator pedal sensor 41 detects the operation of the accelerator pedal of the vehicle V1 by the driver of the vehicle V1, and generates and outputs accelerator pedal operation information indicating the operation of the accelerator pedal. The brake pedal sensor 42 detects the operation of the brake pedal of the vehicle V1 by the driver of the vehicle V1, and generates and outputs brake pedal operation information indicating the operation of the brake pedal. The steering sensor 43 detects the operation of the steering wheel of the vehicle V1 by the driver of the vehicle V1, and generates and outputs steering operation information indicating the operation of the steering wheel.

[0027] The vehicle speed sensor 44 detects the speed of the vehicle V1 and generates and outputs vehicle speed information (vehicle speed pulses) representing the detected speed. The steering angle sensor 45 detects the steering angle (steering angle) of the vehicle V1 and generates and outputs steering angle information representing the detected steering angle. The G sensor 46 detects acceleration acting on the vehicle V1 in a predetermined axial direction and generates and outputs the acceleration detection result as acceleration information. The G sensor 46 may detect acceleration in two mutually orthogonal axial directions or may detect acceleration in three mutually orthogonal axial directions. The GPS sensor 47 receives signals from multiple GPS satellites that form the GPS (Global Positioning System) and generates and outputs vehicle position information based on the received signals. The vehicle position information generated by the GPS sensor 47 represents the current location (current position) of the vehicle V1 using longitude and latitude, or represents the current location of the vehicle V1 using longitude, latitude, and altitude.

[0028] The vehicle sensor information generated by the vehicle sensor unit 40 is information corresponding to the traveling state of the vehicle V1 and includes output information from each sensor within the vehicle sensor unit 40. Therefore, the vehicle sensor information includes accelerator pedal operation information, brake pedal operation information, steering operation information, vehicle speed information, steering angle information, acceleration information, and vehicle position information. However, it is possible that any of this information is not included in the vehicle sensor information. Each sensor within the vehicle sensor unit 40 periodically updates the information it should generate, and the latest vehicle sensor information is sequentially output from the vehicle sensor unit 40. Sensors other than sensors 41 to 47 (for example, distance measurement sensors, temperature sensors, rain sensors, illuminance sensors, shift lever sensors, and door lock sensors) may also be provided in the vehicle sensor unit 40.

[0029] FIG. 5 shows the configuration of the biosensor 60. The biosensor 60 is worn by the user U1 to acquire biometric information from the user U1. The biosensor 60 includes at least an electroencephalogram (EEG) sensor 61 and a heartbeat sensor 62, and the EEG sensor 61 and the heartbeat sensor 62 are worn by the user U1. The EEG sensor 61 detects the brain waves of the user U1 and outputs EEG data indicating the results of the brainwave detection. In other words, the EEG data includes information on the detected brain waves. The heartbeat sensor 62 detects the heartbeat of the user U1 and outputs heartbeat data indicating the results of the heartbeat detection. In other words, the heartbeat data includes information on the detected heartbeat. For example, a headgear-type EEG sensor is used as the EEG sensor 61. For example, a chest belt-type electrocardiogram heartbeat sensor is used as the heartbeat sensor 62. An optical heartbeat sensor can also be used as the heartbeat sensor 62.

[0030] Data obtained from the biosensor 60 worn by the user U1 is referred to as biometric data. The biometric data includes brain wave data and heart rate data, and represents biometric information of the user U1. Depending on the type of biometric information to be acquired or the wearability, other sensors may be included in the biosensor 60 and worn by the user U1. Examples of other sensors include a sweat sensor, a body temperature sensor, a blood pressure monitor, and a NIRS (Near Infrared Spectroscopy) device.

[0031] Based on the estimation result of the emotion of the user U1 (driver), the controller 11 (see FIG. 3) can use the HMI 80 to give an appropriate notification to the user U1 or record necessary information in the recording medium 73. In this case, the controller 11 operates to suppress excessive notification or recording, etc., and to execute them only when truly necessary.

[0032] Below, among the multiple embodiments, several specific operation examples, application techniques, modified techniques, etc. will be described, focusing particularly on the operation of the controller 11. The matters described above in this embodiment are applied to each of the following embodiments unless otherwise specified and unless there is a contradiction. If there are any matters in each embodiment that contradict the matters described above, the description in each embodiment may take precedence. Furthermore, unless there is a contradiction, the matters described in any of the multiple embodiments shown below can also be applied to any of the other embodiments (i.e., any two or more of the multiple embodiments can be combined).

[0033] <<First Example>> A first embodiment will be described. FIG. 6 is a functional block diagram of an on-board control device 10 according to the first embodiment. The on-board control device 10 according to the first embodiment has functional blocks F1 to F6, H1, and H2. The functional blocks F1 to F6 are provided in a controller 11. The controller 11 is a program execution device (computer) capable of executing any program. All or part of the functions of the functional blocks F1 to F6 may be realized by the controller 11 executing a program recorded in the memory 12 or any other recording medium. The functional block H1 is a reference information storage unit that stores reference information REF, and the functional block H2 is a judgment threshold storage unit that stores a judgment threshold TH. The functional blocks H1 and H2 are provided in a non-volatile memory in the memory 12. However, the functional blocks H1 and H2 made of non-volatile memory may be built into the controller 11. The functional blocks F1 to F6 will be described in detail below.

[0034] ---Driving behavior evaluation section F1--- The functional block F1 is a driving behavior evaluation unit. Vehicle sensor information is input to the driving behavior evaluation unit F1 from the vehicle sensor unit 40. The driving behavior evaluation unit F1 executes a driving behavior evaluation process to evaluate the driving behavior of the user U1 (and therefore the driver) based on the vehicle sensor information. The driving behavior of the user U1 is the behavior performed by the user U1 to drive the vehicle V1. The operations input by the user U1 to the accelerator pedal, brake pedal, and steering of the vehicle V1 correspond to the driving behavior of the user U1, and the results of the driving behavior of the user U1 are reflected in the vehicle speed information, steering angle information, and acceleration information (see Figure 4). Therefore, the driving behavior of the user U1 (and therefore the driver) can be evaluated based on the vehicle sensor information.

[0035] The driving behavior evaluation unit F1 generates and outputs driving behavior information DA indicating the evaluation result of the driving behavior. An evaluation period having a length equivalent to a predetermined evaluation time is set in the controller 11. The driving behavior evaluation unit F1 generates driving behavior information DA for the evaluation period by evaluating the driving behavior of the user U1 during the evaluation period based on vehicle sensor information during the evaluation period. The driving behavior information DA includes, for example, all or part of vehicle speed statistical information, steering angle statistical information, acceleration statistical information, and deceleration statistical information. In the following description, it is assumed that the driving behavior information DA includes vehicle speed statistical information, steering angle statistical information, acceleration statistical information, and deceleration statistical information. In the following description, vehicle speed refers to the speed of the vehicle V1, and steering angle refers to the steering angle (steering angle) of the vehicle V1.

[0036] The vehicle speed statistical information represents the statistical quantity of the vehicle speed during the evaluation period. The statistical quantity of the vehicle speed during the evaluation period includes the total sum of the change in the vehicle speed during the evaluation period, as well as the average, maximum, minimum and variance of the vehicle speed during the evaluation period. The driving behavior evaluation unit F1 can derive the vehicle speed statistical information based on the vehicle speed information during the evaluation period. The period P from when the vehicle speed is zero to when the vehicle speed increases and decreases and returns to zero again is A1 may be included in the evaluation period or may be set as the evaluation period. A1 The average and maximum vehicle speeds during the period are displayed.

[0037] The steering angle statistical information represents the statistical quantity of the steering angle during the evaluation period. The statistical quantity of the steering angle during the evaluation period includes the sum of the amount of change in the steering angle during the evaluation period, as well as the average, maximum, minimum and variance of the steering angle during the evaluation period. The driving behavior evaluation unit F1 can derive the steering angle statistical information based on the steering angle information during the evaluation period. The period P from when the steering angle is zero to when the steering angle returns to zero again after increasing and decreasing is A2 may be included in the evaluation period or may be set as the evaluation period. In this case, the period P A2 The average value and maximum value of the steering angle during the period are shown.

[0038] The acceleration statistical information is based on the operation of the accelerator pedal by the user U1, and is an acceleration period P A3 The second velocity is greater than the first velocity. The first velocity may be zero. A3 may be included in or set as the evaluation period. A3 The statistics of the acceleration of the vehicle V1 during the acceleration period P A3 The driving behavior evaluation unit F1 calculates the average, maximum, minimum and variance of the acceleration of the vehicle V1 during the acceleration period P A3 Based on the vehicle speed information or acceleration information during the driving, acceleration statistical information can be derived. The acceleration statistical information indicates the acceleration state of the vehicle V1 (whether sudden acceleration has occurred, etc.).

[0039] The deceleration statistical information is a deceleration period P during which the vehicle speed monotonically decreases from the third speed to the fourth speed based on the operation of the brake pedal by the user U1. A4 The fourth speed represents a statistical quantity of the acceleration of the vehicle V1 during the deceleration period P. The fourth speed is smaller than the third speed. The fourth speed may be zero. A4 may be included in the evaluation period or may be set as the evaluation period. A4 The statistics of the acceleration of the vehicle V1 during the deceleration period P A4The driving behavior evaluation unit F1 calculates the average, maximum, minimum and variance of the acceleration of the vehicle V1 during the deceleration period P A4 The deceleration statistical information can be derived based on the vehicle speed information or acceleration information during the deceleration. The deceleration statistical information indicates the deceleration state of the vehicle V1 (whether or not sudden deceleration has occurred, etc.).

[0040] Here, the driving behavior information DA is calculated as an n-dimensional vector quantity. n represents an integer equal to or greater than 2, and is generally assumed to be sufficiently greater than 2. For example, consider a case where the average, maximum, minimum and variance of the vehicle speed during the evaluation period are included in the driving behavior information DA. In this case, at least "n≧4" holds, and the driving behavior information DA is a vector quantity having the average, maximum, minimum and variance of the vehicle speed as four components. For example, during the acceleration period P A3 is included in the evaluation period, and the acceleration period P A3 Consider a case where the driving behavior information DA includes the average, maximum, minimum and variance of the acceleration of the vehicle V1 during the acceleration period P. In this case, at least "n≧4" is satisfied, and the driving behavior information DA is included in the driving behavior information DA during the acceleration period P. A3 is a vector quantity having the average value, maximum value, minimum value, and variance of the acceleration of vehicle V1 within the vehicle V1 as four components.

[0041] ---Emotion estimation part F2--- The functional block F2 is a feeling estimation unit. Information used for estimating the feeling of the user U1 is input to the feeling estimation unit F2 as estimation information. The feeling estimation unit F2 executes a feeling estimation process to estimate the feeling of the user U1 (and therefore the driver) based on the estimation information.

[0042] The emotion of user U1 estimated by the emotion estimation unit F2 is referred to as estimated emotion EM. The emotion estimation unit F2 generates and outputs information indicating the estimated emotion EM. An estimation period having a length equivalent to a predetermined estimation time is set in the controller 11. The emotion estimation unit F2 identifies the estimated emotion EM for the estimation period by estimating the emotion of user U1 during the estimation period based on the estimation information for the estimation period.

[0043] The emotion estimation unit F2 can perform emotion estimation processing using biometric data obtained from the biometric sensor 60 as estimation information. Known methods can be used to estimate emotion based on biometric data. For example, the method described in International Publication No. 2023 / 127930 proposed by the applicant of the present application can be used. In this case, the emotion estimation unit F2 derives two emotion indices for user U1 based on the biometric data of user U1. One of the emotion indices is the arousal level of the central nervous system (hereinafter referred to as arousal level), and the other is the activity level of the autonomic nervous system (hereinafter referred to as activity level). The emotion estimation unit F2 derives the arousal level of user U1 based on the electroencephalogram data of user U1 and the activity level of user U1 based on the heart rate data of user U1. The emotion estimation unit F2 is provided with an emotion estimation model configured with a calculation formula or conversion data table that identifies emotions from the arousal level and activity level. The emotion estimation unit F2 can estimate the emotion of user U1 by inputting the arousal level and activity level derived as described above into the emotion estimation model. The emotion estimation model is a two-dimensional model with arousal and activity as its two axes, and is created based on medical evidence showing the relationship between arousal and activity and emotions.

[0044] The emotion estimation unit F2 may estimate the emotion of the user U1 in detail based on the facial expression of the user U1 in addition to the biometric data. The emotion estimation unit F2 can detect the facial expression of the user U1 using image recognition based on the facial image of the user U1 included in the in-car camera image. In detecting the facial expression, it is detected which of a plurality of typical facial expressions the facial expression of the user U1 belongs to. The multiple typical facial expressions include a facial expression of joy, a facial expression of anger, a facial expression of sadness, a facial expression of fear, and a facial expression of surprise.

[0045] The controller 11 may be provided with an AI model capable of estimating the user U1's brain wave data and heart rate data from the user U1's facial image. In this case, the brain wave sensor 61 and heart rate sensor 62 can be omitted from the in-vehicle system SYS, and the brain wave data and heart rate data of the user U1 can be estimated by inputting an in-vehicle camera image (in-vehicle image information) to the AI ​​model. The emotion estimation unit F2 can then estimate the user U1's emotion by deriving an emotion index based on the estimated brain wave data and heart rate data. Note that in this specification, AI is an abbreviation for artificial intelligence.

[0046] The emotion estimation method using the derivation of the arousal level and activity level described above is merely one example of an emotion estimation method that the emotion estimation unit F2 can use. The emotion estimation unit F2 can estimate the emotion of user U1 based on estimation information including at least one of biometric data and in-car camera images (face images of user U1). The emotion estimation unit F2 may estimate the emotion of user U1 by further using a microphone signal including a signal component of the user U1's speech sound. The emotion estimation unit F2 may estimate the emotion of user U1 using a known Ekman emotion estimation model based on the in-car camera images. The emotion estimation unit F2 may estimate the emotion of user U1 using any of the methods described in Non-Patent Documents 1 to 4 below. Non-patent literature 1: [online], Hiroshima University, [searched January 8, 2024], Internet<URL:https: / / www.hiroshima-u.ac.jp / news / 57965> Non-patent document 2: [online], Shimadzu Corporation, [searched January 8, 2024], Internet<URL:https: / / www.shimadzu.co.jp / news / press / iynnvrhbb7hb1de-.html> Non-patent document 3: [online], NEC Corporation, [searched January 8, 2024], Internet<URL:https: / / jpn.nec.com / press / 201806 / 20180611_01.html> Non-patent literature 4: [online], Paul Ekman Group, [searched January 8, 2024], Internet<URL:https: / / www.paulekman.com / >

[0047] The emotion of the user U1 estimated by the emotion estimation unit F2 may be one of a plurality of psychological states, including, for example, joy, happiness, anger, sadness, depression, relaxation, calmness, anxiety, fear, and displeasure.

[0048] The emotion estimation unit F2 can assign and set an emotion value to the estimated emotion EM according to the emotion estimation result, and output the estimated emotion EM with the assigned emotion value. Here, the emotion value has a value of 1, 2, or 3. The emotion estimation unit F2 can classify the estimated emotion into one of negative emotion (hereinafter referred to as negative emotion), positive emotion (hereinafter referred to as positive emotion), and other emotions (hereinafter referred to as neutral emotion). The emotion estimation unit F2 assigns and sets an emotion value indicating the classification result to the estimated emotion EM.

[0049] When the emotion estimation unit F2 estimates that the emotion of user U1 corresponds to a psychological state of anger or displeasure, the emotion of user U1 is classified as a negative emotion (i.e., corresponds to a negative emotion). When the emotion of user U1 corresponds to a negative emotion, the emotion estimation unit F2 adds and sets an emotion value of "1" to the estimated emotion EM.

[0050] When the emotion estimation unit F2 estimates that the emotion of user U1 corresponds to a psychological state of fun, joy, or relaxation, the emotion of user U1 is classified as a positive emotion (i.e., corresponds to a positive emotion). When the emotion of user U1 corresponds to a positive emotion, the emotion estimation unit F2 adds and sets an emotion value of "2" to the estimated emotion EM.

[0051] The emotion estimation unit F2 may determine that the emotion of user U1 does not belong to any of the above psychological states. In such cases, an emotion value of "3," indicating that emotion estimation is not possible or that there is no estimated emotion, is added and set to the estimated emotion EM. The emotion value of "3" corresponds to a neutral emotion. In other words, when the emotion estimated by the emotion estimation unit F2 corresponds to a neutral emotion (when classified as a neutral emotion), an emotion value of "3" is set to the estimated emotion EM.

[0052] In this way, the emotion deduction unit F2 determines whether the emotion of user U1 corresponds to a negative emotion based on the estimation information. If the emotion of user U1 corresponds to a negative emotion, the emotion deduction unit F2 outputs an emotion value of "1," and if the emotion of user U1 does not correspond to a negative emotion, the emotion deduction unit F2 outputs an emotion value of "2" or "3."

[0053] ---Selector F3--- The functional block F3 is a selector. Driving behavior information DA and estimated emotion EM are input to the selector F3. The estimated emotion EM input to the selector F3 is added with the emotion value described above. Only the emotion value may be input to the selector F3 from the emotion estimation unit F2. The selector F3 determines whether or not to input the driving behavior information DA from the driving behavior evaluation unit F1 to the functional block F4, depending on the emotion value. The selector F3 inputs the driving behavior information DA to the functional block F4 only when the emotion value is "1."

[0054] ---Difference derivation part F4--- The functional block F4 is a deviation degree derivation unit. When the driving behavior information DA is input to the deviation degree derivation unit F4 from the selector F3, the deviation degree derivation unit F4 executes the deviation degree derivation process. When the driving behavior information DA is not input to the deviation degree derivation unit F4 from the selector F3, the deviation degree derivation process is not executed. Therefore, the deviation degree derivation process is executed only when the estimated emotion EM of the user U1 corresponds to a negative emotion.

[0055] In the deviation degree derivation process, the deviation degree derivation unit F4 refers to the reference information REF stored in the reference information storage unit H1, and compares the driving behavior information DA with the reference information REF to derive the deviation degree DEV therebetween.

[0056] The reference information REF indicates reference driving behavior information and corresponds to the driving behavior information DA when the user U1 has a reference specific emotion. Alternatively, the reference information REF simulates the driving behavior information DA when the user U1 has a reference specific emotion. The reference specific emotion is a positive emotion. Driving behavior information DA when the user U1 has a reference specific emotion may be collected in advance, and the reference information REF may be generated from the collected driving behavior information DA and stored in advance in the reference information storage unit H1. Alternatively, an arbitrary subject may drive the vehicle V1 or another vehicle, and driving behavior information DA when the subject has a reference specific emotion may be collected in advance. In this case, the reference information REF may also be generated from the collected driving behavior information DA and stored in advance in the reference information storage unit H1. In either case, the reference information REF is generated and stored in the reference information storage unit H1 before the deviation degree derivation process is performed. The reference specific emotion may be a neutral emotion. Alternatively, both positive emotions and neutral emotions may be used as the reference specific emotions.

[0057] The deviation DEV is the difference (discrepancy) between the driving behavior information DA and the reference information REF, and may be the absolute value of the difference between the driving behavior information DA and the reference information REF. When the driving behavior information DA and the reference information REF completely match, the deviation DEV is zero. When the driving behavior information DA and the reference information REF do not match, the deviation DEV has a positive value, and the deviation DEV increases as the difference between the driving behavior information DA and the reference information REF increases.

[0058] The reference information REF is also an n-dimensional vector quantity, similar to the driving behavior information DA, so that the difference between the driving behavior information DA and the reference information REF can be defined. When the vectors of the driving behavior information DA and the reference information REF are arranged in an n-dimensional vector space, the Euclidean distance or cosine similarity between these vectors can be obtained as the divergence degree DEV.

[0059] ---Comparison unit F5--- The functional block F5 is a comparison unit. The comparison unit F5 compares the divergence degree DEV derived by the divergence degree derivation unit F4 with the determination threshold TH held by the determination threshold holding unit H2, and generates and outputs a comparison result signal CMP according to their magnitude relationship. The determination threshold TH has a predetermined value greater than 0. The comparison result signal CMP has a value of "0" or "1".

[0060] When "DEV>TH" holds, "CMP = 1", and when "DEV<TH" holds, "CMP = 0". That is, the comparison result signal CMP has a value of "1" when the divergence degree DEV is greater than the determination threshold TH, and has a value of "0" when the divergence degree DEV is less than the determination threshold TH. When "DEV = TH" holds, the value of the comparison result signal CMP may be "0" or "1".

[0061] The user U1 basically performs driving behaviors that focus on safe driving to avoid accidents. During the period when driving behaviors that focus on safe driving are performed, it is expected that the driving behavior information DA will match or be similar to the reference information REF, so "DEV<TH" is likely to hold. Driving behaviors that focus on safe driving can be said to be normal driving behaviors. Then, a comparison result signal CMP of "0" indicates that the user U1 has performed normal driving behaviors, and a comparison result signal CMP of "1" indicates that the user U1 has performed special driving behaviors different from normal driving behaviors. Special driving behaviors are more dangerous than normal driving behaviors. The comparison unit F5 has the function of comparing the divergence degree DEV with the determination threshold TH to determine whether the user U1 has performed special driving behaviors, and reflecting the determination result in the comparison result signal CMP.

[0062] For example, when the vehicle V1 is driven roughly, such as by sudden acceleration or deceleration, the deviation DEV increases, making it easier to derive a comparison result signal CMP of "1." Also, when the vehicle V1 is driven unsteadily or changes lanes frequently, the total amount of change in the steering angle or the variance of the steering angle becomes excessively large, making it easier to derive a comparison result signal CMP of "1."

[0063] ---Specific operation control section F6--- The functional block F6 is a specific operation control unit. The specific operation control unit F6 controls whether or not to execute the specific operation OP depending on the comparison result signal CMP output from the comparison unit F5. The in-vehicle system SYS is provided with a functional unit capable of executing the specific operation OP. The specific operation control unit F6 causes the functional unit to execute the specific operation OP when it receives a comparison result signal CMP of "1". The specific operation control unit F6 prevents the functional unit from executing the specific operation OP when the value of the comparison result signal CMP is "0". For example, the specific operation control unit F6 sends an instruction signal to the functional unit instructing it to execute the specific operation OP when "CMP=1", and does not send the instruction signal to the functional unit when "CMP=0".

[0064] The specific action OP may include an announcing action OP1 that uses an announcing means to notify the user U1 of some kind. The announcing means is a functional unit that executes the announcing action OP1, and includes at least one of a display device 81 or a speaker 82. The announcing by the display device 81 is an announcing that appeals to the user U1's visual sense, and the announcing by the speaker 82 is an announcing that appeals to the user U1's auditory sense. A vibration device that applies vibrations to the user U1 may be included in the HMI 80 (for example, the vibration device may be provided in the seat ST1), in which case the vibration device may be included in the announcing means. An announcing by the vibration device is an announcing that appeals to the user U1's tactile sense.

[0065] The specific action OP may include, instead of or in addition to the announcing action OP1, a recording action OP2 for recording information to be recorded onto the recording medium 73. The recording medium 73 is a functional unit that executes the recording action OP2. The information to be recorded includes at least one of driving behavior information DA and estimated emotion EM (information indicating the estimated emotion EM). The information to be recorded may further include vehicle exterior image information and vehicle interior image information (see FIG. 2) during a period corresponding to the driving behavior information DA and estimated emotion EM.

[0066] ---Operation flowchart--- FIG. 7 shows an operation flowchart of the controller 11. Each process of steps S1 to S4 shown in FIG. 7 is executed by the controller 11. When the vehicle V1 starts, the controller 11 also starts, and the operation of the controller 11 starts from the process of step S1. In step S1, the controller 11 executes a predetermined initialization process related to startup, and during the initialization process, assigns 1 to a variable i that it manages. The variable i has an integer value. After step S1, the process proceeds to step S2.

[0067] After starting up, the controller 11 can sequentially set target periods. Of the target periods that are sequentially set, the ith target period is referred to as the ith target period. In step S2, the controller 11 sets the ith target period, and in the following step S3, the controller 11 executes target processing Q for the ith target period. Thereafter, in step S4, the controller 11 adds 1 to the variable i, and then the process returns to step S2, and the processes from step S2 onwards are repeated.

[0068] Here, each target period has a predetermined time duration, and the lengths of the multiple target periods are the same. However, the lengths of the multiple target periods may vary dynamically. For at least any integer i, the start time of the (i+1)th target period is later than the start time of the ith target period, and the end time of the (i+1)th target period is later than the end time of the ith target period. As shown in FIG. 8(a), the multiple target periods may be non-overlapping periods, in which case the start time of the (i+1)th target period occurs after the end of the ith target period. The end time of the ith target period and the start time of the (i+1)th target period may be the same. Alternatively, as shown in FIG. 8(b), the multiple target periods may overlap each other. In the example of FIG. 8(b), the ith target period and the (i+1)th target period partially overlap each other.

[0069] Fig. 9 shows a flowchart of target process Q according to the first embodiment. Target process Q shown in Fig. 9 is made up of steps S11 to S19. In target process Q, first, controller 11 executes steps S11 and S12. In Fig. 9, it is assumed that step S12 is executed after step S11, but the order in which steps S11 and S12 are executed is arbitrary, and they may be executed simultaneously.

[0070] In step S11, the driving behavior evaluation unit F1 sets the i-th target period as an evaluation period and executes the above-described driving behavior evaluation process for the i-th target period. That is, in step S11, the driving behavior evaluation unit F1 generates driving behavior information DA for the i-th target period by evaluating the driving behavior of the user U1 (and therefore the driver) based on vehicle sensor information for the i-th target period. The driving behavior information DA for the i-th target period is particularly referred to by the symbol "DA[i]." The driving behavior information DA[i] reflects the driving behavior performed by the user U1 in the i-th target period.

[0071] In step S12, the emotion deduction unit F2 sets the i-th target period as the estimation period and executes the emotion deduction process described above for the i-th target period. That is, in step S12, the emotion deduction unit F2 derives (specifies) an estimated emotion EM for the i-th target period by estimating the emotion of user U1 based on the estimation information for the i-th target period. The estimated emotion EM for the i-th target period is particularly referred to by the symbol "EM[i]." The estimated emotion EM[i] is an estimation of the emotion of user U1 for the i-th target period.

[0072] After steps S11 and S12, the process proceeds to step S13. In step S13, the controller 11 determines whether the estimated emotion EM[i] corresponds to a negative emotion. If the estimated emotion EM[i] corresponds to a negative emotion (Y in step S13), the process proceeds from step S13 to step S14. If the estimated emotion EM[i] does not correspond to a negative emotion (N in step S13), the controller 11 terminates the target process Q without performing the processes of steps S14 to S19. In the configuration example of FIG. 6, the process of step S13 is realized using a selector F3. The selector F3 inputs the driving behavior information DA[i] to the deviation degree derivation unit F4 only if the emotion value corresponding to the estimated emotion EM[i] is "1" (i.e., only if the estimated emotion EM[i] is a negative emotion). The process of step S14 is executed by inputting the driving behavior information DA[i] to the deviation degree derivation unit F4.

[0073] In step S14, the deviation degree derivation unit F4 performs a deviation degree derivation process to compare the input driving behavior information DA[i] with reference information REF, which is reference driving behavior information, and derive the deviation degree DEV therebetween. The details of the deviation degree derivation process are as described above. The deviation degree DEV between the driving behavior information DA[i] and the reference information REF is particularly referred to by the symbol "DEV[i]". After step S14, the process proceeds to step S15.

[0074] In step S15, the comparison unit F5 compares the deviation DEV[i] with the judgment threshold TH to determine whether "DEV[i]>TH" is true. If "DEV[i]>TH" is true (Y in step S15), the process proceeds to step S16. If the process proceeds to step S16, the processes of steps S16 and S17 are executed. If "DEV[i]≦TH" is true (N in step S15), the process proceeds to step S18. If the process proceeds to step S18, the processes of steps S18 and S19 are executed. Note that the process may proceed to step S16 when "DEV[i]=TH" is true. The comparison result signal CMP indicating the comparison result between the deviation DEV[i] and the judgment threshold TH is specifically referred to by the symbol "CMP[i]".

[0075] In step S16, the comparison unit F5 determines that the user U1 has performed an unusual driving behavior in the i-th target period and outputs a comparison result signal CMP[i] of "1." The comparison result signal CMP[i] of "1" indicates that the user U1 has performed an unusual driving behavior in the i-th target period. Upon receiving the comparison result signal CMP[i] of "1," in step S17, the specific operation control unit F6 causes the function unit to execute a specific operation OP. The specific operation OP in step S17 includes at least one of the above-mentioned notification operation OP1 and recording operation OP2.

[0076] In step S18, the comparison unit F5 determines that the user U1 performed normal driving behavior in the i-th target period and outputs a comparison result signal CMP[i] of "0." The comparison result signal CMP[i] of "0" indicates that the user U1 performed normal driving behavior in the i-th target period. Upon receiving the comparison result signal CMP[i] of "0," in step S19, the specific operation control unit F6 prevents the function unit from executing the specific operation OP (in practice, it is sufficient to simply not instruct the execution of the specific operation OP). In other words, if the process proceeds to step S19, the specific operation OP is not executed. The target process Q ends via step S18 or S19.

[0077] The following is a supplementary explanation of the notification operation OP1 that can be executed in step S17. In the notification operation OP1 in step S17, the specific operation control unit F6, for example, displays a notification message encouraging safe driving on the display device 81 or outputs a sound from the speaker 82. The specific operation control unit F6 can set the content of the notification message according to the driving behavior information DA[i] (i.e., the notification operation OP1 can notify the user U1 of information according to the driving behavior information DA[i]). For example, if the acceleration period P A3 and the acceleration period P A3 When the average or maximum value of the acceleration of the vehicle V1 during the i-th target period is equal to or greater than a predetermined sudden acceleration threshold, a sudden acceleration message is set as the notification message. The sudden acceleration message includes, for example, the phrase "Please refrain from sudden acceleration" as shown in Figs. 10(a) and 10(b). Alternatively, for example, A4 and the deceleration period P A4 When the absolute value of the average or maximum value of the acceleration of the vehicle V1 during the driving is equal to or greater than a predetermined sudden deceleration threshold, a sudden deceleration message is set as the notification message. The sudden deceleration message may include, for example, the phrase "Avoid sudden braking" as shown in Figures 10(c) and 10(d). In addition, various messages (e.g., "Be careful not to exceed the speed limit" or "Take breaks as appropriate") can be set as the notification message depending on the driving behavior information DA[i].

[0078] The recording operation OP2 that can be executed in step S17 will now be explained in more detail. In the recording operation OP2 in step S17, the specific operation control unit F6 records the information to be recorded on the recording medium 73. The specific operation control unit F6 in step S17 includes at least one of the driving behavior information DA[i] and the estimated emotion EM[i] (information indicating the estimated emotion EM[i]) in the information to be recorded. The information to be recorded may also include date and time information indicating the date and time of the i-th target period and vehicle position information during the i-th target period (see FIG. 4). Furthermore, the information to be recorded may also include vehicle exterior image information and vehicle interior image information during the i-th target period (see FIG. 2).

[0079] In this way, the in-vehicle control device 10 can execute a specific operation OP (notification operation OP1 or recording operation OP2) using a functional unit. At this time, there is concern that performing the notification operation OP1 only because of having negative emotions may result in over-notification, and over-notification may give the user U1 a sense of discomfort or the like. Also, if the recording operation OP2 is performed only because of having negative emotions, the possibility of unnecessary information being recorded also increases. This is because there are drivers who can maintain safe driving even while having negative emotions. Considering this, the controller 11 determines whether the estimated emotion EM corresponds to a specific emotion (here, a negative emotion), and controls whether to execute the specific operation OP through comparison between the driving behavior information DA and the reference information REF when it corresponds to the specific emotion.

[0080] Thereby, the specific operation OP can be performed only when it is determined that the driving behavior of the user U1 is a specific driving behavior different from normal after the estimated emotion EM corresponds to the specific emotion. As a result, excessive execution of the specific operation OP (over-notification or unnecessary information recording) is suppressed, and the specific operation OP can be performed only when it is truly highly necessary. That is, optimization of the execution of the specific operation OP is achieved.

[0081] Specifically, when the estimated emotion EM corresponds to a specific emotion (here, a negative emotion), the controller 11 derives the degree of deviation DEV between the driving behavior information DA and the reference information REF. Then, when "DEV > TH", the controller 11 causes the functional unit to execute the specific operation OP, and when "DEV < TH", it inhibits the functional unit from executing the specific operation OP (S15 to S19).

[0082] <> Thereby, the specific operation OP can be performed only when it is determined that the driving behavior of the user U1 is significantly different from the reference information REF, that is, when it is a specific driving behavior different from normal after the estimated emotion EM corresponds to the specific emotion. As a result, excessive execution of the specific operation OP (over-notification or unnecessary information recording) is suppressed, and the specific operation OP can be performed only when it is truly highly necessary. That is, optimization of the execution of the specific operation OP is achieved.

[0083] Examples of the specific operation OP are an announcing operation OP1 or a recording operation OP2. When the announcing operation OP1 is performed as the specific operation OP, the in-vehicle system SYS can also be called a driving assistance system. When the recording operation OP2 is performed as the specific operation OP, the in-vehicle system SYS can also be called a drive recorder system, and the in-vehicle control device 10 may constitute a drive recorder together with the recording medium 73. The in-vehicle control device 10 constituting the drive recorder can perform, apart from the recording operation OP2, a continuous recording operation for continuously recording outside-vehicle image information in chronological order on the recording medium 73, or an event recording operation for recording outside-vehicle image information on the recording medium 73 when a predetermined event occurs.

[0084] By optimizing the execution of the announcing operation OP1, discomfort to the user U1 that may occur due to excessive announcing is alleviated. By optimizing the execution of the recording operation OP2, unnecessary information recording is suppressed. This contributes to suppressing the required recording capacity and also leads to a reduction in the man-hours required to select truly necessary information. As described above, the recording medium 73 may be a recording medium (hereinafter, specifically referred to as an external recording medium) in an external device (e.g., a server device connected to the Internet) provided outside the vehicle V1. In this case, by optimizing the execution of the recording operation OP2, communication costs when transmitting information to be recorded to the external recording medium via wireless communication are suppressed. If the user U1 is a fleet driver, when an unusual driving behavior is observed, driving behavior information DA, etc. can be left as a log in the external recording medium.

[0085] The estimation information includes at least one of biometric data of the user U1 (driver) and captured image information of the user U1. This allows the emotion of the user U1 to be estimated with a relatively high degree of accuracy. Since the user U1 is photographed by the in-vehicle camera 72, the captured image information of the user U1 is included in the in-vehicle image information. The captured image information of the user U1 includes image information of the face image of the user U1.

[0086] <<Second Example>> The second embodiment will be described. The second embodiment and the third to sixth embodiments described below are based on the first embodiment, and for matters not specifically mentioned in the second to sixth embodiments, the description of the first embodiment also applies to the second to sixth embodiments unless there is a contradiction. However, when interpreting the description of the second embodiment, the description of the second embodiment may take precedence for matters that contradict between the first and second embodiments (the same applies to the third to sixth embodiments described below). As long as there is no contradiction, any two or more of the first to sixth embodiments may be combined.

[0087] FIG. 11 is a functional block diagram of an in-vehicle control device 10 according to the second embodiment. The in-vehicle control device 10 according to the second embodiment has functional blocks F1 to F7, H1, and H2. The functional blocks F1 to F7 are provided in a controller 11. All or part of the functions of the functional blocks F1 to F7 may be realized by the controller 11 executing a program recorded in the memory 12 or any other recording medium. The configuration of FIG. 11 according to the second embodiment is obtained by adding the functional block F7 to the configuration of FIG. 6 shown in the first embodiment. Except for the addition of the functional block F7 and the matters described below accompanying this addition, the configuration and operation of the in-vehicle control device 10 according to the second embodiment are the same as those of the first embodiment. Differences from the first embodiment will now be described.

[0088] The functional block F7 is a reference information update unit. Driving behavior information DA may be input to the reference information update unit F7 from the selector F3. As in the first embodiment, the driving behavior information DA and the estimated emotion EM are input to the selector F3, and the aforementioned emotion value is added to the estimated emotion EM input to the selector F3. Alternatively, only the emotion value may be input to the selector F3 from the emotion estimation unit F2. The selector F3 according to the second embodiment can input the driving behavior information DA from the driving behavior evaluation unit F1 to the deviation deriving unit F4 or the reference information update unit F7 depending on the emotion value. Specifically, the selector F3 according to the second embodiment inputs the driving behavior information DA to the deviation deriving unit F4 when the emotion value is “1,” i.e., when the estimated emotion EM is a negative emotion. The selector F3 according to the second embodiment inputs the driving behavior information DA to the deviation deriving unit F4 when the emotion value is “2,” i.e., when the estimated emotion EM is a positive emotion. The selector F3 according to the second embodiment does not input the driving behavior information DA to either the deviation deriving unit F4 or the reference information updating unit F7 when the emotion value is "3," that is, when the estimated emotion EM is a neutral emotion.

[0089] The reference information update section F7 does not function significantly when the emotion value is "1" or "3." The operation of the controller 11 when the emotion value is "1" or "3" is the same in the first and second embodiments.

[0090] When the emotion value is "2," the reference information update unit F7 updates the reference information REF using the driving behavior information DA input thereto. For example, the reference information update unit F7 calculates the average of the input driving behavior information DA and the reference information REF1 held in the reference information holding unit H1 at that time, and sets the calculated average as the updated reference information REF. The average is a simple average or a weighted average.

[0091] Fig. 12 shows a flowchart of the target process Q according to the second embodiment. The target process Q shown in Fig. 12 is made up of steps S11 to S19 and steps S21 and S22. That is, the target process Q in Fig. 12 can be obtained by adding steps S21 and S22 to the target process Q in Fig. 9.

[0092] In the second embodiment, the processing content of steps S11 to S13 is the same as in the first embodiment. However, the processing content of step S13 in the second embodiment is slightly different from that of the first embodiment, specifically as follows. In step S13, the controller 11 determines whether the estimated emotion EM[i] corresponds to a negative emotion. If the estimated emotion EM[i] corresponds to a negative emotion (Y in step S13), the process proceeds from step S13 to step S14. The operational flow after proceeding to step S14 is the same as that shown in the first embodiment. If the estimated emotion EM[i] does not correspond to a negative emotion (N in step S13), the process proceeds from step S13 to step S21.

[0093] In step S21, controller 11 determines whether estimated emotion EM[i] corresponds to a positive emotion. If estimated emotion EM[i] corresponds to a positive emotion in step S21 (Y in step S21), the process proceeds to step S22. If estimated emotion EM[i] does not correspond to a positive emotion in step S21 (N in step S21), controller 11 ends target process Q without performing any of steps S14 to S19 or step S22. If estimated emotion EM[i] does not correspond to a positive emotion in step S21, this is the case when estimated emotion EM[i] corresponds to a neutral emotion.

[0094] In the configuration example of Fig. 11, the processes of steps S13 and S21 are implemented using a selector F3. If the emotion value corresponding to the estimated emotion EM[i] is "1" (i.e., if the estimated emotion EM[i] is a negative emotion), the selector F3 inputs the driving behavior information DA[i] to a deviation degree derivation unit F4. In response to this input, the processes of steps S14, S15, S16, and S17 or steps S14, S15, S18, and S19 are executed. If the emotion value corresponding to the estimated emotion EM[i] is "2" (i.e., if the estimated emotion EM[i] is a positive emotion), the selector F3 inputs the driving behavior information DA[i] to a reference information update unit F7. In response to this input, the process of step S22 is executed.

[0095] In step S22, the reference information update unit F7 executes an update process to update the reference information REF using the driving behavior information DA[i]. The reference information REF stored in the reference information storage unit H1 immediately before the i-th target period and during the i-th target period is referred to as pre-update reference information REF1. The reference information REF obtained by updating the pre-update reference information REF1 using the driving behavior information DA[i] is referred to as post-update reference information REF2. In the update process, the reference information update unit F7 in step S22 calculates the average of the driving behavior information DA[i] and the pre-update reference information REF1, and sets the calculated average as the post-update reference information REF2. The average is a simple average or a weighted average. The target process Q ends when the process of step S22 is executed.

[0096] As described above, in the second embodiment, similar to the first embodiment, when the estimated emotion EM[i] corresponds to a negative emotion (first specific emotion), the driving behavior information DA[i] is compared with the reference information REF stored in the reference information storage unit H1 during the i-th target period (S14). Then, based on the comparison result, whether or not to perform the specific action OP is controlled (S15-S19). The actions and effects of controlling whether or not to perform the specific action OP are the same as those described in the first embodiment. Additionally, in the second embodiment, when the estimated emotion EM[i] corresponds to a positive emotion (second specific emotion), the reference information REF is updated using the driving behavior information DA[i].

[0097] The driving behavior information DA when the user U1 has positive emotions is driving behavior information DA in a situation where safe driving is likely to be performed, and is suitable as reference information RFF. By providing the reference information update unit F7, the reference information RFF can be updated to appropriate information.

[0098] 12, if the estimated emotion EM[i] does not correspond to a negative emotion in step S13 (N in step S13), the process may always proceed from step S13 to step S22. When this variation is applied, the selector F3 inputs the driving behavior information DA[i] to the reference information update unit F7 when the emotion value corresponding to the estimated emotion EM[i] is "2" or "3." As a result, the update process of step S22 is executed when the estimated emotion EM[i] corresponds to a positive emotion or a neutral emotion.

[0099] In the above explanations of the first and second embodiments, it was assumed that the estimated emotion EM was classified into one of negative emotion, positive emotion, and neutral emotion. However, the estimated emotion EM may also be classified into either negative emotion or emotion other than negative emotion (hereinafter referred to as non-negative emotion) (i.e., classification may be performed using a binary option). In this case, a non-negative emotion is understood to be a positive emotion, or to be either a positive emotion or a neutral emotion.

[0100] <<Third Example>> A third embodiment will now be described. After the controller 11 starts up, the controller 11 may sequentially set target periods according to a predetermined schedule. That is, for example, the controller 11 may set start timings periodically and set a target period starting from each start timing.

[0101] The controller 11 monitors whether a predetermined trigger condition is met, and when the trigger condition is met, it sets a target period and performs the target process Q. In the method in which the start timing is set periodically and repeatedly as described above, the trigger condition is met periodically.

[0102] FIG. 13 is a functional block diagram of an in-vehicle control device 10 according to a third embodiment. The in-vehicle control device 10 according to the third embodiment includes the functional blocks F1 to F7, H1, and H2 shown in the second embodiment, as well as a functional block F8. The functional block F8 is a condition determination unit that determines whether a trigger condition is met. The functional blocks F1 to F8 are provided in the controller 11. All or part of the functions of the functional blocks F1 to F8 may be realized by the controller 11 executing a program recorded in the memory 12 or any other recording medium. The configuration of FIG. 13 according to the third embodiment is obtained by adding the functional block F8 to the configuration of FIG. 11 shown in the second embodiment. However, the functional block F8 may also be added to the configuration of FIG. 6 shown in the first embodiment. Except for the addition of the functional block F8 and the matters described below accompanying this addition, the configuration and operation of the in-vehicle control device 10 according to the third embodiment are the same as those of the first or second embodiment.

[0103] When a trigger condition is satisfied, the controller 11 sets a target period based on the timing at which the trigger condition is satisfied. The start timing of the target period may be the same as the timing at which the trigger condition is satisfied, or may be a predetermined time after the timing at which the trigger condition is satisfied. The start timing of the target period may be a predetermined time before the timing at which the trigger condition is satisfied. The memory 12 is provided with a storage area for temporarily storing vehicle sensor information and estimation information for a certain period of time. The controller 11 can read and refer to vehicle sensor information and estimation information for the past certain period of time from the memory 12 at any time.

[0104] When a target period is set upon the fulfillment of a trigger condition, a target process Q is executed for the target period. The target process Q includes at least the processes of steps S11 and S12 (i.e., a process for evaluating the driving behavior of user U1 and a process for estimating the emotion of user U1). That is, when a target period is set upon the fulfillment of a trigger condition, at least the processes of steps S11 and S12 are executed. Subsequently, whether the processes of steps S14 to S19 or step S22 are executed in the target process Q depends on the content of the estimated emotion EM identified in step S12. When the estimated emotion EM corresponds to a negative emotion, the target process Q further includes a process (processing realized in steps S15 to S19) for controlling whether or not to execute a specific action OP through a comparison with the driving behavior information DA and the reference information REF. When the estimated emotion EM corresponds to a positive emotion, the target process Q may include the update process of step S22 (see FIG. 12 ).

[0105] A method that constantly and repeatedly executes the target process Q is likely to result in a lot of wasted target process Q. By defining appropriate trigger conditions and executing the target process Q when the trigger conditions are met, it is possible to execute the target process Q only at beneficial times (reduce unnecessary execution of the target process Q).

[0106] The trigger condition may be a condition whose success or failure is determined based on vehicle sensor information. A trigger condition whose success or failure is determined based on vehicle sensor information is specifically referred to as trigger condition CND1. When trigger condition CND1 is satisfied, controller 11 may set a target period and perform target process Q.

[0107] The trigger condition CND1 may be satisfied when there is a sudden change in vehicle sensor information. For example, the condition determination unit F8 monitors whether sudden braking or sudden steering has occurred based on the vehicle sensor information, and determines that the trigger condition CND1 is satisfied when it is determined that sudden braking or sudden steering has occurred. Sudden braking refers to a state in which the vehicle speed suddenly decreases (a state in which the acceleration of the vehicle V1 in the traveling direction of the vehicle V1 is negative and the maximum value of the acceleration magnitude during the period in which the vehicle speed decreases is equal to or greater than a predetermined sudden braking threshold). The condition determination unit F8 can determine whether sudden braking has occurred based on the vehicle speed information or acceleration information. Sudden steering refers to a state in which the steering angle suddenly changes (a state in which the change in steering angle per unit time is equal to or greater than a predetermined sudden steering threshold). The condition determination unit F8 can determine whether sudden steering has occurred based on the steering angle information. When sudden braking or sudden steering has occurred, there is a possibility that a dangerous driving behavior that could lead to an accident has been performed, so a target process Q leading to the execution of a specific action OP is executed.

[0108] The trigger condition may be a condition whose success or failure is determined based on the estimation information. A trigger condition whose success or failure is determined based on the estimation information is specifically referred to as trigger condition CND2. The controller 11 may set a target period and perform target process Q when trigger condition CND2 is met. To determine whether trigger condition CND2 is met, condition determination unit F8 uses the emotion estimation unit F2. Specifically, to determine whether trigger condition CND2 is met, the emotion estimation unit F2 periodically performs emotion estimation processing, separate from the emotion estimation processing of step S12 in target process Q. Estimated emotions EM are sequentially identified by the periodically executed emotion estimation processing. The condition determination unit F8 monitors changes in the sequentially identified estimated emotions EM, and determines that trigger condition CND2 is met when a specific change occurs in estimated emotion EM.

[0109] A specific change in the estimated emotion EM may be a change in the estimated emotion EM from a positive emotion to a negative emotion, or a change from a negative emotion to a positive emotion. Alternatively, a specific change in the estimated emotion EM may be a sudden change in the emotion of user U1. For example, the condition determination unit F8 can determine that a sudden change has occurred in the emotion of user U1 when the arousal or activity level of user U1 changes by more than a predetermined threshold in two emotion estimation processes executed adjacently.

[0110] <<Fourth Example>> A fourth embodiment will be described. Referring to FIG. 14, the in-vehicle control device 10 may be wirelessly connected to the communication network NET, and the server device SV may be wired or wirelessly connected to the communication network NET. The in-vehicle control device 10 and the server device SV may be capable of two-way communication via the communication network NET. The server device SV is formed by any one or more computers connected to the communication network NET. The server device SV may be formed using cloud computing. The communication network NET includes the Internet, an intranet, and the like. Some of the above-described functional blocks F1 to F8 and H1 and H2 may be provided in the in-vehicle control device 10, and the remaining functional blocks may be provided in the server device SV. For example, among the functional blocks F1 to F8 and H1 and H2 shown in FIG. 13, the functional blocks F7 and H1 may be provided in the server device SV, and the remaining functional blocks may be provided in the in-vehicle control device 10. Alternatively, the functional blocks may be allocated between the in-vehicle control device 10 and the server device SV in any manner.

[0111] <<Fifth Example>> A fifth embodiment will be described. The deviation deriving unit F4 has the function of determining whether an undesirable driving behavior is being performed by comparing the driving behavior information DA based on vehicle sensor information with reference information REF corresponding to normal driving behavior. As described above, the driving behavior information DA includes vehicle speed statistical information including the average and maximum vehicle speeds, but whether the average and maximum vehicle speeds are desirable or undesirable depends on the upper speed limit set for the road on which the vehicle V1 is traveling. Therefore, the vehicle speed statistical information may include information corresponding to the vehicle speed and the upper speed limit.

[0112] The upper speed limit here is the legal speed limit or the speed limit. Controller 11 can recognize the legal speed limit or the speed limit established for the road on which vehicle V1 is traveling, based on map information and vehicle position information stored in memory 12 or received from an external device (such as the above-mentioned server device SV). Alternatively, if an image of a sign installed on the road on which vehicle V1 is traveling is included in an image captured by the exterior camera, controller 11 can recognize the speed limit established for the road by image recognition of the image captured by the exterior camera.

[0113] Hereinafter, the upper speed limit refers to the legal speed or speed limit established for the road on which the vehicle V1 is traveling. Specifically, the difference obtained by subtracting the average vehicle speed during the evaluation period from the upper speed limit (hereinafter referred to as the first evaluation difference) may be included in the vehicle speed statistical information. Similarly, the difference obtained by subtracting the maximum vehicle speed during the evaluation period from the upper speed limit (hereinafter referred to as the second evaluation difference) and the difference obtained by subtracting the minimum vehicle speed during the evaluation period from the upper speed limit (hereinafter referred to as the third evaluation difference) may be included in the vehicle speed statistical information. In this case, the reference information REF includes first to third reference differences corresponding to the first to third evaluation differences. The jth reference difference simulates the jth evaluation difference when the user U1 has a positive emotion (here, j is 1, 2, or 3). The deviation DEV increases as the absolute value of the difference between the jth evaluation difference and the jth reference difference increases (here, j is 1, 2, or 3).

[0114] <<Sixth Example>> A sixth embodiment will now be described.

[0115] As described above, when the estimated emotion EM corresponds to the first specific emotion, the controller 11 controls whether or not to perform the specific action OP by comparing the driving behavior information DA with the reference information REF. In the above-described embodiments, the first specific emotion is assumed to be a negative emotion, but the first specific emotion may be any predetermined emotion. Generally, not all emotions associated with negative emotions may correspond to the first specific emotion. For example, the first specific emotion may be anger or displeasure, and sadness may be negative but not correspond to the first specific emotion.

[0116] The method shown in the second embodiment includes a method of updating the reference information REF using the driving behavior information DA when the estimated emotion EM corresponds to the second specific emotion. In the second embodiment, the second specific emotion is assumed to be a positive emotion, but the second specific emotion may be a predetermined emotion different from the first specific emotion. In general, not all emotions associated with positive emotions may correspond to the second specific emotion. For example, the second specific emotion may be an emotion of joy, merriment, or relaxation, and an emotion of excitement may be positive but not correspond to the second specific emotion.

[0117] A program that causes a computer device to execute any of the methods described in the embodiments of the present invention, and a non-volatile recording medium on which the program is recorded, are included within the scope of the embodiments of the present invention. The program that causes a computer device to execute any of the methods described in the embodiments of the present invention may be a subprogram incorporated into any main program or called by any main program. The in-vehicle control device 10 is a type of computer device. The method executed by the in-vehicle control device 10 can be referred to as a vehicle operation control method (a vehicle specific operation control method). Any processing in the embodiments of the present invention may be realized by hardware such as a semiconductor integrated circuit, software equivalent to the program, or a combination of hardware and software.

[0118] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above-described embodiments are merely examples of the present invention, and the meanings of the terms of the present invention and each constituent element are not limited to those described in the above-described embodiments. The specific numerical values ​​shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. [Explanation of symbols]

[0119] SYS In-vehicle system U1 User V1 vehicle ST1 seat 10. In-vehicle control device 11 Controller 12 Memory 13 Communications Department 20 Driving control device 30 Actuator section 40 Vehicle sensor unit 60 Biometric Sensor 71 Exterior camera 72 In-car camera 73 Recording Media 80 HMI 81 Display device 82 Speaker 83 Microphone F1 Driving Behavior Evaluation Department F2 Emotion estimation part F3 Selector F4 Deviation degree derivation part F5 Comparison section F6 Specific operation control section F7 Standard information update section F8 Condition judgment section H1 Standard information holding section H2 Judgment threshold holding unit DA Driving behavior information EM Estimated Emotion REF Standard Information DEV Deviation TH judgment threshold CMP comparison result signal SV server device NET communication network

Claims

1. A vehicle operation control method executed by an in-vehicle control device, comprising: Deriving driving behavior information by evaluating a driving behavior of a driver of the vehicle based on vehicle sensor information corresponding to a driving state of the vehicle, and estimating the emotion of the driver based on estimation information; When the estimated emotion corresponds to a specific emotion, the driving behavior information is compared with pre-stored reference information, and whether or not to perform a specific action is controlled based on the comparison result. , A motion control method for a vehicle.

2. When the estimated emotion corresponds to the specific emotion, a degree of deviation between the driving behavior information and the reference information is derived, and if the degree of deviation is greater than a judgment threshold, a functional unit capable of executing the specific action is caused to execute the specific action, and if the degree of deviation is smaller than the judgment threshold, the execution of the specific action by the functional unit is inhibited.

2. The vehicle motion control method according to claim 1.

3. The specific emotion is a first specific emotion, When the estimated emotion during the target period corresponds to the first specific emotion, the driving behavior information during the target period is compared with the reference information, and whether or not to perform the specific action is controlled based on the comparison result. When the estimated emotion during the target period corresponds to a second specific emotion different from the first specific emotion, the reference information is updated using the driving behavior information during the target period.

3. The vehicle operation control method according to claim 1 or 2.

4. When the trigger condition is met, the target process is executed, The target process includes at least a process of evaluating the driving behavior and a process of estimating the driver's emotion, and a process of controlling whether or not the specific action is to be performed when the estimated emotion corresponds to the specific emotion.

3. The vehicle operation control method according to claim 1 or 2.

5. The specific emotion is a negative emotion.

2. The vehicle motion control method according to claim 1.

6. The first specific emotion is a negative emotion, and the second specific emotion is a positive emotion.

4. The vehicle motion control method according to claim 3.

7. The information for estimation includes at least one of biometric data of the driver and captured image information of the driver.

3. The vehicle operation control method according to claim 1 or 2.

8. The specific operation includes a notification operation of notifying the driver of information corresponding to the driving behavior information.

3. The vehicle operation control method according to claim 1 or 2.

9. The specific action includes a recording action of recording at least one of the driving behavior information and the estimated emotion on a recording medium.

3. The vehicle operation control method according to claim 1 or 2.

10. 3. A program for causing a computer to execute the vehicle operation control method according to claim 1.

11. An in-vehicle control device having a controller, the controller derives driving behavior information by evaluating a driving behavior of the driver of the vehicle based on vehicle sensor information corresponding to a driving state of the vehicle, and estimates an emotion of the driver based on estimation information; When the estimated emotion corresponds to a predetermined specific emotion, the controller compares the driving behavior information with pre-stored reference information and controls whether or not to perform a specific action based on the comparison result. ,In-vehicle control device.

12. The vehicle-mounted control device according to claim 11; a human-machine interface capable of executing the specific operation; the specific operation includes a notification operation of notifying the driver of information corresponding to the driving behavior information, The controller controls whether or not the human-machine interface performs the notification action based on the comparison result when the estimated emotion corresponds to the specific emotion. ,In-vehicle systems.

13. The vehicle-mounted control device according to claim 11; a camera installed in the vehicle; a recording medium for recording image information obtained by photographing with the camera, the specific action includes a recording action of recording at least one of the driving behavior information and the estimated emotion on the recording medium, The controller controls whether or not to execute the recording operation on the recording medium based on the comparison result when the estimated emotion corresponds to the specific emotion. ,In-vehicle systems.

Images