Determination device, information processing system, and determination program

Abstract

To provide a technique of appropriately detecting that a dangerous situation occurs in a vehicle.SOLUTION: An exemplary determination device that detects an abnormal running state of a vehicle based on behavior information about behavior of the vehicle determines whether or not a dangerous situation occurs when detecting the abnormal running state, based on driver's emotion information when detecting the abnormal running state.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique for detecting the occurrence of a dangerous situation in a vehicle. [Background technology]

[0002] Conventionally, there is known an information processing system that detects the occurrence of a dangerous situation (a situation on the brink of an accident, a so-called near miss) in a traveling vehicle and performs information processing such as displaying the location where the dangerous situation occurred on a map, thereby contributing to safe driving (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-169885 A Summary of the Invention [Problem to be solved by the invention]

[0004] The occurrence of a near-miss is detected based on behavior information related to the behavior of the vehicle (e.g., vehicle acceleration information, etc.). However, the behavior of the vehicle due to a road bump or the like may be similar to the behavior of the vehicle when a near-miss occurs, and detection based on the behavior information related to the behavior of the vehicle may erroneously detect the road bump or the like as the occurrence of a near-miss.

[0005] In view of the above, an object of the present invention is to provide a technique for appropriately detecting the occurrence of a dangerous situation in a vehicle. [Means for solving the problem]

[0006] An exemplary determination device of the present invention detects an abnormal driving condition of a vehicle based on behavior information regarding the behavior of the vehicle, and determines whether a dangerous situation has occurred when the abnormal driving condition is detected based on emotional information of the driver at the time the abnormal driving condition is detected. Effect of the Invention

[0007] According to the exemplary embodiment of the present invention, when the vehicle is in an abnormal driving state, it is possible to identify whether or not a dangerous situation has occurred in the vehicle, and therefore it is possible to appropriately detect the occurrence of a dangerous situation in the vehicle. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing system according to a first embodiment; [Diagram 2] FIG. 1 is a diagram showing an example of the configuration of a server according to a first exemplary embodiment of the present invention. [Diagram 3] Diagram showing the input / output relationship of the emotion index estimation model [Figure 4] Schematic diagram of the learning device [Diagram 5] A diagram showing an example of a psychological plane table. [Figure 6] FIG. 13 is a diagram showing an example of a determination table; [Figure 7] FIG. 11 is a diagram showing another example of the judgment table; [Figure 8] A diagram showing an example of a database [Figure 9] Another example of a database [Figure 10] FIG. 13 is a diagram showing yet another example of a database. [Figure 11] A diagram showing an example of a near-miss map [Figure 12] Flowchart of information processing executed by the controller [Figure 13] FIG. 13 shows a vehicle and a server according to a second embodiment. [Figure 14] FIG. 13 is a diagram showing an example of the configuration of a server according to a second exemplary embodiment of the present invention. [Figure 15] FIG. 13 is a diagram showing an example of a decision table used in the second embodiment; [Figure 16] 11 is a flowchart of information processing executed by a controller according to the second embodiment. [Figure 17] FIG. 13 is a table for estimating the level of near misses DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.

[0010] First Embodiment [Information Processing System] 1 is a diagram showing an example of the configuration of an information processing system SYS1 according to a first exemplary embodiment of the present invention. The information processing system SYS1 is a system that detects the occurrence of a dangerous situation in a vehicle V1 and executes information processing using the result of the detection.

[0011] As shown in FIG. 1, the information processing system SYS1 includes an in-vehicle device 1, a server 2 (computer), and a driving support device 3.

[0012] The in-vehicle device 1 and the driving support device 3 are mounted on a vehicle V1. In this embodiment, the vehicle V1 also mounts a G (acceleration) sensor SN1, an in-vehicle camera C1, an in-vehicle microphone M1, a GPS sensor SN2, a timing device TM1, and an airbag device AB1 in addition to the in-vehicle device 1 and the driving support device 3. The in-vehicle device 1, the driving support device 3, the G sensor SN1, the in-vehicle camera C1, the in-vehicle microphone M1, the GPS sensor SN2, the timing device TM1, and the airbag device AB1 are connected to an in-vehicle communication network provided inside the vehicle V1.

[0013] The server 2 is disposed outside the vehicle V1 (for example, in a near-miss detection service center building, etc.). The server 2 may be a physical server or a virtual server. The server 2 may be configured by one server or multiple servers.

[0014] The in-vehicle device 1 collects behavior information related to the behavior of the vehicle V1 and data for obtaining an emotion index value of the driver D1, and transmits them to the server 2 via the network N1. In this embodiment, the behavior information related to the behavior of the vehicle V1 is the output of the G-sensor SN1, and the data for obtaining the emotion index value of the driver D1 is an image output from the in-vehicle camera C1 and an audio signal output from the in-vehicle microphone M1.

[0015] The behavior information on the behavior of the vehicle V1 may be the output of a steering angle sensor, the output of a vehicle speed sensor, the output of a brake pedal sensor, or the output of an accelerator pedal sensor, or may be a combination of at least two of the output of the G sensor SN1, the output of the steering angle sensor, the output of the vehicle speed sensor, the output of the brake pedal sensor, and the output of the accelerator pedal sensor. The steering angle sensor is mounted on the vehicle V1, detects the steering angle (steering angle) of the vehicle V1, and generates and outputs steering angle information representing the detected steering angle. The vehicle speed sensor is mounted on the vehicle V1, detects the speed of the vehicle V1, and generates and outputs vehicle speed information (vehicle speed pulse) representing the detected speed. The brake pedal sensor is mounted on the vehicle V1, detects the operation of the brake pedal of the vehicle V1 by the driver D1, and generates and outputs brake pedal operation information indicating the operation of the brake pedal. The accelerator pedal sensor is mounted on the vehicle V1, detects the operation of the accelerator pedal of the vehicle V1 by the driver D1, and generates and outputs accelerator pedal operation information indicating the operation of the accelerator pedal.

[0016] The server 2 detects an abnormal driving state of the vehicle V1 based on behavior information related to the behavior of the vehicle V1, and judges whether or not a dangerous situation has occurred in the vehicle V1 when the abnormal driving state of the vehicle V1 is detected based on the emotion index value of the driver D1 when the abnormal driving state of the vehicle V1 is detected. The server 2 further generates a database related to the judgment result of whether or not a dangerous situation has occurred in the vehicle V1 when the abnormal driving state of the vehicle V1 is detected, and transmits information extracted from the database to the in-vehicle device 1 via the network N1. Although only one vehicle V1 is illustrated in FIG. 1, it is desirable that there are many vehicles V1 equipped with the in-vehicle device 1 capable of communicating with the server 2. If there are many vehicles V1 equipped with the in-vehicle device 1 capable of communicating with the server 2, the database generated by the server 2 can be so-called big data.

[0017] The driving support device 3 uses the database generated by the server 2 to support the driving of the vehicle V1. For example, the driving support device 3 includes a display unit, and when the vehicle V1 approaches a point where a dangerous situation has occurred (near-miss occurrence point), the driving support device 3 displays attention-calling display information, such as a text message saying, "You will soon reach the near-miss occurrence point, so please drive with caution," on the display unit. In addition, for example, the driving support device 3 includes a speaker, and when the vehicle V1 approaches a point where a dangerous situation has occurred (near-miss occurrence point), the driving support device 3 outputs attention-calling audio information, such as a voice message saying, "You will soon reach the near-miss occurrence point, so please drive with caution," from the speaker.

[0018] The G sensor SN1 detects acceleration in a predetermined axial direction applied to the vehicle V1, and generates and outputs the detection result of the acceleration as acceleration information. The G sensor SN1 may detect acceleration in two mutually orthogonal axial directions, or may detect acceleration in three mutually orthogonal axial directions. For example, the G sensor SN1 that detects acceleration in three axial directions is used to detect acceleration in the forward / rearward, left / right, and up / down directions of the vehicle V1, and the acceleration information is used to detect abnormal driving conditions such as sudden deceleration and sudden turns of the vehicle.

[0019] The in-vehicle camera C1 is composed of one or more cameras that capture the interior of the vehicle V1. The in-vehicle camera C1 captures the interior of the vehicle V1 and generates an in-vehicle camera image by capturing the interior of the capture area. The in-vehicle camera C1 is installed so that the face of the driver D1 is located within the capture area of ​​the in-vehicle camera C1.

[0020] The in-vehicle microphone M1 is composed of one or more microphones installed in the cabin of the vehicle V1. The in-vehicle microphone M1 converts ambient sounds around the installation position of the microphone M1 into an audio signal, which is an electrical signal. The in-vehicle microphone M1 is installed so as to be able to pick up the speech of the driver D1.

[0021] The GPS sensor SN2 receives signals from multiple GPS (Global Positioning System) satellites, and generates and outputs vehicle position information based on the reception results. The vehicle position information generated by the GPS sensor SN2 represents the current position of the vehicle V1 by longitude and latitude.

[0022] The timing device TM1 generates and outputs a signal corresponding to the current time. The timing circuit 16 has, for example, a built-in battery, so that it can operate without receiving power from an external source and keep accurate time.

[0023] The airbag device AB1 deploys an airbag when it detects a strong impact due to a vehicle collision, etc. The occurrence of an accident can be determined based on an airbag deployment signal in the airbag device AB1.

[0024] [server] 2 is a diagram showing an example of the configuration of the server 2 according to the first exemplary embodiment of the present invention. The server 2 includes a communication unit 21, a storage unit 22, and a controller 23. The server 2 includes a determination device that detects an abnormal driving state of the vehicle V1 based on behavior information related to the behavior of the vehicle V1, and determines whether or not a dangerous situation has occurred in the vehicle V1 when the abnormal driving state of the vehicle V1 is detected, based on an emotion index value of the driver D1 when the abnormal driving state of the vehicle V1 is detected.

[0025] In addition, the determination device may be configured by cooperation between the on-vehicle block and the server block, in which an on-vehicle block having a part of the determination device is installed in the vehicle V1 and a server block having the remaining part of the determination device is installed in the server 2. In this case, the on-vehicle block and the server block share any information by communication via the network N1.

[0026] The communication unit 21 transmits and receives any signal to and from the in-vehicle device 1. The controller 23 can transmit and receive any information to and from various devices installed in the vehicle V1 using the communication unit 21 and the in-vehicle device 1, but in the following, the description of the communication unit 21 and the in-vehicle device 1 may be omitted.

[0027] The storage unit 22 is configured to have a non-volatile memory such as a ROM (Read only memory) or a flash memory, and a volatile memory such as a RAM (Random access memory). The storage unit 12 stores an emotion index value estimation model 221, a psychological plane table 222, a judgment table 223, a database DB1, a program PG1, and map data MD1.

[0028] 3, the emotion index value estimation model 221 is a trained AI (Artificial Intelligence) model that estimates the emotion index value of any person from the image information and speech information of the person. Therefore, by inputting the image information and speech information of the driver D1 to the emotion index value estimation model 221, the arousal level and activity level indicating the mental state of the driver D1 are estimated by the emotion index value estimation model 221. Note that the emotion index value estimation model 221 may be a trained AI model that estimates the arousal level and activity level of any person from only the image information of the person.

[0029] The emotion index value estimation model 221 is created in a learning device 4 shown in Fig. 4. The learning device 4 may be configured with any one or more servers. A method for creating the emotion index value estimation model 221 in the learning device 4 will be described.

[0030] First, in the data collection step, a plurality of sets of learning data including image information of a subject and the subject's alertness and activity are collected. A large amount of learning data is collected (for example, tens of thousands of sets).

[0031] Image information of the subject includes image information of the subject's face, and is obtained by photographing the subject with any camera. An EEG sensor is attached to the subject. The EEG sensor attached to the subject measures the subject's brainwaves to obtain the subject's brainwave data (measured EEG data). A heart rate sensor is attached to the subject. The heart rate sensor attached to the subject measures the subject's heart rate to obtain the subject's heart rate data (measured heart rate data).

[0032] In the data collection step, a first model is prepared for estimating the level of arousal of a given person based on the measured electroencephalogram data of the person, and the first model is used to estimate (infer) the level of arousal of the subject based on the measured electroencephalogram data of the subject. In the first model, the level of arousal is derived based on the beta / alpha waves of the brainwaves. That is, a value according to the beta / alpha waves of the brainwaves (for example, a value proportional to the beta / alpha waves of the brainwaves) is derived as the level of arousal. The beta / alpha waves of the brainwaves are the ratio of the beta waves of the brainwaves to the alpha waves of the brainwaves. More specifically, for example, the beta / alpha waves of the brainwaves are "INT_β / INT_α" or "PW_β / PW_α". INT_β and INT_α represent the intensities of the beta and alpha waves of the brainwaves, respectively. PW_β and PW_α represent the powers of the beta and alpha waves of the brainwaves, respectively.

[0033] In the data collecting step, a second model is prepared for estimating the activity level of a given person based on the measured heartbeat data of the person, and the second model is used to estimate (infer) the activity level of the subject based on the measured heartbeat data of the subject. In the second model, the activity level is calculated based on the LF (low frequency) component of the time interval between heartbeats (hereinafter referred to as the heartbeat interval) derived from the heartbeat data. The LF component of the heartbeat interval is a frequency component in a predetermined low frequency band (for example, a frequency band from 0.04 to 0.15 Hz) among the frequency components included in the heartbeat interval. For example, the activity level of the subject is derived by calculating the standard deviation in the amplitude of the LF component of the heartbeat interval. At this time, a value according to the standard deviation (for example, a value proportional to the standard deviation) is derived as the activity level. In the amplitude spectrum of the heartbeat interval, the frequency components of the low frequency band of the heartbeat interval consist of 1st to kth frequency components (k is an integer equal to or greater than 2), and the standard deviation corresponds to the standard deviation in the amplitude of the 1st to kth frequency components. The subject's alertness and activity levels estimated by the first and second models are included in the learning data as correct answer data.

[0034] More specifically, a unit period of a certain length of time is set, and image information of the subject obtained by photographing the subject during that unit period and speech information of the subject obtained by recording the subject's speech during that unit period are used as input data, and the subject's alertness and activity during that unit period (alertness and activity after statistical processing such as averaging) are used as correct answer data, and these data are matched together to create learning data.

[0035] That is, the subject's alertness and activity in a certain unit period are estimated by the first and second models based on the measured brain wave data and measured heart rate data of the subject in the unit period. Then, one set of learning data is composed of the image information and speech information of the subject associated with the alertness and activity of the subject. The subject may be the driver D1 itself or may be someone other than the driver D1. There may be multiple subjects. The first and second models may be created in advance by known supervised machine learning. The first and second models may be a common model.

[0036] After the data collection step, a learning step is performed in the learning device 4. In the learning step, learning data is input to the learning device 4. The learning device 4 is provided with an AI model 41 consisting of a neural network. The learning device 4 provides the AI ​​model 41 with image information and speech information of the subject in the learning data as input data, and provides the subject's awakening and activity in the learning data as correct answer data. The AI ​​model 41 estimates (infers) the subject's awakening and activity from the subject's image information. In the learning step, the learning device 4 evaluates the error between the awakening and activity estimated by the AI ​​model 41 and the awakening and activity provided as correct answer data. Then, the learning device 4 executes learning of the AI ​​model 41 so that the error is reduced using a learning algorithm such as an error backpropagation method. The learning here is supervised machine learning, and the parameters (weights, etc.) of the AI ​​model 41 are adjusted in the learning. When a predetermined learning end condition is satisfied, such as the above error converging to a sufficiently small value, the learning ends. After the learning is completed, the AI ​​model 41 is stored in the storage unit 12 as the emotion index value estimation model 221.

[0037] The psychological plane table 222 is data forming the psychological plane PP shown in FIG. 5, and has a data group in which two types of emotion index values ​​(arousal level and activity level) are associated with the corresponding emotion information (emotion type). The psychological plane PP shown in FIG. 5 is a two-dimensional model with two axes of arousal level and activity level. The two-dimensional model is created based on medical evidence (papers, etc.) showing the relationship between arousal level, activity level, and emotion. According to various medical evidence related to psychology, psychology can be estimated based on two types of indexes that indicate the mental and physical state. For this reason, emotion indexes other than arousal level and activity level may be used. Note that a multidimensional model of three or more dimensions may be used as a model for estimating emotion.

[0038] By plotting the arousal and activity of the estimated (calculated) emotion index value on the psychological plane PP, the emotion type assigned to the quadrant in which the plot position exists is estimated as the type of emotion felt by the emotion estimator. In this embodiment, the server 2 does not estimate the emotion of the driver D1, so emotion information may be omitted from the psychological plane table 222. When the emotion type is used for processing, the emotion type estimated using the psychological plane table 222 is used for processing.

[0039] The judgment table 223 will be described later. The database DB1 is a database reflecting the judgment result of whether or not a dangerous situation has occurred in the vehicle V1 when an abnormal driving state of the vehicle V1 is detected, and is a database related to near misses. The memory unit 22 further stores various programs to be executed by the controller 23. The programs stored in the memory unit 22 include the program PG1. The map data MD1 includes road data used to represent road shapes. The map data is divided into predetermined ranges called meshes, and is filed separately for each mesh.

[0040] The controller 23 comprehensively controls the operation of each component in the server 2. The controller 23 includes, as hardware resources, a calculation processing unit including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The controller 23 includes function blocks 231-236.

[0041] The controller 23 is a program execution device (computer) capable of executing any program. The controller 23 executes the program PG1 to realize each function of the controller 23 (including each function of the functional blocks 231 to 236). All operations of the controller 23 described in this embodiment may be operations that are realized by the controller 23 executing the program PG1. The program PG1 may be composed of multiple programs.

[0042] A description will be given of each functional block of the controller 23. Functional blocks 231, 232, 233, 234, 235, and 236 are an acquisition unit, an abnormality detection unit, a near-miss determination unit, a database generation unit, a data output unit, and a near-miss map generation unit, respectively.

[0043] The acquisition unit 231 acquires behavior information related to the behavior of the vehicle V1, data for obtaining the emotion index value of the driver D1, and information used to generate the database DB1. The contents acquired by the acquisition unit 231 are shared by each functional block in the controller 23. The behavior information related to the behavior of the vehicle V1 includes the output of the G sensor SN1. The data for obtaining the emotion index value of the driver D1 includes the in-vehicle camera image information output from the in-vehicle camera C1 and the audio signal output from the in-vehicle microphone M1. The image information of the driver D1 is included in the in-vehicle camera image information, and the controller 23 extracts the image information of the driver D1 from the in-vehicle camera image information and inputs it to the emotion index value estimation model 221. The image information of the driver D1 input to the emotion index value estimation model 221 includes image information of the face of the driver D1. The speech information of the driver D1 is included in the audio signal output from the in-vehicle microphone M1, and the controller 23 extracts the speech information of the driver D1 from the audio signal and inputs it to the emotion index value estimation model 221. The information used to generate database DB1 includes vehicle position information output from GPS sensor SN2, a signal corresponding to the current time output from timing device TM1, and an activation signal output from airbag device AB1 when the airbag is deployed.

[0044] The abnormality detection unit 232 detects an abnormal driving state of the vehicle V1 based on behavior information related to the behavior of the vehicle V1. For example, the abnormality detection unit 232 detects an abnormal driving state of the vehicle V1 when the acceleration indicated by the acceleration information output from the G sensor SN1 falls outside a predetermined range (for example, when the absolute value of the negative acceleration detected by the G sensor SN1 (i.e., the deceleration of the vehicle) becomes equal to or greater than the abnormality determination threshold A1). The abnormality detection unit 232 may detect an abnormal driving state when the brake pedal is suddenly depressed (the depression speed and depression amount are greater than the thresholds), when the steering angle (steering angle) suddenly changes (the steering angle change speed and change amount are greater than the thresholds), etc.

[0045] The near-miss judgment unit 233 judges whether or not a dangerous situation has occurred in the vehicle V1 when an abnormal driving state of the vehicle V1 is detected, using the judgment table 223. The judgment table 223 has a data group that specifies the relationship between behavior information related to the behavior of the vehicle V1, information related to the emotion index value of the driver D1, and events that have occurred in the vehicle V1 (events of dangerous situations, events of non-dangerous situations). The judgment table 223 is configured as a lookup table that uses the behavior information related to the behavior of the vehicle V1 and information related to the emotion index value of the driver D1 as input parameters and outputs the events that have occurred in the vehicle V1.

[0046] Fig. 6 is a diagram showing an example of the judgment table 223. The magnitude of change in the emotion index value may be, for example, the difference in output of the emotion index value estimation model 221 (here, the maximum value of the distance between coordinates consisting of arousal level and activity level at each time point) during the period from when the abnormal driving state of the vehicle V1 is detected by the abnormality detection unit 232 (or from a predetermined time (a suitable time is set based on experiments, etc.) before the detection of the abnormal driving state) until a predetermined time (a suitable time is set based on experiments, etc.) has elapsed. In other words, two coordinates with the maximum distance may be extracted from the multiple coordinates plotted on the psychological plane PP shown in Fig. 5, and the distance between the two extracted coordinates may be used.

[0047] The judgment table 223 is generated based on the characteristic that the greater the deceleration, the higher the possibility of a dangerous situation, and that the more dangerous the situation, the more likely the emotions of the person involved are to fluctuate.

[0048] 6 is used, for example, if the magnitude of the change in the emotion index value is less than B1, the near-miss judgment unit 233 judges that a dangerous situation (near-miss) has not occurred in the vehicle V1 and that the vehicle V1 has traveled on an abnormal road surface. On the other hand, for example, if the deceleration (the absolute value of the negative acceleration detected by the G sensor SN1) is greater than or equal to A3 and less than A4, and the magnitude of the change in the emotion index value is greater than or equal to B1 and less than B2, the near-miss judgment unit 233 judges that a dangerous situation (near-miss) has occurred in the vehicle V1.

[0049] In this example, the near-miss determination unit 233 determines a dangerous situation (near-miss) for the vehicle V1 based on a change in the emotion index value, but it is also possible to determine a dangerous situation (near-miss) for the vehicle V1 based on an emotion type or a change in the emotion type. This emotion type can be estimated, for example, by providing two types of emotion index values ​​(arousal level, activity level) to the psychology plane PP (psychology plane table 222) shown in FIG. 5, as described above.

[0050] In this way, when the vehicle V1 is in an abnormal driving state, the server 2 can identify whether or not a dangerous situation has occurred in the vehicle V1 based on the emotional state of a person (such as a driver) who is correlated with the dangerous situation in the vehicle V1, and can therefore appropriately detect that a dangerous situation has occurred in the vehicle V1. In other words, the server 2 can suppress erroneous detection that a dangerous situation has occurred in the vehicle V1 when a dangerous situation has not occurred in the vehicle V1.

[0051] When a dangerous situation (near miss) occurs in the vehicle V1, it is considered that the driver D1's emotion is more likely to change significantly than the driver D1 to have a specific emotion. Therefore, as described above, the server 2 determines whether or not a dangerous situation has occurred in the vehicle V1 based on the magnitude of the change in the emotion index value, and this is expected to improve the accuracy of identifying whether or not a dangerous situation has occurred in the vehicle V1.

[0052] In addition, since the server 2 can determine an abnormal driving state of the vehicle V1 when the vehicle V1 is not in a dangerous situation, that is, can determine whether the vehicle V1 is driving on an abnormal road surface, it is also possible to generate a database regarding abnormal road surfaces.

[0053] FIG. 7 is a diagram showing another example of the judgment table 223. In the judgment table 223 of FIG. 7, in addition to the magnitude of the change in the emotion index value, the direction of the change in the emotion index value is also defined as the change in the emotion index value. The direction of the change in the emotion index value may be, for example, the direction of the change in the emotion index value in which the difference in the output of the emotion index value estimation model 221 is maximum (here, the maximum value of the distance between the coordinates consisting of the arousal level and the activity level at each time point) during the period from when the abnormal driving state of the vehicle V1 is detected by the abnormality detection unit 232 (or from a predetermined time (a suitable time is set based on experiments, etc.) before the detection of the abnormal driving state) until a predetermined time (a suitable time is set based on experiments, etc.) has elapsed. In other words, the output of the emotion index value estimation model 221 may be set to the direction of a vector connecting the two extracted coordinates (connecting the coordinate with the earlier detection time to the coordinate with the later detection time) by extracting two coordinates with the maximum distance from the multiple coordinates plotted on the psychological plane PP shown in FIG. 5. The starting point of the vector is the coordinate that is plotted earlier in time out of the two extracted coordinates (the coordinates of the arousal level and the arousal level detected earlier), and the end point of the vector is the coordinate that is plotted later in time out of the two extracted coordinates (the coordinates of the arousal level and the arousal level detected later). The direction OR1 in the determination table of Fig. 7 may be, for example, a general direction from a quadrant other than the fourth quadrant of the psychological plane PP toward the fourth quadrant.

[0054] In this way, the server 2 determines whether or not a dangerous situation has occurred in the vehicle V1 based on the magnitude and direction of the change in the emotion index value. This allows the occurrence of a dangerous situation to be determined taking into account the direction of change in the emotional state of the person (driver, etc.) who is correlated with the dangerous situation in the vehicle V1, and is therefore expected to further improve the accuracy of identifying whether or not a dangerous situation has occurred in the vehicle V1.

[0055] The database generation unit 234 generates a database DB1 related to near misses using vehicle position information output from the GPS sensor SN2, a signal corresponding to the current time output from the timing device TM1, and the judgment result of the near miss judgment unit 233.

[0056] 8 is a diagram showing an example of the database DB1. The database DB1 has a data group that specifies the relationship between the point where an event occurred in the vehicle V1, the date and time when the event occurred in the vehicle V1, and the type of the event that occurred in the vehicle V1.

[0057] The database generating unit 234 uses the activation signal output from the airbag device AB1 to distinguish between the occurrence of a dangerous situation in the vehicle V1 and the occurrence of an accident in the vehicle V1, and corrects the database DB1 based on the identification result. This makes it possible to suppress erroneous detection of an accident as a near miss. For example, when the airbag device AB1 outputs an activation signal at xx year xx month xx day xx hour xx minute xx second, the database generating unit 234 corrects the database DB1 shown in FIG. 8 to the database DB1 shown in FIG. 9. Instead of correcting the database DB1 shown in FIG. 8, the database generating unit 234 may directly generate the database DB1 shown in FIG. 9 using the vehicle position information output from the GPS sensor SN2, the signal corresponding to the current time output from the timing device TM1, the determination result of the near miss determination unit 233, and the activation signal output from the airbag device AB1. In this embodiment, the airbag device AB1 detects that an accident has occurred in the vehicle V1. The accident detection means for detecting that an accident has occurred to the vehicle V1 may be, other than the airbag device AB1, an SOS call device that transmits an SOS call (rescue request signal) to an SOS call service center installed outside the vehicle V1 when the driver D1 presses a button.

[0058] The database DB1 may include data on attributes other than the location where the event occurred on the vehicle V1, the date and time when the event occurred on the vehicle V1, and the type of the event that occurred on the vehicle V1. For example, the database DB1 shown in Fig. 10 includes weather information at the location where the event occurred on the vehicle V1, road information at the location where the event occurred on the vehicle V1, behavior information of the vehicle V1 when the event occurred on the vehicle V1, information on the driver D1, vehicle type information of the vehicle V1, and image and audio data around the vehicle V1 when the event occurred on the vehicle V1.

[0059] Weather information at the point where the event occurred in the vehicle V1 can be obtained, for example, from a weather sensor installed near the point where the event occurred in the vehicle V1, a website that provides weather information, etc. Road information at the point where the event occurred in the vehicle V1 includes, for example, dynamic information such as the presence or absence of traffic congestion, and static information such as road width, number of lanes, and legal speed limit. The dynamic information is generated, for example, based on an image captured by an exterior camera that captures the surroundings of the vehicle V1, which is installed in the vehicle V1. The static information may be generated, for example, based on an image captured by the exterior camera, or may be extracted from the map data MD1. Behavior information of the vehicle V1 when the event occurred in the vehicle V1 includes, for example, the output of the G sensor SN1, the output of the steering angle sensor, the output of the vehicle speed sensor, the output of the brake pedal sensor, and the output of the accelerator pedal sensor. Information of the driver D1 includes, for example, the emotion of the driver D1 when the event occurred in the vehicle V1, the gender of the driver D1, and the age of the driver D1. The emotion of the driver D1 when an event occurs in the vehicle V1 is estimated using, for example, the psychological plane table 222 shown in Fig. 4, and the gender and age of the driver D1 are estimated based on, for example, an in-vehicle camera image. The vehicle model information of the vehicle V1 is generated and output, for example, by the in-vehicle device 1 (various information related to the vehicle model, etc., registered (stored) in advance in the in-vehicle device 1). The image and audio data around the vehicle V1 when an event occurs in the vehicle V1 is generated, for example, by an exterior camera that captures the surroundings of the vehicle V1 and an exterior microphone that captures the sounds around the vehicle V1, which are installed on the vehicle V1, based on the image information captured by these exterior cameras and the audio information output from the exterior microphone.

[0060] The data output unit 235 outputs the data extracted from the database DB1 to the in-vehicle device 1 via the communication unit 21. For example, the data output unit 235 outputs information on a point where an event has occurred in the vehicle V1, which corresponds to the vicinity of the current position of the vehicle V1, to the driving support device 3 via the communication unit 21, based on the vehicle position information output from the GPS sensor SN2.

[0061] The near-miss map generating unit 236 generates a near-miss map HP, which is an image in which icons F1, F2, and F3 indicating the points where the event occurred in the vehicle V1 and icons TA1 and TA2 indicating the accident points are superimposed on a map image MP generated from map data MD1, based on information on the points where a dangerous situation event (near-miss event) occurred in the vehicle V1 in the database DB1 (this may be information that integrates information on a plurality of vehicles), as shown in Fig. 11. Note that Fig. 11 is a diagram showing an example of the near-miss map HP.

[0062] The server 2 may not include the near-miss map generating unit 236 and the map data MD1, and a navigation device installed in the vehicle V1 may perform the function of the near-miss map generating unit 236. In addition, a device performing the function of the near-miss map generating unit 236 may be provided in each of the server 2 and the vehicle V1.

[0063] Furthermore, the icons F1 and the like to be superimposed on the near-miss map HP may be selected according to various attributes of the near-miss events contained in the database DB1. For example, a near-miss map HP in which only the icon F2 having the attribute of the evening time period is selected and superimposed on the map image F1 can be expected to enhance the educational effect when used for safety education regarding driving in the evening.

[0064] [Information processing flow chart] Fig. 12 is a flowchart of information processing executed by the controller 23. The information processing may be processing that is realized by the controller 13 executing the above-mentioned program PG1. The information processing shown in Fig. 12 starts when the server 2 is booted.

[0065] First, in step S11, the acquisition unit 131 acquires behavior information on the behavior of the vehicle V1 in the near-miss detection period, data for obtaining the emotion index value of the driver D1 in the near-miss detection period, and information used to generate the database DB1 in the near-miss detection period. The near-miss detection period is an arbitrary period having a predetermined time length (for example, several seconds to several tens of seconds). If no information is transmitted from the in-vehicle device 1 and the acquisition unit 131 cannot acquire information in the near-miss detection period, the acquisition unit 131 updates the near-miss detection period and repeats the same process. Since it is preferable to perform near-miss detection continuously, it is preferable that each near-miss detection period is continuous with no blank period between them. After step S10, the process proceeds to step S20.

[0066] In step S20, the abnormality detection unit 232 attempts to detect an abnormal driving state of the vehicle V1 based on the behavior information related to the behavior of the vehicle V1. If an abnormal driving state is detected, the process proceeds to step S30, and if an abnormal driving state is not detected, the process proceeds to step S70 without updating the database DB1.

[0067] In step S30, the near-miss determination unit 233 determines whether or not a dangerous situation has occurred in the vehicle V1 when the abnormal driving state of the vehicle V1 is detected, based on the emotion index value at that time. After step S30, the process proceeds to step S40.

[0068] In step S40, the database generation unit 234 reflects the judgment result of the near-miss judgment unit 233 in the database DB1. Specifically, if the near-miss judgment unit 233 judges that a dangerous situation occurred in the vehicle V1 when the abnormal driving state of the vehicle V1 was detected, the database generation unit 234 updates the database DB1 and adds the near-miss case to the database DB1. On the other hand, if the near-miss judgment unit 233 judges that a dangerous situation did not occur in the vehicle V1 when the abnormal driving state of the vehicle V1 was detected, the database generation unit 234 does not update the database DB1. After step S40, the process proceeds to step S50.

[0069] In step S50, the database generating unit 234 uses the activation signal (airbag deployment signal) output from the airbag device AB1 when the near-miss case added to the database DB1 to distinguish between the occurrence of a dangerous situation in the vehicle V1 and the occurrence of an accident in the vehicle V1 (deployment of the airbag is distinguished as the occurrence of an accident). After step S50, the process proceeds to step S60.

[0070] In step S60, the database generating unit 234 reflects the identification result in step S50 in the database DB1. Specifically, when the event in which it is determined that a dangerous situation has occurred in the vehicle V1 in the determination process in step S30 is identified as an event in which an accident has occurred in the vehicle V1 in the identification process in step S50, the database generating unit 234 corrects the event type of the corresponding event in the database DB1 to accident occurrence. On the other hand, when the event in which it is determined that a dangerous situation has occurred in the vehicle V1 in the determination process in step S30 is identified as an event in which a dangerous situation has occurred in the vehicle V1 in the identification process in step S50, the database generating unit 234 does not correct the database DB1. Note that the processes of steps S40 to S60 can be performed simultaneously instead of being performed sequentially. After step S60, the process proceeds to step S70.

[0071] In step S70, the data output unit 235 outputs the data extracted from the database DB1 to the in-vehicle device 1 via the communication unit 21, as necessary. After step S70, the process proceeds to step S80.

[0072] In step S80, the near-miss map generating unit 236 generates a near-miss map as necessary, and transmits the near-miss map to the in-vehicle device 1 when a transmission request is received, etc. After step S80, the process returns to step S10.

[0073] <Second embodiment> The second exemplary embodiment of the present invention is an embodiment based on the first embodiment described above. For matters not specifically described in the second embodiment, the description of the first embodiment also applies to the second embodiment unless there is a contradiction. However, when interpreting the description of the second embodiment, the description of the second embodiment may take precedence over any matters that contradict each other between the first and second embodiments.

[0074] In this embodiment, as shown in FIG. 13, the vehicle V1 is equipped with an exterior camera C2 and a distance measurement sensor SN3. The exterior camera C2 is composed of one or more cameras that capture the state outside the vehicle V1. The exterior camera C2 generates an exterior camera image by capturing the state of the capture area. The distance measurement sensor SN3 generates and outputs distance measurement information by performing distance measurement. In the distance measurement sensor SN3, the distance between the vehicle V1 and a solid object (three-dimensional object) located in the distance measurement area around the vehicle V1 is detected, and the orientation of the solid object as viewed from the vehicle V1 is also detected. These detection results are included in the distance measurement information. The distance measurement sensor SN3 may be composed of a LIDAR (Light Detection and Ranging) that performs distance measurement using light, or may be composed of a radar that performs distance measurement using radio waves. The distance measurement sensor SN3 may be composed of a combination of LIDAR and radar.

[0075] FIG. 14 is a diagram showing an example of the configuration of the server 2 according to the second exemplary embodiment of the present invention. The controller 23 has a function block 237. The function block 237 is a scene determination unit. The scene determination unit 237 determines the type of the scene surrounding the vehicle V1 based on the outside camera image. The scene determination unit 237 may determine the type of the scene surrounding the vehicle V1 based on the outside camera image and distance measurement information. By using not only the outside camera image but also the distance measurement information, it is expected that the accuracy of determining the type of the scene surrounding the vehicle V1 can be improved. As the type of the scene surrounding the vehicle V1, for example, a traffic jam scene (determined by, for example, the distance between the vehicle and other vehicles, the relative speed, and the vehicle speed), a scene in which an oncoming right-turning vehicle exists (determined by the direction of the oncoming vehicle, the positional relationship between the intersection and the oncoming vehicle, etc.), a scene in which the vehicle enters an intersection without a traffic light (determined by, for example, the vehicle speed and the distance between the intersection, etc.), etc. are assumed. These determinations are made based on detection information from various on-board sensors that detect the critical travel state, distance measurement information, image recognition processing results of camera images, etc.

[0076] The storage unit 22 stores a plurality of judgment tables 223. As shown in Fig. 15, the plurality of judgment tables 223 are configured by a judgment table 223_1 corresponding to a first scene, a judgment table 223_2 corresponding to a second scene, a judgment table 223_3 corresponding to a third scene, .... Therefore, the number of judgment tables 223 matches the number of scene types judged by the scene judgment unit 237. For these tables, appropriate scenes are selected through experiments or the like, and appropriate values ​​obtained through experiments or the like are set as each judgment data in each table for the selected scene.

[0077] Fig. 16 is a flowchart of information processing executed by the controller 23 according to this embodiment. The information processing may be processing that is realized by the controller 13 executing the above-mentioned program PG1. The information processing shown in Fig. 16 starts when the server 2 is started. Note that the same processes as those in Fig. 12 are given the same step numbers and detailed descriptions are omitted.

[0078] In the flowchart shown in Fig. 16, after step S10, the process proceeds to step S15. In step S15, the scene determination unit 237 determines the type of the scene surrounding the vehicle V1. After step S15, the process proceeds to step S20.

[0079] In step S30, the near-miss determination unit 233 determines whether or not a dangerous situation has occurred in the vehicle V1 by using the determination table 223 (223_1, 223_2, or 223_3) corresponding to the scene determined in step S15.

[0080] In this embodiment, the type of scene surrounding the vehicle V1 is also taken into consideration to determine whether or not a dangerous situation has occurred in the vehicle V1, making it possible to pick up missed detections of the occurrence of a dangerous situation that may occur in the first embodiment, in which whether or not a dangerous situation has occurred in the vehicle V1 is determined based only on behavior information regarding the behavior of the vehicle V1 and the emotion index value of the driver D1.

[0081] The type of scene determined by the scene determination unit 237 may be used to estimate the level (severity) of a near miss. For example, the storage unit 22 stores a near miss level estimation table shown in FIG. 17. The near miss level estimation table shown in FIG. 17 is configured as a lookup table in which the type of scene determined by the scene determination unit 237 and the distance to an obstacle detected by the distance measurement sensor SN3 are input parameters and the near miss level is output. Instead of the distance to an obstacle detected by the distance measurement sensor SN3, the vehicle speed of the vehicle V1 detected by the vehicle speed sensor may be used. For example, the aspect (color, size, line thickness, etc.) of the above-mentioned icon F2 can be changed according to the near miss level to include near miss level information in the near miss map F3.

[0082] <Notes, etc.> Various technical features disclosed in the description of the invention may be modified in various ways without departing from the spirit of the technical creation. Furthermore, the embodiments and modifications disclosed in the description of the invention may be combined to the extent possible.

[0083] In the above-described embodiment, the driver D1's wakefulness and activity level are estimated from the image information and speech information of the driver D1, but the driver D1's wakefulness and activity level may be estimated from the detection result of a biosensor that detects the driver D1's bioinformation. The biosensor includes at least an electroencephalogram sensor and a heartbeat sensor. The electroencephalogram sensor and the heartbeat sensor are attached to the driver D1. The electroencephalogram sensor detects the driver D1's brainwaves and outputs electroencephalogram data indicating the detection result of the brainwaves. That is, the electroencephalogram data includes information on the detected brainwaves. The heartbeat sensor detects the driver D1's heartbeat and outputs heartbeat data indicating the detection result of the heartbeat. That is, the heartbeat data includes information on the detected heartbeat. For example, a headgear-type electroencephalogram sensor is used as the electroencephalogram sensor. For example, a chest belt-type electrocardiogram-type heartbeat sensor is used as the heartbeat sensor. It is also possible to use an optical heartbeat sensor as the heartbeat sensor.

[0084] In the above-described embodiment, the information processing system SYS1 is configured to include the driving assistance device 3. However, the information processing system SYS1 may not be configured to include the driving assistance device 3, and the database DB1 may not be used for driving assistance of the vehicle V1. [Explanation of symbols]

[0085] 1...In-vehicle device 2. Server 3. Driving support device 21. Communications Department 22...Storage section 23. Controller AB1 Airbag device C1...In-car camera C2...Exterior camera D1...Driver M1: In-car microphone N1...Network SN1...G sensor SN2 GPS Sensor SN3... Distance Sensor SYS1: Information processing system TM1...Timekeeping device V1...Vehicle

Claims

1. Detecting an abnormal driving state of the vehicle based on behavior information relating to the behavior of the vehicle; A determination device that determines whether or not a dangerous situation has occurred when the abnormal driving state is detected, based on emotion information of the driver when the abnormal driving state is detected.

2. The determination device according to claim 1 , wherein the emotion information is an emotion index value indicating a biological state correlated with an emotion.

3. The determination device according to claim 2 , wherein the emotion information is a magnitude of change in the emotion index value.

4. 3. The determination device according to claim 2, wherein if a magnitude of the change in the emotion index value is less than a threshold value, it is determined that the vehicle was traveling on an abnormal road surface when the abnormal driving state was detected.

5. The determination device according to claim 2 , wherein the emotion information is a direction of change of the emotion index value.

6. 2. The determination device according to claim 1, further comprising: a determination unit that determines a type of scene based on an image captured by a camera capturing an image of the surroundings of the vehicle; and determines whether or not a dangerous situation has occurred based on a determination criterion corresponding to the type of scene.

7. 2. The determination device according to claim 1, wherein when an accident is detected, the determination device determines that an accident has occurred without determining that a dangerous situation has occurred.

8. The determination device according to claim 2 , wherein the emotion index value is arousal level and activity level indicating a mental state.

9. The determination device according to claim 2 , wherein the emotion information is emotion type information based on an emotion type estimated from an arousal level and an activity level indicating a mental state.

10. An in-vehicle device that collects behavior information related to the behavior of a vehicle and emotion information of a driver; An information processing system including an information processing device, The information processing device includes: Detecting abnormal driving state information and abnormal driving state occurrence position information of the vehicle based on the behavior information; determining whether a dangerous situation has occurred when the abnormal driving state is detected based on the emotion information when the abnormal driving state is detected; An information processing system that generates a database of dangerous situation occurrence location information based on abnormal driving condition occurrence location information corresponding to the determination result that the dangerous situation has occurred.

11. An in-vehicle device that collects behavior information related to the behavior of a vehicle and emotion information of a driver; An information processing device; A driving assistance device installed in the vehicle, The information processing device includes: Detecting abnormal driving state information and abnormal driving state occurrence position information of the vehicle based on the behavior information; determining whether a dangerous situation has occurred in the vehicle when the abnormal driving state is detected based on the emotion information when the abnormal driving state is detected; generating a database of dangerous situation occurrence location information based on abnormal driving state occurrence location information corresponding to the determination result that the dangerous situation has occurred; The driving assistance device includes: An information processing system that provides driving support for the vehicle by utilizing the dangerous situation occurrence location information in the database.

12. Detecting an abnormal driving state of the vehicle based on behavior information relating to a behavior of the vehicle; determining whether a dangerous situation has occurred when the abnormal driving state is detected based on emotion information of the driver when the abnormal driving state is detected; A judgment program that causes a computer to execute the above.

Images