Estimation device, estimation method, and program

Abstract

To improve a degree of freedom in information to be outputted for estimating a state of a driver through a response of the driver.SOLUTION: An estimation device 10 is equipped with an output part 120, a detection part 140, and an estimation part 160. The output part 120 outputs demand response information requiring a response to a driver of a moving body. The detection part 140 detects the response of the driver to the demand response information. The estimation part 160 estimates a state of the driver on the basis of at least one of appropriateness or timing of the response.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an estimation device, an estimation method, and a program. [Background technology]

[0002] There is technology that can predict the drowsiness and fatigue level of drivers of vehicles and other vehicles, and provide warnings as needed.

[0003] Patent Document 1 describes a method for predicting a driver's risk of drowsiness based on their answers to questions about diet and sleep. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2019-61480 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, according to the technology described in Patent Document 1, the driver would have to answer the same questions repeatedly asked by the driver assistance device each time. As a result, there was a possibility that the driver would become bored with the questions or give the same pattern of answers each time, leading to a decrease in prediction accuracy.

[0006] One example of a problem that this invention aims to solve is to increase the degree of freedom of the information output to estimate the driver's state through the driver's response. [Means for solving the problem]

[0007] The invention described in claim 1 is, An output unit that outputs response-required information to the driver of a moving object, A detection unit for detecting the driver's response to the aforementioned information requiring a response, An estimation unit that estimates the state of the driver based on at least one of the appropriateness and timing of the response, and It is an estimation device.

[0008] The invention according to claim 13 is An estimation method executed by one or more computers, comprising An output step of outputting response-required information for requesting a response from the driver of the moving body, A detection step of detecting the driver's response to the output of the response-required information, And an estimation step of estimating the state of the driver based on at least one of the appropriateness and timing of the response. It is an estimation method.

[0009] The invention according to claim 14 is A program for causing a computer to execute the estimation method according to claim 13.

Brief Description of Drawings

[0010] [Figure 1] It is a block diagram illustrating the functional configuration of an estimation device according to an embodiment. [Figure 2] It is a flowchart illustrating the flow of an estimation method according to an embodiment. [Figure 3] It is a diagram illustrating a computer for realizing an estimation device. [Figure 4] It is a diagram showing a first example of the relationship between driving load and the processing performed by the estimation device. [Figure 5] It is a diagram showing a second example of the relationship between driving load and the processing performed by the estimation device.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and the description will be omitted as appropriate.

[0012] (Embodiment) Figure 1 is a block diagram illustrating the functional configuration of the estimation device 10 according to this embodiment. The estimation device 10 according to this embodiment comprises an output unit 120, a detection unit 140, and an estimation unit 160. The output unit 120 outputs response-required information that requests a response from the driver of a moving object. The detection unit 140 detects the driver's response to the response-required information. The estimation unit 160 estimates the driver's state based on at least one of the appropriateness and timing of the response.

[0013] In this embodiment, the mobile body is not particularly limited, but may be a vehicle, an aircraft, or a ship. Examples of vehicles include trains, four-wheeled vehicles, two-wheeled vehicles, etc. Examples of aircraft include drones, aircraft, etc. The driver of the mobile body may be on board the mobile body or may be driving the mobile body remotely. The estimation device 10 according to this embodiment may or may not be mounted on the mobile body.

[0014] Information requiring a response may include, for example, a question directed at the driver. Examples of questions include inquiries, suggestions, and appeals. The driver's response to information requiring a response may be an answer to the question, or it may be an acknowledgment. Examples of answers to questions may include affirmative responses such as "yes" or "uh-huh," negative responses such as "no" or "no," or responses other than affirmation or negation. The driver's response to information requiring a response may include auditory responses such as vocalizations, or physical responses such as nodding.

[0015] The output of information requiring a response may be, for example, audio output from a speaker or image output from a display.

[0016] The detection unit 140 can detect the driver's response by means of sound detection using a microphone, image detection using a camera, motion detection using sensors, etc.

[0017] The estimation unit 160 estimates the driver's state based on at least one of the appropriateness and timing of the response. Therefore, the driver's state can be estimated through a wide variety of questions and other means, which offer a high degree of freedom in the information requiring a response.

[0018] The driver's condition refers to, for example, the driver's level of fatigue or safety margin. Fatigue may include the degree of drowsiness. In the following, the description of fatigue can be reinterpreted as a description of safety margin by reversing the order of the fatigue levels.

[0019] Figure 2 is a flowchart illustrating the flow of the estimation method according to this embodiment. The estimation method according to this embodiment is executed by one or more computers. The estimation method includes an output step S10, a detection step S20, and an estimation step S30. In the output step S10, response-requiring information is output, requesting a response from the driver of the moving object. In the detection step S20, the driver's response to the output of the response-requiring information is detected. In the estimation step S30, the driver's state is estimated based on at least one of the appropriateness and timing of the response.

[0020] The estimation method according to this embodiment can be performed by the estimation device 10 according to this embodiment.

[0021] As described above, according to this embodiment, the estimation unit 160 estimates the driver's state based on at least one of the appropriateness and timing of the response. Therefore, the driver's state can be estimated through a variety of questions and the like.

[0022] (Example 1) The estimation device 10 according to Embodiment 1 has the same configuration as the estimation device 10 according to the embodiment. In the estimation device 10 according to this embodiment, the estimation unit 160 estimates the driver's state based at least on the response timing. The functional configuration of the estimation device 10 according to this embodiment is illustrated in Figure 1. It will be described in detail below.

[0023] In this embodiment, the estimation unit 160 determines the response time to the output of the response-requiring information based on the timing of the response. The estimation unit 160 then estimates the driver's state based on the response time.

[0024] The response time is, for example, the time from when the output unit 120 outputs information requiring a response until the detection unit 140 detects the driver's response. If the information requiring a response is output as voice, the response time is specifically the time from when the output, such as a question, ends until the driver's response begins. The response time can be the time until a voice response is uttered, or it can be the time until an action such as nodding begins.

[0025] The estimation unit 160 determines that the driver is fatigued if the response time is longer than a predetermined time T. The estimation unit 160 further estimates that the driver's fatigue level is higher the greater the response time exceeds time T. Time T is not particularly limited, but for example, it is between 0 seconds and 10 seconds. The estimation unit 160 can output one or more of the following as estimation results: information indicating whether or not the driver is fatigued, the driver's fatigue level, and the driver's margin level. The fatigue level may be a numerical value or symbol indicating the intensity of fatigue. The margin level may be a numerical value or symbol indicating the magnitude of the margin.

[0026] The estimation unit 160 may also determine the driver's state using the average of multiple response times. That is, the output unit 120 outputs multiple response-requiring information, the detection unit 140 detects the driver's response to each response-requiring information, and the estimation unit 160 identifies the response time for each response. The estimation unit 160 then calculates the average of the identified multiple response times and estimates the driver's state by comparing the average with time T.

[0027] The estimation unit 160 may determine the fatigue level using reference information that shows the relationship between response time or the average response time and the fatigue level. The reference information is stored in the storage unit 100 in advance and can be read and used by the estimation unit 160. The reference information may be a table or a mathematical formula, etc. The estimation unit 160 extracts the fatigue level corresponding to the determined response time or the average response time from the reference information and uses the extracted fatigue level as the driver's fatigue level. The storage unit 100 may be provided in the estimation device 10 or may be provided outside the estimation device 10. If the storage unit 100 is provided inside the estimation device 10, for example, the storage unit 100 may be implemented using a storage device 1080, which will be described later.

[0028] For example, the output unit 120 outputs a question as information requiring a response. If the driver hesitates and cannot answer immediately, it is determined that the driver is highly fatigued. In that case, the output unit 120 further outputs a message prompting the driver to take a break. On the other hand, if the driver can answer immediately, it is determined that the driver is not highly fatigued. In that case, for example, the output unit 120 may continue the conversation based on the answer. For example, suppose the output unit 120 outputs the question "What did you eat last night?" as information requiring a response, and the driver immediately answers "Omurice." In that case, the estimation unit 160 determines that the driver is not highly fatigued, and based on the keywords in the answer, the output unit 120 may further output information such as "Speaking of omurice, here are some recommended restaurants..." Also, if the estimation unit 160 determines that the driver is not highly fatigued, the output unit 120 may continue with questions or conversations to identify the driver's needs and personality. According to the estimation device 10, it is possible to estimate the driver's state through a natural conversation-like interaction between the estimation device 10 and the driver. The estimation unit 160 may estimate the driver's state each time the detection unit 140 detects the driver's response to the output of the output unit 120. However, the estimation unit 160 may not estimate the driver's state even if the detection unit 140 detects the driver's response to the output of the output unit 120.

[0029] The hardware configuration of the estimation device 10 is described below. Each functional component of the estimation device 10 may be implemented by hardware that realizes each functional component (e.g., hardwired electronic circuits), or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it). The case in which each functional component of the estimation device 10 is implemented by a combination of hardware and software will be further explained below.

[0030] Figure 3 illustrates a computer 1000 for implementing the estimation device 10. Computer 1000 is any computer. For example, computer 1000 may be an SoC (System on Chip), a Personal Computer (PC), a server machine, a tablet terminal, or a smartphone. Computer 1000 may be a dedicated computer designed to implement the estimation device 10, or it may be a general-purpose computer.

[0031] Computer 1000 includes a bus 1020, a processor 1040, memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. Bus 1020 is a data transmission path for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data to and from each other. However, the method of connecting the processor 1040 and the other components is not limited to bus connection. The processor 1040 is a variety of processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or FPGA (Field-Programmable Gate Array). Memory 1060 is a main memory device implemented using RAM (Random Access Memory), etc. Storage device 1080 is an auxiliary storage device implemented using a hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

[0032] The input / output interface 1100 is an interface for connecting the computer 1000 with input / output devices. For example, input devices such as keyboards, microphones, cameras, and sensors, and output devices such as displays and speakers are connected to the input / output interface 1100. The method by which the input / output interface 1100 connects to the input and output devices may be wireless or wired.

[0033] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method by which the network interface 1120 connects to the network may be wireless or wired.

[0034] The storage device 1080 stores program modules that realize each functional component of the estimation device 10. The processor 1040 reads each of these program modules into the memory 1060 and executes them to realize the function corresponding to each program module.

[0035] The following describes in detail each functional component of the estimation device 10 according to this embodiment.

[0036] The output unit 120 determines the response information to be output based on at least one of the current location, weather, date, time, and destination of the moving object. The output unit 120 may also select the response information to be output from a plurality of response information pre-held in the storage unit 100 accessible from the output unit 120, based on at least one of the location, weather, date, time, and destination of the moving object. In this case, each of the plurality of response information is pre-associated with one or more combinations of a specific location, specific weather, specific date, specific time, and specific destination. The output unit 120 extracts the response information corresponding to one or more combinations of the location, weather, date, time, and destination of the moving object from the plurality of response information when outputting the response information. In doing so, it identifies the response information to be output.

[0037] Alternatively, the output unit 120 may determine the response information to be output using a model that takes at least one of the moving object's position, weather, date, time, and destination as input and outputs the response information. This model may be, for example, a machine learning-trained model and may include a neural network. The response information may be generated by the model.

[0038] The response information output by the output unit 120 can be information related to at least one of the current location of the moving object, weather, date, time, and destination. For example, the output unit 120 may select response information such as "It's been a long time since we've had such nice weather, hasn't it?" based on the current and past weather. For example, the output unit 120 may select response information such as "It's gotten quite cold, hasn't it? Do you like winter?" based on the current date and temperature. For example, the output unit 120 may select response information such as "Are you getting hungry?" based on the current time. For example, the output unit 120 may select response information such as "Shall we go into the restaurant on the right up ahead?" based on the current location of the moving object. For example, the output unit 120 may select response information such as "What would you like to eat at the restaurant at your destination?" based on the destination. However, the output unit 120 may also output response information that is not related to any of the current location of the moving object, weather, date, time, or destination, such as "What did you eat last night?" The information requiring a response may be questions designed to estimate the driver's personality, such as "Do you prefer dogs or cats?" or "Do you eat udon or ramen more often?". The output unit 120 may also generate information requiring a response by combining multiple questions.

[0039] The output unit 120 can acquire information indicating the current position of the mobile object from the GNSS (Global Navigation Satellite System) receiver mounted on the mobile object. The output unit 120 can acquire past and present weather information, as well as future weather forecast information, from, for example, a weather information server via a communication network. If the mobile object is navigating towards a destination, the output unit 120 can acquire information indicating the mobile object's destination from the mobile object's navigation system or autonomous driving system, etc.

[0040] The output unit 120 can output a variety of questions by determining the response information to be output based on at least one of the current location of the moving object, weather, date, time, and destination. Therefore, it is possible to avoid the user becoming bored with the questions and the decrease in estimation accuracy due to patterned answers.

[0041] The output unit 120 outputs information requiring a response. For example, the output unit 120 outputs sound data to a speaker connected to the estimation device 10 to output the information requiring a response as an audio signal. Alternatively, the output unit 120 outputs image data to a display connected to the estimation device 10 to display the information requiring a response. The image data may be a still image or data that constitutes a video. These speakers and displays are provided so that the driver can recognize the output information. For example, the speakers and displays are installed inside a mobile vehicle.

[0042] The output unit 120 may further identify advertising information to be output based on the driver's response, and output the identified advertising information when the estimated driver's condition meets predetermined conditions. The response may be, for example, the answer to a question to identify the personality. Personality includes preferences, strengths and weaknesses, etc. Alternatively, the response may be the answer to the most recent information requiring a response, and may indicate the driver's mood or state, such as being hungry or what they want to do. When predetermined conditions are met, for example, when the driver's fatigue level is not high, specifically when the estimation unit 160 determines that the driver is not fatigued, when the estimated driver's fatigue level is less than a predetermined fatigue level, or when the estimated driver's stamina level is above a predetermined stamina level.

[0043] For example, each of the multiple pieces of advertising information pre-stored in the memory unit 100 is associated with information indicating a specific personality. The output unit 120 extracts advertising information corresponding to the personality estimated based on the content of the driver's response. In this way, the output unit 120 can select and output advertising information from among the multiple pieces of advertising information according to the driver's preferences. Alternatively, the output unit 120 may further identify and output advertising information based on the location of the moving object. In addition, the output unit 120 may generate and output advertising information based on map information or the like.

[0044] The output of advertising information, like the output of information requiring a response, can be, for example, audio output from a speaker or image output from a display.

[0045] The detection unit 140 detects the driver's response. The detection unit 140 may detect answers to questions, etc., as the driver's response, or it may detect the driver's nods in response to information requiring a response. The detection unit 140 can acquire sound data, including the driver's voice, etc., from a microphone connected to the estimation device 10. This microphone is installed, for example, on the dashboard of the driver's seat of a mobile vehicle, on the steering wheel, etc. The detection unit 140 can detect the driver's response to information requiring a response by analyzing the acquired sound data using existing technology.

[0046] In addition, the detection unit 140 may acquire image data including at least a portion of the driver from a camera connected to the estimation device 10. This image data may constitute a video. The detection unit 140 recognizes the driver's behavior by analyzing the acquired image data using existing technology. The detection unit 140 detects the driver's nodding or head shaking behavior as a response to information requiring a response.

[0047] A moving object may contain other people besides the driver. The method by which the detection unit 140 identifies a response from the driver is described below. The detection unit 140 detects, for example, whether or not there is mouth movement of the person sitting in the driver's seat using an image. It then identifies the sound data obtained when the mouth of the person sitting in the driver's seat is moving. It then assumes that the driver's voice is included in that sound data. Alternatively, the detection unit 140 may identify the driver's voiceprint based on the driver's utterance identified in this way. In that case, the driver's voice can be identified from the sound data without using an image. In addition, the detection unit 140 may identify the driver's voice by identifying the location of voice generation using sound data from multiple microphones. For example, among the multiple microphones, the voice that is acquired more loudly in the sound data obtained from the microphone closest to the driver's seat can be identified as the driver's voice.

[0048] The detection unit 140 may detect a nod in the image only if it cannot detect an audible response. Also, if the detection unit 140 cannot detect a response from the driver within a predetermined time after outputting the response request information, the output unit 120 may output the same response request information again, or another response request information such as "Are you listening?", to prompt the driver to respond. Alternatively, if the detection unit 140 cannot detect a response from the driver within a predetermined time after outputting the response request information, the estimation unit 160 may determine that the driver is highly fatigued. In this case, the estimation unit 160 sets a predetermined fatigue level as the driver's fatigue level.

[0049] The estimation unit 160 determines the response time based on the output timing of the response information and the driver's response timing to that information. The response time is as described above. A signal indicating the output timing of the response information can be obtained from the output unit 120. In addition, if the response information is output as audio, the output timing of the response information can be determined from the sound data of the microphone via the detection unit 140. If the response information is output as an image, the output timing of the response information can be determined from the image data of the camera via the detection unit 140. On the other hand, the estimation unit 160 determines the driver's response timing based on the detection result of the detection unit 140. The estimation unit 160 determines the time from the output timing of the response information to the driver's response timing to that information as the response time.

[0050] As described above, the estimation unit 160 estimates the driver's state by comparing the identified response time with a predetermined time T. Alternatively, 160 estimates the driver's state using the identified response time and the reference information described above. Depending on the estimation result of the estimation unit 160, the output unit 120 may further output information to the driver encouraging them to take a break, such as "Please take a break." For example, if the estimation unit 160 determines that the driver is fatigued, and the driver's fatigue level is above a predetermined fatigue level, or the driver's margin of safety is below a predetermined margin of safety, the output unit 120 outputs information encouraging the driver to take a break. On the other hand, if the estimation unit 160 determines that the driver is not fatigued, and the driver's fatigue level is below a predetermined fatigue level, or the driver's margin of safety is above a predetermined margin of safety, the output unit 120 does not output information encouraging the driver to take a break.

[0051] The output of information prompting a break, like the output of information requiring a response, can be, for example, audio output from a speaker or image output from a display.

[0052] <Output timing and estimated timing> In the estimation device 10, at least one of the timing for the output unit 120 to output response information and the timing for the estimation unit 160 to estimate the driver's state may be determined based on at least one of the following: the current position of the moving body, the surrounding conditions of the moving body, the conditions inside the moving body, and the movement of the moving body. Specifically, the magnitude of the driving load is determined based on at least one of the following: the current position of the moving body, the surrounding conditions of the moving body, the conditions inside the moving body, and the movement of the moving body. Then, at least one of the timing for the output unit 120 to output response information and the timing for the estimation unit 160 to estimate the driver's state is determined based on the magnitude of the driving load.

[0053] Figure 4 shows a first example of the relationship between the operating load and the processing performed by the estimation device 10. If response information is output when the operating load is high while the driver is operating the vehicle, there is a possibility that the response delay may occur due to the effects of the operating load rather than the driver's fatigue. To suppress such effects, it is preferable that the output unit 120 outputs response information only during periods when the operating load is low. Alternatively, it is preferable that the output unit 120 outputs response information during periods when the operating load is moderate, while the estimation unit 160 does not perform driver state estimation using the response to that response information.

[0054] In the example shown in this figure, the output unit 120 outputs response-required information when the operating load index or level is less than the second threshold. The estimation unit 160 then estimates the driver's state based on the driver's response to that response-required information. The output unit 120 does not output response-required information when the operating load index or level is greater than or equal to the second threshold but less than the first threshold. Alternatively, the output unit 120 may output response-required information when the operating load index or level is greater than or equal to the second threshold but less than the first threshold, in which case the estimation unit 160 does not perform driver state estimation using the response to that response-required information. Furthermore, the output unit 120 does not output response-required information when the operating load index or level is greater than or equal to the first threshold. Thus, it is preferable to determine the timing of the output of response-required information by the output unit 120 and the timing of the estimation by the estimation unit 160 using that response-required information according to the operating load.

[0055] The method for determining the operating load is described below. Below, an example is described in which the output unit 120 determines the operating load, but the determination of the operating load may be performed by the output unit 120 or by the estimation unit 160 in a similar manner.

[0056] The output unit 120 can identify the operating load based on at least one of the following: the current position of the mobile body, the conditions around the mobile body, the conditions inside the mobile body, and the operation of the mobile body. The operating load can also be described, for example, as the difficulty of operation.

[0057] The output unit 120 can determine the driving load based on the current location of the moving object as follows: The output unit 120 acquires map information of the area around the moving object from, for example, the storage unit 100. The output unit 120 also acquires the current location of the moving object from the GNSS receiver mounted on the moving object. The output unit 120 then identifies the type of area the moving object is moving through by comparing the current location with the map information. The output unit 120 then determines the driving load according to the location of the moving object. For example, if the moving object is a vehicle, the driving load is determined to be greater when it is located at an intersection, highway, multi-lane road, narrow road, curve, busy area, parking lot, or toll booth than when it is located elsewhere.

[0058] The output unit 120 can determine the operating load based on the surrounding conditions of the mobile vehicle as follows: The output unit 120 acquires information indicating the surrounding conditions of the mobile vehicle from cameras and sensors (such as millimeter-wave sensors and LiDAR) mounted on the mobile vehicle. It then determines that the operating load is greater than in other cases when the area around the mobile vehicle is congested, the road surface is uneven, or the road surface is wet. The output unit 120 also acquires past and present weather information from a server that provides weather information. Alternatively, the output unit 120 determines the current weather from information obtained from sensors installed on the mobile vehicle. It then determines that the operating load is greater than in other cases when it is currently raining, snowing, sleeting, or when fog is present. The output unit 120 may also determine the magnitude of the operating load according to the amount of rainfall or snowfall. Furthermore, based on the current time, the output unit 120 determines that the operating load is greater at times when the surroundings are dark, such as at night, in the evening, or at dawn, than at other times. The output unit 120 may also determine the magnitude of the operating load according to the darkness.

[0059] The output unit 120 can determine the operating load based on the conditions inside the mobile vehicle as follows. The output unit 120 acquires information indicating the conditions inside the vehicle from cameras installed to capture images of the inside of the mobile vehicle and microphones to acquire sounds inside the mobile vehicle. The output unit 120 then determines that the operating load is greater than at other times, for example, when a conversation is taking place inside the mobile vehicle.

[0060] The output unit 120 can determine the driving load based on the movement of the moving body as follows: The period from when the navigation device or the like starts guiding the vehicle to turn right or left at an intersection until the vehicle completes the turn, or the period during which the vehicle's speed is above a predetermined speed, may be determined to have a greater driving load than other periods. The output unit 120 can acquire information such as the steering state and speed of the vehicle from the vehicle's control device or various sensors.

[0061] The operating load can be indicated, for example, by a level or index that shows the magnitude of the load. For example, the level or index of the operating load can also be calculated based on a combination of multiple elements as described above (such as the current position of the moving body, the conditions around the moving body, the conditions inside the moving body, and the movement of the moving body). For example, the output unit 120 calculates a score indicating the height of the operating load for each element, and uses the calculated value obtained using the multiple scores (for example, a sum, a weighted sum, or a value obtained as an average) as the operating load index. Alternatively, the operating load index can be converted to an operating load level by classifying it into levels based on the range to which the index belongs.

[0062] In the estimation device 10, the timing at which the output unit 120 outputs information requiring a response and the timing at which the estimation unit 160 estimates the driver's state may be determined using the driver's biometric information. The biometric information includes information such as pulse rate, eye movements, blinking, facial expressions, and body movements. By outputting information requiring a response at a timing based on the biometric information, unnecessary questions to the driver can be reduced, and the state can be estimated effectively.

[0063] For example, the driver's seat and steering wheel are equipped with one or more sensors (such as pulse sensors and vibration sensors) to acquire the driver's biometric information. Cameras are also provided around the driver's seat to capture images of the driver's face and eyes. Furthermore, the driver may wear a device (such as a wristband or goggles) to acquire biometric information while driving. The output unit 120 can acquire the driver's biometric information from these sensors, cameras, and devices. The output unit 120 then analyzes the acquired biometric information and outputs a response-required information triggered, for example, when fatigue is suspected. Examples of situations that may indicate fatigue include when the most recent average pulse rate changes by more than a predetermined value compared to the average pulse rate within a predetermined period immediately after starting to drive, when the eyes dart around, when blinking occurs at a frequency exceeding a predetermined rate, when the driver has a tired expression, or when the driver moves their body at a frequency exceeding a predetermined rate. Note that a tired expression can be detected by analyzing facial images using existing technology. Furthermore, the output unit 120 may change the content of the response-required information output according to the analyzed state of the driver. For example, if the driver is analyzed to be drowsy, it may ask, "Are you feeling sleepy?", or if the driver is analyzed to be tense, it may ask, "Why don't you take a deep breath?"

[0064] For example, the output unit 120 determines at predetermined intervals whether or not to output information requiring a response after the driver starts driving. Then, according to the method described above, if it is possible to output information requiring a response, the output unit 120 outputs the information requiring a response.

[0065] <Response time correction> The estimation unit 160 may correct the response time based on at least one of the following: the current position of the moving object, the conditions around the moving object, the conditions inside the moving object, and the movement of the moving object, and estimate the driver's state based on the corrected response time. Specifically, the estimation unit 160 identifies the magnitude of the driving load based on at least one of the following: the current position of the moving object, the conditions around the moving object, the conditions inside the moving object, and the movement of the moving object. The estimation unit 160 also corrects the response time based on the magnitude of the driving load. Then, the estimation unit 160 estimates the driver's state based on the corrected response time. The method for identifying the driving load is as described above.

[0066] When the operating load is heavy, it is expected that the response time will be longer than usual. Therefore, by correcting the response time according to the weight of the operating load and then estimating the driver's condition, the estimation accuracy can be improved.

[0067] Figure 5 shows a second example of the relationship between the operating load and the processing performed by the estimation device 10. In this example, the output unit 120 outputs response information when the operating load index or level is less than the second threshold. The estimation unit 160 then estimates the driver's state based on the response time to the response information, as described above. At this time, no correction is made to the response time based on the operating load. The output unit 120 also outputs response information when the operating load index or level is greater than or equal to the second threshold and less than the first threshold. The estimation unit 160 then identifies the response time to the response information and further corrects that response time. For example, the estimation unit 160 performs the correction by multiplying the identified response time by a correction coefficient. The estimation unit 160 estimates the driver's state by comparing the corrected response time with a predetermined time T. The correction coefficient is, for example, 0.5 or more and 0.8 or less. The correction coefficient may be a predetermined fixed value or a value corresponding to the magnitude of the operating load. The output unit 120 does not output response information when the operating load index or level is equal to or greater than the first threshold.

[0068] In addition, the estimation unit 160 may change the time T or reference information used for estimation according to the operating load. In this case, the storage unit 100 stores in advance information indicating the time T for each operating load or reference information for each operating load. The estimation unit 160 then reads the time T or reference information corresponding to the operating load from the storage unit 100 and uses it for estimation.

[0069] In addition, the estimation unit 160 may correct the response time for each type of information requiring a response. This is because the standard response time may vary depending on the content of the question, as the time it takes to think can differ. For example, statistical values ​​(e.g., deviations) of the response times of multiple drivers for each type of information requiring a response can be obtained in advance and stored in the memory unit 100. Then, the estimation unit 160 corrects the response time for each type of information requiring a response based on these statistical values. The estimation unit 160 estimates the driver's state by comparing the corrected response time with a predetermined time T.

[0070] <Reference time T S > It is assumed that there are some individual differences in response time to questions, etc. Therefore, the estimation unit 160 sets a reference time T based on the driver's past response times. S The driver's state may be estimated based on a comparison with the driver's response time in this instance. Reference time T S For example, this is a time period determined based on the timing of responses detected by the detection unit 140 during the period from when the driver started driving the mobile object until a predetermined time has elapsed. The predetermined time is, for example, 5 minutes or more and 30 minutes or less. During the period when a long time has not elapsed since the start of driving, the driver can be considered not to be fatigued. Therefore, the response time during such a period is set to the reference time T. S By using it in this way, the estimation accuracy can be improved.

[0071] Once the driver starts operation, the output unit 120 outputs the first response-required information before a predetermined time has elapsed. The estimation unit 160 then determines the response time for that response-required information and sets the reference time T SStore it as such. Then, while the driver continues to drive, each time the response time is specified, the estimation unit 160 subtracts the reference time T S from the specified response time. By doing so, the estimation unit 160 calculates the difference between the response time and the reference time T S . The estimation unit 160 estimates that the higher the calculated difference, the higher the driver's fatigue level. Note that the reference time T S may be used as the predetermined time T described above.

[0072] As another example, the reference time T S may be a statistical value of the driver's past response times for each position of the moving body, for each time, or for each combination of the position and time of the moving body. The statistical value is, for example, an average value.

[0073] For example, if there is a position where the driver often boards the moving body, the output unit 120 is set to output the response-required information at that position. Then, when the estimation unit 160 specifies the response time, it associates the response time with the position and stores it in the storage unit 100. Further, the estimation unit 160 calculates the average value of the plurality of response times obtained at that position as the reference time T S for that position. Then, when the response-required information is output at that position and the response time is specified next, the estimation unit 160 compares the specified response time with the reference time T S for that position as described above, and estimates the driver's state. Since the driving load is likely to be the same at the same position, the estimation accuracy can be improved by such a method.

[0074] For example, if there is a time period when the driver often boards the moving body on a daily basis, the output unit 120 is set to output the response-required information during that time period. Note that the time period means a period between a certain time and a certain time. Then, when the estimation unit 160 specifies the response time, it associates the response time with the time period and stores it in the storage unit 100. Further, the estimation unit 160 calculates the average value of the plurality of response times obtained during that time period as the reference time T SIt is calculated as follows. Then, when response information is output during that time period and the response time is determined, the estimation unit 160 calculates the determined response time and the reference time T for that time period, in the same manner as described above. S By comparing these, the driver's condition is estimated. Since the driving load is likely to be the same during the same time period, this method can improve the accuracy of the estimation.

[0075] For example, if a person frequently passes through the same location at the same time for commuting to work or school, the output unit 120 is configured to output response information at that time and location. The estimation unit 160 then identifies the response time and stores it in the storage unit 100, associating it with the time and location combination. The estimation unit 160 also uses the average value of multiple response times obtained at that time and location as a reference time T for that time and location combination. S It is calculated as follows. Then, when response information is output for that time period and location, and the response time is determined, the estimation unit 160 calculates the reference time T for the combination of the determined response time and that time period and location, in the same manner as described above. S By comparing these, the driver's condition is estimated. Since the driving load is likely to be the same at the same time and location, this method can improve the accuracy of the estimation.

[0076] Furthermore, the memory unit 100 contains a reference time T specified for each driver. S Statistical values ​​may be stored. For example, the memory unit 100 further stores information indicating a personal identifier and a voiceprint in association, and the driver's identity is determined by the voiceprint. Instead of a voiceprint, the individual may be identified by facial recognition. For example, each time the driver drives the vehicle, the reference time T as described above is stored. S The driver's personal identifier is identified. Then, the reference time T of the driver's personal identifier is stored in the memory unit 100. S The statistics for the newly identified reference time T S It is updated using the driver's reference time T. The estimation unit 160 uses the driver's reference time T. S The statistical values ​​are based on the reference time T.S It can be used instead, and the driver's condition can be estimated in the same way as described above.

[0077] In addition, the estimation unit 160 may further estimate the driver's condition using the driver's biometric information. The biometric information is as described above. For example, the estimation unit 160 calculates a first score indicating fatigue level or margin using response time, and calculates a second score indicating fatigue level or margin using biometric information. The estimation result may then be a calculated value obtained using the first score and the second score (for example, a value obtained as a sum, weighted sum, or average). For example, as described above, the estimation unit 160 raises the second score higher than in other cases when fatigue is suspected.

[0078] The estimation unit 160 may further estimate the driver's condition using the driving duration, the cumulative value of an index indicating the driving load, etc. The estimation unit 160 estimates that the longer the driving duration, the higher the degree of fatigue. Also, it estimates that the larger the cumulative value of the index indicating the driving load from the start of driving, the higher the degree of fatigue. The index indicating the driving load is calculated by the output unit 120 or the estimation unit 160 at predetermined intervals while the driver is driving.

[0079] According to this embodiment, the same actions and effects as in the embodiment can be obtained.

[0080] (Example 2) The estimation device 10 according to Embodiment 2 has the configuration of the estimation device 10 according to the embodiment. The estimation device 10 according to this embodiment is the same as the estimation device 10 according to Embodiment 1, except for the points described below. In the estimation device 10 according to this embodiment, the estimation unit 160 estimates the driver's state based on the appropriateness of the response. Here, appropriateness of the response is, for example, at least one of the smoothness of the response and the correctness of the response. Smoothness of the response is, for example, whether there is hesitation or not, and whether there is any rephrasing. Correctness of the response is, for example, whether it is a correct or incorrect answer.

[0081] The estimation based on the smoothness of the response is described below. As described in Example 1, the output unit 120 outputs information requiring a response. The detection unit 140 then uses the sound data acquired from the microphone to detect the driver's response. Here, the detection unit 140 further detects whether or not the driver hesitates. Hesitation includes vocalizations such as "uh," "um," "uh," "um," and "well." The detection unit 140 also detects whether or not the driver corrects themselves when responding. The detection unit 140 can detect the presence or absence of hesitation and correction by analyzing the sound data using existing technology.

[0082] The estimation unit 160 obtains information from the detection unit 140 indicating whether or not there is hesitation and whether or not there is repetition. Based on the information obtained from the detection unit 140, the estimation unit 160 estimates that the driver's fatigue level is higher when hesitation is identified than when no hesitation is identified. Furthermore, based on the information obtained from the detection unit 140, the estimation unit 160 estimates that the driver's fatigue level is higher when repetition is identified than when no repetition is identified.

[0083] The following describes estimation based on the correctness of the response. When estimation based on the correctness of the response is performed, the output unit 120 outputs a question for which the correct answer is already known as response information. Examples of response information output by the output unit 120 include "What is 12 minus 4?", "What day of the week is it today?", "Please tell me your mobile phone number," and "Please tell me the month of this car's inspection." Response information includes questions that ask for information pre-registered in the estimation device 10, or questions with generally known answers. The information pre-registered in the estimation device 10 includes information about the driver and information about the moving object.

[0084] When the output unit 120 outputs a response-requiring information selected from multiple response-requiring information items previously stored in the storage unit 100, a correct answer is pre-associated with each response-requiring information item. Furthermore, when response-requiring information is generated by the output unit 120, the correct answer to that information is also generated. Note that multiple correct answers may exist for a single response-requiring information item.

[0085] When the output unit 120 outputs information requiring a response, the detection unit 140 detects the driver's response to that information. At that time, the detection unit 140 also identifies the content of the response. The detection unit 140 can identify the content of the response by, for example, analyzing the driver's voice obtained from the microphone using existing technology. The estimation unit 160 obtains information indicating the content of the response from the detection unit 140. The estimation unit 160 then determines whether the content of the response matches the correct answer to the outputted information requiring a response. If the content of the response matches the correct answer to the outputted information requiring a response, the estimation unit 160 identifies the response as correct. If the content of the response does not match the correct answer to the outputted information requiring a response, the estimation unit 160 identifies the response as incorrect. The estimation unit 160 estimates that the driver's fatigue level is higher when the response is identified as incorrect than when it is identified as correct.

[0086] Furthermore, the estimation device 10 according to this embodiment may also estimate the driver's state based on the response time, etc., as described in Embodiment 1. For example, the estimation unit 160 estimates the response time T r , the duration of operation from the start of operation T d The system identifies the cumulative operating load R from the start of operation, the hesitation score S1, the repetition score S2, and the correct / incorrect score S3. If there is hesitation, S1=1; if there is no hesitation, S1=0. If there is repetition, S2=1; if there is no repetition, S2=0. If the response is incorrect, S3=1; if the response is correct, S3=0. The estimation unit 160 then determines the fatigue index = a × T r +b×T dThe fatigue index is calculated using the relationship shown as "+c×R+d×S1+e×S2+f×S3". Here, a, b, c, d, e, and f are predetermined constants. Zero or any positive real number can be arbitrarily applied as the constant. In addition, the estimation unit 160 may further estimate the driver's condition using the biological information described in Example 1.

[0087] According to this embodiment, the same actions and effects as in the embodiment can be obtained.

[0088] The embodiments and examples described above with reference to the drawings are illustrative examples of the present invention, and various other configurations can also be adopted.

[0089] Examples of reference formats are provided below. 1. An output unit that outputs response-required information to the driver of a moving object, A detection unit for detecting the driver's response to the aforementioned information requiring a response, The system includes an estimation unit that estimates the driver's state based on at least one of the appropriateness and timing of the response. Estimation device. 2. In the estimation device described in 1., The output unit determines the response information to be output based on at least one of the current location of the moving object, weather, date, time, and destination. Estimation device. 3. In the estimation device described in 1. or 2., The estimation unit determines the response time to the output of the response-requiring information based on the timing of the response, The driver's state is estimated based on the response time. Estimation device. 4. In the estimation device described in 3., The detection unit detects the driver's response to the information requiring a response. Estimation device. In the estimation apparatus described in 5.3 or 4, The estimation unit, The response time is corrected based on at least one of the following: the current position of the moving body, the conditions around the moving body, the conditions inside the moving body, and the movement of the moving body. The driver's state is estimated based on the corrected response time. Estimation device. 6. In the estimation device described in any one of 3. to 5., The estimation unit estimates the driver's state based on a comparison of a reference time based on the driver's past response times with the driver's current response time. Estimation device. In the estimation device described in 7.6, The aforementioned reference time is the time determined based on the timing of the response detected by the detection unit during the period from when the driver started operating the mobile body until a predetermined time has elapsed. Estimation device. In the estimation device described in 8.6, The aforementioned reference time is a statistical value of the driver's past response time for each location of the moving object, each time period, or each combination of the location of the moving object and each time period. Estimation device. 9. In the estimation device described in any one of 1. to 8., The estimation unit estimates the driver's state based on the appropriateness of the response. The appropriateness of the response is at least one of the smoothness of the response and the correctness of the response. Estimation device. 10. In the estimation device described in any one of items 1 to 9, Based on at least one of the following: the current position of the mobile body, the conditions surrounding the mobile body, the conditions inside the mobile body, and the movement of the mobile body, the output unit determines the timing for outputting the response information and the estimation unit determines the timing for estimating the driver's state. Estimation device. 11. In the estimation device described in any one of 1. to 10., Using the driver's biometric information, the system determines at least one of the timing at which the output unit outputs the response-required information and the timing at which the estimation unit estimates the driver's state. Estimation device. 12. In the estimation device described in any one of 1. to 11., The output unit is, Based on the content of the above response, the advertising information to be output is identified, When the estimated state of the driver meets predetermined conditions, the identified advertising information is output. Estimation device. 13. An estimation method performed by one or more computers, An output step that outputs response-required information requesting a response from the driver of a moving object, A detection step for detecting the driver's response to the output of the information requiring a response, The estimation step includes estimating the driver's state based on at least one of the appropriateness and timing of the response. Estimation method. 14. A program that causes a computer to execute the estimation method described in 13. [Explanation of symbols]

[0090] 10 Estimation device 100 Storage section 120 Output section 140 Detection unit 160 Estimation Department 1000 calculator 1020 Bus 1040 processor 1060 memory 1080 Storage Devices 1100 Input / Output Interface 1120 Network Interface

Claims

1. An output unit that outputs response-required information to the driver of a moving object, A detection unit for detecting the driver's response to the aforementioned information requiring a response, The system includes an estimation unit that determines the response time to the output of the response-requiring information based on the timing at which the response is detected, and estimates the driver's state based on the response time. The estimation unit estimates the driver's state based on a comparison between a reference time, which is the time determined by the detection unit based on the timing of the response detected during the period from when the driver started operating the mobile body until a predetermined time has elapsed, and the driver's current response time. Estimation device.

2. In the estimation device according to claim 1, The output unit determines the response information to be output based on at least one of the current location of the moving object, weather, date, time, and destination. Estimation device.

3. In the estimation device according to claim 1, The detection unit detects the driver's response to the information requiring a response. Estimation device.

4. In the estimation device according to claim 1, The estimation unit, The response time is corrected based on at least one of the following: the current position of the moving body, the conditions around the moving body, the conditions inside the moving body, and the movement of the moving body. The driver's state is estimated based on the corrected response time. Estimation device.

5. In the estimation device according to claim 1, Based on at least one of the following: the current position of the mobile body, the conditions surrounding the mobile body, the conditions inside the mobile body, and the movement of the mobile body, the output unit determines the timing for outputting the response information and the estimation unit determines the timing for estimating the driver's state. Estimation device.

6. In the estimation device according to claim 1, Using the driver's biometric information, the system determines at least one of the timing at which the output unit outputs the response-required information and the timing at which the estimation unit estimates the driver's state. Estimation device.

7. In the estimation device according to any one of claims 1 to 6, The output unit is, Based on the content of the above response, the advertising information to be output is identified, When the estimated state of the driver meets predetermined conditions, the identified advertising information is output. Estimation device.

8. An estimation method performed by one or more computers, An output step that outputs response-required information requesting a response from the driver of a moving object, A detection step for detecting the driver's response to the output of the information requiring a response, The estimation step includes determining the response time to the output of the response-requiring information based on the timing at which the response was detected, and estimating the driver's state based on the response time. In the estimation step, the driver's state is estimated based on a comparison between a reference time, which is the time determined based on the timing of the response detected in the detection step during the period from when the driver started operating the mobile body until a predetermined time has elapsed in the past, and the driver's current response time. Estimation method.

9. A program that causes a computer to execute the estimation method described in claim 8.

Images