Awakening state determination system
The system optimizes battery consumption in mobile terminals by adjusting the frequency of information transmission based on monitoring importance, addressing the issue of rapid power depletion in wakefulness determination systems, ensuring continuous functionality during and after automatic driving control.
Patent Information
- Application Number
- JP2021053252
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-26
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-03-26
AI Technical Summary
Existing systems that determine a driver's wakefulness state using a mobile terminal increase processing load and rapidly deplete battery power, especially when the importance of monitoring is low, which can lead to the battery running out of power prematurely.
A system that adjusts battery consumption based on the importance of monitoring the driver's wakefulness by varying the frequency of information transmission processes, such as image acquisition, depending on factors like the predicted end of automatic driving control, driver alertness, and battery status, to conserve power.
The system effectively prevents rapid battery depletion by optimizing power consumption, allowing continuous wakefulness determination and ensuring the mobile terminal can maintain functionality during and after automatic driving control.
Smart Images

Figure 0007585926000001 
Figure 0007585926000002 
Figure 0007585926000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a system for determining a driver's wakefulness state. [Background technology]
[0002] Conventionally, devices have been proposed that use a camera (driver monitor camera) to capture an image of the face of a vehicle driver and determine whether the driver is awake based on the image data acquired by the camera (hereinafter referred to as "conventional devices") (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2014-115983 A Summary of the Invention
[0004] Furthermore, conventionally, an automatic driving device configured to execute automatic driving control has been proposed. Automatic driving control is a control that automatically changes the vehicle speed and steering angle of the steering wheel without the driver's driving operation. The level of automatic driving control (so-called automatic driving level) is classified into several levels 0 to 5 by the Society of Automotive Engineers (SAE). In recent years, level 3 automatic driving control has started to be put into practical use. Level 3 is a level at which an automatic driving device executes all driving operations under a specific environment, and is a level at which the driver is required to perform driving operations when a request for switching driving operations (hereinafter simply referred to as a "driving operation switching request") occurs. Therefore, level 3 automatic driving control is sometimes referred to as "conditional automatic driving control."
[0005] At level 3, if the autonomous driving device cannot continue autonomous driving control (if a request to take over driving operations occurs), the driver must take over driving operations. Therefore, the driver is not permitted to fall asleep. On the other hand, the driver is permitted to look away from the vehicle. Therefore, while the autonomous driving device is performing level 3 autonomous driving control, the driver can perform various actions (such as operating a mobile device or reading a book).
[0006] For example, assume that a driver operates a mobile terminal (e.g., a smartphone) while automatic driving control is being executed. A mobile terminal is usually equipped with a camera. Therefore, a configuration in which an image of the driver acquired by the mobile terminal is used to determine whether the driver is in an awake state is also considered to be effective. Therefore, the inventor of the present application is considering a system in which information acquired by the mobile terminal is used to determine whether the driver is in an awake state. However, in such a system, the processing load of the mobile terminal becomes large, and there is a possibility that the remaining power of the battery of the mobile terminal will be rapidly reduced.
[0007] One of the objectives of the present disclosure is to provide a technology that can adjust the degree of battery consumption of a mobile device depending on the situation in a system that determines whether a driver is awake using information acquired on a mobile device.
[0008] In one or more embodiments, the wakefulness determination system includes: A control device (10) that executes an awake state determination process for determining whether a driver (DR) of a vehicle (VA) is in an awake state; a mobile terminal (110) carried by the driver and equipped with a battery (304), the mobile terminal acquiring information used in the wakefulness state determination process at the mobile terminal and executing an information transmission process for transmitting the acquired information to the control device as terminal side information; Equipped with. The control device is configured to execute the wakefulness state determination process by using the terminal side information. The mobile terminal is configured to execute the information transmission process so that when the level of importance of monitoring the driver is lower than a predetermined level, the degree of consumption of power from the battery is smaller than when the level of importance is higher than the predetermined level.
[0009] According to the above configuration, when the importance of monitoring the driver is low, consumption of battery power is reduced, and therefore a rapid decrease in the remaining battery power can be prevented.
[0010] In one or more embodiments, the mobile terminal is configured to execute the information transmission process when it is estimated that there is a high possibility that the battery's power will run out quickly, so that the degree of consumption is smaller than when it is estimated that there is a low possibility that the battery's power will run out quickly.
[0011] According to the above configuration, the mobile terminal can continue the information transmission process while preventing the battery from running out (or while extending the time until the battery runs out). Therefore, the control device can continue the wakefulness state determination process by using the terminal side information transmitted by the information transmission process.
[0012] In one or more embodiments, the wakefulness state determination system further includes a first imaging unit (76) mounted on the vehicle and configured to capture an image of the driver. The mobile terminal includes a second imaging unit (202) configured to capture an image of the driver. The control device includes: Calculating a degree of wakefulness of the driver based on an image acquired by the first imaging unit or the second imaging unit; Determining the level of importance based on the degree of arousal. It is structured as follows.
[0013] For example, if the driver feels drowsy, there is a possibility that the driver will fall asleep in the near future. In such a situation, the level of importance of monitoring the driver is high. According to the above configuration, the control device can determine the level of importance of monitoring the driver based on the degree of wakefulness.
[0014] In one or more embodiments, the control device is configured to execute an automatic driving control for automatically performing a driving operation of the vehicle, and the control device is configured to execute the wakefulness state determination process by using the terminal side information during execution of the automatic driving control.
[0015] In one or more embodiments, the control device is configured to determine the level of importance based on a predicted time (Tf) until the autonomous driving control is terminated.
[0016] For example, when the automatic driving control is terminated, the driver needs to take over the driving operation. If the driver is dozing off, the driver cannot take over the driving operation. Therefore, when the time until the automatic driving control is terminated is predicted to be short, the level of importance of monitoring the driver is high. In contrast, when the time until the automatic driving control is terminated is predicted to be long, the driving operation of the vehicle is performed by the automatic driving control, so the level of importance of monitoring the driver is relatively low. In this way, the control device can determine the level of importance of monitoring the driver based on the predicted time until the automatic driving control is terminated.
[0017] In one or more embodiments, when the automatic driving control is terminated, the mobile terminal: When there is a high possibility that the automatic driving control will be resumed, the information transmission process is continued; When the possibility of restarting the automatic driving control is low, the information transmission process is terminated. It is structured as follows.
[0018] According to the above configuration, when there is a high possibility that the autonomous driving control will be resumed, the mobile terminal continues the information transmission process. When the autonomous driving control is resumed, the control device can immediately resume the wakefulness determination process using the terminal side information.
[0019] In one or more embodiments, when there is a high possibility of resuming the automatic driving control, the mobile terminal: A process in which the degree of consumption is small among the information transmission processes is continued; The process having a large degree of consumption among the information transmission processes is terminated. It is structured as follows.
[0020] According to the above configuration, the mobile terminal terminates the information transmission process that consumes a large amount of battery power, so that it is possible to prevent the remaining battery power from being rapidly reduced. Since the mobile terminal continues the information transmission process that consumes a small amount of battery power, when the automatic driving control is resumed, the control device can immediately resume the wakefulness determination process by using the terminal side information.
[0021] In one or more embodiments, the control device may be implemented by a microprocessor programmed to perform one or more functions described herein. In one or more embodiments, the control device may be implemented in whole or in part by hardware configured by one or more application specific integrated circuits, i.e., ASICs. In the above description, the names and / or symbols used in the embodiments are added in parentheses to components corresponding to one or more embodiments described below. However, each component is not limited to the embodiment defined by the name and / or symbol. [Brief description of the drawings]
[0022] [Figure 1] 1 is a schematic configuration diagram of an awakening state determination system according to a first embodiment. [Diagram 2] FIG. 2 is an external view of the mobile terminal according to the first embodiment. [Diagram 3] 1 is a schematic configuration diagram of a mobile terminal according to a first embodiment. [Figure 4] FIG. 13 is a diagram showing a situation in which a driver operates a mobile terminal while automatic driving control is being executed. [Diagram 5] 5 is an example of an image captured by the front camera of a mobile terminal in the situation of FIG. 4. [Figure 6]FIG. 2 is a sequence diagram of the automatic driving device (automatic driving ECU) and the mobile terminal in the first embodiment. [Figure 7] 7 is a flowchart showing a routine executed in step 607 of FIG. 6 in the first embodiment. [Figure 8] 7 is a flowchart showing a routine executed in step 607 of FIG. 6 in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0023] First Embodiment As shown in FIG. 1, the awake state determination system according to the first embodiment (hereinafter, sometimes referred to as the "first system") includes an automatic driving device 100 mounted (applied) on a vehicle VA, and a mobile terminal 110.
[0024] (Configuration of the automatic driving device) The autonomous driving device 100 includes an autonomous driving ECU 10, an engine ECU 20, a brake ECU 30, a steering ECU 40, a meter ECU 50, and a navigation ECU 60. These ECUs are connected to each other via a controller area network (CAN) 90 so as to be able to transmit and receive data to and from each other.
[0025] ECU is an abbreviation of Electronic Control Unit, and is an electronic control circuit having a microcomputer as a main component. The microcomputer includes a CPU, ROM, RAM, an interface, a non-volatile memory, etc. The CPU realizes various functions by executing instructions (routines, programs) stored in the ROM. For example, an autonomous driving ECU 10 includes a microcomputer including a CPU 10a, a ROM 10b, a RAM 10c, an interface (I / F) 10d, a non-volatile memory 10e, etc.
[0026] The engine ECU 20 is connected to an engine actuator 21. The engine actuator 21 includes a throttle valve actuator that changes the opening degree of a throttle valve of an internal combustion engine 22. The engine ECU 20 can change the torque generated by the internal combustion engine 22 by driving the engine actuator 21. The torque generated by the internal combustion engine 22 is transmitted to the driving wheels via a transmission 23 and a driving force transmission mechanism. Therefore, the engine ECU 20 can control the driving force of the vehicle VA by controlling the engine actuator 21.
[0027] If the vehicle VA is a hybrid vehicle, the engine ECU 20 can control the driving force generated by either or both of the "internal combustion engine and the electric motor" as the driving source. Furthermore, if the vehicle VA is an electric vehicle, the engine ECU 20 can control the driving force generated by the electric motor as the driving source.
[0028] The brake ECU 30 is connected to a brake actuator 31. The brake actuator 31 includes a known hydraulic circuit. The hydraulic circuit includes a reservoir, an oil pump, various valve devices, and the like. The brake actuator 31 adjusts the hydraulic pressure (i.e., braking pressure) supplied to the wheel cylinder 32 in response to an instruction from the brake ECU 30. The frictional braking force generated on the wheel changes in response to the braking pressure. Therefore, the brake ECU 30 can control the braking force of the vehicle VA by controlling the brake actuator 31.
[0029] The steering ECU 40 is connected to an assist motor (M) 41. The assist motor 41 is incorporated in a steering mechanism 42. The steering mechanism 42 is a mechanism for steering the steered wheels by rotating the steering wheel SW. The steering mechanism 42 includes a steering wheel SW, a steering shaft US connected to the steering wheel SW, and a steering gear mechanism (not shown). The steering ECU 40 detects the steering torque input to the steering wheel SW by the driver using a steering torque sensor (not shown) provided on the steering shaft US, and drives the assist motor 41 based on this steering torque. The steering ECU 40 applies a steering torque (steering assist torque) to the steering mechanism 42 by driving the assist motor 41, thereby assisting the steering operation of the driver.
[0030] In addition, the steering ECU 40 drives the assist motor 41 in accordance with a steering command (including an automatic steering torque, described later) from the automatic driving ECU 10 while the automatic driving control described below is being executed. This automatic steering torque is different from the steering assist torque described above, and represents a torque that is applied to the steering mechanism 42 by a steering command from the automatic driving ECU 10 without requiring a steering operation by the driver. The automatic steering torque changes the steering angle of the steered wheels of the vehicle VA.
[0031] The meter ECU 50 is connected to an alarm device 51. The alarm device 51 includes a buzzer and a display device. The display device is, for example, a multi-information display provided in front of the driver's seat. In response to an instruction from the autonomous driving ECU 10, the meter ECU 50 can sound a buzzer to alert the driver or display an alert mark (warning lamp) on the display device.
[0032] The navigation ECU 60 is connected to a GPS receiver 61 and a map storage unit 62. The GPS receiver 61 receives a GPS signal for detecting the "latitude and longitude" of the location where the vehicle VA is located. The map storage unit 62 stores map information. The map information includes road information. The road information includes the road type, road width, road curvature, etc. In this example, the road type is either a motorway (expressway) or a general road. The navigation ECU 60 performs various calculation processes based on the latitude and longitude of the location where the vehicle VA is located and the map information, etc., and displays the position of the vehicle VA on the map on a touch panel (not shown).
[0033] The vehicle speed sensor 71 detects the traveling speed (vehicle speed) of the vehicle VA, and outputs a signal indicating the vehicle speed SPD to the automatic driving ECU 10.
[0034] The acceleration sensor 72 acquires information related to the acceleration occurring in the vehicle VA (hereinafter referred to as "first acceleration information"), and outputs a signal representing the first acceleration information to the autonomous driving ECU 10. The first acceleration information includes an acceleration ax1 which is the acceleration in the forward / backward direction (forward / backward acceleration) of the vehicle VA, an acceleration ay1 which is the acceleration in the lateral direction (lateral acceleration) of the vehicle VA, and an acceleration az1 which is the acceleration in the vertical direction of the vehicle VA.
[0035] The surrounding sensor 73 is adapted to obtain information about the road around the vehicle VA (including the lane in which the vehicle VA is traveling) and information about three-dimensional objects existing on the road. The three-dimensional objects include, for example, moving objects such as pedestrians, four-wheeled vehicles and two-wheeled vehicles, and fixed objects such as guardrails and fences. Hereinafter, these three-dimensional objects will be simply referred to as "objects." The surrounding sensor 73 includes a radar sensor 74 and a camera sensor 75.
[0036] The radar sensor 74, for example, emits millimeter wave band radio waves (hereinafter referred to as "millimeter waves") to a surrounding area including at least the area in front of the vehicle VA, and receives millimeter waves (i.e., reflected waves) reflected by objects present within the emission range. The radar sensor 74 then determines whether or not an object is present, and calculates information indicating the relative relationship between the vehicle VA and the object. The information indicating the relative relationship between the vehicle VA and the object includes the distance between the vehicle VA and the object, the orientation (or position) of the object with respect to the vehicle VA, and the relative speed between the vehicle VA and the object. Hereinafter, the information indicating the relative relationship between the vehicle VA and the object will be simply referred to as "object information."
[0037] The camera sensor 75 photographs the scenery ahead of the vehicle VA to obtain image data. Based on the image data, the camera sensor 75 recognizes the left and right lane lines (e.g., left and right white lines) that define the driving lane, and calculates parameters indicating the shape of the driving lane (e.g., curvature) and parameters indicating the positional relationship between the vehicle VA and the driving lane. The parameter indicating the positional relationship between the vehicle VA and the driving lane is, for example, the distance from the left or right lane line to the center position of the vehicle VA in the vehicle width direction. The information obtained by the camera sensor 75 is referred to as "lane information." The camera sensor 75 may be configured to determine the presence or absence of an object based on the image data and calculate object information.
[0038] The surrounding sensor 73 outputs information regarding the surrounding conditions of the vehicle VA, including "object information and lane information", to the autonomous driving ECU 10 as "vehicle surrounding information".
[0039] The driver monitor camera (first imaging unit) 76 is disposed above the column cover (so-called steering column cover) of the steering wheel SW. The driver monitor camera 76 captures an image of the driver to obtain image data, and outputs the image data to the automatic driving ECU 10. The position and orientation of the driver monitor camera 76 are set so that the entire face of the driver can be captured. Hereinafter, the image data obtained by the driver monitor camera 76 is referred to as the "first image."
[0040] The driver monitor camera 76 may be set at another position as long as it can capture the face of the driver inside the vehicle VA. For example, the driver monitor camera 76 may be provided near the dashboard or the windshield.
[0041] Furthermore, the autonomous driving ECU 10 is connected to a communication unit 80. The communication unit 80 includes an "antenna unit and a communication processing unit" (not shown). The communication unit 80 is configured to be capable of transmitting and receiving information to and from the mobile terminal 110 through wireless communication.
[0042] The communication unit 80 may be configured to be able to communicate with devices other than the mobile terminal 110. The communication unit 80 may communicate with VICS (registered trademark) (Vehicle Information and Communication System) to receive road traffic information from VICS (registered trademark). The road traffic information includes information on congested sections, information on road closure sections, and the like.
[0043] The automatic driving switch 81 is a switch operated by the driver. By operating the automatic driving switch 81, the driver can set the operation state of the automatic driving control, which will be described later.
[0044] (Autonomous Driving Control Overview) Hereinafter, the autonomous driving ECU 10 is simply referred to as "ECU 10." The ECU 10 is configured to execute autonomous driving control. In this example, the autonomous driving level is level 3. The autonomous driving control includes speed control and steering control.
[0045] An example of speed control is adaptive cruise control (hereinafter, simply referred to as "ACC"). ACC itself is publicly known (see, for example, JP 2014-148293 A, JP 2006-315491 A, and the specification of Japanese Patent No. 4172434).
[0046] ACC includes two types of control: constant speed cruise control and preceding vehicle following control. Constant speed cruise control is a control that adjusts the speed of the vehicle VA so that the traveling speed of the vehicle VA matches a target speed Vset without the need to operate the accelerator pedal or brake pedal. Preceding vehicle following control is a control that makes the vehicle VA follow the preceding vehicle (the vehicle to be followed) while maintaining the inter-vehicle distance Dset between the vehicle VA and the preceding vehicle (the vehicle to be followed) at a target inter-vehicle distance Dset without the need to operate the accelerator pedal or brake pedal. The vehicle to be followed is a vehicle that is traveling in the area ahead of the vehicle VA and immediately before the vehicle VA.
[0047] The ECU 10 judges whether or not a vehicle to be followed exists based on the vehicle surroundings information. If the ECU 10 judges that a vehicle to be followed does not exist, the ECU 10 executes constant speed traveling control. The ECU 10 controls the engine actuator 21 using the engine ECU 20 to control the driving force so that the vehicle speed SPD coincides with the target speed Vset, and controls the brake actuator 31 using the brake ECU 30 to control the braking force as necessary.
[0048] On the other hand, when the ECU 10 determines that a vehicle to be followed is present, it executes control to follow the preceding vehicle. The ECU 10 calculates a target inter-vehicle distance Dset by multiplying the target inter-vehicle time tw by the vehicle speed SPD. The target inter-vehicle time tw is set using a non-illustrated inter-vehicle time switch. The ECU 10 controls the engine actuator 21 using the engine ECU 20 to control the driving force so that the inter-vehicle distance between the vehicle VA and the vehicle to be followed coincides with the target inter-vehicle distance Dset, and controls the brake actuator 31 using the brake ECU 30 to control the braking force as necessary.
[0049] One example of steering control is lane keeping assist control (hereinafter, simply referred to as "LKA"). LKA itself is publicly known (see, for example, JP 2008-195402 A, JP 2009-190464 A, JP 2010-6279 A, and Japanese Patent No. 4349210, etc.).
[0050] LKA is a control that applies automatic steering torque to the steering mechanism 42 to change the steering angle of the steering wheels of the vehicle VA so that the vehicle VA travels along a target driving line that is set using either a lane marking or the driving trajectory of the preceding vehicle (i.e., the preceding vehicle trajectory), or both.
[0051] The ECU 10 sets a target driving line based on vehicle surroundings information. The target driving line is, for example, a line connecting the center positions in the road width direction between the left and right lane lines (i.e., the center line of the lane in which the vehicle VA is traveling), or the trajectory of the preceding vehicle. The ECU 10 calculates an automatic steering torque for driving the vehicle VA along the target driving line. The ECU 10 controls the assist motor 41 using the steering ECU 40 so that the actual torque generated by the assist motor 41 coincides with the automatic steering torque. This changes the steering angle of the steering wheels of the vehicle VA.
[0052] The ECU 10 can execute the automatic driving control under a specific situation. The ECU 10 executes the automatic driving control when a predetermined automatic driving execution condition is satisfied. For example, the automatic driving execution condition is satisfied when all of the following conditions A1 to A3 are satisfied. (Condition A1): The vehicle VA is traveling on a highway. (Condition A2): The vehicle speed SPD is equal to or less than a predetermined speed threshold SPDth. (Condition A3): The driver presses the automatic driving switch 81 to set the operation state of the automatic driving control from the OFF state to the ON state.
[0053] The speed threshold SPDth may be set to a relatively small value. In this case, the situation where the autonomous driving execution condition is satisfied means that the vehicle VA is traveling in a congested section. In this situation, the ECU 10 may execute the autonomous driving control.
[0054] The autonomous driving execution condition is not limited to the above example, and may be other conditions. The road information stored in the map storage unit 62 may include information indicating a section where level 3 autonomous driving control is possible (hereinafter referred to as an "autonomous driving section"). In this case, the ECU 10 acquires the road information via the navigation ECU 60. When the driver presses the autonomous driving switch 81 while the vehicle VA is traveling in an autonomous driving section, the ECU 10 may determine that the autonomous driving execution condition is satisfied.
[0055] In another example, the ECU 10 may acquire traffic information (including information on congested sections) via the communication unit 80. In this configuration, when the driver presses the automatic driving switch 81 while the vehicle VA is traveling through a congested section, the ECU 10 may determine that the automatic driving execution condition is satisfied.
[0056] When a predetermined automatic driving end condition is satisfied, the ECU 10 ends the automatic driving control. For example, the automatic driving end condition is satisfied when at least one of the following conditions B1 to B4 is satisfied. (Condition B1): The vehicle VA arrives at the exit of a freeway. (Condition B2): The vehicle speed SPD becomes greater than the speed threshold SPDth. (Condition B3): The driver presses the automatic driving switch 81 to set the operation state of the automatic driving control from the on state to the off state. (Condition B4): An event has occurred that makes it impossible to continue autonomous driving control (e.g., a system error). The autonomous driving end condition is not limited to the above example, and may be another condition.
[0057] (Mobile device configuration) The mobile terminal 110 is a device carried by the driver, and in this example, is a smartphone as shown in Fig. 2. The mobile terminal 110 may be another device (for example, a tablet terminal or a wearable terminal) as long as it is carried by the driver.
[0058] The mobile terminal 110 includes a display 201. The display 201 is a touch panel type display, and is capable of detecting a touch operation on the display surface (operation surface) of the display 201.
[0059] Mobile terminal 110 includes a front camera (second imaging unit) 202 on the surface facing display 201. Mobile terminal 110 may include a rear camera on the surface opposite display 201 (that is, the back surface).
[0060] Furthermore, the mobile terminal 110 includes an acceleration sensor 203 therein. The acceleration sensor 203 acquires information on the acceleration occurring in the mobile terminal 110 (referred to as "second acceleration information"). The second acceleration information includes information on the acceleration in each of the "x-axis, y-axis, and z-axis directions" shown in FIG. 2. The x-axis is a coordinate axis extending in the horizontal direction of the mobile terminal 110. The y-axis is a coordinate axis extending in the vertical direction of the mobile terminal 110. The z-axis is a coordinate axis extending in the vertical direction of the mobile terminal 110. Hereinafter, the acceleration in the x-axis direction is referred to as "acceleration ax2", the acceleration in the y-axis direction is referred to as "acceleration ay2", and the acceleration in the z-axis direction is referred to as "acceleration az2".
[0061] The mobile terminal 110 further includes a first volume button 204a, a second volume button 204b, and a home button 205. When the driver presses the first volume button 204a, the volume of the mobile terminal 110 increases, and when the driver presses the second volume button 204b, the volume of the mobile terminal 110 decreases. The home button 205 is a button for temporarily closing various running applications and displaying a home screen on the display 201.
[0062] 3, the mobile terminal 110 includes a CPU 301, a memory 302, and a non-volatile memory 303. The CPU 301, the memory 302, the non-volatile memory 303, the display 201, the front camera 202, and the acceleration sensor 203 are connected to each other via an internal bus 310.
[0063] The CPU 301 includes at least one processor or circuit. The memory 302 includes, for example, a RAM. The non-volatile memory 303 includes, for example, a flash memory and a ROM. The CPU 301 uses the memory 302 as a working memory to execute instructions (routines, programs) stored in the non-volatile memory 303.
[0064] Furthermore, the mobile terminal 110 includes a battery 304. The above-mentioned components of the mobile terminal 110 operate using power supplied from the battery 304.
[0065] (Outline of wakefulness state determination process) As shown in Fig. 4, the driver DR may operate the mobile terminal 110 while the ECU 10 is executing the automatic driving control. Therefore, the ECU 10 determines whether the driver DR is in an awake state by using information acquired by the mobile terminal 110 while the automatic driving control is being executed. Hereinafter, such a process is referred to as "awake state determination process." Furthermore, the information acquired by the mobile terminal 110 is referred to as "terminal side information."
[0066] For the above configuration, as shown in Fig. 3, the non-volatile memory 303 of the mobile terminal 110 stores a monitoring application 330. The monitoring application 330 is a program for monitoring the state of the driver DR.
[0067] CPU 301 executes monitoring application 330 to acquire information to be used in the wakefulness state determination process, and transmits the acquired information as terminal-side information to autonomous driving device 100. Hereinafter, such processing will be simply referred to as "information transmission processing."
[0068] In this example, CPU 301 executes information transmission processing as follows. CPU 301 acquires an image of the face of driver DR (hereinafter referred to as "second image") from front camera 202. Fig. 5 is an example of second image 500 acquired by front camera 202. CPU 301 transmits second image 500 to automatic driving device 100 as terminal side information.
[0069] The ECU 10 receives the terminal side information via the communication unit 80. The ECU 10 executes an awakening state determination process based on the second image 500. Specifically, the ECU 10 determines the awakening degree of the driver DR according to a known method. In this example, the awakening degree includes a first degree, a second degree, and a third degree. The first degree indicates that the driver DR is in an awakening state. The second degree indicates that the driver DR is not in a drowsy state, but is feeling drowsy. The third degree indicates that the driver DR is in a drowsy state.
[0070] For example, the degree of wakefulness is determined based on at least one of the following: eye closure rate, blinking frequency, eye open state (distance between upper eyelid and lower eyelid), eye movement, and the presence or absence of yawning behavior, etc. The eye closure rate will be briefly described below.
[0071] The ECU 10 extracts the eyes of the driver DR from the second image 500 by inputting the second image 500 to a classifier for eye detection or by performing template matching on the second image 500. The ECU 10 determines whether the eyes are closed based on, for example, the aspect ratio of the extracted eye portion. The ECU 10 calculates the ratio of time the eyes are closed within a predetermined time as an eye closure rate. If the eye closure rate is equal to or less than a first threshold, the ECU 10 determines that the degree of wakefulness of the driver DR is a first degree. If the eye closure rate is greater than the first threshold and equal to or less than a second threshold, the ECU 10 determines that the degree of wakefulness of the driver DR is a second degree. If the eye closure rate is greater than the second threshold, the ECU 10 determines that the degree of wakefulness of the driver DR is a third degree.
[0072] When the ECU 10 determines that the wakefulness level of the driver DR is the third level, the ECU 10 causes the notification device 51 to execute a predetermined attention-calling process. Specifically, the ECU 10 uses the meter ECU 50 to sound a buzzer and to display a warning lamp on the display device.
[0073] In addition to the above-mentioned attention-calling process, the ECU 10 may execute the following process. The ECU 10 may determine that the automatic driving control cannot be continued and notify the driver DR of a request to change driving operations. Furthermore, the ECU 10 may gradually decelerate the vehicle VA and stop the vehicle VA in a safe position (for example, a road shoulder or an emergency parking area).
[0074] (Overview of operation) In the above configuration, the mobile terminal 110 executes the information transmission process, so the processing load of the mobile terminal 110 is large. Therefore, there is a possibility that the remaining power of the battery 304 will be rapidly reduced. Therefore, the first system adjusts the degree of consumption of the power of the battery 304 of the mobile terminal 110 according to the situation.
[0075] For example, when the automatic driving control is terminated, the driver DR needs to take over the driving operation. If the driver is dozing, the driver DR cannot take over the driving operation. Therefore, when it is predicted that the time until the automatic driving control is terminated is short, it is important to monitor the driver DR. On the other hand, when it is predicted that the time until the automatic driving control is terminated is long, the driving operation of the vehicle VA is performed by the automatic driving control, so it is relatively less important to monitor the driver DR.
[0076] In another example, when the driver DR feels drowsy, there is a possibility that the driver DR will fall asleep in the near future. In such a situation, it is important to monitor the driver DR. On the other hand, when the driver DR is awake, it is relatively less important to monitor the driver DR.
[0077] Taking the above into consideration, the ECU 10 determines whether or not the first condition is satisfied. In this example, the first condition is a condition for determining whether the level of importance of monitoring the driver DR is higher than a predetermined level, and is satisfied when at least one of the following conditions C1 and C2 is satisfied. (Condition C1): A predicted time Tf until the automatic driving control ends is within a predetermined time threshold Tth. (Condition C2): The degree of wakefulness of the driver DR is the second degree or the third degree.
[0078] For example, the ECU 10 calculates the predicted time Tf based on the distance from the current position of the vehicle VA to the exit of the motorway, and thereby determines whether the condition C1 is satisfied. This is because when the vehicle VA arrives at the exit of the motorway, the condition B1 is satisfied (i.e., the automatic driving control is terminated). When the vehicle VA is traveling in a congested section, the ECU 10 may calculate the predicted time Tf based on the distance from the current position of the vehicle VA to the end of the congested section. This is because when the vehicle VA leaves the congested section, the vehicle speed SPD becomes greater than the speed threshold SPDth, and thus the condition B2 is satisfied. In this way, the ECU 10 can determine whether the level of importance of monitoring the driver DR is higher or lower than a predetermined level based on the predicted time Tf.
[0079] Furthermore, the ECU 10 calculates the degree of wakefulness of the driver DR based on the second image 500, and determines whether the condition C2 is satisfied. In this manner, the ECU 10 can determine whether the level of importance of monitoring the driver DR is higher or lower than a predetermined level, based on the degree of wakefulness of the driver DR.
[0080] The ECU 10 transmits a processing instruction signal instructing the contents of the information transmission processing to the mobile terminal 110. The contents of the processing instruction signal are changed depending on whether the first condition is satisfied. The mobile terminal 110 executes the information transmission processing according to the processing instruction signal.
[0081] When the first condition is met, the level of importance of monitoring the driver DR is higher than a predetermined level. Therefore, the mobile terminal 110 executes a first process as an information transmission process. The first process is a process of acquiring a second image 500 and transmitting the second image 500 to the automatic driving device 100 as terminal side information every time a first time T1 elapses. The first time T1 is set to a relatively short time. Therefore, when the first condition is met, the mobile terminal 110 transmits the terminal side information (second image 500) to the automatic driving device 100 at a high frequency.
[0082] When the first condition is not satisfied, the level of importance of monitoring the driver DR is lower than a predetermined level. Therefore, the mobile terminal 110 executes the second process as the information transmission process. The second process is a process of acquiring the second image 500 and transmitting the second image 500 to the automatic driving device 100 as terminal side information every time the second time T2 elapses. The second time T2 is longer than the first time T1. Therefore, when the first condition is not satisfied, the mobile terminal 110 transmits the terminal side information (the second image 500) to the automatic driving device 100 less frequently than when the first condition is satisfied. The second process is a process that imposes a smaller processing load on the mobile terminal 110 than the first process. That is, the second process is a process that consumes less power from the battery 304 than the first process.
[0083] In this way, when the first condition is not met (i.e., the importance of monitoring the driver DR is lower than a predetermined level), the mobile terminal 110 executes an information transmission process so that the degree of power consumption of the battery 304 is reduced compared to when the first condition is met (i.e., the level of importance is higher than a predetermined level).
[0084] (Example of operation) The operation of the automatic driving device 100 (specifically, the ECU 10) and the mobile terminal 110 will be specifically described with reference to FIG.
[0085] When the driver DR gets into the vehicle VA, a wireless connection is established between the ECU 10 and the mobile terminal 110. For example, the wireless connection between the ECU 10 and the mobile terminal 110 may be established by pairing using Bluetooth (registered trademark). As a result, the ECU 10 and the mobile terminal 110 start communication via the communication unit 80 (step 601).
[0086] After that, the driver DR drives the vehicle VA. The vehicle VA travels on a motorway at a vehicle speed SPD equal to or less than a speed threshold SPDth. In this situation, the driver DR presses the automatic driving switch 81. The ECU 10 determines that the automatic driving execution conditions are met, and starts automatic driving control (level 3) (step 602). The ECU 10 transmits a start instruction signal to the mobile terminal 110 (step 603). The start instruction signal is a signal that instructs the start of the monitoring application 330.
[0087] In response to the start instruction signal, the mobile terminal 110 displays an approval screen on the display 201 (step 604). The approval screen is a screen for approving the start of the monitoring application 330. When the driver DR presses a predetermined approval button on the approval screen, the mobile terminal 110 starts the monitoring application 330 (step 605). The mobile terminal 110 transmits initial information to the ECU 10 (step 606). The initial information includes a signal notifying that the monitoring application 330 has been started, and the second image 500.
[0088] The ECU 10 executes a routine in FIG. 7, which will be described later, using the second image 500 included in the initial information to determine the content of the information transmission process (step 607). The ECU 10 transmits a processing instruction signal instructing the content of the determined information transmission process to the mobile terminal 110 (step 608). The mobile terminal 110 executes the information transmission process in accordance with the processing instruction signal (step 609). The mobile terminal 110 transmits terminal side information (in this example, the second image 500) to the ECU 10 (step 610). The ECU 10 executes an awake state determination process based on the terminal side information (step 611).
[0089] The processes of steps 607 to 611 are repeatedly executed until the automatic driving control ends. When these processes are repeatedly executed, the ECU 10 uses the information on the wakefulness degree of the driver DR determined in step 611 in the process of step 607.
[0090] After that, the above-mentioned automatic driving end condition is satisfied. The ECU 10 ends the automatic driving control (step 612). Then, the ECU 10 transmits an end instruction signal to the mobile terminal 110 (step 613). The end instruction signal is a signal that instructs the end of the monitoring application 330. In response to the end instruction signal, the mobile terminal 110 ends the monitoring application 330 (step 614).
[0091] Next, the process executed by the ECU 10 (specifically, the CPU 10a) in step 607 in FIG. 6 will be described with reference to FIG.
[0092] 7, the ECU 10 starts the process from step 700 and proceeds to step 701 to determine whether or not the first condition is satisfied. If the first condition is satisfied, the ECU 10 determines "Yes" in step 701 and proceeds to step 702 to determine the first process as the information transmission process. After that, the ECU 10 proceeds to step 795 to end this routine. As a result, the ECU 10 transmits a process instruction signal to the mobile terminal 110 in step 608 to cause the mobile terminal 110 to execute the first process.
[0093] On the other hand, if the first condition is not satisfied, the ECU 10 judges "No" in step 701 and proceeds to step 703 to determine the second process as the information transmission process. After that, the ECU 10 proceeds to step 795 to end this routine. As a result, the ECU 10 transmits a process instruction signal to the mobile terminal 110 in step 608 to cause the mobile terminal 110 to execute the second process.
[0094] The first system having the above configuration has the following effects. The mobile terminal 110 changes the contents of the information transmission process according to the level of importance of monitoring the driver DR. Specifically, when the level of importance of monitoring the driver DR is higher than a predetermined level, the mobile terminal 110 executes the first process as the information transmission process. In contrast, when the level of importance of monitoring the driver DR is lower than the predetermined level, the mobile terminal 110 executes the second process as the information transmission process. The second process is a process that consumes less power of the battery 304 than the first process. In this way, when the importance of monitoring the driver DR is low, the consumption of power of the battery 304 is suppressed. Therefore, it is possible to suppress a sudden decrease in the remaining power of the battery 304.
[0095] (Variation 1 of the first system) The monitoring application 330 may further include a function of detecting operation information. The operation information is information related to an operation by the driver DR on the mobile terminal 110. The mobile terminal 110 transmits the operation information to the ECU 10 as terminal side information. The ECU 10 may use the operation information to determine whether the driver DR is in an awake state.
[0096] Specifically, the mobile terminal 110 executes the monitoring application 330 to detect the following first to third operations: Furthermore, the mobile terminal 110 obtains information regarding the time at which each operation was detected. (First operation): An operation of touching a finger to the display 201 and then immediately removing the finger from the display 201 (a so-called tap operation). (Second operation): An operation of moving a finger on the display 201 while keeping the finger in contact with the display 201 (a so-called swipe operation). (Third operation): An operation of changing the distance between two fingers while keeping the two fingers in contact with the display 201 (a so-called pinch out operation or pinch in operation).
[0097] 6, if the driver DR performs an operation on the approval screen, and no operation by the driver DR is detected for a certain period of time, this is likely to mean that the driver DR is not awake (i.e., the driver DR is dozing off). In consideration of this, the ECU 10 may determine whether the driver DR is awake based on the period during which no operation by the driver DR is detected.
[0098] Specifically, as long as at least one of the first to third operations is detected within a predetermined time threshold pth, the ECU 10 determines that the driver DR is in an awake state. On the other hand, when the time period during which none of the first to third operations is detected is equal to or longer than the time threshold pth, the ECU 10 determines that the driver DR is not in an awake state.
[0099] In such a configuration, the first process may be a process of acquiring operation information and transmitting the operation information as terminal-side information to the automatic driving device 100 every time a first time T1 elapses. The second process may be a process of acquiring operation information and transmitting the operation information as terminal-side information to the automatic driving device 100 every time a second time T2 elapses. Even in this configuration, the second process is a process that consumes less power from the battery 304 than the first process.
[0100] In another example, the first process may be a process of acquiring the second image 500 and transmitting the second image 500 to the automatic driving device 100 as terminal-side information every time the first time T1 elapses. The second process may be a process of acquiring operation information and transmitting the operation information to the automatic driving device 100 as terminal-side information every time the first time T1 elapses. When the operation information is acquired at the same frequency, the process of acquiring the operation information is a process that imposes a smaller processing load on the mobile terminal 110 than the process of acquiring an image. Even in this configuration, the second process is a process that consumes less power from the battery 304 than the first process.
[0101] The mobile terminal 110 may transmit information on operations other than the first operation to the third operation as terminal-side information (operation information) to the ECU 10. The mobile terminal 110 may transmit information on the operation of the first volume button 204a, the operation of the second volume button 204b, and the operation of the home button 205 as terminal-side information to the ECU 10.
[0102] (Variation 2 of the First System) The mobile terminal 110 may transmit the second acceleration information as terminal side information to the ECU 10. When the driver DR is moving the mobile terminal 110, there is a high possibility that the driver DR is in an awake state. In consideration of this, the ECU 10 may use the second acceleration information to determine whether the driver DR is in an awake state.
[0103] The second acceleration information includes acceleration components caused by the movement (swaying, vibration, etc.) of the vehicle VA. In consideration of this, the ECU 10 converts three acceleration component values in the coordinate system of the mobile terminal 110 (the xyz coordinate system shown in FIG. 2) into three acceleration component values in the coordinate system of the vehicle VA. That is, the ECU 10 converts the second acceleration information (i.e., the acceleration ax2, the acceleration ay2, and the acceleration az2) into an acceleration ax3 in the forward / rearward direction of the vehicle VA, an acceleration ay3 in the lateral direction of the vehicle VA, and an acceleration az3 in the up / down direction of the vehicle VA. If there is a difference between the first acceleration information and the converted second acceleration information (ax3, ay3, and az3), it can be determined that the driver DR intentionally moves the mobile terminal 110. The ECU 10 determines that the driver DR is in an awake state if at least one of the following conditions D1 to D3 is satisfied. (Condition D1):|ax1-ax3|≧axth (Condition D2):|ay1-ay3|≧ayth (Condition D3): |az1-az3|≧azth Here, axth, ayth, and azth are predetermined thresholds for determining whether the driver DR is moving the mobile terminal 110, respectively.
[0104] On the other hand, if none of the conditions D1 to D3 is satisfied, the ECU 10 determines that the driver DR is not in an awake state.
[0105] In such a configuration, the first process may be a process of acquiring second acceleration information every time the first time T1 elapses and transmitting the second acceleration information as terminal-side information to the automatic driving device 100. The second process may be a process of acquiring second acceleration information every time the second time T2 elapses and transmitting the second acceleration information as terminal-side information to the automatic driving device 100. Even in this configuration, the second process is a process that consumes less power from the battery 304 than the first process.
[0106] (Variation 3 of the First System) The terminal side information may include at least one of the second image 500, the operation information, and the second acceleration information. Therefore, the ECU 10 may determine whether the driver DR is in an awake state by using at least one of the second image, the operation information, and the second acceleration information. The first process may be a process of acquiring at least one of the second image 500, the operation information, and the second acceleration information. The second process may be a process of acquiring at least one of the second image 500, the operation information, and the second acceleration information, as long as the second process is a process that consumes less power of the battery 304 than the first process.
[0107] <Second embodiment> Next, an awakening state determination system according to a second embodiment (hereinafter, may be referred to as a "second system") will be described. In this embodiment, the mobile terminal 110 changes the contents of the information transmission process according to whether or not there is a possibility that the power of the battery 304 will run out quickly.
[0108] For example, when the remaining power of the battery 304 is low, if the mobile terminal 110 executes a process with a high processing load as an information transmission process, the power of the battery 304 may run out quickly. Taking this into consideration, the ECU 10 determines whether or not the second condition is satisfied. In this example, the second condition is satisfied when it is estimated that there is a high possibility that the power of the battery 304 will run out quickly, and is satisfied when at least one of the following conditions E1 to E3 is satisfied. (Condition E1): The remaining power of the battery 304 is smaller than a predetermined remaining power threshold Rath. (Condition E2): The magnitude (absolute value) of the amount of decrease per unit time of the remaining power of the battery 304 is greater than a predetermined change amount threshold Dath. (Condition E3): The temperature of the mobile terminal 110 is greater than a predetermined temperature threshold value Tath.
[0109] When the second condition is met (i.e., it is estimated that there is a high possibility that the power of the battery 304 will run out quickly), the mobile terminal 110 executes an information transmission process so that the degree of consumption of the power of the battery 304 is reduced compared to when the second condition is not met (i.e., it is estimated that the above possibility is low).
[0110] (Example of operation) Next, the process executed by the ECU 10 (specifically, the CPU 10a) in step 607 in Fig. 6 will be described with reference to Fig. 8. In step 607, the ECU 10 executes a routine in Fig. 8 instead of the routine in Fig. 7. The ECU 10 executes a routine not shown to repeatedly acquire information on the remaining amount of power in the battery 304, information on the temperature of the mobile terminal 110, and the like from the mobile terminal 110.
[0111] The ECU 10 starts the process from step 800 in FIG. 8 and proceeds to step 801 to determine whether or not the above condition C1 is satisfied. Now, it is assumed that the condition C1 is satisfied. In this case, the ECU 10 judges "Yes" in step 801 and proceeds to step 802 to determine whether or not the second condition is satisfied. If the second condition is not satisfied, the ECU 10 judges "No" in step 802 and proceeds to step 804 to determine the first process as the information transmission process. Thereafter, the ECU 10 proceeds to step 895 and ends this routine. As a result, the ECU 10 transmits a process instruction signal to the mobile terminal 110 in step 608 to cause the mobile terminal 110 to execute the first process.
[0112] On the other hand, if the second condition is satisfied, the ECU 10 judges "Yes" in step 802 and proceeds to step 803 to notify the driver DR of a driving operation changeover request. This is because, when the predicted time Tf until the end of the automatic driving control is short and there is a high possibility that the power of the battery 304 will run out quickly, it is better to have the driver DR take over the driving operation as soon as possible. After that, the ECU 10 proceeds to step 895 and ends this routine. In this case, the ECU 10 proceeds directly from step 607 to step 612 in the routine of FIG. 6 to end the automatic driving control.
[0113] On the other hand, it is assumed that the condition C1 is not satisfied in step 801. In this case, the ECU 10 judges "No" in step 801 and proceeds to step 811 to judge whether or not the above-mentioned condition C2 is satisfied. If the condition C2 is satisfied, the ECU 10 judges "Yes" in step 811 and proceeds to step 812 to judge whether or not the second condition is satisfied. If the second condition is not satisfied, the ECU 10 judges "No" in step 812 and proceeds to step 814 to determine the first process as the information transmission process. After that, the ECU 10 proceeds to step 895 and ends this routine. As a result, the ECU 10 transmits a process instruction signal to the mobile terminal 110 in step 608 to cause the mobile terminal 110 to execute the first process.
[0114] On the other hand, if the second condition is satisfied, the ECU 10 judges "Yes" in step 812 and proceeds to step 813 to determine the second process as the information transmission process. After that, the ECU 10 proceeds to step 895 to end this routine. As a result, the ECU 10 transmits a process instruction signal to the mobile terminal 110 in step 608 to cause the mobile terminal 110 to execute the second process.
[0115] Assume that the condition C2 is not satisfied in step 811. In this case, the ECU 10 judges "No" in step 811 and proceeds to step 821 to judge whether the second condition is satisfied. If the second condition is not satisfied, the ECU 10 judges "No" in step 821 and proceeds to step 823 to determine the third process as the information transmission process. After that, the ECU 10 proceeds to step 895 and ends this routine. As a result, the ECU 10 transmits a processing instruction signal to the mobile terminal 110 in step 608 to cause the mobile terminal 110 to execute the third process. In this example, the third process is a process of acquiring the second image 500 and transmitting the second image 500 to the automatic driving device 100 as terminal side information every time the third time T3 elapses. The third time T3 is longer than the second time T2. Therefore, the third process is a process that imposes a smaller processing load on the mobile terminal 110 than the second process. That is, the third process consumes less power from the battery 304 than the second process.
[0116] On the other hand, when the second condition is satisfied, the ECU 10 judges "Yes" in step 821 and proceeds to step 822 to determine the fourth process as the information transmission process. After that, the ECU 10 proceeds to step 895 and ends this routine. As a result, the ECU 10 transmits a processing instruction signal to the mobile terminal 110 in step 608 to cause the mobile terminal 110 to execute the fourth process. In this example, the fourth process is a process of acquiring the second image 500 and transmitting the second image 500 to the automatic driving device 100 as terminal side information every time the fourth time T4 elapses. The fourth time T4 is longer than the third time T3. Therefore, the fourth process is a process that imposes a smaller processing load on the mobile terminal 110 than the third process. That is, the fourth process is a process that consumes less power from the battery 304 than the third process.
[0117] The second system having the above configuration has the following effects. The mobile terminal 110 changes the content of the information transmission process according to whether or not there is a possibility that the power of the battery 304 will run out quickly. Specifically, when it is estimated that there is a high possibility that the power of the battery 304 will run out quickly, the mobile terminal 110 executes the information transmission process so that the degree of consumption of the power of the battery 304 is smaller than when the possibility is estimated to be low. With this configuration, the mobile terminal 110 can continue the information transmission process while preventing the power of the battery 304 from running out (or while extending the time until the power of the battery 304 runs out). The ECU 10 can continue the awake state determination process by using the terminal side information transmitted by the information transmission process.
[0118] (Variation 1 of the second system) When the second condition is satisfied, the ECU 10 may notify the driver DR of a message encouraging the driver DR to charge the mobile terminal 110 using the notification device 51. The mobile terminal 110 may also notify the driver DR of the message.
[0119] (Variation 2 of the second system) As in the first embodiment, the first process may be a process of acquiring at least one of the second image 500, the operation information, and the second acceleration information. The second process may be a process of acquiring at least one of the second image 500, the operation information, and the second acceleration information, as long as the process consumes less power of the battery 304 than the first process. The third process may be a process of acquiring at least one of the second image 500, the operation information, and the second acceleration information, as long as the process consumes less power of the battery 304 than the second process. The fourth process may be a process of acquiring at least one of the second image 500, the operation information, and the second acceleration information, as long as the process consumes less power of the battery 304 than the third process.
[0120] (Variation 3 of the second system) When the second condition is satisfied and there is a high possibility that the automatic driving control will be executed for a long period of time, the ECU 10 may select, as the information transmission process, a process that consumes the least amount of power from the battery 304.
[0121] <Third embodiment> Next, an awakening state determination system according to a third embodiment (hereinafter, may be referred to as a "third system") will be described. In this embodiment, when the autonomous driving control is terminated, the mobile terminal 110 continues or terminates the monitoring application 330 according to the likelihood of resuming the autonomous driving control.
[0122] 6, when the ECU 10 ends the autonomous driving control in step 612 of the routine in Fig. 6, the mobile terminal 110 ends the monitoring application 330 in step 614. In this way, in a situation where the autonomous driving control is not being executed, it is preferable for the mobile terminal 110 to end the monitoring application 330 from the viewpoint of not reducing the power of the battery 304.
[0123] However, there are cases where the automatic driving control is resumed within a relatively short time after the automatic driving control is terminated. An example of such a case is when the vehicle VA enters a second congested section relatively quickly after leaving a first congested section. Another example is when the vehicle VA enters another motorway again within a relatively short time after leaving a motorway. In these cases, the processes of steps 603 and 604 in FIG. 6 are performed again, which is annoying for the driver DR. Furthermore, it takes time to start the monitoring application 330 again.
[0124] In consideration of the above, the ECU 10 may determine the possibility of restarting the autonomous driving control in step 612 of Fig. 6. When the ECU 10 determines that the possibility of restarting the autonomous driving control is low, the ECU 10 terminates the monitoring application 330. In contrast, when the ECU 10 determines that the possibility of restarting the autonomous driving control is high, the ECU 10 continues the monitoring application 330.
[0125] For example, the ECU 10 may acquire destination information of the vehicle VA from the navigation ECU 60. When the vehicle VA is scheduled to enter a motorway again based on the destination information of the vehicle VA, the ECU 10 determines that there is a high possibility that the automatic driving control will be resumed.
[0126] For example, the ECU 10 may receive road traffic information from VICS (registered trademark) via the communication unit 80. When there is a high possibility that the vehicle VA will enter a congested section again within a relatively short time, the ECU 10 determines that there is a high possibility that the automatic driving control will be resumed.
[0127] The third system having the above configuration has the following advantages. When there is a high possibility of resuming the automatic driving control, the driver DR does not need to perform an operation to resume the information transmission process (the operation of step 604). The driver DR does not feel bothered. This improves convenience. Furthermore, when the automatic driving control is resumed, the ECU 10 can immediately resume the wakefulness determination process using the terminal side information.
[0128] (Variation 1 of the third system) When the mobile terminal 110 determines that the automatic driving control is likely to be resumed, the mobile terminal 110 may change the operating state of the monitoring application 330. It is assumed that the mobile terminal 110 acquires the second image 500, the operation information, and the second acceleration information as the terminal side information. When the automatic driving control is likely to be resumed, the mobile terminal 110 continues the process of the information transmission process that consumes a small amount of power from the battery 304, and ends the process of the information transmission process that consumes a large amount of power from the battery 304. Specifically, the mobile terminal 110 continues the process of acquiring the operation information and the second acceleration information, and ends only the process of acquiring the second image 500. The load of the process of acquiring the second image 500 is larger than the load of the process of acquiring other information (the operation information and the second acceleration information). Therefore, the mobile terminal 110 ends only the process of acquiring the second image 500, and thus it is possible to prevent the remaining power of the battery 304 from being rapidly reduced. On the other hand, since the mobile terminal 110 continues the process of acquiring the operation information and the second acceleration information, when the automatic driving control is resumed, the ECU 10 can immediately resume the wakefulness determination process by using the terminal side information (the operation information and the second acceleration information). Furthermore, the driver DR does not need to perform an operation for resuming the information transmission process (the operation of step 604). This improves convenience.
[0129] (Variation 2 of the third system) When there is a high possibility that the automatic driving control will be resumed, the mobile terminal 110 may stop only the process of acquiring the second image 500, the operation information, and the second acceleration information while keeping the monitoring application 330 running. This makes it possible to more effectively suppress the consumption of the remaining power of the battery 304.
[0130] The present disclosure is not limited to the above-described embodiments, and various modifications can be made within the scope of the present disclosure.
[0131] The ECU 10 may determine whether or not the condition C2 is satisfied based on an image acquired by the driver monitor camera 76 (that is, the first image).
[0132] The mobile terminal 110 may execute the process of step 607 in Fig. 6. In this case, the mobile terminal 110 acquires the road information and traffic information from the ECU 10.
[0133] Another ECU may be disposed between the automatic driving ECU 10 and the mobile terminal 110. The other ECU may execute a process for determining whether the driver DR is in an awake state. [Explanation of symbols]
[0134] 10...automatic driving ECU, 20...engine ECU, 30...brake ECU, 40...steering ECU, 50...meter ECU, 60...navigation ECU, 100...automatic driving device, 110...mobile terminal.
Claims
1. A control device that executes an awakening state determination process to acquire an awakening level of a driver of a vehicle; a portable terminal carried by the driver and including a battery and a second imaging unit, the portable terminal executing an information transmission process to transmit an image acquired by the second imaging unit as information to be used in the wakefulness state determination process to the control device as terminal side information; Equipped with The control device includes: Executing the wakefulness state determination process using the terminal side information; when the degree of wakefulness acquired by the wakefulness state determination process is a first degree indicating that the driver is in an wakeful state, it is determined that the level of importance of monitoring the driver is lower than a predetermined level, and when the degree of wakefulness acquired by the wakefulness state determination process is either a second degree indicating that the driver is not in a drowsy state but is in a state of feeling drowsy or a third degree indicating that the driver is in a drowsy state, it is determined that the level of importance is higher than the predetermined level, The mobile terminal includes: When it is determined that the level of importance is higher than the predetermined level, a first process is executed as the information transmission process, in which the image is acquired by the second imaging unit every time a first time period (T1) elapses and the image is transmitted to the control device as the terminal side information; When it is determined that the level of importance is lower than the predetermined level, a second process is executed as the information transmission process, in which the image is acquired by the second imaging unit and the image is transmitted to the control device as the terminal side information every time a second time (T2) longer than the first time (T1) elapses. It was configured as follows: Awakening state determination system.
2. The wakefulness state determination system according to claim 1, The mobile terminal includes: Even if the level of importance is determined to be higher than the predetermined level, When it is estimated that there is a high possibility that the power of the battery will be depleted quickly, the second process is executed as the information transmission process; When it is estimated that there is a low possibility that the power of the battery will run out quickly, the first process is executed as the information transmission process. It was configured as follows: Awakening state determination system.
3. The wakefulness state determination system according to claim 1, The control device includes: The vehicle is configured to execute an automatic driving control for automatically performing a driving operation of the vehicle, When the predicted time (Tf) until the end of the autonomous driving control is within a predetermined time threshold (Tth), it is determined that the level of importance is higher than the predetermined level; If the predicted time (Tf) is not within the time threshold (Tth), the importance level is determined to be lower than the predetermined level. It was configured as follows: Awakening state determination system.
4. 4. The wakefulness state determination system according to claim 3, When the automatic driving control is terminated, the mobile terminal When there is a high possibility that the automatic driving control will be resumed, the information transmission process is continued; When the possibility of restarting the automatic driving control is low, the information transmission process is terminated. It was configured as follows: Awakening state determination system.
Citation Information
Patent Citations
Driver warning device
JP2014115983A
Apparatus and method for predicting alertness level of motor vehicle driver
JP2018518205A
Drive shift control device and drive shift control program
JP2019038471A
Awakening support device, program, and awakening support system
JP2019109779A
Drowsiness detection system, drowsiness detection data generation system, drowsiness detection method, computer program, and detection data
JP2019154613A
