System and method for monitoring driver glance behavior

Abstract

To provide a method, a system, and a device for a driver monitoring system.SOLUTION: A driver monitoring system includes an internal vehicle camera, an internal vehicle display, and an electronic control unit (ECU). When a driver enters a vehicle, a prompt (e.g., consent) may be displayed to the driver to provide driver input. In response to the driver providing the driver input, the ECU may adjust the permissible glance time. The ECU may be configured to use the camera to determine that the driver's eye position is in a location where the driver's gaze direction is not oriented toward the vehicle's direction of travel for a period exceeding the adjusted glance time. Accordingly, the ECU may be configured to initiate countermeasures to facilitate a corrective response from the driver.SELECTED DRAWING: Figure 2A

Description

[Background technology]

[0001] Field

[0002] FIELD OF THE DISCLOSURE The present disclosure relates generally to systems and methods for monitoring vehicle drivers, and more particularly to systems and methods for managing driver glance behavior.

[0003] 2. Description of Related Art

[0004] Automobiles are operated by drivers in a variety of conditions. There are a variety of distractions within and around the vehicle and roadway that can draw the driver's attention away from the vehicle's direction of travel. A distracted driver increases the likelihood of creating unsafe driving conditions, which can result in potential harm to the driver, other vehicle occupants, occupants in nearby vehicles, and / or pedestrians. Modern vehicles often include driver camera monitoring systems that allow the driver to look away for a short period of time (e.g., four seconds or less) before receiving a warning that the driver needs to look back at the road. However, drivers can become frustrated with the frequency of such warnings in certain driving conditions.

[0005] It is therefore desirable to provide a system and method for monitoring vehicle driver behavior to enhance road and vehicle safety, while at the same time improving the user experience and safety of the vehicle. Summary of the Invention

[0006] Generally, one aspect of the subject matter described in this disclosure may be embodied in a system for monitoring a vehicle driver. The system includes a camera operably disposed in the vehicle and configured to monitor an eye position of the vehicle driver while the vehicle is being operated. The system further includes a display operably disposed in the vehicle. The system further includes an electronic control unit connected to the camera and the display. The electronic control unit is configured to operate the display to display a request to the vehicle driver prompting the vehicle driver to provide driver input. The electronic control unit is further configured to adjust the allowable glance time from a default glance time to an adjusted glance time in response to the vehicle driver providing the driver input. The electronic control unit is further configured to determine an eye position of the vehicle driver such that the vehicle driver's gaze direction is not directly directed in the direction of vehicle travel while the vehicle is being operated. The electronic control unit is further configured to determine that a period of time during which the vehicle driver's gaze direction is not directly directed in the direction of vehicle travel exceeds the adjusted glance time. The electronic control unit is further configured to initiate countermeasures to prompt a response from the vehicle driver in response to determining that the period of time during which the vehicle driver's gaze direction is not directed directly in the direction of vehicle travel exceeds the adjusted glance time.

[0007] In another aspect, the subject matter may be embodied in a method for managing a driver's glance behavior. The method includes operating a display to display a request to a vehicle driver prompting the vehicle driver to provide driver input. The method further includes, in response to the vehicle driver providing the driver input, adjusting the allowable glance time from a default glance time to an adjusted glance time (e.g., a time longer than the default glance time). The method further includes determining that the vehicle driver's eyes are positioned such that the vehicle driver's gaze direction is not directly directed in the direction of vehicle movement while the vehicle is being operated. The method further includes determining that a period of time during which the vehicle driver's gaze direction is not directly directed in the direction of vehicle movement exceeds the adjusted glance time. The method further includes, in response to determining that the period of time during which the vehicle driver's gaze direction is not directly directed in the direction of vehicle movement exceeds the adjusted glance time, initiating a countermeasure to encourage a response from the vehicle driver.

[0008] In another aspect, the subject matter may be embodied in a non-transitory computer-readable medium storing content that causes one or more computing systems to perform an automated operation. The automated operation includes operating, by the one or more computing systems, a display to display a request to the vehicle driver prompting the vehicle driver to provide driver input. The automated operation includes adjusting, by the one or more computing systems, an allowable glance time from a default glance time to an adjusted glance time in response to the vehicle driver providing the driver input. The automated operation includes determining, by the one or more computing systems, that the vehicle driver's eyes are positioned such that the vehicle driver's gaze direction is not directly directed in the direction of vehicle travel while the vehicle is being operated. The automated operation includes determining, by the one or more computing systems, that a period of time during which the vehicle driver's gaze direction is not directly directed in the direction of vehicle travel exceeds the adjusted glance time. The automatic operation includes initiating, by one or more computing systems, countermeasures to prompt a response from the vehicle operator in response to determining that a period of time during which the vehicle operator's gaze direction is not directed directly in the direction of vehicle travel exceeds an adjusted glance time.

[0009] Optionally, these and other embodiments may include one or more of the following features.

[0010] In various aspects, the driver input includes an acceptance of the agreement by the vehicle driver.

[0011] In various aspects, the driver input includes driver attitude information.

[0012] In various aspects, the electronic control unit is further configured to determine a vehicle crash or safety risk and adjust the allowable glance time based on the driver and / or vehicle crash or safety risk. The adjustments to the allowable glance time based on the vehicle crash or safety risk may be made in real time to reduce the allowable glance time while the vehicle is being operated by the vehicle driver. The adjustments to the allowable glance time may be user-adjusted by a vehicle owner (e.g., a parent). The allowable glance time may also be password-protected or locked by the vehicle owner to improve safety.

[0013] In various aspects, the electronic control unit is further configured to determine a geographic location of the vehicle and adjust the allowable glance time based on the geographic location of the vehicle.

[0014] In various aspects, the electronic control unit is further configured to adjust the allowable glance time based on the experience level of the vehicle operator.

[0015] In various aspects, the electronic control unit is further configured to adjust the allowable glance time based on the temporal data.

[0016] In various aspects, the electronic control unit is further configured to adjust the allowable glance time based on real-time vehicle data received from vehicle sensors.

[0017] Various aspects are described in a step-by-step manner. However, the methods described herein may be performed continuously while the vehicle is operating, using a rolling window over incoming vehicle and / or driver data. [Brief explanation of the drawings]

[0018] Other systems, methods, features, and advantages of the present invention will become apparent to one of ordinary skill in the art upon examination of the following figures and detailed description. Components shown in the drawings are not necessarily to scale and may be exaggerated to more fully illustrate the important features of the present invention.

[0019] [Figure 1] FIG. 1 is a block diagram of an exemplary driver monitoring system according to an embodiment of the present invention.

[0020] [Figure 2A] FIG. 2A shows an exemplary diagram of the positioning of one or more internal cameras of the driver monitoring system of FIG. 1 within a vehicle, with the vehicle driver's gaze direction facing in the direction of vehicle travel, according to an embodiment of the present invention. [Figure 2B] FIG. 2B shows an exemplary diagram of the positioning of one or more internal cameras of the driver monitoring system of FIG. 1 within a vehicle where the vehicle driver's gaze direction is not directed in the direction of vehicle movement, according to an embodiment of the present invention.

[0021] [Figure 3] FIG. 3 is a flow diagram of an exemplary process for managing driver glance behavior using the driver monitoring system of FIG. 1 according to an embodiment of the present invention.

[0022] [Figure 4] FIG. 4 is a flow diagram of an exemplary process for managing driver glance behavior using the driver monitoring system of FIG. 1 according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0023] Disclosed herein are systems, vehicles, and methods for monitoring a vehicle driver and managing the driver's glance behavior. The driver monitoring system may include a vehicle inward-facing camera (e.g., inside the cabin facing the driver), a vehicle display, and a processing unit configured to execute instructions to monitor the driver and / or provide driver alerts. The driver monitoring system may monitor the driver's eye and / or head positioning to determine whether the driver's eyes are directed toward the road. The driver monitoring system may allow the driver to look away for a short period of time (also referred to herein as an acceptable glance time), e.g., between 1 and 6 seconds, before activating a driver assistance alert prompting the driver to look ahead or forward.

[0024] Aspects of the subject matter described in this disclosure may be implemented to achieve one or more of the following advantages: A driver monitoring system may vary the allowable glance time based on various factors, including whether the driver accepts an agreement in exchange for longer glances in lower-risk contexts. An "agreement" is a prompt at the beginning of a drive in which the driver promises to minimize or eliminate their involvement in non-driving-related tasks in exchange for longer allowable glance times on highways or other roadways. More generally, the driver may agree not to engage in non-driving-related tasks in high-risk contexts in exchange for longer glances in low-risk contexts. The driver monitoring system may, upon agreement, tighten glance limits in high-risk contexts in exchange for longer glances in lower-risk contexts. By providing the vehicle driver with an incentive for longer allowable glance times as measured by the driver monitoring system, the driver monitoring system tends to reduce the driver's involvement in non-driving-related tasks.

[0025] In various aspects, the acceptable glance time may also be based on the driver's age, health, eyesight, time of day, traffic congestion, road quality, speed limits, weather conditions, roadway hazards, vehicle type, number of passengers in the vehicle, and / or combinations thereof. As an example, if the driver's age is over 60, the acceptable glance time is reduced. If there is a large amount of traffic congestion, the acceptable glance time is reduced. If only the driver is present in the vehicle, the acceptable glance time is increased.

[0026] For example, a driver monitoring system may be used in L2 (Level 2) driving automation, where the purpose of the prompts is to promote driving task engagement and driver roadway monitoring for hazards. The driver monitoring system may address both driver distraction mitigation and L2 driving automation engagement. The driver monitoring system may reduce driver engagement in non-driving-related tasks in complex, high-risk roadway environments. The driver monitoring system may reduce driver non-use or aversion to driver monitoring technology by providing longer limits on allowable glance times.

[0027] The driver monitoring system may utilize supervised machine learning predictive models to take into account vehicle data, survey data, driver behavior data, different environmental factors, and / or environmental conditions. The driver monitoring system may utilize a global positioning system (GPS) unit to detect location data, including the vehicle's current location, to determine various vehicular traffic, pedestrian traffic, and / or surrounding environmental event information. In this manner, the driver monitoring system may consider various environmental and / or temporal factors, such as time of day, location, weather, and / or other factors, when determining an appropriate and acceptable glance duration. This allows for a more precise and accurate understanding of different risk levels at different locations, times, etc., to accurately determine an appropriate and acceptable glance duration.

[0028] Other benefits and advantages include the use of artificial intelligence, including machine learning algorithms with models that determine crash or safety risk. By determining crash or safety risk, the driver monitoring system predicts risk levels and determines appropriate acceptable glance times. For example, the driver monitoring system may alert the driver to promote a corrective response from the driver to prevent risky driving behavior and encourage the driver to look ahead. In another example, the driver monitoring system may control various aspects of the vehicle to prevent risky driving behavior and / or encourage the driver to look ahead. The driver monitoring system may learn from each instance of driver assistance warnings being issued and the driver's response thereto.

[0029] Various aspects refer to "soft" and "hard" countermeasures. Generally, "soft" countermeasures refer to those that encourage or attempt to shape behavior through suggestions or prompts, while "hard" refers to those that directly modify driver behavior (e.g., pedal force, feature availability, etc.) through the vehicle interface.

[0030] 1 is a block diagram of a monitoring system 100 (e.g., a driver monitoring system). The monitoring system 100 may be retrofitted to, connected to, include, or contained within a vehicle 102. The monitoring system 100 may connect to, be coupled to, or include an external database 104. The monitoring system 100 may have a network 106 that links the external database 104 with the vehicle 102. The network 106 may be a local area network (LAN), a wide area network (WAN), a cellular network, the Internet, or a combination thereof that couples, connects, and / or otherwise communicates between the vehicle 102 and the external database 104.

[0031] Monitoring system 100 may monitor the driver's eye and / or head positioning to determine whether the driver's eyes are facing the road. Driver monitoring system 100 may allow the driver to look away for a predetermined glance time before activating a driver assistance alert that prompts the driver to look ahead or ahead.

[0032] The monitoring system 100 may include, be retrofitted to, or otherwise connected to a vehicle 102. A vehicle 102 is a vehicle capable of transporting people, goods, or permanently or temporarily attached equipment. The vehicle 102 may be a self-propelled, wheeled vehicle, such as a passenger car, sport utility vehicle, truck, bus, van, or other motor-, battery-, or fuel-cell-powered vehicle. For example, the vehicle 102 may be an electric vehicle, a hybrid vehicle, a hydrogen fuel cell vehicle, a plug-in hybrid vehicle, or any other type of vehicle having a fuel cell stack, a motor, and / or a generator. Other example vehicles include bicycles, trains, airplanes, or boats, and any other form of vehicle capable of transportation. The vehicle 102 may be semi-autonomous or autonomous. That is, the vehicle 102 may operate on its own, without human input. An autonomous vehicle may have and use one or more sensors and / or navigation units to drive autonomously.

[0033] The monitoring system 100 includes one or more processors, such as an electronic control unit (ECU) 108, and a memory 110. The monitoring system 100 may include other components, such as a navigation unit 112, one or more sensors 114, including one or more external cameras 116a, one or more internal cameras 116b, a vehicle speed sensor 118, and / or other sensors 136, a network access device 120, a user interface 122, and / or an output device 124. The other sensors 136 may include radar, ultrasonic, LiDAR, microphones, etc. The monitoring system 100 may connect to, be coupled to, and / or include one or more vehicle components, such as a motor and / or generator 126, an engine 128, a battery 130, a transmission 132, and / or a battery management control unit (BMCU) 134.

[0034] The ECU 108 may be implemented as a single ECU or multiple ECUs. The ECU 108 may be electrically connected to some or all of the other components in the vehicle 102, such as the motor and / or generator 126, the transmission 132, the engine 128, the battery 130, the battery management control unit (BMCU) 134, the memory 110, the network access device 120, and / or one or more sensors 114. The ECU 108 may monitor the vehicle 102 and / or the driver of the vehicle 102. The ECU 108 may include one or more processors or controllers specifically designed to monitor the driver's eye and / or head positioning. The ECU 108 may issue a driver assistance warning in response to the driver glancing away from the road ahead for longer than a predetermined period of time. The ECU 108 may be configured to determine an appropriate maximum allowable glance duration based on various factors, as described herein.

[0035] The ECU 108 may interface with various countermeasures to encourage the driver to maintain focus on the road ahead, thereby reducing risky driving. The ECU 108 may employ "soft" countermeasures such as prompting the driver to look at the road or encouraging the use of automated driving features to maintain a safe distance. The ECU 108 may employ "hard" countermeasures such as reducing the speed of the vehicle 102 and / or increasing the following distance of the vehicle 102 relative to a lead vehicle.

[0036] The ECU 108 may further use driver data, vehicle data, environmental data, and / or temporal data to help determine a crash or safety risk. The maximum allowable glance time may be reduced due to an increased crash or safety risk. For example, the ECU 108 may analyze vehicle data, which may include vehicle 102 parameters (e.g., speed, GPS location, acceleration, etc.) and / or environmental parameters external to the vehicle 102 (e.g., surrounding vehicles, buildings, pedestrians, weather, road conditions, etc.), to determine an appropriate maximum allowable glance time. For example, the maximum allowable glance time may be reduced in heavy traffic and / or general roads that typically have more intersections, pedestrians, etc., while the maximum allowable glance time may be increased in light traffic and / or highways or freeways that typically have fewer intersections, pedestrians, etc. If the maximum allowable glance time is exceeded, the ECU 108 may issue a driver assistance warning, prompting the driver to look ahead to mitigate the risky driving behavior. In various aspects, the ECU 108 may utilize machine learning algorithms to determine crash or safety risk. The ECU 108 may be coupled to a memory 110 and may execute instructions stored in the memory 110.

[0037] Memory 110 is coupled to ECU 108 and may store instructions executed by ECU 108. Memory 110 may include one or more of random access memory (RAM) or other volatile or non-volatile memory. Memory 110 may be a non-transitory memory or data storage device, such as a hard disk drive, a solid-state disk drive, a hybrid disk drive, or other suitable data storage, and may further store machine-readable instructions that may be loaded and executed by ECU 108. Additionally, memory 110 may be used to record and store data before, after, and / or during the occurrence of a risky driving behavior.

[0038] The monitoring system 100 may include a user interface 122. The monitoring system 100 may display one or more notifications on the user interface 122. The one or more notifications on the user interface 122 may notify a vehicle occupant when to initialize or activate the monitoring system 100. The user interface 122 may include input / output devices that receive user input from user interface elements, buttons, dials, a microphone, a keyboard, or a touchscreen. For example, the user interface 122 may receive user input that may include a confirmation regarding consent acceptance. For example, other configurations may include preferences regarding the magnitude or type of countermeasures to mitigate risky driving behavior. The user interface 122 may provide output to an output device 124, such as a display, a speaker, audio and / or visual indicators, or a refreshable Braille display. For example, the user interface 122 may display a warning, alert, or specific countermeasure being taken by the ECU 108.

[0039] The monitoring system 100 may include a network access device 120. The network access device 120 may include one or more communication ports or channels, such as a WiFi unit, a Bluetooth unit, a radio frequency identifier (RFID) tag or reader, or a cellular network unit that accesses a cellular network (such as 3G, 4G, or 5G). The network access device 120 may transmit data to and receive data from the external database 104. For example, the ECU 108 may communicate with the external database 104 via the network 106 to obtain a baseline model and / or predictive algorithms that take into account behavior and / or objects at the current location of the vehicle 102. The monitoring system 100 may use the baseline model and / or predictive algorithms to determine a collision or safety risk for the vehicle 102.

[0040] The network access device 120 may transmit data to and receive data from a user device 140 (e.g., a smartphone, tablet, personal computer, etc.), which may be located remotely from the vehicle 102. The monitoring system 100 may display one or more notifications on the user device 140, as well as the user interface 122. Thus, the user device 140 may have a user interface by which a user may communicate with the ECU 108 from a remote location.

[0041] Network access device 120 may transmit data to and receive data from other databases 142 using, for example, vehicle-to-vehicle (V2V) communications, vehicle-to-everything (V2X), and / or vehicle-to-infrastructure (V2I) communications. For example, monitoring system 100 may enable vehicle 102 to exchange vehicle data with a second vehicle using V2V communications techniques. Monitoring system 100 may use V2I and / or V2X communications techniques to receive location data such as traffic congestion, weather reports, bridge clearance levels, traffic light status, and / or crime data that informs monitoring system 100 of conditions at or near the location of vehicle 102 or the location to which vehicle 102 is headed. Thus, vehicle 102 may communicate with another vehicle or network via network access device 120 using vehicle-to-vehicle or vehicle-to-infrastructure communications.

[0042] The monitoring system 100 may further include a navigation unit 112. The navigation unit 112 may be integrated into the vehicle 102 or may be a separate unit connected to the vehicle 102, such as a personal device with navigation capabilities. When the navigation unit 112 is separate from the vehicle 102, the navigation unit 112 may communicate with the vehicle 102 via the network access device 120. In some implementations, the vehicle 102 may include a GPS unit instead of the navigation unit 112 to detect location data including the current location and date / time information of the vehicle 102. In that regard, the ECU 108 may perform the functions of the navigation unit 112 based on data received from the GPS unit. At least one of the navigation unit 112 or the ECU 108 may predict or suggest a route plan including a start location and a destination location. The navigation unit 112 or the ECU 108 may perform the navigation function. Navigation functions may include, for example, predicting a route and routing, providing navigation instructions, and receiving user input such as confirmation of the predicted route and routing or destination. Other information, for example, the current speed of the vehicle 102, may be extrapolated, interpreted, or otherwise calculated from data obtained from the navigation unit.

[0043] The navigation unit 112 may provide and obtain navigation map information for the vehicle 102, including location data, which may include a current location, a starting location, a destination location, and / or a route between the starting location or the current location and the destination location. The navigation unit 112 may include a memory (not shown) that stores the route data. The navigation unit 112 may receive data from other sensors capable of detecting data corresponding to the location information. For example, the other sensors may include a gyroscope or an accelerometer.

[0044] Navigation map information may include entity information, which may include locations or points of interest, such as government buildings, private businesses, schools, tourist attractions, or other points of interest. These various entities may be factors in determining collision or safety risks.

[0045] The surveillance system 100 may further include one or more sensors 114. The one or more sensors 114 may include one or more external cameras 116a, one or more internal cameras 116b, a vehicle speed sensor 118, and / or other sensors 136.

[0046] The one or more external cameras 116a may be positioned along the exterior of the vehicle 102, such as on the roof, trunk, sides, and / or front of the vehicle 102. The location of the external cameras 116a may vary depending on the type of vehicle, among other factors. Various views of the surrounding environment may be used to form a panoramic or 360-degree image of the surrounding environment outside the vehicle 102, enabling the surveillance system 100 to detect other vehicles and / or objects outside the vehicle 102, such as vehicles in front of, beside, or behind the vehicle 102. The one or more external cameras 116a may capture image data including a single frame or image or continuous video of the surrounding environment outside the vehicle 102.

[0047] 2A , the one or more internal cameras 116b may include multiple cameras positioned within the vehicle 102 to capture various fields of view within the cabin of the vehicle 102. The one or more internal cameras 116b may be positioned within the vehicle 102, such as on the rearview mirror 202, in or on the dashboard 204, on or near the visor, or any other suitable location that may capture the face / body of the driver and / or passengers. The one or more internal cameras 116b may capture images including a single frame or image or continuous video of the interior of the vehicle 102 or the environment within the vehicle 102.

[0048] In one example, the ECU 108 may utilize the internal camera 116b as an eye tracking device to monitor the vehicle driver's eye position while the vehicle is being operated. For example, one or more internal cameras 116b may be used to measure the driver's eye position (e.g., gaze point) and the driver's eye movement (e.g., eye movement relative to the driver's head). This may be achieved by utilizing a facial imaging camera, which may be positioned inside the vehicle at any location in front of (directly or peripherally) the vehicle driver. Exemplary locations for the facial imaging camera may include on the rearview mirror 202, on the dashboard 204, on the steering wheel mounting stem, or the like. The one or more internal cameras 116b may be configured to capture images or video of the vehicle driver's face while driving, and the ECU 108 is configured to extract the driver's eye position from the images / video.

[0049] In various aspects, the driver's eye movements are determined by light (such as infrared light) reflected from the cornea of ​​the eye, which is detected by a suitable electronic device or optical sensor (which may be part of the one or more internal cameras 116b). Information regarding the eye movements may be utilized (e.g., by the ECU 108 shown in FIG. 1 , which is associated with the one or more internal cameras 116b) to determine the driver's eye rotation based on changes in the reflected light. In various aspects, the ECU 108 is configured to estimate the vehicle driver's eye position while the vehicle is being operated by monitoring the driver's head position. For example, in response to the ECU 108 detecting via the one or more internal cameras 116b that the driver's head has tilted downward (e.g., see FIG. 2B), it may be assumed that the driver is looking down.

[0050] The one or more sensors 114 may include a vehicle speed sensor 118. The vehicle speed sensor 118 may measure the amount of rotation of multiple wheels to determine whether the vehicle 102 is stationary and / or parked.

[0051] The one or more sensors 114 may include other sensors 136 that measure road conditions, weather, ambient light around the vehicle 102, and / or other environmental factors that may be used to determine collision or safety risk. The other sensors 136 may include one or more external ultrasonic sensors positioned outside and / or within the vehicle 102 but pointed outward to capture various views of the surrounding environment outside the vehicle 102. The ultrasonic sensors may be used to detect other vehicles and / or objects outside the vehicle 102, such as vehicles in front of, beside, or behind the vehicle 102. The other sensors 136 may include gyroscopes and / or accelerometers that measure the attitude and / or acceleration of the vehicle 102.

[0052] The monitoring system 100 may include an output device 124. The output device 124 may be an audio indicator, a visual indicator, a communication device, or other output device. For example, an audio or visual indicator may be used to sound or flash an alarm, respectively.

[0053] The monitoring system 100 may connect to, be coupled to, and / or include one or more vehicle components. The one or more vehicle components may include a motor and / or generator 126. The motor and / or generator 126 may convert electrical energy into mechanical power, such as torque, and may convert mechanical power into electrical energy. The motor and / or generator 126 may be connected to a battery 130. The motor and / or generator 126 may convert energy from the battery 130 into mechanical power and provide energy back to the battery 130, for example, via regenerative braking. In some implementations, the vehicle 102 may include one or more additional power-generating devices, such as an engine 128 or a fuel cell stack (not shown). The engine 128 burns fuel to provide power instead of and / or in addition to the power provided by the motor and / or generator 126.

[0054] The battery 130 may be connected to the motor and / or generator 126 and may provide electrical energy to and receive electrical energy from the motor and / or generator 126. The battery 130 may include one or more rechargeable batteries.

[0055] The BMCU 134 may be connected to the battery 130 and may control and manage the charging and discharging of the battery 130. For example, the BMCU 134 may use battery sensors to measure parameters used to determine the state of charge (SOC) of the battery 130. The BMCU 134 may control the battery 130.

[0056] The one or more vehicle components may include a transmission 132. The transmission 132 may have different gears and / or modes, such as park, drive, and / or neutral, and may shift between different gears. The transmission 132 manages the amount of power provided to the wheels of the vehicle 102, taking into account the amount of speed.

[0057] The monitoring system 100 may include or be connected to an external database 104. A database is, for example, any collection of pieces of information organized for search and retrieval by a computer. A database may be organized in tables, schemas, queries, reports, or any other data structure. A database may use any number of database management systems. The external database 104 may include a third-party server or website that stores or provides information. The information may include real-time information, periodically updated information, or user-entered information. A server may be a computer in a network used to provide services, such as access to files or sharing of peripherals, to other computers in the network.

[0058] The external database 104 may store personalized driver data, such as driver behavior history, driver survey data, etc. The external database 104 may be updated and / or provide updates in real time. The external database 104 may store and / or provide driver data to the ECU 108. The external database 104 may also store environmental factors, such as weather information or date and time information, and provide the environmental factors to the ECU 108 to assist in determining crash or safety risk. The weather information may include temperature, weather, road conditions, precipitation, and / or other weather factors that may affect crash or safety risk. For example, cold weather, precipitation, and / or driving at night may increase crash or safety risk, while sunny weather and / or driving during the daytime may decrease crash or safety risk. Thus, the ECU 108 may be configured to adjust the allowable glance duration based on temporal data (e.g., date, time of year, etc.). As another example, if personalized driver data indicates that a driver has historically engaged in risky driving behavior in a particular geographic location, monitoring system 100 may determine and / or be more likely to determine a higher crash or safety risk as the driver approaches that geographic location. Monitoring system 100 may reduce the allowable glance time in real time as vehicle 102 approaches an area of ​​high crash or safety risk.

[0059] 3 is a flow diagram of an exemplary process 300 for managing driver glance behavior. One or more appropriately programmed computers or one or more data processing devices, such as the ECU 108 of the monitoring system 100 of FIG. 1, may implement the process 300.

[0060] With combined reference to FIGS. 1 and 3, the ECU 108 may detect that the ignition of the vehicle 102 is turned on (302).

[0061] The ECU 108 may operate the output device 124 (e.g., a display) to indicate the request to the vehicle operator to prompt the vehicle operator to provide driver input (304).

[0062] The ECU 108 may receive input from the vehicle driver in real time (306). The vehicle driver may provide driver input via the user interface 122. In various aspects, the request displayed in step 304 is an agreement, and the driver input is an acceptance of the agreement. For example, the driver may select an icon, toggle, button, or the like that confirms the agreement. Thus, upon entering the vehicle and / or pressing the ignition switch, the monitoring system 100 provides a prompt that may be displayed on the cluster or center stack. The prompt may include the driver's agreement to safe driving behavior (e.g., staying within speed limits, not engaging in non-driving-related tasks, etc.) in exchange for a longer allowable glance time from the driver camera monitoring system or a shorter glance time during high-risk scenarios.

[0063] In various aspects, the prompted agreement may, at a minimum, include a written statement of the driver's responsibility to minimize the driver's performance of non-driving-related tasks and variations in acceptable glance duration depending on the context. The prompted agreement may use messaging that uses social norms (e.g., "Safe drivers do not perform non-driving-related tasks in these contexts") or safety (e.g., "Last year, [numerous people] died in crashes due to driver distraction"). The prompted agreement may include a survey to determine appropriate glance permissions for the driver. The prompted agreement may include a training module that guides the driver on how to manage their glances given the context prior to the agreement.

[0064] In various aspects, the driver input may include data representing the driver's attitudes regarding risky driving behavior, their views on social norms for risky driving behavior, their sensation-seeking assessment, and / or their impulsivity assessment. For example, the request may include a survey asking the driver to provide personalized driver data indicative of the driver's personality and / or attitudes.

[0065] The ECU 108 may adjust the allowable glance time from the default glance time to an adjusted glance time (308). In various aspects, the adjusted glance time is longer than the default glance time. Step 308 may be performed in response to the vehicle operator providing driver input in step 306.

[0066] The ECU 108 may determine (310) that the vehicle driver's glance time exceeds an allowable glance time while the driver is operating the vehicle. In other words, the ECU 108 may determine that a period of time during which the vehicle driver's gaze direction is not directed in the direction of vehicle movement exceeds an adjusted glance time. For example, the ECU 108 may determine that the vehicle driver's eyes are positioned such that the driver's gaze direction is not directed in the direction of vehicle movement while the vehicle is being operated. For example, FIG. 2B illustrates the vehicle driver looking downward at the user interface 122, resulting in the vehicle driver's eyes being positioned such that the vehicle driver's gaze direction is not directed in the direction of vehicle movement while the vehicle is being operated. In contrast, FIG. 2A illustrates the vehicle driver looking forward through the windshield, resulting in the vehicle driver's eyes being positioned such that the vehicle driver's gaze direction is directed (e.g., straight ahead) in the direction of vehicle movement while the vehicle is being operated. The ECU 108 may use one or more internal cameras 116b to monitor the vehicle driver's gaze direction in real time while the vehicle driver is operating the vehicle.

[0067] In response to the vehicle driver maintaining an eye position that is not directed toward the direction of vehicle travel for a period that exceeds the adjusted glance time, the ECU 108 may initiate a countermeasure to encourage a response from the vehicle driver (312). In various aspects, the ECU 108 may be configured to implement a “soft” countermeasure, for example, by displaying an alert or prompting a behavior. In other words, the monitoring system 100 may operate or control one or more vehicle components in response to detecting that the driver's glance time has exceeded the allowable glance time. For example, with momentary reference to FIG. 1 , the ECU 108 may send an alert signal to the user interface 122 and / or the output device 124 to encourage the driver to pay attention to the roadway. The alert signal may be a visual signal (e.g., a light on the vehicle dashboard cluster), an audio signal, and / or a haptic signal (e.g., a vibration in the driver's seat and / or steering wheel that provides a haptic alert to the driver).

[0068] Exemplary soft countermeasures may include displaying a message to the driver such as "Keep your eyes on the road ahead." Soft countermeasures may encourage the driver to use driver convenience systems (e.g., adaptive cruise control, lane keeping assist, etc.). Soft countermeasures may include other warnings, alerts, HMI (human-machine interface), etc. as desired.

[0069] The monitoring system 100 may determine whether the vehicle driver responded to the countermeasure within a predetermined period of time. The monitoring system 100 may store personalized driver data, including a history of the vehicle driver's responses to past countermeasures activated by the monitoring system 100. Generally, the monitoring system 100 may penalize drivers who do not comply with the countermeasures with a shorter allowable glance time, while rewarding compliant drivers with a less stringent (i.e., longer) allowable glance time. The allowable glance time may be changed in real time.

[0070] 4 is a flow diagram of an exemplary process 400 for managing driver glance behavior. One or more appropriately programmed computers or one or more data processing devices, such as the ECU 108 of the monitoring system 100 of FIG. 1, may implement the process 300. With reference to FIG. 4, elements with similar element numbering as depicted in FIG. 3 are intended to be the same or similar and are not necessarily repeated for clarity.

[0071] 1 and 4 , the ECU 108 may determine a vehicle collision or safety risk (410). The ECU 108 may determine the magnitude of the collision or safety risk at a particular geographic location of the vehicle 102 (or a geographic location the vehicle 102 is approaching) based on environmental characteristics (e.g., other vehicles, speed, past collision data, etc.). The ECU 108 may adjust the acceptable glance time based on the estimated collision or safety risk (e.g., a decrease in collision or safety risk may increase the acceptable glance time). For example, a younger driver may receive less acceptable glance time from a prompt agreement due to the higher risk within that driver demographic. Additionally, the acceptable glance time may vary based on the following distance from the lead vehicle or the vehicle's relative speed (e.g., a decrease in following distance decreases the acceptable glance time). The collision or safety risk may be based on past collision data at a particular intersection or geographic location. The collision or safety risk may vary based on temporal data (e.g., date and time or time of year). For example, the collision or safety risk may be higher at night when the driver's visibility is limited. The collision or safety risk may vary based on other environmental characteristics, such as surrounding vehicles, weather, etc.

[0072] The ECU 108 may adjust 412 the allowable glance time based on the collision or safety risk determined in step 410. This adjustment may occur in real time while the vehicle is moving on the roadway. In this manner, the allowable glance time may be personalized in real time based on driver data, vehicle data, and / or other geographic and / or temporal data. The ECU 108 may operate the monitoring system 100 (e.g., by controlling when to activate countermeasures) based on the adjusted allowable glance time.

[0073] The monitoring system 100 may determine the magnitude of a crash or safety risk at a particular location based on environmental characteristics (e.g., other vehicles, speed, past crash or safety data, etc.). The monitoring system 100 may include personalization to adjust the allowable glance time or the allowable glance time variance during the prompt agreement. For example, younger drivers (e.g., ages 18-25) may receive a smaller allowable glance time during the prompt agreement due to the higher risk within that driver demographic. Thus, the ECU 108 may be configured to adjust the allowable glance time based on the vehicle driver's experience level. Additionally, the variance may vary based on the following distance from the lead vehicle or the vehicle's relative speed. Furthermore, these context variables may serve as indicators of driver misuse of this feature, causing the glance time tolerance to be reverted to its original state.

[0074] The ECU 108 may be configured to determine the vehicle's geographic location and adjust the allowable glance time based on the vehicle's geographic location. For example, the allowable glance time may be increased on highways and / or freeways with fewer intersections, pedestrians, and / or distractions. In contrast, the allowable glance time may be limited and / or reduced on local roads with more intersections, pedestrians, and / or distractions. Furthermore, certain geographic locations may be known and / or expected to involve lower or higher risk, and the allowable glance time may be adjusted accordingly.

[0075] Adherence to the prompted agreement during a drive can be used to provide feedback at the end of the drive, leveraging the prompted agreement, and adjustments can be implemented in the future based on that adherence. Generally, drivers who do not comply with the agreement can receive a more severe (i.e., shorter) acceptable glance, while drivers who comply can be given a less severe (i.e., longer) acceptable glance.

[0076] In various aspects, personalization can be performed by using a suitable learning algorithm, for example, reinforcement learning (RL). If feedback based on context variables confirms that the driver adheres to safe driving behavior, the driver can be rewarded with a longer off-road glance time. If feedback based on context variables indicates that the driver misuses the system by engaging in risky driving behavior, the off-road glance time can be reduced (negative or zero rewards are used in RL).

[0077] The allowable glance time may also be used to signal ADAS (Advanced Driver Assistance Systems) warnings. One example is an LDA (Lane Departure Alert) visual warning. For example, if the driver is currently allowed a longer allowable glance time for non-driving-related tasks, the monitoring system 100 may cause the LDA visual warning to remain on the screen for a longer period of time to allow the driver additional time to view the LDA visual warning.

[0078] In various aspects, the system's algorithm determines an acceptable glance time for a given driver, which may depend on the driver's visual scan pattern relative to the road environment, such that a "good" visual scan pattern is given a longer acceptable glance time, and a "poor" visual scan pattern is given a shorter acceptable glance time. A good visual scan pattern may be determined by a wider visual and horizontal gaze variance relative to the traffic environment, a higher frequency of looking at the rearview and side mirrors, etc. A poor visual scan pattern may be determined by a limited visual and horizontal gaze variance relative to the traffic environment (e.g., a visual angle deviation greater than 30 degrees), a lower frequency of looking at the rearview and side mirrors, etc.

[0079] The improvement of favorable glances can be confirmed / correlated with the observation of vehicle behavior so that the consistency of safe behavior can be rewarded by giving the driver even more time to look away from the road. Whether through more complex glance algorithms implemented in the DMS (dealer management system) or by safe driving confirmation (feedback at the end of the drive), the driver can be rewarded (or deterred). While confirmation of safe driving behavior may be most relevant to promoting safe driving, measuring the quality of glances or visual scanning is not only a difficult problem but also tends not to produce the desired results (see the visual scanning patterns of a driver who is not consciously driving, as if on autopilot).

[0080] The preferred embodiments of the present invention have been disclosed in an illustrative manner. Accordingly, the terms employed throughout should be read as non-limiting. Those skilled in the art will be able to conceive of minor modifications to the teachings herein, but it should be understood that what is intended to be limited within the scope of the patent granted herein are all such embodiments that reasonably fall within the scope of the advancement to the art to which this specification contributes, and that the scope should not be limited except in light of the appended claims and their equivalents.

Claims

1. A system for monitoring a vehicle driver, comprising: a camera operatively disposed in a vehicle and configured to monitor the eye position of the vehicle driver while the vehicle is being operated; a display operatively disposed in the vehicle; an electronic control unit connected to the camera and the display; The electronic control unit comprises: operating the display to display a request to the vehicle operator prompting the vehicle operator to provide driver input; adjusting an allowable glance time from a default glance time to an adjusted glance time in response to the vehicle operator providing the driver input; determining that the eye position of the vehicle driver is such that the vehicle driver's gaze direction is not directed toward a direction of travel of the vehicle while the vehicle is being operated; determining a period of time during which the vehicle driver's gaze direction is not directed in the direction of travel of the vehicle exceeds the adjusted glance time; In response to determining that the period of time during which the vehicle driver's gaze direction is not directed in the direction of travel of the vehicle exceeds the adjusted glance time, the system is configured to initiate countermeasures to encourage a response from the vehicle driver.

2. The system of claim 1 , wherein the driver input includes an acceptance of an agreement by the vehicle driver.

3. The system of claim 1 , wherein the driver input includes driver attitude information.

4. The electronic control unit further comprises: determining a crash or safety risk of the vehicle; The system of claim 1 , configured to adjust the allowable glance time based on the crash or safety risk of the vehicle.

5. The electronic control unit further comprises: determining a geographic location of the vehicle; The system of claim 1 , configured to adjust the allowable glance time based on the geographic location of the vehicle.

6. The system of claim 1 , wherein the electronic control unit is further configured to adjust the allowable glance time based on an experience level of the vehicle operator.

7. The system of claim 1 , wherein the electronic control unit is further configured to adjust the allowable glance time based on temporal data.

8. The system of claim 1 , wherein the electronic control unit is further configured to adjust the allowable glance time based on real-time vehicle data received from vehicle sensors.

9. 1. A method for managing driver glance behavior, comprising: operating the display to display a request to the vehicle operator prompting the vehicle operator to provide driver input; adjusting an allowable glance time from a default glance time to an adjusted glance time in response to the vehicle operator providing the driver input; determining an eye position of the vehicle driver such that the vehicle driver's gaze direction is not directed toward a direction of travel of the vehicle while the vehicle is being operated; determining a period of time during which the vehicle driver's gaze direction is not directed in the direction of travel of the vehicle exceeds the adjusted glance time; In response to determining that the period of time during which the vehicle driver's gaze direction is not directed in the direction of travel of the vehicle exceeds the adjusted glance time, initiating a countermeasure to encourage a response from the vehicle driver; A method comprising:

10. The driver input is The driver of the vehicle agrees to the agreement; or The method of claim 9 , further comprising at least one of driver attitude information.

11. determining a crash or safety risk of said vehicle; adjusting the allowable glance time based on the crash or safety risk of the vehicle; The method of claim 9 further comprising:

12. determining a geographic location of the vehicle; adjusting the allowable glance time based on the geographic location of the vehicle; The method of claim 9 further comprising:

13. The method of claim 9 , further comprising adjusting the allowable glance time based on an experience level of the vehicle operator.

14. The method of claim 9 , further comprising adjusting the allowable glance time based on temporal data.

15. The method of claim 9 , further comprising adjusting the allowable glance time based on real-time vehicle data received from vehicle sensors.

16. A non-transitory computer-readable medium having stored thereon content that causes one or more computing systems to perform an automated operation, the automated operation comprising at least: operating, by the one or more computing systems, the display to display a request to the vehicle operator prompting the vehicle operator to provide driver input; adjusting, by the one or more computing systems, an allowable glance time from a default glance time to an adjusted glance time in response to the vehicle operator providing the driver input; determining, by the one or more computing systems, that the vehicle driver's eyes are positioned such that the vehicle driver's gaze direction is not directed toward a direction of travel of the vehicle while the vehicle is being operated; determining, by the one or more computing systems, that a period of time during which the vehicle driver's gaze direction is not directed in the direction of travel of the vehicle exceeds the adjusted glance time; In response to determining that the period of time during which the vehicle driver's gaze direction is not directed in the direction of travel of the vehicle exceeds the adjusted glance time, initiating, by the one or more computing systems, countermeasures to prompt a response from the vehicle driver; 1. A non-transitory computer-readable medium comprising:

17. The automatic operation is determining a crash or safety risk of the vehicle by the one or more computing systems; adjusting, by the one or more computing systems, the acceptable glance time based on the crash or safety risk of the vehicle; 20. The non-transitory computer-readable medium of claim 16, further comprising:

18. 20. The non-transitory computer-readable medium of claim 17, wherein the adjustment by the one or more computing systems regarding the allowable glance time based on the crash or safety risk of the vehicle is made in real time while the vehicle is being operated by the vehicle driver to reduce the allowable glance time.

19. The automatic operation is determining a geographic location of the vehicle by the one or more computing systems; adjusting, by the one or more computing systems, the allowable glance time based on the geographic location of the vehicle; 20. The non-transitory computer-readable medium of claim 16, further comprising:

20. 17. The non-transitory computer-readable medium of claim 16, wherein the automatic operation further comprises adjusting, by the one or more computing systems, the allowable glance time based on at least one of an experience level of the vehicle driver, temporal data, or vehicle data received from a vehicle sensor.

Images