Detecting driver attention using heatmaps

Abstract

The present application relates to a technology for monitoring driver attention in a vehicle. A driver distraction system collects vehicle data and scene information from the vehicle traveling on a route. The vehicle data and scene information are then processed to generate a reference heat map. Simultaneously, the driver distraction system can capture the driver's gaze to track the driver's gaze direction and duration while driving the vehicle on the route. The gaze direction and duration are processed to generate a driver gaze heat map. By analyzing the driver gaze heat map and the reference heat map, the driver distraction level of the driver in the vehicle is determined, and based on the driver distraction level, a suggestion or reminder warning is output to the driver of the vehicle.

Description

Technical Field

[0001] The present application relates generally to driver distraction, and more particularly to monitoring a driver's attention while operating a vehicle. Background Art

[0002] Driver distraction is an increasingly common cause of traffic accidents, particularly with the increasing prevalence of devices like mobile devices, which divert drivers' attention from the road. Monitoring and preventing driver distraction is crucial not only for ensuring a safe driving environment for distracted drivers, but also for other nearby drivers who may be affected by the distracted driver. Vehicles with driver monitoring capabilities enable them to take measures to prevent or assist in preventing accidents caused by driver distraction. For example, a warning system can be activated to alert the driver of distraction, or automated features such as braking and steering can be activated to control the vehicle until the driver is no longer distracted. These warning and preventative monitoring systems can detect driver distraction by assessing the driver's head posture and gaze angle. However, these systems typically analyze data frame by frame and often do not consider other information such as road conditions and driver intent. Summary of the Invention

[0003] According to one aspect of the present application, a computer-implemented method for monitoring driver attention in a vehicle is provided, comprising: collecting vehicle data and scene information from the vehicle traveling on a route, wherein the vehicle data and the scene information are processed to generate a reference heat map; capturing the driver's gaze to track the driver's gaze direction and duration while driving the vehicle on the route, wherein the gaze direction and duration are processed to generate a driver gaze heat map; analyzing the driver gaze heat map and the reference heat map to determine a driver distraction level of the driver in the vehicle; and outputting recommendations to the driver of the vehicle based on the driver distraction level.

[0004] Optionally, in any of the above aspects, it also includes: using a convolutional neural network (CNN) including one or more convolutional layers and at least one fully connected layer to generate the reference heat map from the scene information and the vehicle data, wherein the reference heat map identifies one or more areas in the scene information where the driver needs to pay attention to safe driving; generating the driver gaze heat map based on the gaze direction and duration of the driver when driving the vehicle, wherein the driver gaze heat map identifies one or more areas in the scene information that the driver views during the duration.

[0005] Optionally, in any of the above aspects, tracking the gaze direction and duration includes: the one or more sensors capturing an image sequence of the driver within the duration; estimating the driver's gaze direction based on the gaze extracted from the image sequence; and superimposing the gaze direction onto the scene information to form the driver's gaze heat map.

[0006] Optionally, in any of the above aspects, collecting the vehicle data includes: recording the position of the vehicle during driving on the route; retrieving historical vehicle data representing the vehicle data within a predetermined time period from a database; collecting the scene information includes: acquiring street images in real time from the one or more sensors or online maps.

[0007] Optionally, in any of the above aspects, the vehicle data includes at least one of vehicle speed data, turn indication data, lighting condition data, brake usage data, steering wheel usage data, global positioning system (GPS) data, accelerator pedal usage data and navigation route data; the scene information includes at least one of scene images, online street maps, radar maps, weather, traffic and road conditions.

[0008] Optionally, in any of the above aspects, the reference heat map is further combined with the driver gaze heat map to form a driver gaze heat map, wherein the driver attention heat map is used to determine whether the driver distraction level meets or exceeds a threshold attention level.

[0009] Optionally, in any of the above aspects, when the threshold is met or exceeded, the recommendation is a warning to focus the driver's gaze direction on one or more areas of the driver's gaze heat map based on a value associated with each of the one or more areas, wherein the value represents the level of attention required by the driver for each of the one or more areas.

[0010] Optionally, in any of the above aspects, the one or more processors are located in the vehicle or communicatively coupled to the vehicle.

[0011] According to another aspect of the present application, a system for monitoring driver attention in a vehicle is provided, comprising one or more vehicle sensors for: collecting vehicle data and scene information from the vehicle traveling on a route, wherein the vehicle data and the scene information are processed to generate a reference heat map; capturing the driver's gaze to track the driver's gaze direction and duration when driving the vehicle on the route, wherein the gaze direction and duration are processed to generate a driver gaze heat map; one or more processors for: analyzing the driver gaze heat map and the reference heat map to determine a driver distraction level of the driver in the vehicle; and outputting recommendations to the driver of the vehicle based on the driver distraction level.

[0012] According to another aspect of the present application, a non-transitory computer-readable medium is provided, storing computer instructions for monitoring driver attention in a vehicle, wherein the computer instructions, when executed by one or more processors, cause the one or more processors to perform the following steps: collecting vehicle data and scene information from the vehicle traveling on a route; processing the vehicle data and scene information to generate a reference heat map; capturing the driver's gaze to track the driver's gaze direction and duration when driving the vehicle on the route, wherein the gaze direction and duration are processed to generate a driver gaze heat map; analyzing the driver gaze heat map and the reference heat map to determine a driver distraction level of the driver in the vehicle; and outputting recommendations to the driver of the vehicle based on the driver distraction level.

[0013] This summary is provided to introduce some concepts further described below in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all of the shortcomings identified in the background. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present aspects are illustrated by way of example and not limitation in the accompanying figures, in which like references indicate similar elements.

[0015] Figure 1A A driver distraction system according to an embodiment of the present technology is shown.

[0016] Figure 1B Shown according to Figure 1A Detailed example of a driver distraction system.

[0017] Figure 2 An exemplary flowchart of obtaining driver distraction information according to an embodiment of the present application is shown.

[0018] Figure 3A Shown according to Figure 2 Example of creating a reference heatmap.

[0019] Figure 3B Shown according to Figure 3A An exemplary CNN of an embodiment.

[0020] Figure 4 An exemplary embodiment of generating a driver gaze heat map based on the gaze direction and duration of a vehicle driver is shown.

[0021] Figure 5 Shows the use of Figure 3A and Figure 4 Driver attention network based on the reference heatmap and driver gaze heatmap.

[0022] Figure 6 An exemplary in-cabin vehicle setup for providing a warning to the driver is shown.

[0023] 7A to 7C An exemplary flowchart for monitoring driver attention in a vehicle according to an embodiment of the present application is shown.

[0024] Figure 8 A computing system on which embodiments of the present application can be implemented is shown. DETAILED DESCRIPTION

[0025] The present application, which generally relates to driver attention detection, will now be described with reference to the accompanying drawings.

[0026] This technology involves monitoring a driver's attention while driving a vehicle to prevent or reduce driver distraction. Scene information and vehicle data are collected via one or more vehicle sensors, and a reference heat map is generated based on the collected information and data. The reference heat map indicates areas within the scene information where the driver should pay attention, thereby enhancing safe driving. In one embodiment, the vehicle data is used to determine the driver's intent. For example, the driver's intent can be determined by analyzing vehicle status such as the navigation program, speed, steering wheel angle, and accelerator / brake pedals. In another embodiment, the driver's gaze direction and duration are determined, and a gaze track is generated. Because the gaze track is correlated with the scene information, it can represent the actual driver's attention area, and the gaze track can be generated in the form of a driver gaze heat map. The reference heat map and the driver gaze heat map are simultaneously processed using deep learning or rule-based methods to determine the driver's attention in the vehicle. If the system determines that the driver is distracted (e.g., not paying attention to a high-risk area or an area requiring attention), the driver is alerted to the surrounding environment or road conditions through warnings or recommendations.

[0027] It should be understood that the present application embodiments can be implemented in many different ways, and the scope of the claims should not be interpreted as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make the present application thorough and complete, and to fully convey the present application embodiment concepts to those skilled in the art. In fact, the present application is intended to cover substitutes, modifications and equivalents of these embodiments included in the spirit and scope of the present application as defined by the appended claims. In addition, in the following detailed description of the present application embodiments, many specific details are set forth in order to provide a thorough understanding. However, it is clear to those skilled in the art that the present application embodiments can be implemented without these specific details.

[0028] Figure 1A A driver distraction system according to an embodiment of the present technology is shown. Driver distraction system 106 is shown installed or otherwise included in vehicle 101, which also includes a cabin in which driver 102 may be seated. Driver distraction system 106, or one or more portions thereof, may be implemented by an in-cabin computer system and / or a mobile computing device, such as, but not limited to, a smartphone, tablet, notebook computer, laptop computer, and / or the like.

[0029] According to some embodiments of the present technology, the driver distraction system 106 obtains (or collects) current data of the driver 102 of the vehicle 101 from one or more sensors. In other embodiments, the driver distraction system 106 also obtains (or collects) additional information about the driver 102 from one or more databases 140, because the additional information is related to features such as the driver's facial features, historical head posture and eye gaze information. The driver distraction system 106 analyzes the current data and / or additional information of the driver 102 of the vehicle 101, thereby identifying the driver's head posture and eye gaze. In one embodiment, the driver distraction system 106 also monitors and collects vehicle data and scene information, as described below. This type of analysis can be performed using one or more computer-implemented neural networks and / or some other computer-implemented models, as explained below.

[0030] like Figure 1A As shown, the driver distraction system 106 is communicatively coupled to a capture device 103 that can be used to acquire current data of the driver of the vehicle 101 as well as vehicle data and scene information. In one embodiment, the capture device 103 includes sensors and other devices for acquiring current data of the driver 102 of the vehicle 101. The captured data can be processed by a processor 104 that includes hardware and / or software for detecting and tracking driver movement, head posture, and gaze direction. Figure 1BAs described in detail, the capture device may further include one or more cameras, microphones, or other sensors to capture data. In another embodiment, the capture device 103 may capture the scene in front of the vehicle on its route (e.g., surrounding environment and / or scene information). The front sensor may include a radar sensor, a laser sensor, a lidar sensor, an optical imaging sensor, etc. It should be understood that the sensors may also cover the side, rear, and top (upward and downward) of the vehicle 101.

[0031] In one embodiment, the capture device 103 may be external to the driver distraction system 106 (e.g., Figure 1A ) or may be included as part of the driver distraction system 106. Figure 1B Additional details of the driver distraction system 106 are described according to some embodiments of the present technology.

[0032] Still refer to Figure 1A The driver distraction system 106 is also shown as being communicatively coupled to various types of vehicle-related sensors 105 included within the vehicle 101. Such sensors 105 may include, but are not limited to, a speedometer, a global positioning system (GPS) receiver, and a clock. The driver distraction system 106 is also shown as being communicatively coupled to one or more communication networks 130, which enable access to one or more databases 140 and / or other types of data storage. The databases 140 and / or other types of data storage may store vehicle data for the vehicle 101. Examples of such data include, but are not limited to, driving record data, driving performance data, driver's license type data, driver's facial features, driving head posture, driver's gaze, etc. Such data may be stored in a local database or other data storage located within the vehicle 101. However, the data may also be stored in one or more databases 140 or other data storage located remotely from the vehicle 101. Thus, such databases 140 or other data storage may be communicatively coupled to the driver distraction system via one or more communication networks 130.

[0033] The communication network 130 may include a data network, a wireless network, a telephone network, or any combination thereof. It is contemplated that the data network may be any local area network (LAN), metropolitan area network (MAN), wide area network (WAN), a public data network (e.g., the Internet), a short-range wireless network, or any other suitable packet-switched network. In addition, the wireless network can be a cellular network, etc., and can adopt various technologies, including enhanced data rates for global evolution (EDGE), general packet radio service (GPRS), global system for mobile communications (GSM), Internet protocol multimedia subsystem (IMS), universal mobile telecommunications system (UMTS), etc., and any other suitable wireless medium, such as world-wide interoperability for microwave access (WiMAX), long term evolution (LTE) network, code division multiple access (CDMA), wideband code division multiple access (WCDMA), wireless fidelity (Wi-Fi), wireless LAN (WLAN), Bluetooth, Internet protocol (IP) data broadcast, satellite, mobile ad-hoc network (MANET), etc., or any combination thereof. The communication network 130 can be connected to the communication device 102 ( Figure 1B ) etc. to enable communication between the driver distraction system 106 and the database 140 and / or other data storage.

[0034] Although Figure 1AWhile the embodiments of the present invention are described with reference to vehicle 101, it should be understood that the disclosed technology can be employed in a wide range of technical fields and is not limited to vehicles. For example, in addition to vehicles, the disclosed technology can also be used in virtual or augmented reality devices or simulators, where head pose and gaze estimation, vehicle data, and / or scene information may be required.

[0035] Now refer to Figure 1B Additional details of the driver distraction system 106 according to some embodiments of the present technology are described. The driver distraction system 106 includes a capture device 103, one or more processors 108, a vehicle system 104, a navigation system 107, a machine learning engine 109, an input / output (I / O) interface 114, a memory 116, visual / audio alerts 118, a communication device 120, and a database 140 (which may also be part of the driver distraction system).

[0036] Capture device 103 can be responsible for using one or more capture devices, such as sensor 103A, camera 103B or microphone 103C, in the driver's cab, according to the driver's motion and / or audio data monitoring and identification driver behavior (comprising distraction) captured. In one embodiment, capture device 103 is positioned to capture the motion of driver's head and face, and in other implementations, the motion of driver's trunk and / or driver's limbs and hands also can be captured. For example, detection and tracker 108A, head pose estimator 108B and gaze direction estimator 108C can monitor the driver's motion captured by capture device 103, to detect specific posture, such as head pose, or whether people are watching towards a specific direction.

[0037] Still other embodiments include capturing audio data via microphone 103C, either together with or independently of driver movement data. The captured audio may be, for example, an audio signal of driver 102 captured by microphone 103C. The audio may be analyzed to detect various characteristics that may vary depending on the driver's state. Examples of such audio characteristics include the driver's voice, passenger voices, music, and the like.

[0038] Although capture device 103 is shown as a single device with multiple components, it should be understood that each component (e.g., sensor, camera, microphone, etc.) can be a separate component located in a different area of ​​vehicle 101. For example, sensor 103A, camera 103B, microphone 103C, and depth sensor 103D can each be located in a different area of ​​the vehicle's cabin. In another example, the various components of capture device 103 can be part of another component or device. For example, camera 103B and visual / audio 118 can be part of a mobile phone or tablet (not shown) placed in the vehicle's cabin, while sensor 103A and microphone 103C can be separately located in different locations in the vehicle's cabin.

[0039] The detection and tracker 108A monitors the facial features of the driver 102 captured by the capture device 103, and then these features can be extracted after detecting the driver's face. The term facial features includes, but is not limited to, points around the eyes, nose, and mouth areas, as well as points outlining the contour portions of the detected face of the driver 102. Based on the monitored facial features, the initial positions of one or more eye features of the driver's 102 eyeball can be detected. The eye features can include the iris and the first and second corners of the eye. Thus, for example, detecting the position of each of the one or more eye features includes detecting the position of the iris, detecting the position of the first corner of the eye, and detecting the position of the second corner of the eye.

[0040] The head pose estimator 108B uses the monitored facial features to estimate the head pose of the driver 102. As used herein, the term "head pose" describes an angle involving the relative orientation of the driver's head relative to the plane of the capture device 103. In one embodiment, the head pose includes the yaw and pitch angles of the driver's head relative to the plane of the capture device. In another embodiment, the head pose includes the yaw, pitch, and roll angles of the driver's head relative to the plane of the capture device.

[0041] The gaze direction estimator 108C estimates the gaze direction (and gaze angle) of the driver. In operation of the gaze direction estimator 108C, the capture device 103 may capture an image or set of images (e.g., of the driver of a vehicle). The capture device 103 may send the images to the gaze direction estimator 108C, where the gaze direction estimator 108C detects facial features from the images and tracks (e.g., over time) the driver's gaze. One type of gaze direction estimator is Eye tracking system.

[0042] In another embodiment, the gaze direction estimator 108C can detect eyes from the captured image. For example, the gaze direction estimator 108C can rely on the center of the eye to determine the gaze direction. In short, it can be assumed that the driver is looking forward relative to the orientation of his or her head. In some embodiments, the gaze direction estimator 108C provides more accurate gaze tracking by detecting pupil or iris position or using a geometric model based on the estimated head pose and the detected iris and the position of each of the first and second eye corners. Pupil and / or iris tracking enables the gaze direction estimator 108C to detect gaze direction independent of head pose. Drivers typically visually scan their surroundings with little or no head movement (e.g., glancing left or right (or up or down) to better see items or objects outside their direct line of sight). These visual scans are often directed at objects on or near the road (e.g., viewing road signs, pedestrians near the road, etc.) and objects within the vehicle cabin (e.g., viewing console readouts such as the speedometer, operating a radio or other dashboard device, or viewing / operating a personal mobile device). In some cases, the driver may glance at some or all of these objects with minimal head movement (e.g., out of the corner of his or her eye). By tracking the pupil and / or iris, the gaze direction estimator 108C can detect glances up, down, and to the side that would not be detected in a system that simply tracks head position.

[0043] In one embodiment, the gaze direction estimator 108C can cause the processor 108 to determine a gaze direction (e.g., of the vehicle operator's gaze) based on detected facial features. In some embodiments, the gaze direction estimator 108C receives a series of images (and / or videos). The gaze direction estimator 108C can detect facial features in multiple images (e.g., a series of images or an image sequence). Thus, the gaze direction estimator 108C can track gaze directions over time and store such information in a database 140, etc.

[0044] In addition to performing the above-mentioned gesture and gaze detection, the processor 108 may also include an image corrector 108D, a video enhancer 108E, a video scene analyzer 108F and / or other data processing and analysis to determine the scene information captured by the capture device 103.

[0045] Image corrector 108D receives the captured data and can perform corrections, such as stabilizing the video. For example, bumps in the road may cause the data to be jittery, blurry, or distorted. The image corrector can stabilize the image to prevent horizontal and / or vertical jitter, and / or can correct for translation, rotation, and / or scaling.

[0046] The video enhancer 108E can perform additional enhancement or processing in situations where lighting is poor or data compression is high. Video processing and enhancement can include, but is not limited to, gamma correction, dehazing, and / or deblurring. Other video processing enhancement algorithms can be used to reduce noise in low-light video input, and then contrast enhancement techniques, such as, but not limited to, tone mapping, histogram stretching and equalization, and gamma correction, can be used to restore visual information in low-light video.

[0047] Video scene analyzer 108F can identify the video content from capture device 103. For example, the video content can include a scene or a series of scenes from the front camera 103B in the vehicle. The analysis of the video can involve a variety of technologies, including but not limited to low-level content analysis such as feature extraction, structural analysis, object detection and tracking, and high-level semantic analysis such as scene analysis, event detection and video mining. For example, by identifying the content of the incoming video signal, it can be determined whether the vehicle 101 is traveling along a highway or within a city, whether there are any pedestrians, animals or other objects / obstacles on the road, etc. By performing image processing (for example, image correction, video enhancement, etc.) while or before performing image analysis (for example, video scene analysis, etc.), the image data can be prepared in a specific manner for the type of analysis being performed. For example, image correction for reducing blur can more accurately perform video scene analysis by clearing the appearance of edge lines used for object identification.

[0048] The vehicle system 104 may provide signals corresponding to any state of the vehicle, the vehicle's surroundings, or outputs from any other information source connected to the vehicle. Vehicle data outputs may include, for example, analog signals (e.g., flow rate), digital signals provided by a single information source (e.g., a clock, a thermometer, a position sensor (e.g., a global positioning system (GPS) sensor), etc.), and digital signals transmitted via a vehicle data network (e.g., an engine controller area network (CAN) bus that can transmit engine-related information, a climate control CAN bus that can transmit climate control-related information, and a multimedia data network that can transmit multimedia data between multimedia components in the vehicle). For example, the vehicle system 104 may retrieve from the engine CAN bus the current speed of the vehicle estimated by wheel sensors, the power state of the vehicle via the vehicle's battery and / or power distribution system, the ignition state of the vehicle, etc.

[0049] The navigation system 107 of the vehicle 101 can generate and / or receive navigation information, such as location information (e.g., via a GPS sensor and / or other sensors 105), route guidance, traffic information, point-of-interest (POI) identification, and / or provide other navigation services to the driver. In one embodiment, the navigation system or a portion of the navigation system is remote from the vehicle 101 and communicatively coupled to the vehicle 101.

[0050] The input / output interface 114 supports providing information to a user and / or other components or devices using various input / output devices. Examples of input devices include a keyboard, a microphone, a touch-enabled entity (e.g., a capacitor or other sensor for detecting physical touch), a camera (e.g., which can use visible or non-visible wavelengths, such as infrared frequencies, to identify motion as gestures that do not involve touch), etc. Examples of output devices include visual / audio alerts 118, such as displays, speakers, etc. In one embodiment, the I / O interface 114 receives driver motion data and / or audio data of the driver 102 from the capture device 103. The driver motion data can be related to the eyes and face of the driver 102, etc., and can be analyzed by the processor 108.

[0051] The data collected by driver distraction system 106 can be stored in database 140, memory 116, or any combination thereof. In one embodiment, the collected data comes from one or more sources external to vehicle 101. The stored information can be data related to driver distraction and safety, such as information captured by capture device 103. In one embodiment, the data stored in database 140 can be a collection of data collected for one or more drivers of vehicle 101. In one embodiment, the collected data is head pose data of the driver of vehicle 101. In another embodiment, the collected data is gaze direction data of the driver of vehicle 101. The collected data can also be used to generate datasets and information that can be used to train models for machine learning, such as machine learning engine 109.

[0052] In one embodiment, the memory 116 may store instructions executable by the processor 108, the machine learning engine 109, and programs or applications (not shown) that may be loaded and executed by the processor 108. In one embodiment, the machine learning engine 109 includes executable code stored in the memory 116 that may be executed by the processor 108 and selects one or more machine learning models stored in the memory 116 (or database 140). The machine models may be developed using well-known and conventional machine learning and deep learning (e.g., convolutional neural network (CNN) implementations) techniques, as described in detail below.

[0053] Figure 2 An exemplary flowchart for obtaining driver distraction information according to an embodiment of the present application is shown. In an embodiment, the flowchart may be a computer-implemented method executed at least in part by hardware and / or software components shown in the various figures and as described herein. In one embodiment, the disclosed process may be performed by Figure 1A and Figure 1B In one embodiment, a software component executed by one or more processors (eg, processor 108 or processor 802) performs at least a portion of the process.

[0054] The process of obtaining driver distraction information (results) is shown as being divided into two parts: part 201A describes capturing driver 102 information to generate a driver gaze heat map, and part 201B describes collecting scene information and vehicle data to generate a reference heat map. For ease of description, the process is divided into two parts, but during implementation, the two parts can be implemented in parallel or simultaneously.

[0055] In the process described in section 201A, at step 202, visual information of the driver (e.g., an image of the driver) is captured using one or more capture devices 103 to determine head pose and gaze estimation. The capture device 103 may include, but is not limited to, sensors facing the driver (e.g., infrared sensors, ultrasonic sensors, image sensors, depth sensors, etc.), cameras, microphones, etc. The driver's visual information is extracted from the capture device 103 and sent to the processor 108 for detection and tracking by the detector and tracker 108A, estimation of head pose by the head pose estimator 108B, and / or estimation of gaze direction by the gaze direction estimator 108C, wherein the gaze direction (angle) and duration are estimated.

[0056] In one embodiment, the driver's gaze direction and duration are monitored in real time and can be interpreted as a three-dimensional (3D) vector or horizontal and vertical angles. However, any number of methods (such as geometric methods or deep learning methods) can be used to estimate the gaze direction. Geometric methods use the geometric relationship between near-infrared light and the reflection of near-infrared light on the driver's iris and eye center to obtain the gaze direction. Examples include using near-infrared (NIR) illumination to create corneal reflections and tracking them relative to the pupil center to estimate two-dimensional (2D) and 3D models of the gaze vector. These methods require polynomial or geometric approximations of the eye to obtain the gaze direction or gaze point. Another technique includes an appearance-based method that extracts content information from the eye area image, such as local features, shape, and texture of the eye area, to estimate the gaze direction. In addition, deep learning methods use neural networks, such as convolutional neural networks (CNNs), to extract information from appearance information. CNNs are able to learn multi-level nonlinear transformations of input data (e.g., face images) using labeled examples through gradient descent-based optimization. CNNs consist of fully parameterized (trainable) filters that convolve the input to produce feature maps, nonlinear layers, pooling / downsampling layers that downsample the feature maps, and fully connected layers.

[0057] Using any of the techniques described above (or techniques known in the art) to estimate the driver's gaze direction and duration, a heat map may be generated at step 204. Figure 3B Explaining heatmap generation

[0058] The process shown in section 201B uses one or more capture devices 103 to capture scene information and vehicle data. In one embodiment, at step 206A, scene information is collected. Scene information may include information obtained from the vehicle's surroundings, such as road conditions (e.g., paved, dirt, potholes, hills, snow, water, etc.), streets (e.g., street names, intersections, crossroads, etc.), maps and landscapes of the surrounding environment (e.g., determined by images or online maps, radar maps, etc.), weather (e.g., hot, cold, snow, rain, dry, etc.), time of day (e.g., morning or evening, etc.), lighting conditions (e.g., sunlight, darkness, etc.), information obtained by tracking or navigation systems (e.g., GPS, etc.), etc. The capture devices 103 used to capture scene information may include, but are not limited to, lidar sensors, cameras, infrared sensors, radar sensors, laser rangefinders, video sensors, etc.

[0059] In one embodiment, in step 206B, vehicle data is collected. The vehicle data includes, but is not limited to, speed, position, direction, tire pressure, amount of turning data, trip duration data, previous vehicle data (or historical data), headlight usage, lane change data, steering data, and turn signal information. The capture device 103 used to capture vehicle data may include, but is not limited to, a compass, a navigation system, a position and direction sensor, an image sensor, a geographic information sensor, a steering wheel sensor, a brake sensor, a pedal sensor, a speedometer, a date and time sensor, a telemetry sensor, and the like. In one embodiment, in step 206B, the collected vehicle data 206B may indicate the driver's intention or action. For example, if the steering wheel sensor provides an output indicating that the steering wheel is turning rapidly, and the brake sensor indicates that the brakes are being pressed firmly, this combination may indicate that the driver is attempting to avoid hitting an object by braking and turning around the object.

[0060] The driver's "intention" or "action" can also be based on the driver's historical data (for example, information of a single driver or multiple different drivers), real-time vehicle data and / or a combination of real-time vehicle data and historical data collected over a period of time, or learn from these data. The collected historical data can identify the pattern or action sequence that indicates or reflects the driver's intention when performing various actions or during some activities. The historical data can also include information about the driver's years of experience, type of experience, traffic accident history, entries in employment records, speeding and / or other traffic violation history, the number, severity and length of various driver behaviors identified before. In one embodiment, the driver distraction system 106 can collect the driving profile of a specific driver and store it in the database 140 or memory 116 as part of the driver's history. For example, the driver's profile can also include the speed at which the driver accelerates or decelerates, the usual speed under specific driving conditions such as night driving, driving in rain or snow, the frequency of the driver turning or crossing the center line, etc. In one embodiment, the historical data can be retrieved from the database 140 or memory 116, for example, from the database of the driver's insurance company. The historical data may also include information about one or more driving routes, such as road profiles, road topology, the number of stops associated with the driving route, the number of turns associated with the driving route, and the like.

[0061] In step 206A, the capture device 103 collects scene information, and in step 206B, the capture device 103 collects vehicle data. After collection, the scene information and vehicle data are extracted and can be sent to at least one of the vehicle system 104, the navigation system 107, and the processor 108. In one embodiment, the extracted image can be corrected by the image corrector 108D, the video enhanced by the video enhancer 108E, and the video scene analyzed by the video scene analyzer 108F for additional processing. After collecting the scene information and vehicle data, a reference heat map can be generated (step 208). The reference heat map will be discussed in detail below and includes areas associated with safe driving that require lower driver attention, and areas associated with distracted driving that require higher driver attention. In one embodiment, weights or scores are associated with the areas.

[0062] For example, during inclement weather, rush hour, or nighttime, various areas of the surrounding environment may require higher levels of driver attention than areas of the surrounding environment during good weather, off-peak hours, and daytime. These areas can be considered safe driving areas where lower levels of driver distraction may occur, and distracted driving areas where higher levels of driver distraction may occur. In another example, a driver may require higher levels of attention when driving along a winding road or highway than when driving along a straight road or a cul-de-sac. In this case, a driver driving along a winding road or highway may have areas with higher levels of driver distraction, while a driver driving along a straight road or a cul-de-sac may have areas with lower levels of driver distraction. These driver attention areas can be identified in the reference heat map, as described below.

[0063] In step 210, at least one processor, such as processor 108, processes the driver gaze heat map generated in step 204 and the reference heat map generated in step 208. In one embodiment, the driver gaze heat map and the reference heat map can be merged (or fused) and processed to identify areas where the driver's attention is required (or not required). For example, the driver gaze heat map and the reference heat map can be input into a deep learning network (e.g., an encoder-decoder network) for processing to generate a reference heat map. Figure 3A and Figure 3B The deep learning network and reference heatmaps are described.

[0064] Processing the output of the driver gaze heat map and the reference heat map in step 210 produces a result in step 212 representing the driver distraction level. Based on the result, the driver's attention may be shifted to different areas of the output. In one embodiment, the result can be output as a visual and / or audio suggestion or warning. For example, a visual warning can be turned on, such as a light emitting diode (LED) under the windshield, or a road scene can be displayed on the screen, suggesting that the driver pay attention to the highlighted area. Another example includes a voice suggestion or warning. Such suggestions or warnings can include beeps or verbal announcements issued through a speaker to indicate where the driver should pay attention. The driver distraction results are described in detail below.

[0065] Figure 3A Shown according to Figure 2 Example of creating a reference heat map. In one embodiment, the reference heat map 208 is generated using a combination of collected scene information 206A and vehicle data 206B processed by a deep learning network (e.g., CNN 306). In one embodiment, CNN 306 is an encoder-decoder type CNN. The encoder portion of CNN 306 (where input is received—in this case, the input is scene information and vehicle data) extracts high-level information (objects such as cars, crosswalks, traffic lights, etc.) from low-level image information (such as solid color pixels). In particular, the encoder-decoder type CNN 306 can include three types of networks: convolution, pooling, and batch normalization. The convolution layer extracts local features, the pooling layer downsamples the feature map and propagates special invariant features to deeper layers, and the batch normalization layer normalizes the distribution of the training data to accelerate learning. When encoding is complete, the feature map in the original scene is extracted and the output is adjusted to obtain a feature vector. In order to obtain the driver's intention from the vehicle data, the vehicle data (collected as a vector) is cascaded to the feature vector (cascaded with the fully connected layer). After that, the feature vectors are resized again to obtain feature maps. After encoding, the decoder part (where the output is generated - in this case, the reference heatmap) takes the high-level information and maps it into another mid-level information to transform the information into an image of the same size so that objects with the same label are also assigned the same color (for heatmaps). Specifically, the decoder upsamples the image to the same size as the image input to the encoder part and performs another convolution. This convolution adds geometric details to the feature map to make up for any loss in the pooling layer during encoding, resulting in a smoother, less jagged output. Figure 3B CNN 306 is further described.

[0066] The reference heat map 208 can be generated in real time and use different colors, indicators (e.g., numbers, pointers, etc.) or other identification techniques to identify areas. The areas identified on the reference heat map 208 can correspond to areas that require different levels or degrees of driver attention. For example, the heat map 208 can include areas with weighted driver attention levels, wherein areas with positive values ​​indicate that more driver attention is required, while areas with negative values ​​require less driver attention. (It should be understood that in the illustrated embodiment, grayscale is used instead of color). In one example, the color scale 308A can use red to indicate areas that require a higher level of driver attention, and can use green to indicate areas that require a lower level of driver attention. Because specific areas require different levels of attention, the colors and / or indicators on the reference heat map 208 can change to reflect the necessary level or degree of attention of the driver. For example, an area including a traffic light can be turned red or have an indicator (e.g., an arrow) that diverts the driver's attention to observe the traffic light within a predetermined time. In another example, when a pedestrian crosses the street, the driver's attention will be diverted to follow the pedestrian. Areas where pedestrians are walking can be highlighted using a color (e.g., red) or can be highlighted using an indicator. Similarly, areas where a lower level of driver attention is required, such as signs or billboards, can be highlighted using green. In some cases, the driver's attention level or degree can be based on a user profile or the route the vehicle is traveling. In other cases, the driver's attention level or degree can be determined by known or recognized areas that typically require driver attention (e.g., crosswalks, winding roads, etc.).

[0067] Sensor measurements used to collect scene information 206A include images or video (e.g., a set of images) of the environment surrounding vehicle 101. The images or video may be captured by a camera, for example, facing outward from the front of vehicle 101 (e.g., through the front windshield, mounted on the exterior of the vehicle, etc.). The camera's field of view includes all or a portion of the road in front of vehicle 101 (e.g., the foreground in front of the vehicle, the lane the vehicle is traveling in, adjacent lanes, etc.), and may optionally include areas to the sides of vehicle 101 (e.g., the driver's side, the passenger's side), the area behind vehicle 101, the area above vehicle 101 (e.g., through the sunroof), or any other suitable portion of the environment outside vehicle 101. In one embodiment, the images may be used to optically identify and / or track the vehicle's position within the environment (e.g., to locate the vehicle), the vehicle's position within a lane, the vehicle's position relative to other objects (e.g., vehicles, pedestrians, etc.), etc. The images may also be used to determine the driving environment, such as whether there are objects in front of or beside vehicle 101, whether the road is hilly or has curves, etc. The images may also be used to determine, verify and / or confirm other factors, such as drifting within a lane on the road, although this information may also be available through vehicle data.

[0068] Images and videos can be captured by any number of different sensors in any number of different formats. For example, images can be visual spectrum images, hyperspectral images, IR images, or samples taken at any other suitable wavelength. Images can be recorded using ambient light, light emitted by a light emitter (e.g., light emitted by an LED flash), or any other suitable light source. Images can be stereo images (e.g., recorded by a stereo camera), single images, or any other suitable image. As described above, images can also be combined into a sequence of images to form a video. In the illustrated example, the image (represented by collected scene information 206A) is a street scene of a road and buildings captured by a forward-facing camera connected to vehicle 101. In the captured image, the street scene also includes people on the sidewalk (on both sides of the road), traffic lights, signs, and crosswalks. The image can represent a street scene corresponding to a map, such as online map 302 generated by navigation system 107. In one embodiment, navigation system 107 overlays navigation information onto the image. For example, the image can be a view of a location on online map 302 (represented as a black dot on the map), and turn signals (represented by curved arrows) can be overlaid onto the image.

[0069] In one embodiment, sensor measurements include vehicle data 206B collected by sensors on vehicle 101. Vehicle data 206B may be received from vehicle 101 via a vehicle data bus, wirelessly transmitted from vehicle 101, or otherwise received from vehicle 101. Vehicle data 206B may include, but is not limited to, control panel signals (e.g., whether the radio is changing, whether the user interface is being used, etc.), steering wheel signals (e.g., number of touches on the steering wheel, high pressure areas, high temperature areas), seat sensor signals (e.g., driver weight distribution or changes over time, etc.), or any other suitable set of measurements. Sensor measurements may also include any other vehicle information obtained from a vehicle computer or communication / network bus and / or one or more sensors located on vehicle 101 (e.g., brake sensors, pedal sensors, positioning sensors, odometer sensors, speed sensors, tire pressure sensors, lane departure sensors, following distance sensors, roll stability sensors), and the sensors described above.

[0070] Figure 3B Shown according to Figure 3A In the example shown, CNN 306A includes CNN layers 3081, 3082, ... 308 N , feature maps 3101, 3102, ... 310 Nand a 1×1 CNN layer 312 coupled to or including a pixel downsampler 312 and a heatmap generator 316. A CNN 306 that has been previously trained receives one or more images 305 to detect various spatial and / or object location regions within each image 305. The images 305 are received by the CNN 306A. For example, the images may include scenes, e.g. Figure 3A The scene information 206A in the image has roads, buildings and other identifiable objects (e.g., other cars, people, animals, etc.). CNN 306A passes through convolutional layers 3081, 3082...308 N Perform convolution calculations to generate feature maps 3101, 3102, ... 310 N Feature graphs 3101, 3102, ... 310 N The vector is extracted and output to obtain a feature vector. The vector is cascaded to the feature vectors including vehicle speed, turning state, braking state, navigation instructions, etc., and the feature vector is adjusted to obtain feature maps 3101, 3102...310 N Subsequently, the 1×1 convolutional layer 312 transforms the feature maps 3101, 3102, ... 310 N Narrowed down to a single decision point based on each pixel, referred to herein as a per-pixel decision point. A classifier (not shown) uses the per-pixel decision point to determine whether a particular pixel belongs to a particular object class, such as the "crosswalk" object class.

[0071] In one embodiment, a pixel downsampler 314 is used to reduce the number of per-pixel decision points to be evaluated. The pixel downsampler 314 downsamples the per-pixel decision points, for example, by 4×. Then, for each pixel that is downsampled (e.g., 4×), a heat map generator 316 determines whether the downsampled per-pixel decision point belongs to a particular target class. Next, the heat map generator 316 generates a heat map 318 that identifies, on a per-pixel basis, regions in the image 305 that include objects corresponding to the target class. In one embodiment, the heat map 318 is evaluated to identify regions corresponding to different objects (e.g., pedestrians, roads, crosswalks, traffic lights, vehicles, etc.). Thus, different image target classes for different object types can be identified in a region of the image 305. For example, a region on the reference heat map 208 may have per-pixel points determined to correspond to a first target class corresponding to a pedestrian and a second target class corresponding to a crosswalk on a road.

[0072] Figure 4An exemplary embodiment of generating a driver gaze heat map based on the gaze direction and duration of a vehicle driver is shown. As shown, an environment 402 (e.g., a street scene represented by scene information) is captured by a capture device 103 coupled to a vehicle 101. The driver's gaze direction and duration can be monitored while the driver 102 is driving the vehicle 101 along a route. For example, a detector and tracker 108A, a head pose estimator 108B, and / or a gaze direction estimator 108C can be used to process the captured gaze direction and duration. As described above, any number of gaze estimation techniques can be used to track the driver's gaze direction and duration.

[0073] In one embodiment, the processed gaze direction and duration can generate a gaze track (represented by a solid black line) that represents the area of ​​the environment 402 observed by the driver 102 of the vehicle 101. For example, the driver's gaze direction corresponding to one or more broad areas (e.g., up, down, etc.) and the time spent gazing at the broad areas can be tracked and used to generate the gaze track. More specifically, the gaze direction and duration can be expressed according to the following formula: G t =[(x, y), t], where (x, y) represents the two-dimensional (2D) position of the gaze in the environment 402, and t represents the timestamp during which the gaze direction is collected. Using this information, a gaze trajectory can be drawn relative to the environment 402 during a period of time (duration) from time t1 to time t2. It should be understood that when the vehicle 101 travels along the route, multiple gaze directions and duration information of the driver 102 can be collected in different time periods and different environments. Optionally, in one embodiment, the gaze trajectory can be filtered and denoised from the environment 402 using a Kalman filter, a moving average filter, etc. to generate a filtered gaze trajectory 404. The filtered gaze trajectory can then be applied to the environment 402 to generate a driver gaze heat map 406. In one embodiment, for each gaze direction and duration in the gaze trajectory, the gaze direction and duration are accumulated using a Gaussian (or normal) distribution and recorded on the gaze heat map. Therefore, the driver 102 can designate areas with more gaze tracks as locations that need to be viewed more.

[0074] Driver's gaze heat map 406 provides the statistical information about the driver's gaze direction in a period of time (duration).For example, the driver's gaze heat map 406 can indicate the zone (it can be associated with particular class or object) that the driver in the duration that the driver watches when driving is gazed at.In one embodiment, driver's gaze heat map 406 is the visual representation of environment (for example, street scene), and wherein the region of highlighting (shown with gray scale) shows the position that receives driver 102 most attention.These highlighted regions can take the form of traditional heat map, wherein, for example, colors such as red, orange illustrate the zone of higher driver's attention level, and colors such as blue, green illustrate the zone of lower driver's attention level.But, can use any form of visual representation, for example shape, sign, image etc.In other embodiments, the region of highlighting can be weighted with the value of indication driver's attention level.Should be understood that different from reference heat map 208, with reference to the position that the regional representation in the heat map 208 needs driver's specific attention level, the driver's gaze region in the heat map 406 represents driver's attention level based on gaze direction and duration.

[0075] Figure 5 Shows the use of Figure 3A and Figure 4 The driver attention network 500 takes the reference heat map 208 and the driver gaze heat map 406 as input and uses a deep learning network (e.g., Siamese neural network) to ) combines the inputs to generate a driver attention result (AttentionResult) 508. The Siamese neural network determines the similarity between two comparable inputs—in this case, the similarity between the reference heatmap 208 and the driver gaze heatmap 406.

[0076] Siamese neural network has two comparable subnets 502A and 502B, wherein, subnets have identical parameters and weights. Reference heat map 208 is input in the first subnet 502A, and the first subnet 502A comprises a sequence of convolutional layers, pooling layers, fully connected layers and eigenvectors R. Eigenvectors R represent the encoding of input (i.e., reference heat map). In one embodiment, reference heat map 208 is used as ground truth. Similarly, driver's gaze heat map 406 is input in the second subnet 502B, and the second subnet 502B comprises a sequence of convolutional layers, pooling layers, fully connected layers and eigenvectors G. Eigenvectors G represent the encoding of input (i.e., driver's gaze heat map). Then, two eigenvectors R and G can be evaluated using loss functions etc. to determine the distance between the two encodings. As described above, the results are merged or fused to form driver's attention heat map 506, and Siamese network 500 outputs driver's attention results 508. An example of a Siamese neural network is described in “Siamese Neural Networks for One-Shot Image Labeling” by Koch et al., Department of Computer Science, University of Toronto.

[0077] The driver attention result 508 generally identifies whether the driver is driving attentively or distracted. Based on the driver attention result 508, the driver distraction system 106 may continue to further monitor the driver 102 and / or provide warnings or suggestions. In one embodiment, if the attention result 508 indicates that the driver is driving attentively, the driver distraction system 106 may continue to monitor the driver 102 of the vehicle (step 512) until the driving is completed. In another embodiment, if the attention result 508 indicates that the driver 102 is driving distracted, a warning or suggestion may be issued to the driver 102 (step 514), such as Figure 6 As shown in the example of . Warnings or advisories can use visual warnings and / or audio warning signals to attract the driver's attention. When visual warnings such as flashing lights or written messages on display 602 are turned on, the driver should pay attention to these warnings. Other visual warnings can also be displayed in various other forms such as arrows, pointers or any other indicators. Visual warnings can be displayed on any number of different media, such as on a head-up display or on the display of a mobile device. Audio warnings can also be played alone or in conjunction with visual warnings to attract the driver's attention. For example, the speakers used for playback in the vehicle can play a loud sound or announce an advisory similar to a hotel fire alarm.

[0078] In one embodiment, the driver distraction level can be measured according to a score or rating scale. For example, the driver distraction level can be determined based on a scale of 1 to 100. A score of 1 can indicate that the driver 102 is not distracted or is driving attentively, while a score of 100 can indicate that the driver 102 is completely distracted or is not paying attention to areas in the environment that have been determined to require high levels of driver attention. Scores between 1 and 100 can be quantified as low, moderate, or severe driver attention or distraction indications.

[0079] In another embodiment, the score can be determined based on the amount of overlap between the driver's gaze and an object in the environment (as viewed by the driver in the vehicle) or based on the overlap between the driver's expected gaze and the expected location of the object in the environment. In one example, the driver distraction system 106 can identify objects in the vehicle's surroundings, determine the direction and duration of the driver's gaze, optionally determine the driver's field of view (FoV), and determine a distraction score based on the amount of the driver's gaze or the overlap of the field of view with the identified object. In another example, the score can be determined based on the rate of change of the driver's gaze on the object and the duration. For example, the score can be increased based on a rate below a threshold level, which indicates that the driver is not adequately scanning the environment while operating the vehicle, but is instead looking at a fixed point (e.g., may be distracted by thoughts).

[0080] In one embodiment, score (or rating scale) can be based on the profile and / or historical data of specific driver.For example, specific driver should know that when driving at high speed on a curved road, may encounter problems, also should know to stop carefully at a stop sign.Other embodiments can simply be graded according to scenario events or environment.For example, if the traffic light is red, and the driver distraction system 106 determines that the driver 102 is watching the rear cabin of the vehicle 101, then high score can be associated with the driver attention result 508, so that early warning is provided to the driver 102, to stop at the traffic light.In other embodiments, score can be updated according to the scene information, vehicle data and real-time events collected or the gaze direction and duration of previous activity.

[0081] 7A to 7C An exemplary flow chart for monitoring driver attention in a vehicle according to an embodiment of the present application is shown. In an embodiment, the flow chart may be a computer-implemented method executed at least in part by hardware and / or software components shown in the various figures and as described herein. In one embodiment, the disclosed process may be performed by Figure 1A and Figure 1B In one embodiment, a software component executed by one or more processors (eg, processor 108 or processor 802) performs at least a portion of the process.

[0082] refer to Figure 7A In step 702, the driver distraction system 106 collects scene information 206A and vehicle data 206B using one or more sensors to monitor the driver's attention and distraction. The scene information represents the environment surrounding the vehicle while traveling along the route. For example, the environment may include a scene captured by a camera coupled to the vehicle or the current road conditions. In one embodiment, in step 702A, the collected scene information 206A and vehicle data 206B are processed to generate a reference heat map 208. The reference heat map 208 can be generated from the scene information 206A and the vehicle data 206B using a CNN having one or more convolutional layers and at least one fully connected layer, so that the reference heat map 206A identifies one or more areas in the scene information 206A where the driver's attention is required for safe driving.

[0083] At step 704, the driver distraction system 106 also captures visual information 202 (e.g., driver posture and / or gaze direction and duration) of the driver 102 of the vehicle 101. In one embodiment, at step 704A, the gaze direction and duration are processed to generate a driver gaze heat map 406. The driver gaze heat map 406 can be generated based on the gaze direction and duration of the driver 102 while driving the vehicle 101, such that the driver gaze heat map 406 identifies one or more areas of the scene information that the driver is looking at during a duration.

[0084] At step 706 , the driver distraction system 106 analyzes the reference heat map 208 and the driver gaze heat map 406 to determine a level of driver distraction (or attention) of the driver 102 in the vehicle 101. In one embodiment, at step 706A, the reference heat map 208 and the driver gaze heat map 406 are merged or fused to form the driver attention heat map 506.

[0085] Then, at step 708 , the driver attention heat map 506 is analyzed to output warnings or recommendations to the driver 102 of the vehicle 101 based on the driver's distraction level.

[0086] Steering Figure 7B , describes a sub-process for collecting scene information 206A and vehicle data 206B. At step 702B, the location of vehicle 101 is recorded while traveling along the route. The location of vehicle 101 can be determined using any number of different techniques, such as using navigation system 107, which may include GPS. Then, at step 702C, historical vehicle data representing vehicle data over a predetermined time period can be retrieved from a database, and at step 702D, scene information can be collected. In one embodiment, collecting scene information includes collecting street imagery in real time from one or more sensors or online maps.

[0087] exist Figure 7C , a sub-process for capturing driver posture and gaze is described. In step 704B, tracking the gaze direction and duration of the driver 102 includes capturing a sequence of images of the driver 102 over a duration (e.g., a time period). In one embodiment, one or more sensors in the vehicle, such as a camera, are used to capture images of the driver. In step 704C, the gaze direction of the driver 102 is estimated based on the posture and gaze extracted from the image sequence, and in step 704D, the gaze direction is superimposed on the scene information 206A to form the gaze heat map 406.

[0088] Figure 8 800 can be programmed (e.g., via computer program code or instructions) to further enhance driver safety using the driver behavior (e.g., distraction) detection described herein, and includes a communication mechanism, such as bus 810, for transferring information between other internal components and external components of the computer system 800. In one embodiment, the computer system 800 is Figure 1A Computer system 800, or a portion thereof, constitutes a means for performing one or more steps for further ensuring driver safety using driver distraction detection.

[0089] The bus 810 includes one or more parallel conductors for transmitting information, enabling rapid transfer of information between devices coupled to the bus 810. One or more processors 802 are coupled to the bus 810 for processing information.

[0090] One or more processors 802 perform a set of operations on information (or data) specified by a computer program code, wherein the computer program code relates to using driver distraction detection to further ensure driver safety. Computer program code is a set of instructions or statements that provide instructions for the operation of a processor and / or computer system to perform specified functions. For example, the code can be written in a computer programming language that is compiled into the processor's native instruction set. The code can also be written directly using the native instruction set (e.g., machine language). The set of operations includes importing information from bus 610 and loading the information onto bus 810. Each operation in the set of operations that the processor can perform is represented to the processor by information called an instruction (e.g., an opcode of one or more digits). The sequence of operations (e.g., an opcode sequence) to be performed by the processor 802 constitutes processor instructions, also known as computer system instructions or simply computer instructions.

[0091] The computer system 800 also includes a memory 804 coupled to the bus 810. The memory 804, such as a random access memory (RAM) or any other dynamic storage device, stores information including processor instructions for further ensuring driver safety using driver distraction detection. The computer system 800 can change the information stored in the dynamic memory. The RAM supports the storage and retrieval of information units stored at locations called memory addresses independently of information at adjacent addresses. The processor 802 also uses the memory 804 to store temporary values ​​during the execution of processor instructions. The computer system 800 also includes a read-only memory (ROM) 806 or any other static storage device coupled to the bus 810 for storing static information. A non-volatile (persistent) storage device 808, such as a magnetic disk, optical disk, or flash memory card, is also coupled to the bus 810 for storing information including instructions.

[0092] In one embodiment, information, including instructions for using a head pose and gaze estimator to further ensure distracted driver safety, is provided to bus 810 for use by the processor from an external input device 812, such as a keyboard, microphone, infrared (IR) remote control, joystick, game pad, stylus, touch screen, head-mounted display, or sensor operated by a human user. A sensor detects conditions in its vicinity and converts these detections into physical expressions compatible with measurable phenomena used to represent information in computer system 800. Other external devices coupled to bus 810, primarily for human interaction, include a display device 814 for presenting text or images; a pointing device 816, such as a mouse, trackball, cursor direction keys, or motion sensor, for controlling the position of a small cursor image presented on display 814 and issuing commands associated with graphical elements presented on display 814; and one or more cameras 884 for capturing, recording, and storing one or more still images and / or moving images (e.g., video, movie, etc.), which may also include audio recordings.

[0093] In the illustrated embodiment, dedicated hardware, such as an application specific integrated circuit (ASIC) 820, is coupled to bus 810. The dedicated hardware is used to perform operations that processor 802 does not have the speed to execute for its dedicated purpose.

[0094] The computer system 800 also includes a communication interface 870 coupled to the bus 810. The communication interface 870 provides one-way or two-way communication coupling to various external devices operated by their own processors. Typically, the coupling is to a network link 878 connected to a local network 880, to which various external devices such as servers or databases can be connected. Alternatively, the link 878 can be directly connected to an internet service provider (ISP) 884 or a network 890, such as the internet. The network link 878 can be wired or wireless. For example, the communication interface 870 can be a parallel port, a serial port, or a universal serial bus (USB) port on a personal computer. In some embodiments, the communication interface 870 is an integrated services digital network (ISDN) card, a digital subscriber line (DSL) card, or a telephone modem, which provides an information communication connection to a corresponding type of telephone line. In some embodiments, the communication interface 870 is a cable modem that converts signals on the bus 810 into signals for communication connections via coaxial cables or into optical signals for communication connections via fiber optic cables. As another example, the communication interface 870 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN (e.g., Ethernet). A wireless link can also be implemented. For a wireless link, the communication interface 870 sends and / or receives electrical, acoustic, or electromagnetic signals, including infrared and optical signals, which carry information streams such as digital data. For example, in a wireless handheld device, such as a handheld phone such as a cell phone, the communication interface 870 includes a radio frequency band electromagnetic transmitter and receiver called a wireless transceiver. In some embodiments, the communication interface 870 of a mobile device such as a mobile phone or tablet computer can be connected to a communication network to further ensure the safety of distracted drivers using a head posture and gaze estimator.

[0095] Network link 878 typically uses a transmission medium to provide information through one or more networks to other devices that use or process the information. For example, network link 878 may connect to a host computer 882 or to equipment 884 operated by an ISP through a local network 880. The ISP equipment 884, in turn, provides data communication services through the public global packet-switched communications network now commonly referred to as the Internet 890.

[0096] A computer connected to the Internet, referred to as a server host 882, hosts processes that provide services in response to information received over the Internet. For example, the server host 882 hosts processes that provide information representing video data for presentation on the display 814. It is contemplated that the components of the system 800 can be deployed in various configurations within other computer systems, such as a host 882 and a server 882.

[0097] At least some embodiments of the present application involve implementing some or all of the techniques described herein using computer system 800. According to one embodiment of the present application, these techniques are performed by computer system 800 in response to processor 802 executing one or more sequences of one or more processor instructions contained in memory 804. Such instructions, also known as computer instructions, software, and program code, may be read into memory 804 from another computer-readable medium, such as storage device 808 or network link 878. Execution of the sequences of instructions contained in memory 804 causes processor 802 to perform one or more method steps described herein.

[0098] It should be understood that the present subject matter can be implemented in many different ways and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to make the present subject matter complete and to fully convey the present application to those skilled in the art. In fact, the present subject matter is intended to encompass substitutes, modifications, and equivalents of these embodiments within the scope and spirit of the present subject matter defined by the appended claims. Moreover, in the following detailed description of the present subject matter, many specific details are set forth to provide a thorough understanding of the present subject matter. However, it is clear to those skilled in the art that the present subject matter can be implemented without these specific details.

[0099] The present invention is described in detail herein by the flow chart and / or block diagram of the method, device (system) and computer program product that provide in conjunction with the embodiment of the present application.It should be understood that the combination of each square frame of flow chart and / or block diagram and the square frame in flow chart and / or block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of general-purpose computer, special-purpose computer or other programmable data processing device to generate machine, so, the instruction executed by the processor of computer or other programmable instruction execution device creates the mechanism for realizing the function / action specified in one or more frames of flow chart and / or block diagram.

[0100] Computer-readable non-transitory media include various types of computer-readable media, including magnetic storage media, optical storage media, and solid-state storage media, and specifically do not include signals. It should be understood that the software can be installed in a device and sold with the device. Alternatively, the software can be obtained and loaded into the device, including via optical disk media or any other means from a network or distribution system, including, for example, from a server owned by the creator of the software or from a server not owned by the creator of the software but used by the creator of the software. For example, the software can be stored on a server for distribution over the Internet.

[0101] Computer-readable storage media does not include propagating signals, is accessible by a computer and / or processor, and includes volatile and non-volatile removable and / or non-removable internal and / or external media. Various types of storage media accommodate stored data in any suitable digital format for a computer. Those skilled in the art will appreciate that other types of computer-readable media, such as zip drives, solid-state drives, magnetic tapes, flash memory cards, flash drives, cartridges, etc., may be used to store computer-executable instructions for performing the novel methods (actions) of the disclosed architecture.

[0102] The terms used herein are for the purpose of describing particular aspects only and are not intended to limit this application. Unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" as used herein include the plural meaning. It should also be understood that the terms "comprises" and / or "comprising" as used in this specification are used to indicate the presence of stated features, integers, steps, operations, elements, and / or parts, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or combinations thereof.

[0103] The description of the present application is presented only for illustration and description purposes and is not intended to be exhaustive or to limit the present application in any disclosed manner. Without departing from the scope and spirit of the present application, it will be clear to those of ordinary skill in the art that various modifications and variations will be apparent to those of ordinary skill in the art. Various aspects of the present application are selected and described to better explain the principles and practical applications of the present application and to enable those of ordinary skill in the art to understand the present application and the various modifications suitable for the intended specific use.

[0104] For the purposes of this document, each process associated with the disclosed technology can be performed serially by one or more computing devices. Each step in a process can be performed by the same or different computing devices used in other steps, and each step does not necessarily have to be performed by a single computing device.

[0105] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as exemplary ways of implementing the claims.

Claims

1. A computer-implemented method for monitoring driver attention in a vehicle, characterized in that include: collecting vehicle data and scenario information from said vehicles traveling on a route; capturing a driver's gaze to track a direction and duration of the driver's gaze while driving the vehicle on the route; generating a reference heat map from the scene information and the vehicle data using a convolutional neural network (CNN) comprising one or more convolutional layers and at least one fully connected layer, wherein the reference heat map identifies one or more areas of the scene information requiring driver attention for safe driving; generating the driver gaze heat map based on the gaze direction and duration of the driver while driving the vehicle, wherein the driver gaze heat map identifies one or more areas of the scene information viewed by the driver during the duration; Encoding the reference heat map through a first sub-network to generate a feature vector R, wherein the first sub-network is a deep learning network; Encoding the driver's gaze heat map through a second sub-network to generate a feature vector G, wherein the second sub-network is a deep learning network; determining a driver distraction level of the driver in the vehicle based on a distance between a feature vector R and a feature vector G; An advisory is output to the driver of the vehicle based on the driver distraction level.

2. The computer-implemented method of claim 1 , wherein: Tracking the gaze direction and duration includes: One or more sensors capture a sequence of images of the driver over the duration; estimating the gaze direction of the driver based on the gaze extracted from the image sequence; The gaze direction is superimposed on the scene information to form the driver gaze heat map.

3. The computer-implemented method of claim 1 , wherein: Collecting the vehicle data includes: recording the position of the vehicle during travel on the route; retrieving historical vehicle data representing said vehicle data over a predetermined time period from a database; Collecting the scene information includes: Street imagery is acquired in real time from one or more sensors or online maps.

4. The computer-implemented method according to any one of claims 1 to 3, wherein: The vehicle data includes at least one of vehicle speed data, turn indication data, light condition data, brake usage data, steering wheel usage data, global positioning system (GPS) data, accelerator pedal usage data, and navigation route data; The scene information includes at least one of a scene image, an online street map, a radar map, weather, traffic, and road conditions.

5. The computer-implemented method according to any one of claims 1 to 3, wherein: Also included is merging the reference heat map with the driver gaze heat map to form a driver attention heat map, wherein the driver attention heat map is used to determine whether the driver distraction level meets or exceeds a threshold attention level.

6. The computer-implemented method of claim 5, wherein: When the threshold is met or exceeded, the recommendation is a warning to focus the driver's gaze direction on one or more areas of the driver's gaze heat map based on a value associated with each of the one or more areas of the driver's gaze heat map, wherein the value represents the driver's required attention level for each of the one or more areas of the driver's gaze heat map.

7. The computer-implemented method according to any one of claims 1 to 3 and 6, characterized in that The method is implemented by the one or more processors, which are located in or communicatively coupled to the vehicle.

8. A system for monitoring driver attention in a vehicle, comprising: One or more sensors for: collecting vehicle data and scene information from the vehicles traveling on a route, wherein the vehicle data and the scene information are processed to generate a reference heat map; capturing a driver's gaze to track a gaze direction and duration of the driver while driving the vehicle on the route, wherein the gaze direction and duration are processed to generate a driver gaze heat map; One or more processors for: generating the reference heat map from the scene information and the vehicle data using a convolutional neural network (CNN) comprising one or more convolutional layers and at least one fully connected layer, wherein the reference heat map identifies one or more areas of the scene information requiring driver attention for safe driving; generating the driver gaze heat map based on the gaze direction and duration of the driver while driving the vehicle, wherein the driver gaze heat map identifies one or more areas of the scene information that the driver is looking at during the duration; Encoding the reference heat map through a first sub-network to generate a feature vector R, wherein the first sub-network is a deep learning network; Encoding the driver's gaze heat map through a second sub-network to generate a feature vector G, wherein the second sub-network is a deep learning network; determining a driver distraction level of the driver in the vehicle based on a distance between a feature vector R and a feature vector G; An advisory is output to the driver of the vehicle based on the driver distraction level.

9. The system according to claim 8, characterized in that Tracking the gaze direction and duration includes: The one or more sensors capture a sequence of images of the driver over the duration; estimating the gaze direction of the driver based on the gaze extracted from the image sequence; The gaze direction is superimposed on the scene information to form the driver gaze heat map.

10. The system according to claim 8, wherein: Collecting the vehicle data includes: recording the position of the vehicle during travel on the route; retrieving historical vehicle data representing said vehicle data over a predetermined time period from a database; Collecting the scene information includes: Street images are acquired in real time from the one or more sensors or an online map.

11. The system according to any one of claims 8 to 10, characterized in that The vehicle data includes at least one of vehicle speed data, turn indication data, light condition data, brake usage data, steering wheel usage data, global positioning system (GPS) data, accelerator pedal usage data, and navigation route data; The scene information includes at least one of a scene image, an online street map, a radar map, weather, traffic, and road conditions.

12. The system according to any one of claims 8 to 10, characterized in that The one or more processors are further configured to merge the reference heat map with the driver gaze heat map to form a driver attention heat map, wherein the driver attention heat map is used to determine whether the driver distraction level reaches or exceeds a threshold attention level.

13. The system according to claim 12, wherein: When the threshold is met or exceeded, the recommendation is a warning to focus the driver's gaze direction on one or more areas of the driver's gaze heat map based on a value associated with each of the one or more areas of the driver's gaze heat map, wherein the value represents the driver's required attention level for each of the one or more areas of the driver's gaze heat map.

14. The system according to any one of claims 8 to 10 and 13, characterized in that The one or more processors are located in or communicatively coupled to the vehicle.

15. A computer-readable medium, characterized in that storing computer instructions for monitoring driver attention in a vehicle, the computer instructions, when executed by one or more processors, causing the one or more processors to perform the following steps: collecting vehicle data and scene information from the vehicles traveling on a route, wherein the vehicle data and the scene information are processed to generate a reference heat map; capturing a driver's gaze to track a gaze direction and duration of the driver while driving the vehicle on the route, wherein the gaze direction and duration are processed to generate a driver gaze heat map; generating the reference heat map from the scene information and the vehicle data using a convolutional neural network (CNN) comprising one or more convolutional layers and at least one fully connected layer, wherein the reference heat map identifies one or more areas of the scene information requiring driver attention for safe driving; generating the driver gaze heat map based on the gaze direction and duration of the driver while driving the vehicle, wherein the driver gaze heat map identifies one or more areas of the scene information that the driver is looking at during the duration; Encoding the reference heat map through a first sub-network to generate a feature vector R, wherein the first sub-network is a deep learning network; Encoding the driver's gaze heat map through a second sub-network to generate a feature vector G, wherein the second sub-network is a deep learning network; determining a driver distraction level of the driver in the vehicle based on a distance between a feature vector R and a feature vector G; An advisory is output to the driver of the vehicle based on the driver distraction level.

16. The computer-readable medium of claim 15, wherein: Tracking the gaze direction and duration includes: One or more sensors capture a sequence of images of the driver over the duration; estimating the gaze direction of the driver based on the gaze extracted from the image sequence; The gaze direction is superimposed on the scene information to form the driver gaze heat map.

17. The computer-readable medium of claim 15, wherein: Collecting the vehicle data includes: recording the position of the vehicle during travel on the route; retrieving historical vehicle data representing said vehicle data over a predetermined time period from a database; Collecting the scene information includes: Street-level imagery is acquired in real time from one or more sensors.

18. The computer-readable medium according to any one of claims 15 to 17, wherein: The vehicle data includes at least one of vehicle speed data, turn indication data, light condition data, brake usage data, steering wheel usage data, global positioning system (GPS) data, accelerator pedal usage data, and navigation route data; The scene information includes at least one of a scene image, an online street map, a radar map, weather, traffic, and road conditions.

19. The computer-readable medium according to any one of claims 15 to 17, wherein: The one or more processors are further caused to perform the step of merging the reference heat map with the driver gaze heat map to form a driver attention heat map, wherein the driver attention heat map is used to determine whether the driver distraction level meets or exceeds a threshold attention level.

20. The computer-readable medium of claim 19, wherein: When the threshold is met or exceeded, the recommendation is a warning to focus the driver's gaze direction on one or more areas of the driver's gaze heat map based on a value associated with each of the one or more areas of the driver's gaze heat map, wherein the value represents the driver's required attention level for each of the one or more areas of the driver's gaze heat map.

21. The computer-readable medium according to any one of claims 15 to 17 and 20, wherein: The one or more processors are located in or communicatively coupled to the vehicle.

Images