Method for operating a driver monitoring system for a vehicle and driver monitoring system

Abstract

A method for operating a driver monitoring system for a vehicle, comprising the following steps: Receiving a plurality of gaze target points for a driver of the vehicle; inputting the plurality of gaze target points into a predictive distraction probability model to obtain a predictive distraction distribution; Determine whether one or more meaningful gaze locations exist in the plurality of gaze target points; Comparing the predictive distraction distribution to a predictive distraction distribution threshold if one or more meaningful gaze locations are present in the plurality of gaze target points; and Warn the driver when the predictive distraction distribution reaches or exceeds the predictive distraction distribution threshold; wherein the predictive distraction probability model includes developing a graphical representation of the plurality of gaze target points; wherein the graphical representation is a histogram graph of the plurality of gaze target points; wherein the graphical representation is used to develop a gaze gray-level matrix that captures a plurality of gaze transitions with the plurality of gaze target points; wherein the method further comprises the step of factoring the gaze gray value matrix using a non-negative matrix factorization (NMF) technique to develop a plurality of NMF factors.

Description

[0001] The field of the invention relates to driver monitoring and, in particular, to eye pattern monitoring for motorists.

[0002] US 2018 / 0 053 103 A1 describes systems and methods for managing visual assignment that use models to determine states based on visual data and, based on this, output feedback based on the determined states. Visual data is first acquired by a system for managing visual assignment. The visual data comprises eye image sequences of a person in a particular state, e.g., performing a task or activity. Visual features are identified from the visual data so that gaze information, including direction and duration, can be calculated. The visual data, information derived therefrom, and / or other contextual data are input to the models corresponding to the states to calculate probabilities that the particular state the person is in is one of the modeled states.Based on the state with the highest probability, optimal feedback, such as a warning or instruction, is issued to connected devices, systems or objects.

[0003] EP 2 201 496 B1 describes an apparatus for determining an inattentive state. The apparatus comprises an observed quantity detecting means for detecting information about the driver's line of sight and at least one of either information about driving operations or information about the driving state as the observed quantity; and an inattentive state determining means for determining the driver's inattentive state based on the stochastic relationship between the inattentive state and the observed quantity, as well as the observed quantity detected using the observed quantity detecting means. Furthermore, the inattentive state determining means calculates the probability that the driver is driving inattentively and determines that the driver is driving inattentively if the calculated probability exceeds a threshold.

[0004] EP 2 347 400 B1 describes a method for combining sensor data acquired by at least two sensors coupled to at least one application, wherein at least one of the sensors provides driver-related sensor data of a driver-related behavior and at least one external sensor provides driver-independent sensor data. The sensor data of the at least two sensors are combined because the respective measurement errors of the data with respect to the at least one application are temporally uncorrelated.

[0005] Promoting driver attention and concentration are desirable, but false positives and excessive threat or potential threat alerts can overwhelm a driver. It is beneficial to warn drivers about potential threats; however, it may be more beneficial to warn drivers about potential threats they are not visually aware of. This involves aligning threat detection procedures with assessing whether the threat is visually perceived.

[0006] It can therefore be viewed as a task to warn vehicle drivers of potential threats that they may not visually notice.

[0007] According to the invention, a method of operating a driver monitoring system for a vehicle is provided, comprising the steps of: receiving a plurality of gaze target points for a driver of the vehicle; inputting the plurality of gaze target points into a predictive distraction probability model to obtain a predictive distraction distribution; determining whether one or more meaningful gaze locations are present in the plurality of gaze target points; comparing the predictive distraction distribution to a predictive distraction distribution threshold if one or more meaningful gaze locations are present in the plurality of gaze target points;and warning the driver when the predictive distraction distribution reaches or exceeds the predictive distraction distribution threshold, wherein the predictive distraction probability model includes developing a graphical representation of the plurality of gaze target points, the graphical representation being a histogram graph of the plurality of gaze target points, the graphical representation being used to develop a gaze gray-level matrix capturing a plurality of gaze transitions with the plurality of gaze target points. The method further comprises the step of factoring the gaze gray-level matrix using a non-negative matrix factorization (NMF) technique to develop a plurality of NMF factors.

[0008] According to various embodiments, this method may further include any of the following features or any technically conceivable combination of some or all of these features: • the predictive distraction distribution is a graphical or numerical data scheme that represents an accident or near-accident probability based on past data; • Combining the predictive distraction distribution with a predictive saliency distribution if one or more meaningful gaze locations are missing from the plurality of gaze target points; • Determining a current driving state and using the current driving state as input when combining the predictive saliency distribution with the predictive distraction distribution; • the predictive distraction probability model includes a hidden Markov model (HMM); • the predictive distraction probability model involves using an observation matrix of event probabilities for given gaze locations to obtain the predictive distraction distribution; • the step of incorporating the gaze gray value matrix into a Hidden Markov Model (HMM); • the insertion takes place in a sliding window manner in one-stage steps; • Deriving a plurality of features from the plurality of non-negative matrix factorization (NMF) factors; and / or • Comparing the plurality of features with an observation matrix of event probabilities for given gaze locations and gaze transitions to obtain the predictive distraction distribution.

[0009] According to the invention, a driver monitoring system for a vehicle is provided, comprising: a driver-side camera and an electronic control unit, ECU, operatively coupled to the driver-side camera, the ECU configured to obtain a plurality of gaze target points for a driver of the vehicle from the driver-side camera; input the plurality of gaze target points into a predictive distraction probability model to obtain a predictive distraction distribution; determine whether one or more meaningful gaze locations are present in the plurality of gaze target points; compare the predictive distraction distribution to a predictive distraction distribution threshold if one or more meaningful gaze locations are present in the plurality of gaze target points;and warn the driver when the predictive distraction distribution reaches or exceeds the predictive distraction distribution threshold, wherein the predictive distraction probability model includes developing a graphical representation of the plurality of gaze target points, the graphical representation being a histogram graph of the plurality of gaze target points, the graphical representation being used to develop a gaze gray-level matrix capturing a plurality of gaze transitions with the plurality of gaze target points, the ECU being configured to factor the gaze gray-level matrix using a non-negative matrix factorization, NMF, technique to develop a plurality of NMF factors.;

[0010] One or more embodiments are described below in conjunction with the accompanying drawings, in which like designations indicate like elements, and wherein: Fig. 1 is a block diagram depicting one embodiment of a driver monitoring system that may utilize the methods disclosed herein; Fig. 2 different viewing locations in and around the vehicle from the system of Fig. 1 illustrates; Fig. 3 illustrates a still image of a predictive saliency distribution according to an embodiment; Fig. 4 illustrates another still image of two predictive saliency distributions according to an embodiment; Fig. 5 is a flowchart of one embodiment of a method for operating a driver monitoring system, which may be used in conjunction with the driver monitoring system of Fig. 1 was described; Fig. 6 is a flowchart showing substeps of one of the steps of the method of Fig. 5 shows; Fig. 7 is a graph of gaze locations over time, which is compared with the Fig. 5 and Fig. 6 illustrated method can be used; Fig. 8 shows a predictive deflection distribution according to an embodiment; Fig. 9 is a histogram graph of gaze location counts compared with the Fig. 5 and Fig. 6 illustrated method can be used; Fig. 10 is another histogram graph of gaze location counts, which is compared with the Fig. 5 and Fig. 6 illustrated method can be used; Fig. 11 matrices that are related to the Fig. 5 and Fig. 6 illustrated methods can be used; Fig. 12 is a flowchart illustrating the steps involved in the gaze saliency distribution analysis in the context of the driver monitoring system of Fig. 1 describes; and Fig. 13 is a flowchart illustrating the steps involved in the sensor saliency distribution analysis in the context of the driver monitoring system of Fig. 1 describes.

[0011] The system and methods described below include driver distraction prediction for a driver monitoring system using time-series gaze pattern analysis. The gaze pattern analysis results in a predictive distraction distribution that can be used to determine whether to provide a warning to the driver. The predictive distraction distribution can be a graphical or numerical data schema representing a crash or near-crash probability for a given gaze location or gaze transition based on past data. In some embodiments, the gaze pattern analysis can be combined with a predictive saliency distribution to estimate or rate potential threats to the vehicle. The predictive saliency distribution is a spatiotemporal camera-based predictive distribution over threats that other drivers would likely visually attend to.The present systems and methods can help detect distraction more efficiently than other driver monitoring systems through more complex pattern analysis while reducing false alarms.

[0012] With reference to Fig. 1, an operating environment is shown that includes a driver monitoring system 10 that can be used to implement the methods disclosed herein. The driver monitoring system 10 generally includes sensors 22-32, a forward-facing camera 34, a driver-side camera 36, ​​a GNSS receiver 38, a wireless communication device 40, other vehicle system modules (FSMs) 50-58, and an electronic control unit (ECU) 60. The driver monitoring system 10 further includes a constellation of global navigation satellite system (GNSS) satellites 68, one or more wireless carrier systems 70, a land communications network 76, a computer or server 78, and a backend device 80. It is understood that the disclosed methods can be used with any number of different systems and are not specifically limited to the operating environment shown herein.The following paragraphs provide a brief overview of such a driver monitoring system 10; however, other systems not shown herein may also utilize the disclosed methods. It should also be noted that the driver monitoring system 10 and methods may be used with any type of vehicle, including classic passenger cars, sport utility vehicles (SUVs), crossover vehicles, trucks, vans, buses, recreational vehicles (RVs), motorcycles, etc. These are only a few of the possible applications, as the driver monitoring system and methods described herein are not limited to the embodiment in FIG. Fig. 1 and can be implemented with any number of different vehicles.

[0013] Any number of different sensors, components, devices, modules, systems, etc. can provide information, data, and / or other inputs to the driver monitoring system 10. These include, for example, the Fig. 1, as well as others known in the art but not shown here. It should be noted that the host vehicle sensors, cameras, object detection sensors, GNSS receivers, ECUs, MMSs, and any other components that are part of and / or utilized by the driver monitoring system 10 may be integrated in hardware, software, firmware, or a combination thereof. These components may directly sense or measure the conditions for which they are provided, or they may indirectly evaluate such conditions based on information provided by other sensors, components, devices, modules, systems, etc. Further, these components may be directly coupled to a control unit or ECU 60, indirectly coupled via other electronic devices, a communications bus, network, etc., or coupled to another arrangement in accordance with the prior art.These components may be integrated into another vehicle component, device, module, system, etc. (e.g. sensors that are already part of an active safety system, a traction control system (TCS), an electronic stability control (ESC) system, an anti-lock braking system (ABS), etc.), they may be standalone components (as shown schematically in . Fig. 1) or they may be deployed according to a different arrangement. In some cases, multiple sensors may be deployed to detect a single parameter (e.g., to provide redundancy). It should be noted that the preceding scenarios represent only some of the possibilities, as any suitable arrangement or architecture may be used to perform the methods described herein.

[0014] The host vehicle sensors 22-30 may include any type of sensing or other component that provides the present systems and methods with data or information regarding the performance, state, and / or condition of the vehicle 12. Information from the host vehicle sensors 22-30 may be used to derive information regarding the tasks or threats at hand (e.g., whether the host vehicle 12 is accelerating toward a potential threat, road conditions, etc.). According to the non-limiting example provided in Fig. 1, the host vehicle sensors include host vehicle speed sensors 22-28 and a dynamic sensor unit 30.

[0015] The host vehicle speed sensors 22-28 provide speed measurements to the system 10 indicative of the rotational speed of the wheels and thus the overall speed or velocity of the vehicle. In one embodiment, individual wheel speed sensors 22-28 are coupled to each of the vehicle's four wheels and separately provide speed measurements indicative of the rotational speed of the corresponding wheel (e.g., by counting pulses at a rotating wheel or wheels). Those skilled in the art will recognize that these sensors may operate according to optical, electromagnetic, or other technologies, and that the speed sensors 22-28 are not limited to any particular type of speed sensor.In another embodiment, the speed sensors may be coupled to specific parts of the vehicle, such as a transmission output shaft or behind the speedometer, and generate speed readings from these measurements. It is also possible to derive or calculate speed readings from acceleration readings (those skilled in the art will be aware of the relationship between speed and acceleration readings). In another embodiment, the speed sensors 22-28 determine the vehicle speed relative to the ground by directing radar, laser, and / or other signals toward the ground and analyzing the reflected signals, or by using feedback from a navigation unit having global positioning system (GPS) capabilities (e.g., GNSS 38). It is possible that the speed readings may be provided to the system 10 by another module, subsystem, system, etc.such as a powertrain or engine control module or a brake control module. However, other known speed sensing techniques may be used.

[0016] The dynamic sensor unit 30 provides the system with dynamic measurements related to the various dynamic conditions occurring in the vehicle, such as acceleration and yaw rate. The unit 30 may include any combination of sensors or sensor elements that sense or measure vehicle dynamics, and it may be in a separate housing or packaged in a single unit. According to one embodiment, the dynamic sensor unit 30 is an integrated inertial measurement unit (IMU) that includes a yaw rate sensor, a lateral acceleration sensor, and a longitudinal acceleration sensor. Some examples of suitable acceleration sensor types include microelectromechanical system (MEMS)-type sensors and tuning fork-type sensors, although any type of acceleration sensor may be used.Depending on the specific requirements of the system, the acceleration sensors may be single- or multi-axis sensors capable of sensing acceleration and / or deceleration, detecting the magnitude and / or direction of acceleration as a vector quantity, directly detecting or measuring acceleration, calculating or inferring acceleration from other measurements such as vehicle speed measurements, and / or providing G-force acceleration, to name a few possibilities. Although the dynamic sensor unit 30 is shown as a separate unit, it is possible that this unit or elements thereof may be integrated into another unit, device, module, system, etc.

[0017] The object detection sensor 32 provides the system 10 with sensor readings and object data related to vehicles, pedestrians, or other nearby objects or threats surrounding the vehicle 12. The object sensor readings may represent the presence, position, speed, and / or acceleration of neighboring vehicles, as well as those of pedestrians and other nearby objects. This data may be absolute (e.g., an object speed or acceleration relative to the ground or other frame of reference) or the data may be relative (e.g., an object speed or acceleration relative to the host vehicle). While only one object detection sensor 32 is schematically illustrated, in some embodiments, multiple object detection sensors are included to monitor various positions around the vehicle 12.Each of the object detection sensors may be a single sensor or a combination of sensors and may include one or more radar devices, laser devices, lidar devices, ultrasonic devices, night vision devices, other known devices, or combinations thereof. In one advantageous embodiment, the object detection sensor 32 is a radar sensor or a lidar sensor. In another advantageous embodiment, the object detection sensor 32 is a penetrating radar sensor.

[0018] Of course, other vehicle sensors could be used in addition to or instead of the sensors described above to provide information regarding the condition of the vehicle 12. Some possible examples may include a V2X communication unit to provide information regarding other vehicles, infrastructure, or pedestrians (e.g., V2V, V2I, or V2P); an environmental sensor to provide measurements regarding external weather conditions or other environmental events; steering angle sensors; accelerator and brake pedal sensors; dynamics sensors; and gear selection sensors, to name a few. Further, some implementations of the present systems and methods may not include all of the vehicle sensors or other components described herein.

[0019] A surround camera 34 and a vehicle-mounted camera 36 may be used to provide surround camera images and information related to gaze patterns of the driver of the vehicle 12, respectively. In one advantageous embodiment, the surround camera 34 is a forward-facing camera that receives camera images of the environment in front of the vehicle 12. However, the camera 34 may point in other directions, and the methods may assess erroneous threats in other areas surrounding the vehicle (e.g., with a rearview camera when the vehicle 12 is reversing). The surround camera 34 and / or the vehicle-mounted camera 36 may be directly or indirectly connected to the ECU 60 for processing inputs from the cameras. Cameras 34, 36 may be any suitable camera type (e.g., charge-coupled device (CCD), complementary metal-oxide semiconductor (CMOS), etc.).) and may include any suitable lens known in the art, so they are not limited to any particular type, make, or model. In one particular embodiment, both cameras 34, 36 are mounted on glasses worn by the driver of the vehicle 12. In another embodiment, the cameras 34, 36 are integrated into a single camera module mounted near or on the windshield or rearview mirror of the vehicle 12. In some embodiments, only one camera may be used to obtain both the surround camera images and the driver view images. Other camera configurations are certainly possible, such as mounting the surround camera 34 on the exterior of the vehicle 12 and mounting the driver-side camera 36 near the rearview mirror, to name a few examples.Some non-limiting examples of possible embodiments or features that may be used with cameras 34, 36 include: infrared LEDs for night vision; wide-angle or fisheye lenses; surface-mounted, flush-mounted, license plate-mounted, or side-mounted cameras; stereoscopic arrays with multiple cameras; cameras integrated into taillights, brake lights, or other components at the rear of the vehicle; and wired or wireless cameras, to name a few possibilities. Adaptations to the methods described herein to account for different camera types and / or positions of cameras 34, 36 may be made offline before executing the method in real-time or near-real-time.

[0020] The cameras 34, 36 may provide a plurality of images (e.g., acquired from streaming video or other captured video) to the ECU 60, which may then process the images to create a predictive distraction distribution, a predictive saliency distribution, and / or a gaze tracking probability distribution, as described in detail below. In one embodiment, the cameras 34, 36 continuously send video data to the ECU 60 while the vehicle's ignition or primary drive system is on or activated. The video data may be interlaced or progressive scan video data or interlaced scan video data for the ECU 60. The ECU 60 may then decode, transform, or otherwise process the video data so that the video encoded in the data may be appropriately processed and utilized by the various methods described herein.Other image processing may be performed by the processor of the ECU 60 or another processing device in the vehicle 12.

[0021] As discussed further below, through the use of image processing techniques, the processor may detect specific objects, such as an impending threat to the vehicle 12, to which the driver may not be paying attention. In one embodiment, the ECU 60 may utilize image processing software that can distinguish specific objects in the captured images and, by analyzing a series of images, optionally in combination with information from one or more vehicle sensors such as sensor 32, determine a position, distance, speed, and / or acceleration of such a unique threat or objects relative to the vehicle 12.

[0022] Each of the devices 22-36 may be independent, as in Fig. 1, or they may be integrated or enclosed in another device, unit, or module (e.g., some of the sensors 22-28 could be housed in an inertial measurement unit (IMU), the camera 34 could be integrated into an active safety system, etc.). Furthermore, each of the devices 22-36 may be dedicated, as in Fig. 1, or they may be part of or shared with other systems or subsystems in the vehicle (e.g., the camera 34 and / or some of the sensors 22-30 could be part of a semi-autonomous driving system). The video input and / or sensor input devices 22-36 may be provided directly to the ECU 60 or indirectly via any other device, module, and / or system generally known in the art. Accordingly, the devices 22-36 are not limited to the schematic representation in Fig. 1 or the above exemplary descriptions, nor are they limited to any particular embodiment or arrangement, as long as they can be used with the method described herein.

[0023] The global navigation satellite system (GNSS) receiver 38 receives radio signals from a constellation of GNSS satellites 68. The GNSS receiver 38 may be configured to comply with and / or operate in accordance with specific regulations or laws of a given geopolitical region (e.g., country). The GNSS receiver 38 may be configured for use with various GNSS implementations, including the Global Positioning System (GPS) for the United States, the BeiDou Navigation Satellite System (BDS) for China, the Global Navigation Satellite System (GLONASS) for Russia, Galileo for the European Union, and various other navigation satellite systems. For example, the GNSS receiver 38 may be a GPS receiver capable of receiving GPS signals from a constellation of GPS satellites 68.And in another example, the GNSS receiver 38 may be a BDS receiver that receives a plurality of GNSS (or BDS) signals from a constellation of GNSS (or BDS) satellites 68. In any implementation, the GNSS receiver 38 may include at least a processor and memory, including non-transitory computer-readable memory for storing instructions (software) accessible to the processor for performing the processing performed by the receiver 38.

[0024] The GNSS receiver 38 can be used to provide navigation and other position-related services to the vehicle driver.

[0025] Navigation system information, such as information about upcoming events that may affect travel, may be presented on the display 50 or presented verbally, as is the case with turn-by-turn navigation. The navigation services may be provided using a dedicated on-vehicle navigation module (which may be part of the GNSS receiver 38 and / or integrated as part of the wireless communication device 40 or other FSM), or some or all of the navigation services may be provided via the vehicle communication device 40 (or other telematics-enabled device) installed in the vehicle, with the position or location information being sent to a remote location to provide the vehicle with navigation maps, map annotations (points of interest, restaurants, etc.), route calculations, and the like.The position information may be communicated to the vehicle backend device 80 or another remote computing system, such as a computer 78, for other purposes, such as for training purposes in developing the predictive distraction distribution and / or the predictive saliency distribution, as discussed below.

[0026] The wireless communication device 40 is capable of communicating data via short-range wireless communications (SRWC) or via cellular network communications through the use of a cellular chipset 44, as depicted in the illustrated embodiment. In one embodiment, the wireless communication device 40 is a central vehicle computer used to perform at least a portion of the methods discussed below. In the illustrated embodiment, the wireless communication device 40 includes an SRWC circuit 42, a cellular chipset 44, a processor 46, a memory 48, and antennas 43 and 45.In one embodiment, the wireless communication device 40 may be a stand-alone module, or in other embodiments, the device 40 may be integrated or included as part of one or more other vehicle system modules, such as a central stack module (CSM), a body control module (BCM), an infotainment module, a head unit, and / or a gateway module. In some embodiments, the device 40 may be implemented as an OEM-installed (embedded) or aftermarket device installed in the vehicle. In some embodiments, the wireless communication device 40 includes a telematics unit (or telematics control unit) capable of performing cellular communications using one or more cellular carrier systems 70.The telematics unit may be integrated with the GNSS receiver 38 so that, for example, the GNSS receiver 38 and the wireless communication device (or telematics unit) 40 are directly connected to each other rather than being connected via communication bus 59.

[0027] In some embodiments, wireless device 40 may be configured to wirelessly communicate according to one or more short-range wireless communications (SRWC), such as one of Wi-Fi™, WiMAX™, Wi-Fi Direct™, other IEEE 802.11 protocols, ZigBee™, Bluetooth™, Bluetooth™ Low Energy (BLE), or near-field communication (NFC). As used herein, Bluetooth™ refers to any Bluetooth™ technologies, such as Bluetooth Low Energy™ (BLE), Bluetooth™ 4.1, Bluetooth™ 4.2, Bluetooth™ 5.0, and other Bluetooth™ technologies that may be developed. As used herein, Wi-Fi™ or Wi-Fi™ technology refers to any Wi-Fi™ technologies, such as IEEE 802.11b / g / n / ac or another IEEE 802.11 technology. The short-range wireless communication (SRWC) circuit 42 enables the wireless communication device 40 to transmit and receive SRWC signals, such as BLE signals.The SRWC circuitry may enable device 40 to connect to another SRWC device. Additionally, in some embodiments, the wireless communication device may include a cellular chipset 44, thereby enabling the device to communicate via one or more cellular protocols, such as those used by the cellular carrier system 70. In these cases, the wireless communication device becomes a user equipment (UE) usable for conducting cellular communications over the cellular carrier system 70.

[0028] The wireless communication device 40 may enable the vehicle 12 to communicate with one or more remote networks (e.g., one or more networks at the backend device 80 or computers 78) via packet-switched data communications. This packet-switched data communication may be performed using a non-vehicle wireless access point connected to a land network via a router or modem. When used for packet-switched data communication, such as TCP / IP, the communication device 40 may be configured with a static IP address or may be configured to automatically receive an assigned IP address from another device on the network, such as a router or from a network address server. Packet-switched data communications may also be performed using a cellular network accessible by the device 40.The communication device 40 may communicate data via the wireless carrier system 70 via the cellular chipset 44. In such an embodiment, radio transmissions may be used to establish a communication channel, such as a voice channel and / or a data channel, with the wireless carrier system 70 so that voice and / or data transmissions may be sent and received over the channel.

[0029] Processor 46 may be any type of device capable of processing electronic instructions, including microprocessors, microcontrollers, host processors, control units, vehicle communication processors, and application-specific integrated circuits (ASICs). It may be a dedicated processor used only for communication device 40, or it may be shared with other vehicle systems. Processor 46 executes various types of digitally stored instructions, such as software or firmware programs, stored in memory 48 that enable device 40 to provide a wide variety of services. For example, processor 46 may execute programs or process data to perform at least some of the methods discussed herein.The memory 48 can be a temporarily operating memory, any non-transitory computer-readable medium, or another type of memory. For example, the memory can be any of a number of different types of RAM (random access memory, including various types of dynamic RAM (DRAM) and static RAM (SRAM)), ROM (read-only memory), solid-state drives (SSDs) (including other solid-state memory such as solid-state hybrid hard drives (SSHDs)), hard disk drives (HDDs), magnetic or optical disk drives. Similar components, such as those previously described (processor 46 and / or memory 48, as well as SRWC circuit 42 and cellular chipset 44) can be included in another control module and / or various other FSMs that typically include such processing / memory capabilities, such as an ECU 60.

[0030] The wireless communication device 40 is connected to the bus 59 and can receive sensor data from one or more vehicle sensors 22-32 and / or the cameras 34, 36, and thereafter, the vehicle 12 can send this data (or other data derived from or based on this data) to other devices or networks, including the vehicle backend device 80. However, in some embodiments, all or some of the data is processed by the ECU 60 or another module. In an advantageous embodiment, real-time or near-real-time processing is performed entirely via the ECU 60 to avoid processing delays. However, training for the methods can be processed in whole or in part using a computer 78 and / or the backend device 80 (including servers 82 and databases 84).

[0031] The vehicle electronics 20 also includes a number of vehicle user interfaces that provide vehicle occupants with a means of providing and / or receiving information, including the visual display 50, pushbutton(s) 52, microphone 54, audio system 56, and / or haptic feedback device 58. As used herein, the term "vehicle user interface" broadly includes any suitable form of electronic device, including both hardware and software components, located on the vehicle 12 that enables the vehicle user to communicate with or through a component of the vehicle. The vehicle user interfaces 50-54 are also on-board sensors that can receive input from a user or other sensory information.The pushbutton(s) 52 enable manual user input to the communication device 40 to provide other data, response, or control inputs. The audio system 56 provides audio output to a vehicle occupant and may be a dedicated, standalone system or part of the primary vehicle audio system. According to the particular embodiment shown herein, the audio system 56 is operably coupled to both the vehicle bus 59 and an entertainment bus (not shown) and may provide AM, FM, and satellite radio, CD, DVD, and other multimedia functionality. This functionality may be provided in conjunction with or independently of an infotainment module. The audio system 56 may be used to provide directional audible awareness when a driver of the vehicle 12 should be alerted to a potential threat.The microphone 54 provides an audio input for the wireless communication device 40 to enable the driver or another passenger to provide voice commands and / or make hands-free calls via the wireless carrier system 70. For this purpose, it can be connected to an automated vehicle-owned speech processing unit using human-machine interface (HMI) technology, which is known in the art. The visual display or touchscreen 50 is preferably a graphic display and can be used to provide a variety of input and output functions. The display 50 can be a touchscreen on the instrument panel, a head-up display that is reflected by the windshield, or a projector that can project graphics for viewing by a vehicle passenger.For example, in one embodiment, the display 50 is an augmented reality display shown through the windshield of the vehicle 12. The haptic feedback device 58 may be used to provide tactile sensations to the driver of the vehicle 12. In this embodiment, the haptic feedback device 58 is a seat 90. The regions 92, 94 may be activated, for example, to warn a driver of the vehicle 12 that there is a potential threat toward the corresponding side of the vehicle. Various other vehicle user interfaces may also be used, as the interfaces of . Fig. 1 are only an example of a specific implementation.

[0032] Accordingly, if a driver of the vehicle 12 becomes distracted, he or she may be warned using the one or more vehicle user interfaces, as discussed further below.

[0033] The ECU 60 controls various components of the driver monitoring system 10 and accomplishes some, if not all, of the real-time or near-real-time vehicle-based processing required to perform the methods herein. Accordingly, the ECU 60 may receive feedback or information from numerous sources, such as the sensors 22-32 and cameras 34, 36, and then use this feedback or information to effectively monitor the driver and / or potential threats to the vehicle 12. The ECU 60 may be considered a control unit, control module, etc., and may include any selection of electronic processing devices, memory devices, input / output (I / O) devices, and / or other known components, and may perform various control and / or communication-related functions.In an exemplary embodiment, the ECU 60 includes an electronic storage device 62 that stores sensor readings (e.g., sensor readings from the sensors 22-32), image or video information (e.g., image or video transmission from the cameras 34, 36), lookup tables or other data structures (e.g., one or more matrices or models, as described below), algorithms (e.g., the algorithms embodied in the form of the methods described below), etc. The storage device 62 may maintain a buffer consisting of collected data over a predetermined period of time or in predetermined circumstances (e.g., a driver's gaze targets, sensor readings, etc.). The storage device 62, or just a portion thereof, may be implemented or maintained in the form of an electronic data structure, as is known in the art. The ECU 60 also includes an electronic processing device 64 (e.g.,a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), etc.) that executes instructions for software, firmware, programs, algorithms, scripts, etc. stored in the storage device 62 that may partially govern the processes and methods described herein. .

[0034] Depending on the specific embodiment, the ECU 60 may be a standalone vehicle electronic module (e.g., a dedicated or specific driver monitoring control unit), it may be integrated or included within a vehicle electronic module (e.g., a video control unit), it may be part of a larger network or system (e.g., an active safety system), or it may be a slave unit implementing low-level controls based on a supervisory vehicle control unit, to name a few possibilities. Accordingly, the ECU 60 is not limited to any particular embodiment or arrangement and may be used by the present method to control one or more aspects of driver monitoring system 10 operation.The driver monitoring system 10 and / or the ECU 60 may also include a calibration file, which is a setup file that defines commands given to control components such as the display 50, the audio system 56, and / or the haptic feedback device 58.

[0035] The wireless carrier system 70 may be any suitable cellular telephone system. The carrier system 70 is shown including a cellular tower 72; however, the carrier system 70 may include one or more of the following components (e.g., depending on the cellular technology): cellular towers, base transceiver stations, mobile switching centers, base station controllers, evolved nodes (e.g., eNodeBs), mobility management units (MMEs), serving and PGN gateways, etc., as well as other network components required to connect the wireless carrier system 70 to the land network 76 or to connect the wireless carrier system to user equipment (UEs, e.g., which may include a telematics unit in the vehicle 12). The carrier system 70 may implement any suitable communication technology, including GSM / GPRS technology, CDMA or CDMA2000 technology, LTE technology, etc.

[0036] In addition to using the wireless carrier system 70, another wireless carrier system in the form of satellite communications may be used to provide unidirectional or bidirectional communications with the vehicle. This may be accomplished using one or more communications satellites (not shown) and an uplink transmitting station (not shown). Unidirectional communications may, for example, be satellite radio services, where programming content (news, music, etc.) is received from the uplink transmitting station, packaged for upload, and then transmitted to the satellite, which broadcasts the programming to subscribers. Bidirectional communications may, for example, be satellite telephony services, which utilize the one or more communications satellites to relay telephone communications between the vehicle 12 and the uplink transmitting station.If used, this satellite telephony can be used either in addition to or instead of the wireless carrier system 70.

[0037] The land network 76 may be a conventional land-based telecommunications network connected to one or more landline telephones and connecting the wireless carrier system 70 to the vehicle backend device 80. For example, the land network 76 may include a public switched telephone network (PSTN), such as that used to provide wired telephony, packet-switched data communications, and the Internet infrastructure. One or more segments of the land network 76 could be implemented using a standardized wired network, a fiber or other optical network, a cable network, power lines, other wireless networks such as wireless local area networks (WLANs) or networks providing wireless broadband access (BWA), or any combination thereof.

[0038] The computers 78 (only one is shown) may be any of a number of computers accessible via a private or public network, such as the Internet. In one embodiment, each such computer 78 may be used for one or more purposes, such as training and initially creating the predictive distraction probability model and / or the predictive saliency distribution. Other such accessible computers 78 may be, for example: a client computer that may be used by the vehicle owner or other subscribers for purposes such as accessing or receiving vehicle data, setting or configuring subscriber preferences, or controlling vehicle functions; or a third-party repository from or to which vehicle data or other information is provided, whether by communicating with the vehicle 12, the backend device 80, or both.A computer 78 may also be used to provide Internet connectivity, such as DNS services, or as a network address server using DHCP or another suitable protocol to assign an IP address to the vehicle 12.

[0039] The vehicle backend device 80 is located remotely from the vehicle 12. The backend device 80 may be configured to provide the vehicle electronics 20 with a number of different system backend functions through the use of one or more electronic servers 82 and, in many cases, may provide processing capabilities for the initial training of the models described herein, while most of the real-time or near-real-time processing occurs at the vehicle 12, such as with ECU 60. The backend device 80 may be a physical call center, or it could be a cloud-based server or the like. The backend device 80 includes vehicle backend servers 82 and databases 84, which may be stored on a variety of storage devices.The vehicle backend device 80 may include any or all of these various components, and preferably, all of the various components are coupled together by a wired or wireless local area network. The backend device 80 may receive and transmit data via a modem connected to the land network 76. Data transmissions may also occur through wireless systems such as IEEE 802.11x, GPRS, and the like. Those skilled in the art will recognize that, although only one backend device 80 and one computer 78 are illustrated in the exemplary embodiment, a plurality of remote devices 80 and / or computers 78 may be used. Furthermore, a plurality of backend devices 80 and / or computers 78 may be geographically distributed, and they may each coordinate information and services with one another.

[0040] The servers 82 may be computers or other computing devices that include at least one processor and at least one memory. The processors may be any type of device capable of processing electronic instructions, including microprocessors, microcontrollers, host processors, control units, vehicle communication processors, and application-specific integrated circuits (ASICs). The processors may be dedicated processors used only for the server 82, or they may be shared with other systems. The at least one processor may execute various types of digitally stored instructions, such as software or firmware, that enable the servers 82 to provide a wide variety of services. This software may be stored in computer-readable memory and may be any suitable non-transitory computer-readable medium.The memory may be, for example, any of a number of different types of RAM (random access memory, including various types of dynamic RAM (DRAM) and static RAM (SRAM)), ROM (read-only memory), solid-state drives (SSDs) (including other solid-state storage such as solid-state hybrid hard drives (SSHDs)), hard disk drives (HDDs), magnetic or optical disk drives. For network communications (e.g., intranet communications, Internet communications including Internet connections), the servers may include one or more network interface cards (NICs) (including wireless NICs (WNICs)) that may be used to transport data to and from the computers. These NICs may enable the one or more servers 82 to connect to each other, to databases 84, or to other network devices, including routers, modems, and / or switches.In a particular embodiment, the NICs (including WNICs) of the servers 82 may enable SRWC connections to be established and / or may include Ethernet (IEEE 802.3) ports to which Ethernet cables can be connected, providing a data connection between two or more devices. The backend device 80 may include a number of routers, modems, switches, or other devices that may be used to provide network capabilities, such as connecting to the land network 76 and / or cellular carrier system 70.

[0041] The databases 84 may be stored on a variety of devices, such as a powered cache or any suitable non-transitory computer-readable medium. The memory may be, for example, any of a number of different types of RAM (random access memory, including various types of dynamic RAM (DRAM) and static RAM (SRAM)), ROM (read-only memory), solid-state drives (SSDs) (including other solid-state storage such as solid-state hybrid hard drives (SSHDs)), hard disk drives (HDDs), magnetic or optical disk drives, storing some or all of the software required to perform various external device functions discussed herein.One or more databases 84 at the backend device 80 may store various information and may include a database for storing information related to the creation of the predictive distraction probability model and / or the predictive saliency distribution.

[0042] Fig. Figure 2 illustrates a view of the interior of the vehicle 12 from the driver monitoring system of Fig. 1, which shows a variety of different exemplary gaze locations 102-126 that can be used with the present systems and methods. The gaze locations 102-126 correspond to the gaze targets of the driver of the vehicle. When the driver-side camera 36 receives information regarding the driver's gaze targets, these gaze targets can be correspondingly assigned to the gaze targets 102-126 using the ECU 60 or the like. The gaze locations 102-126 are schematically shown in Fig. 2 with dashed circles or outlines. However, it is understood that the gaze locations may be differently sized (e.g., larger or smaller than illustrated), differently shaped (e.g., the gaze locations may be more amorphously shaped to correspond to specific vehicle features without any gaps between the gaze locations), differently categorized (e.g., they could represent different areas of the vehicle, or there may be more or fewer locations than illustrated), etc.

[0043] Gaze locations 102-126 represent areas to which the driver is likely to visually attend. Although other examples are certainly possible, in a particular example, the gaze locations include: a right window gaze location 102, a right mirror gaze location 104, a right forward gaze location 106, a rearview mirror gaze location 108, a passenger gaze location 110, a left window gaze location 112, a left mirror gaze location 114, a left forward gaze location 116, an interior object gaze location 118, an instrument cluster gaze location 120, a forward gaze location 122, a center console gaze location 124, and a cell phone gaze location 126. Other gaze location types, shapes, etc. are also possible, and it may even be possible for some gaze locations to partially overlap. In the example shown, the driver's gaze target points are obtained by the driver-side camera 36 and then assigned to the various gaze locations 102-126 or correlated with them in some other way.These gaze targets / gaze locations can be used in the procedures described in detail below.

[0044] In addition to the gaze target points / gaze locations 102-126, Fig. 3 and Fig. 4 schematically illustrates various embodiments of a threat detection distribution 200, 202, 204 that may be used with the present systems and methods. In particular, the depicted threat detection distributions 200, 202, 204 are predictive saliency distributions 210, which in some embodiments may be used to supplement, corroborate, etc., gaze analyses. The threat detection distribution 200 is a predictive saliency distribution 210 superimposed on an analysis image 212 of the environment outside the vehicle 12, such as that taken by the surround camera 34. The predictive saliency distribution 210 is a spatiotemporal camera-based predictive distribution of threats and related to threats that other drivers would likely visually observe. The predictive saliency distributions 210 are highlighted in this example because a gaze target point 214 is identified by a potential threat (ieTarget vehicle 216). Due to the divergence between the predictive saliency distribution 210 and the gaze target point estimate 214, such as a gaze tracking probability distribution obtained from the driver-side camera 36, ​​the systems and methods may warn the driver of the vehicle 12 of the potential threat or target vehicle 216. The predictive saliency distribution 210 includes a high warning zone 218, a moderate warning zone 220, and a low warning zone 222. In some embodiments, the high warning zone 218 may be colored red or the like to represent the highest estimated risk radius, the moderate warning zone 220 may transition from red to orange or yellow to represent a moderate estimated risk radius, and the low warning zone 222 may transition to green or blue to represent a low estimated risk radius.

[0045] The predictive saliency distribution 210 is similar to a dynamic thermal map or the like, which changes according to movement of the vehicle 12 and / or movement of objects or threats in the environment. The evolution of the predictive saliency distribution is described in more detail below. In the illustrations of Fig. 3 and Fig. 4, the predictive saliency distribution 210 includes the first threat 216, and then each zone 218-222 in each of the distributions 200, 202, 204 changes or deforms as the vehicle moves toward the intersection 224, which is further described in Fig. 4 is shown. Fig. 4 has a new analysis image 226, which is created at a later time than the analysis image 212 of Fig. 3 is recorded. In Fig. 4, the threat detection distributions 202, 204, given the gaze target point estimate 214, may indicate areas that the driver of the vehicle 12 should pay attention to (e.g., the predictive saliency distributions 210). The methods 1200, 1300 described below describe the development of the various distributions, which are schematically shown in Fig. 3 and Fig. 4, in more detail.

[0046] Fig. 5 illustrates a method 500 for operating a driver monitoring system for a vehicle using the system 10 and the vehicle 12 described above with reference to Fig. 1-4. It is understood that the steps of each method described herein are not necessarily presented in any particular order, and that performing some or all of the steps in an alternative order or concurrently is possible and contemplated. Furthermore, it is likely that the methods could be implemented in other systems that differ from the driver monitoring system 10 described in Fig. 1, and the description of the methods in the context of the system 10 is only an example.

[0047] Step 502 involves receiving a plurality of gaze target points for a driver of the vehicle 12. The gaze target points may be in the form of eye tracking data received at the ECU 60 from the driver-side camera 36. In some embodiments, the eye tracking data may represent a plurality of X,Y coordinates. These may be estimated or projected onto the scene in front of the vehicle 12, as schematically represented by points 230 in Fig. 3 and Fig. 4. The gaze target points can also be assigned to the gaze locations 102-126 shown in Fig. 2. Accordingly, the gaze target points may be gaze target point estimates from various gaze locations in and around the vehicle 12. The gaze target points may be obtained by analyzing multiple frames from the driver-side camera video. In a particular embodiment, the frame rate for the driver-side camera 36 is approximately 1 / 10 of a second.

[0048] Step 504 involves inputting the plurality of gaze target points from step 502 into a predictive distraction probability model to obtain a predictive distraction distribution. The predictive distraction distribution is a graphical or numerical data schema representing a crash or near-crash probability based on past data. To obtain the predictive distraction distribution, the gaze target points are processed using a predictive distraction probability model. The predictive distraction probability model may be implemented in a number of ways, one of which is generally described in the breakout substeps of Fig. 6 is illustrated.

[0049] Fig. 6 illustrates a number of substeps 504 1 -504 5 . Substeps 504 1 -504 3represent an initial training and testing period, which usually takes place before the entire method 500 is executed, wherein the substeps 504 4 -504 5 represent the classification aspect of step 504, which operates in real-time or near-real-time, in conjunction with method 500. However, it is possible and contemplated to perform more training and testing operations externally (e.g., via computer 78 and backend device 80) and send updates from this training to vehicle 12 for processing with ECU 60 with the updated information in real-time or near-real-time.

[0050] Substep 504 1involves obtaining historical data from an event database. In one embodiment, the event database includes labeled historical data from drivers. The labeling may include event instances (crash or near miss) and a baseline (normal) for given features or gaze locations. The baseline includes normal driving at specific gaze locations, whereas the crash / near miss includes gaze locations observed or otherwise estimated when the driver was in a crash (or within a specific time period (e.g., 10 seconds) before the crash) or near miss scenario. Whether an event is labeled as a near miss may depend on the desired sensitivity of the system. For example, near miss instances may occur when the driver performs a corrective maneuver to avoid a crash (e.g., swerving or emergency braking).In a particular embodiment, the prior data was obtained or otherwise sourced from the Virginia Tech Transportation Institute (VTTI) 100-car data. Additionally, in this embodiment, the crash / near-miss determination may follow the VTTI standards and statistical analysis parameters for determining the crash / near-miss and baseline. In some embodiments, prior data from substep 504 may be used. 1 from the driver of vehicle 12 or other vehicle drivers.

[0051] Substep 504 2 involves extracting features from the training data. In a first embodiment, the feature extraction step involves extracting gaze target points, such as gaze locations 102-126, which are Fig. 2, which can be predefined for the model. Fig. 7 and Fig. 8 illustrate a time series gaze pattern analysis that can be used with this substep. Fig. 7 is a graph 700 of time on the x-axis in 1 / 10 units of a second (total 3 seconds) and the Fig. Figure 2 illustrates gaze locations 102-126. X1 represents no video data, and X2 represents cases where the driver's eyes are closed. Fig. 8 has a corresponding x-axis representing time in 1 / 10 units of a second (3 seconds in total) with the accident or near-accident probability on the y-axis. Fig. 7 illustrates that the driver begins to look at a left mirror view location 114 (section 702), then an interior object view location 118 (section 704), the rearview mirror view location 108 (section 706), the forward view location 122 (section 708), and finally the left window view location 112 (section 710). Fig. 8 is then a predictive distraction distribution 800 for the gaze target points that are in Fig. 7. The changes in the predictive deflection distribution 800 generally follow the changes in the graphical representation of Fig. 7 (e.g., variations that occur at approximately seven-tenths of a second, sixteen-tenths of a second, twenty-tenths of a second, twenty-five-tenths of a second). The predictive distraction distribution 800 may also consider how long a driver maintains a gaze target point. For example, segment 704 in Fig. 7 a gaze target point at an interior object gaze location 118. The longer the driver maintains this gaze target point, the greater the probability of an accident / near miss. This is reflected by the increasing gradient in the corresponding section in the predictive distraction distribution 800. Other exemplary variations, patterns, etc. are certainly possible and depend on the training data.

[0052] In a second embodiment, the feature extraction sub-step 504 includes 2 Extracting features such as gaze targets and transitions between the gaze targets. To implement the second embodiment, features such as gaze locations and transitions are extracted using a graphical analysis, a gray-value matrix, and / or a non-negative matrix factorization (NMF) technique. Graphical representations of this second embodiment of feature extraction are shown in the Fig. 9, Fig. 10 and Fig. 11. In this embodiment, the gaze target points from previous data are analyzed in one-second steps for graphical analysis, which includes two gaze histograms 900, 1000, each in Fig. 9 and Fig. 10. In histograms 900, 1000, gaze locations 102-126 are on the x-axis (where X1 represents no video data and X2 represents closed eyes), and the count values ​​for each gaze location are plotted along the y-axis. In this example, each histogram 900, 1000 represents one second, with a new frame taken every tenth of a second (i.e., ten counts per histogram). Other example time periods and frame rates are certainly possible, as are alternative graphical formats.

[0053] The feature number values ​​in this second embodiment of sub-step 504 2are used to obtain a gray value matrix. Following the example above, the histograms 900,1000 are used to obtain a gaze gray value matrix 1100, which is Fig. 11. In this example grayscale matrix 1100, gaze locations 102-126, X1, X2 are plotted against gaze locations 102-126, X1, X2. Grayscale matrix 1100 records gaze transitions with gaze target points. As shown, the highest frequency range 1102 occurs at 122, 122 because the driver spent the majority of this 1-2 second period looking toward forward gaze location 122. Range 1104 represents a count frequency or estimated count frequency of transitions of the driver's gaze target point from forward gaze location 122 to interior object gaze location 118, with range 1106 representing the opposite transition (i.e., from interior object gaze location 118 to forward gaze location 122). As shown, grayscale matrix 1100 includes other gaze locations and gaze transitions.The structure of the grayscale matrix may differ from the example grayscale matrix 1100, and its shape and configuration will change depending on a number of factors, such as the number of gaze locations, analysis timing, etc.

[0054] The gray value matrix 1100 in this second embodiment of sub-step 504 2 can be used to derive a progression of a variety of features, including gaze locations and gaze transitions, that are more likely to lead to a crash / near miss. In one example, an additional level of feature extraction is achieved using a non-negative matrix factorization (NMF) technique. Fig. Figure 11 also shows three submatrices 1110, 1112, 1114 resulting from factoring the gray value matrix 1100 using the NMF technique. However, it should be noted that more submatrices with more features are likely to be generated, but in Fig. 11 illustrates only three. NMF may be particularly useful for developing submatrices 1110, 1112, 1114 given their tendency to cluster input data. The high-frequency region in each submatrix may be used as the extracted feature for step 504. 2 used. In this example, sub-matrix 1110 has a high-frequency region 1120 corresponding to the left window gaze location 112, 112. Sub-matrix 1112 has a high-frequency region 1122 corresponding to the right window gaze location 102, 102. Sub-matrix 1114 has a high-frequency region 1124 corresponding to the left mirror gaze location 114, 114. The NMF technique can help infer new features from the input prior data, such as gaze transitions that occur at the intersection of two different gaze locations 102-126, X1, X2.

[0055] Substep 504 3involves developing an observation matrix of event probabilities for the given features (e.g., gaze locations and / or gaze transitions), which in substep 504 2 extracted. The observation matrix contains feature probabilities from the prior data, which are used to train the Hidden Markov Model (HMM) used for the real-time / quasi-real-time aspects of method 500. The observation matrix size may include a number of features in each row, including two columns for the probability of a given observed gaze target point at an event (e.g., crash and / or near miss) and the probability of a given observed gaze target point at the baseline (e.g., normal driving or no crash / near miss). Following the first embodiment described above, the observation matrix would be Fig. 7 and Fig. 8 include the gaze locations with the following exemplary probabilities, as shown in Table I below. Table I P(beob.|Ereignis) P(beob.|Baseline) Mobiltelefon 126 0,0450 0,0003 Mittelkonsole 124 0,0203 0,0043 Augen geschlossen X2 0,0457 0,0063 Vorwärts 122 0,4737 0,8313 • • • • • • • • • Rechter Spiegel 104 0,0037 0,0050 Rechtes Fenster 102 0,0187 0,0200

[0056] For example, the value P(obs.|event) for the mobile phone gaze location 126 is relatively high compared to other gaze locations (e.g., a crash / near-miss event may be more likely at a mobile phone gaze target point). The value P(obs.|baseline), on the other hand, is relatively low for the mobile phone gaze target point 126 because these are generally less frequent gaze target points for drivers. These probabilities, and accordingly the observation matrices, change depending on previous data and training and may be updated regularly (e.g., by communicating new training data from the backend device 80 to the ECU 60 via the wireless communication device 40). The observation matrix for the second embodiment may be similar to the observation matrix in Table I above, but it includes the features resolved by the gray-value matrix 1100 rather than including only gaze locations.The observation matrix can represent clusters of viewpoints and the frequency of occurrence of each cluster relative to the other clusters.

[0057] Substeps 504 4 and 504 5 represent the real-time or quasi-real-time aspects of the predictive distraction probability model. Substep 504 4 involves extracting features from the eye tracking data. This can be achieved using the gaze target points obtained in step 502. Similar to the first exemplary embodiment with sub-step 504 3 The gaze target points can be correlated and analyzed with respect to the gaze locations 102-126 (with X1 and X2). Following the second exemplary embodiment with step 504 3 The gaze target points can be correlated and analyzed to determine gaze locations and gaze transitions.

[0058] In substep 504 5A predictive distraction distribution is obtained. In one embodiment, the observation matrix can then be used in conjunction with an HMM to obtain the predictive observation distribution for the real-time or quasi-real-time gaze targets obtained from the driver. The observation matrix can accordingly be used to perform a prediction on input test data using a forward prediction algorithm. In one embodiment, Bayesian updating is used, and the forward prediction algorithm (e.g., whether it is in an event state or baseline state) is calculated based on the gaze location and the previous data using the following equation: P(statet0|observationt0)=α∗P(observationt0|statet0)∗∑statetprevious probability of a state

[0059] This can then be followed by a recursive algorithm in which the following equation (Equation 2) is repeated in fixed time steps: P(statet+1|observation1:t+1)=α∗P(observationt+1|statet+1)∑statet(statet+1|statet1)∗P(statet|observation1:t)

[0060] Where α is the normalization coefficient; P(observation t+1 |Condition t+1 ) is obtained from the observation matrix; P(state t+1 |Condition t ) is the transition probability for moving from the previous state to a new state (i.e., the probability of moving from the previous state predicted in the previous time step to another state (which can be an event or a baseline)); and P(state t |Observation 1:t) is the estimated state probability in the previous step. This Bayesian update, when applied to the time-series gaze target point data, can yield the probability that an accident / near-miss (i.e., event state) exists.

[0061] The predictive probability distribution may be the result of the Bayesian update computed above, or in some embodiments, a predicted probability obtained at least partially using data from the observation matrix may be analyzed for a specified time window (e.g., 1 second) to obtain the predicted probability distribution. In one advantageous embodiment (the first exemplary embodiment above), the HMM takes gaze target points as input and predicts the current state (e.g., event or baseline; accident / near miss or normal). For each gaze target point input, the probability value is searched in the observation matrix and then multiplied by the transition probability to obtain the predicted state for a given gaze location.In another advantageous embodiment (the second exemplary embodiment above), the gray value matrix 1100 is introduced into the HMM in a sliding window manner in single-stage steps. The HMM then provides the distraction probability distribution (e.g., the probability of an accident / near-miss). The sliding window approach in single-stage steps can help to progressively update the parameters of the predictive distraction probability model.

[0062] To access procedure 500 in Fig. Returning to Figure 5, the predictive distraction distribution may be compared in step 504 to a predictive distraction distribution threshold in step 506. Depending on how the predictive distraction distribution is structured, step 506 asks whether the predictive distraction distribution does not meet or exceed the predictive distraction distribution threshold. In one embodiment, step 506 asks whether the predictive distraction distribution value is greater than a predictive distraction distribution threshold. It should be understood that the terms for comparing steps such as "less than" or "more than" are open-ended, so they may each include "less than or equal to" or "more than or equal to," and this will depend on the determined parameter values ​​in the desired implementation.The predictive distraction distribution threshold may be a dynamic threshold derived or based at least in part from past data. In a more specific embodiment, the predictive distraction distribution threshold is a heuristically derived threshold based at least in part on the current gaze location or gaze transition, as described in more detail below. In a more specific embodiment, the predictive distraction distribution threshold is a probability of 0.4-0.6 (particularly 0.5) for certain gaze locations, which is again described in more detail below. For example, the threshold may be lower for gaze locations such as mobile phone gaze location 126 and higher for gaze locations such as forward gaze location 122.The threshold analysis in step 506 may help determine when method 500 will trigger system 10 to warn the driver of vehicle 12. If the predictive distraction distribution does not meet or exceed the threshold, method 500 may return to step 502 to continue monitoring the driver.

[0063] If the predictive distraction distribution meets or exceeds the threshold, the method proceeds to step 508 to determine whether one or more meaningful gaze locations exist.

[0064] Step 508 involves determining whether one or more meaningful gaze locations are present in the plurality of received gaze targets from the driver of the vehicle 12. This step may look at gaze locations within an analyzed time period (e.g., the last second, although other time periods are possible). If one or more meaningful gaze locations are present, the method proceeds to step 510 to alert the driver. If one or more meaningful gaze locations are not present, the method proceeds to step 512 to evaluate the current driving condition. In one embodiment, the determination of meaningful gaze locations depends on the prediction probability. For example, if the prediction probability or accuracy is higher than a certain amount (e.g., 85-90%) for a given gaze location, then this may be considered a meaningful gaze location.In another embodiment, the meaningful gaze locations include the mobile phone gaze location 126, closed eyes X2, the interior object gaze location 118, the passenger gaze location 110, and the center console gaze location 124. In another embodiment, if the most recent gaze locations predominantly include the following gaze locations, it may be determined that one or more meaningful gaze locations are not present: the forward gaze location 122, the rearview mirror gaze location 108, the left and right window gaze locations 102, 112, and the left and right mirror gaze locations 104, 114.

[0065] Step 510 involves warning the driver of the vehicle 12. Various warnings are provided, such as with the display 50. In one advantageous embodiment, the display 50 is an augmented reality display that highlights or provides some type of visual indication to the driver that attention should be directed to a different location than where they are currently focused (e.g., a potential threat is highlighted on the augmented reality display or a display in the vehicle 12). In another embodiment, a directional audible cue is provided using the audio system 56. For example, audible cues for directional audible awareness may be provided to help indicate where a detected threat is generally located. In yet another embodiment, a haptic feedback device 58 is used to alert the driver.For example, areas 92, 94 in seat 90 may be activated to warn a driver of vehicle 12 that there is a potential threat toward the corresponding side of the vehicle. Other MMS-based warnings are certainly possible, as are various other warnings. For example, an autonomous driving action or the like may be performed to help avoid the threat. Generally, the warning in step 510 helps to reduce or mitigate predictive driver distraction.

[0066] If it is determined in step 508 that one or more meaningful gaze locations are not present, the method may proceed to step 512 to evaluate the current driving condition of the vehicle 12. This step may provide vehicle-related contextual information to strengthen prediction accuracy. For example, the method 500 may not proceed if a certain driving condition exists (e.g., if the vehicle 12 is stopped, in reverse, in a parking space, in a driveway, etc.). For example, if the vehicle 12 is stopped at a traffic light, the method 500 may wait until the vehicle 12 is moving before alerting the driver.

[0067] Step 512 combines the predictive distraction distribution determination with a predictive saliency distribution. In some embodiments, the current driving state determined in step 510 may also be used as an input in this step. For example, the predictive saliency distribution calculations summarized below may only be performed when the vehicle 12 is in motion. The method 500 may rely on a predictive saliency distribution rather than just the predictive distraction distribution to determine whether to issue a warning to the driver (step 508). Step 512 may involve generating a gaze saliency distribution (method 1200 in Fig. 12) and / or the generation of a sensor saliency distribution (method 1300 in Fig. 13).

[0068] Both methods 1200, 1300 include similar steps to generate the predictive saliency distribution used in each method. As mentioned above, the predictive saliency distribution is a spatiotemporal camera-based predictive distribution about threats that other drivers would likely visually attend to. In some embodiments, training to initially develop one or more aspects of the predictive saliency distribution is accomplished at least in part using computers 78 and backend device 80, with information about threats a driver is likely to attend to then being sent locally to vehicle 12 for real-time or near-real-time processing.Accordingly, the predictive saliency distribution may be constructed using a lookup table, an image matching algorithm, or other collection of particular threats, and these threats may be weighted or otherwise ranked (e.g., vehicle backing out of parking space, as with threat vehicle 216 in FIG. Fig. 3, oncoming traffic or intersections 224, as in Fig. 3 and Fig. 4, or other potential threats, including but not limited to objects or pedestrians in the road, collision risks, road features such as sharp curves, etc.). Model training may be used to develop the threat set, for example, by observing drivers and recording gaze patterns at certain objects that arise during driving. For example, if a majority of drivers visually attend to an object or threat type, that object or threat type may be included in the set. Weighting and / or ranking may be accomplished using various techniques, including weighting by distance, speed, acceleration, etc., using data obtained from sensors 22-32, camera 34, or another source.

[0069] To develop the predictive saliency distribution, steps 1202, 1302 include obtaining a plurality of initial environment camera images before obtaining the analysis environment camera image. The plurality of initial environment camera images and the analysis environment camera image are preferably consecutive images obtained from video input from the environment camera 34 or extracted in another way. The number of initial environment camera images may depend on the batch size to be propagated in a neural network, as described in more detail below. In an advantageous embodiment, the number of initial environment camera images is fifteen, with the sixteenth image being the analysis environment camera image, such as image 212 shown in Fig. 3. Once the plurality of initial environment camera images are obtained, each analysis image can proceed sequentially after the initial batch. For example, the waiting time is not affected by the analysis image 226 shown in Fig. 4, since a sufficient batch size has already been obtained, and the subsequent processing of the single analysis environment image can be processed after the initial environment camera images have been processed.

[0070] Steps 1204, 1304, in one embodiment, include performing optical flow analysis on the initial environment camera images. In one embodiment, the optical flow analysis includes an image comparison of each of the initial environment camera images. The optical flow analysis helps encode information related to relative motion in the area in front of the vehicle 12, or another area being monitored applying the method 1200, 1300. In one particular example, OpenCV Deepflow is used in steps 1204, 1304. In other embodiments, variant energy minimization or another type of image comparison optical flow analysis is used.

[0071] Steps 1206, 1306, in one particular embodiment, involve semantic segmentation of the analysis environment camera image. The semantic segmentation analysis may provide landscape information and may output various regions, structures, segments, shapes, etc., which are used to generate the predictive saliency distribution. The semantic segmentation may use any operable algorithm or segmentation technique and will likely depend on the desired segmentation output structures. In some embodiments, each individual initial environment image is also analyzed using semantic segmentation analysis. In such an embodiment, a composite sequence of 16 segmented frames is input in steps 1208, 1308. However, alternatively numbered sequences are certainly possible (e.g.,the initial batch may have more or less sequential image frames).

[0072] Steps 1208, 1308 involve generating the predictive saliency distribution. Steps 1208, 1308 take inputs from the analysis environment camera image, the results of the optical flow analysis from steps 1204, 1304, and the results of the semantic segmentation analysis from steps 1206, 1306. In one embodiment, a neural network is used to create the predictive saliency distribution, and the predictive saliency distribution is a probability distribution function indicating potential threat areas in the analysis environment image that other drivers would likely be aware of. In one example, as in Fig. 3 and Fig. As illustrated in Figure 4, the predictive saliency distribution 210 is a heatmap that dynamically highlights different zones in the sequential images, with the distribution 210 changing or deforming as the relative positions of different threats in the images change. The predictive saliency distribution 210 may also be represented in various other forms, such as numerically, graphically, or using another distribution function model.

[0073] Both methods 1200, 1300 use the predictive saliency distribution generated in steps 1202-1208 and 1302-1308, respectively, as input. Methods 1200, 1300 differ in that, in addition to the predictive saliency distribution, a gaze target point estimate (method 1200: steps 1210-1212) and a threat-weighted occupancy probability distribution (method 1300: steps 1310-1312) are used as inputs. Accordingly, method 1200 is more focused on gaze saliency, while method 1300 is more focused on sensor saliency, with each method capable of generating an alert in the overall method 500 (step 510).

[0074] Step 1210 of method 1200 includes receiving eye tracking data from the driver of the vehicle. This may be achieved using the gaze target points received in step 502 of method 500, for example, with the driver-side camera 36. In some embodiments, the eye tracking data may represent multiple X,Y coordinates. These may be estimated or projected onto the scene in front of the vehicle 12, as schematically represented by points 230 in Fig. 3 and Fig. 4. A gaze target point estimation may be obtained by analyzing multiple frames from the driver-side camera video. In some embodiments, the frame rate for the driver-side camera 36 is higher (e.g., approximately every 1 / 10 of a second) than the frame rate for the surround camera 34, so that more data for the gaze target point estimation can be obtained.

[0075] Step 1212 of method 1200 includes determining a gaze tracking probability distribution using the eye tracking data from step 1210. The gaze tracking probability distribution is a gaze target point estimate that may represent collections or groups of coordinated eye movements (e.g., a model distribution across the scene). In one embodiment, a 2D Hidden Markov Model (HMM) is used to determine the gaze tracking probability distribution from the received eye tracking data. The 2D HMM may be an advantageous model in sequence-based image analysis.

[0076] Step 1214 of method 1200 involves generating a homographic projection to match the gaze tracking probability distribution and the analysis surround camera image. The homographic projection accordingly matches the input from the surround camera 34 and the driver-side camera 36. The processing attributes and algorithms involved in generating the homographic projection depend on various factors, such as the mounting arrangement of each of the cameras 34, 36, the type of cameras, the dimensions of the images, etc. Generating the homographic projection in step 1214 enables a more efficient and accurate comparison with the predictive saliency distribution calculated in step 1208.

[0077] Step 1216 includes determining a gaze saliency divergence between the predictive saliency distribution determined in steps 1202-1208 and the gaze tracking probability distribution determined in steps 1210-1214. The greater the divergence, the more likely a driver is not paying attention to a saliency threat to the vehicle 12. In an advantageous embodiment, step 1216 includes calculating the Kullback Leibler (KL) divergence between the predictive saliency distribution and the gaze tracking probability distribution. Combining the KL divergence (the gaze saliency divergence) with the neural network for the predictive saliency distribution may enable more complex approximations and more accurate determinations of false threat detection. Other methods for determining the divergence in step 1216 include:without limitation, sampling saliency, histogram analysis, pixel linearity, analyzing the area under a ROC (Receiver Operating Characteristic) curve, or any other operable method.

[0078] Step 1218 of method 1200 includes comparing the gaze saliency divergence determined in step 1216 to a gaze saliency divergence threshold. In one embodiment, step 1218 asks whether the gaze saliency divergence is greater than a predictive gaze saliency divergence threshold. It should be understood that the terms for comparing steps such as "less than" or "more than" are open-ended, such that they may each include "less than or equal to" or "more than or equal to," and this will depend on the determined parameter scores in the desired implementation. The gaze saliency divergence threshold may be a dynamic threshold derived, at least in part, from or based on prior data.In a more specific embodiment, the gaze saliency divergence threshold is a heuristically derived threshold based at least in part on the current saliency and / or gaze pattern. For example, if the predictive saliency distribution indicates a potential threat toward the perimeter (e.g., oncoming traffic from a side street), but the driver is looking toward the center, the threshold may be lower. In contrast, there is a significant tendency for drivers to stare toward the horizon. If the predictive saliency distribution indicates a potential threat on the road ahead of the vehicle while the driver is looking toward a peripheral region, the threshold may be higher. Accordingly, the gaze saliency threshold may be adjustable depending on the nature of the threat, the position of the driver's gaze in the driving environment, or other factors.Advantageously, the gaze saliency threshold is developed such that a high-probability saliency prediction (e.g., zones 218, 220 in the predictive saliency distribution 210) with a low-probability gaze target point estimate triggers the system 10 to warn the driver of the vehicle 12. Thus, after step 1218, the method 1200 returns to step 510 of the method 500 to warn the driver if the gaze saliency divergence is greater than the gaze saliency divergence threshold. In such a situation, the driver may be distracted, tired, or inattentive.

[0079] As described above, both methods 1200, 1300 use the predictive saliency distribution generated in steps 1202-1208 and 1302-1308, respectively, as input. The method 1300 in Fig. 13 differs from procedure 1200 in Fig.12 in that the method 1300 uses a threat-weighted occupancy probability distribution (method 1300: steps 1310-1312) as input instead of the gaze target point estimate (method 1200: steps 1210-1212).

[0080] Step 1310 of method 1300 involves receiving external sensor readings. This may be accomplished using the object detection sensor 32, which is advantageously a radar sensor or a lidar sensor. In a particular embodiment, the sensor readings received in step 1310 are object detection readings from a penetrating radar sensor. The representation of information from the sensor readings may be provided in a number of different operable forms. For example, a Markov Random Field (MRF) model may be used to estimate an occupancy grid using sensor readings from the object detection sensor 32, which may be filtered and / or smoothed.

[0081] Step 1312 of method 1300 includes determining a threat-weighted occupancy probability distribution from one or more of the sensor readings obtained in step 1310. Following the example provided above, the occupancy grid may be used to at least partially determine the threat-weighted occupancy probability distribution. The occupancy grid may be developed using an MRF model, with each grid cell generally representing a threat location, with one or more aspects such as inertia, relative velocity, etc., represented in a different dimension (e.g., along the Z-axis, with the location characterized via X, Y coordinates; some embodiments may have three or more dimensions).Accordingly, in this embodiment, the occupancy grid is the threat-weighted occupancy probability distribution; however, other methods for generating the threat-weighted occupancy probability distribution are possible. For example, sensor data may be provided in other coordinate schemes or in other formats more suitable for different distribution types. In addition, step 1312 may use information such as host vehicle speed, as indicated by readings from speed sensors 22-28, or information from other system components to assist in generating the threat-weighted occupancy probability distribution.

[0082] Step 1314 of method 1300 involves generating an alignment projection to align the threat-weighted occupancy probability distribution and the analysis surround camera image obtained in step 1302. In one embodiment, the alignment projection is a homographic projection, although other alignment techniques are possible and may depend on the type of sensor 32. The alignment projection accordingly aligns the input from the surround camera 34 and the object detection sensor 32. The processing attributes and algorithms involved in generating the projection depend on various factors, such as the mounting arrangement of the camera 34, the type of sensor 32, the dimensions of the images, the range of the sensor 32, etc. Generating the homographic projection in step 1314 enables a more efficient and accurate comparison with the predictive saliency distribution calculated in step 1308.

[0083] Step 1316 involves determining a sensor saliency divergence between the predictive saliency distribution determined in steps 1302-1308 and the threat-weighted occupancy probability distribution determined in steps 1310-1314. The greater the divergence, the more likely anomalous environmental sensor indications are. In some cases, the object detection sensor 32 may indicate ordinary objects or maneuvers that were not triggered or made risky with the predictive saliency distribution. These anomalies could be helpful when training or developing the predictive saliency distribution, alerting a driver regarding misaligned sensors (e.g., with a high-probability saliency prediction and a low-probability threat-weighted occupancy detection), and / or alerting a driver regarding a low-probability saliency threat (e.g.,one that most drivers would not judge) but is always risky, as indicated by a threat-weighted occupancy detection with high probability. In an advantageous embodiment, step 1316 includes calculating the Kullback Leibler (KL) divergence between the predictive saliency distribution and the threat-weighted occupancy probability distribution. Combining the KL divergence (the sensor saliency divergence) with the neural network for the predictive saliency distribution may enable more complex approximations and more accurate determinations of erroneous threat detection. Other methods for determining the divergence in step 1316 include, but are not limited to, sample saliency, histogram analysis, pixel linearity, analyzing the area under a ROC (Receiver Operating Characteristic) curve, or any other operable method.

[0084] Step 1318 of method 1300 includes comparing the sensor saliency divergence determined in step 1316 to a sensor saliency divergence threshold. In one embodiment, step 1318 asks whether the sensor saliency divergence is greater than a predictive sensor saliency divergence threshold. Again, it should be understood that the terms for comparing steps such as "less than" or "more than" are open-ended, such that they may each include "less than or equal to" or "more than or equal to," and this will depend on the determined parameter scores in the desired implementation. As with the predictive distraction distribution threshold and the gaze saliency divergence threshold, the sensor saliency divergence threshold may be a dynamic threshold derived, at least in part, from or based on prior data.In a more specific embodiment, the sensor severity divergence threshold is a heuristically derived threshold based at least in part on the current severity and / or sensor measurements. For example, if a penetrating radar object detection sensor 32 indicates that a cyclist is approaching the vehicle from behind a hedge on the side of the vehicle 12, but the predictive severity distribution does not indicate a risk, the threshold could be lower. The threshold may be higher for more salient threats directly in front of the vehicle. Accordingly, the sensor severity threshold may be adjustable depending on the type of threat, the type of sensor, or other factors. Advantageously, the sensor severity threshold is designed such that a low-probability severity prediction (e.g.,Zone 222 in the predictive saliency distribution 210) with a threat-weighted occupancy estimate with high probability triggers the system 10 to warn the driver of the vehicle 12. Accordingly, after step 1318, the method 1300 returns to step 510 of the method 500 to warn the driver if the gaze saliency divergence is greater than the gaze saliency divergence threshold.

Claims

[1] A method for operating a driver monitoring system for a vehicle, comprising the following steps: Receiving a plurality of gaze target points for a driver of the vehicle; inputting the plurality of gaze target points into a predictive distraction probability model to obtain a predictive distraction distribution; Determine whether one or more meaningful gaze locations exist in the plurality of gaze target points; Comparing the predictive distraction distribution to a predictive distraction distribution threshold if one or more meaningful gaze locations are present in the plurality of gaze target points; and Warn the driver when the predictive distraction distribution reaches or exceeds the predictive distraction distribution threshold; wherein the predictive distraction probability model includes developing a graphical representation of the plurality of gaze target points; wherein the graphical representation is a histogram graph of the plurality of gaze target points; wherein the graphical representation is used to develop a gaze gray-level matrix that captures a plurality of gaze transitions with the plurality of gaze target points; wherein the method further comprises the step of factoring the gaze gray value matrix using a non-negative matrix factorization (NMF) technique to develop a plurality of NMF factors. [2] The method of claim 1, wherein the predictive distraction distribution is a graphical or numerical data schema representing a crash or near-crash probability based on past data. [3] The method of claim 1, further comprising the step of combining the predictive distraction distribution with a predictive saliency distribution when one or more meaningful gaze locations are not present in the plurality of gaze target points. [4] The method of claim 3, further comprising the step of determining a current driving condition and using the current driving condition as input in combining the predictive saliency distribution with the predictive distraction distribution. [5] The method of claim 1, wherein the predictive distraction probability model includes a Hidden Markov Model (HMM). [6] The method of claim 1, wherein the predictive distraction probability model includes using an observation matrix of event probabilities for given gaze locations to obtain the predictive distraction distribution. [7] The method of claim 1, further comprising the step of incorporating the gaze gray value matrix into a Hidden Markov Model (HMM). [8] The method of claim 7, wherein the introduction is carried out in a sliding window manner in single-stage steps. [9] The method of claim 1, further comprising the step of deriving a plurality of features from the plurality of non-negative matrix factorization (NMF) factors. [10] The method of claim 9, further comprising the step of comparing the plurality of features with an observation matrix of event probabilities for given gaze locations and gaze transitions to obtain the predictive distraction distribution. [11] Driver monitoring system for a vehicle, comprising: a driver-side camera; and an electronic control unit (ECU) operably coupled to the driver-side camera, the ECU configured to obtain a plurality of gaze targets for a driver of the vehicle from the driver-side camera; input the plurality of gaze targets into a predictive distraction probability model to obtain a predictive distraction distribution; determine whether one or more meaningful gaze locations are present in the plurality of gaze targets; the predictive distraction distribution with a predictive Distraction distribution threshold when one or more meaningful gaze locations are present in the plurality of gaze target points; and warn the driver when the predictive distraction distribution reaches or exceeds the predictive distraction distribution threshold; wherein the predictive distraction probability model includes developing a graphical representation of the plurality of gaze target points; wherein the graphical representation is a histogram graph of the plurality of gaze target points; wherein the graphical representation is used to develop a gaze gray-level matrix that captures a plurality of gaze transitions with the plurality of gaze target points; wherein the ECU is configured to factorize the gaze gray value matrix using a non-negative matrix factorization, NMF, technique to develop a plurality of NMF factors.

Images