Driver attention determination using gaze detection

Abstract

This disclosure provides systems, methods, and apparatus for vehicle driving assistance systems that support image processing for vehicle monitoring operations. In a first aspect, a monitoring method includes: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to the field of view of the user; determining a set of regions of interest based on the second image data; determining a gaze direction of the user based on the first image data; and determining a concentration score based on a correspondence between the set of regions of interest and the gaze direction. Other aspects and features are also claimed and described.

Description

Technical Field

[0001] Aspects of the present disclosure relate generally to driver-operated vehicles and, more particularly, to methods and systems suitable for providing driving assistance to a driver of a vehicle. Background Art

[0002] Vehicles come in many shapes and sizes, are propelled by a variety of propulsion technologies, and carry cargo including humans, animals, or objects. These machines have enabled cargo to be moved across long distances, to be moved at higher speeds, and to be moved at greater speeds than human effort could move. Vehicles were originally driven by humans to control the speed and direction at which cargo arrived at their destination. Human operation of vehicles has caused many unfortunate accidents caused by collisions between vehicles, vehicles and objects, vehicles and humans, or vehicles and animals. As research into vehicle automation has progressed, a variety of driver assistance systems have been produced and introduced. These driver assistance systems include navigation guidance through GPS, adaptive cruise control, lane change assistance, collision avoidance systems, night vision, parking assistance, and blind spot detection. Summary of the invention

[0003] The following summarizes some aspects of the present disclosure to provide a basic understanding of the technology discussed. This summary is not an exhaustive overview of all the expected features of the present disclosure, and is neither intended to identify the key or important elements of all aspects of the present disclosure, nor to delineate the scope of any or all aspects of the present disclosure. The sole purpose of this summary is to present some concepts of one or more aspects of the present disclosure in a summarized form as a prelude to more specific embodiments presented later.

[0004] The operator of a vehicle may be distracted, which is a factor in many vehicle crashes. Driver distraction can include changing the radio station, observing events outside the vehicle, and using electronic devices. Sometimes, situations result in situations that even a focused driver cannot recognize in time to prevent a vehicle crash. Aspects of the present disclosure provide improved systems for assisting a driver in a vehicle with enhanced situational awareness while traveling on a road.

[0005] Example embodiments provide driver behavior monitoring systems and methods that can detect driver inattention regardless of whether the driver's inattention is the result of distraction or untrained vehicle operation. The driver behavior analysis results can be combined with other vehicle sensor information (e.g., engine information, speed sensor, location information, etc.) to determine the occurrence and location of an accident, the near miss of an accident, and / or score the performance of the vehicle operator. The system can report the accident to authorities, emergency responders, or other nearby drivers based on the location of the car by utilizing global positioning system (GPS) information and / or wireless multicast technology.

[0006] In some embodiments, the system can use a map of high-risk areas in a traffic scene, wherein the map provides a means for identifying areas that a focused driver should observe in order to safely navigate the traffic scene. An eye monitor can be used to record where the driver is looking. This driver data can be aggregated over time and compared with a risk map over time to determine whether the driver's gaze pattern is consistent with sufficient situational awareness. The risk map models the scene around the vehicle and indicates high-risk areas, which refer to areas that may represent danger to the vehicle. High-risk areas may include: road users (e.g., bicycles, motorcycles, pedestrians, etc.) that may pose risks given their positioning and movement; and entrances, exits, or edges of buildings, vegetation, or other structures that may hide road users. The risk map can be generated by a machine learning model that is trained to identify such areas in a traffic scene.

[0007] In certain embodiments, aspects of the present disclosure use images or image sequences from a forward-looking camera to generate a map of a high-risk area. Sensors that provide depth information, such as a stereo camera or a laser radar, can be used to correlate image positioning with positioning in the world. Images from an inward-facing camera can be used to monitor the driver's eyes and determine the gaze direction. The driver's concentration can be measured by the proportion of time spent looking at the risk area provided by the map. If this proportion drops below a certain threshold, the driver can be warned by an audio-visual component, and if the driver's concentration does not improve, further action can be taken.

[0008] In one aspect of the present disclosure, a method for vehicle monitoring includes: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to the user's field of view; determining a set of regions of interest based on the second image data; determining a gaze direction of the user based on the first image data; and determining a concentration score based on the correspondence between the set of regions of interest and the gaze direction.

[0009] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations including: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to the field of view of the user; determining an area of ​​interest based on the second image data; determining a gaze direction of the user based on the first image data; and determining a concentration score based on a correspondence between the area of ​​interest and the gaze direction.

[0010] In an additional aspect of the present disclosure, a device includes: a component for receiving first image data from a first camera oriented in a first direction having a first field of view facing a user; a component for receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to the user's field of view; a component for determining an area of ​​interest based on the second image data; a component for determining a gaze direction of the user based on the first image data; and a component for determining a concentration score based on the correspondence between the area of ​​interest and the gaze direction.

[0011] In an additional aspect of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to the field of view of the user; determining a region of interest based on the second image data; determining a gaze direction of the user based on the first image data; and determining a concentration score based on a correspondence between the region of interest and the gaze direction.

[0012] In an additional aspect of the present disclosure, a method for vehicle monitoring includes: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user operating a vehicle; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to a second field of view of a surrounding environment of the vehicle; determining a risk map based on the second image data; determining a gaze direction of the user based on the first image data; determining a correspondence between the risk map and the gaze direction; and controlling the vehicle based on the correspondence between the risk map and the gaze direction.

[0013] In an additional aspect of the present disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations including: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user operating a vehicle; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to a second field of view of a surrounding environment of the vehicle; determining a risk map based on the second image data; determining a gaze direction of the user based on the first image data; determining a correspondence between the risk map and the gaze direction; and controlling the vehicle based on the correspondence between the risk map and the gaze direction.

[0014] In an additional aspect of the present disclosure, an apparatus includes: a component for receiving first image data from a first camera oriented in a first direction having a first field of view facing a user operating a vehicle; a component for receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to a second field of view of a surrounding environment of the vehicle; determining a risk map based on the second image data; a component for determining a gaze direction of the user based on the first image data; a component for determining a correspondence between the risk map and the gaze direction; and a component for controlling the vehicle based on the correspondence between the risk map and the gaze direction.

[0015] In an additional aspect of the present disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations including: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user operating a vehicle; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to a second field of view of a surrounding environment of the vehicle; determining a risk map based on the second image data; determining a gaze direction of the user based on the first image data; determining a correspondence between the risk map and the gaze direction; and controlling the vehicle based on the correspondence between the risk map and the gaze direction.

[0016] In an additional aspect of the present disclosure, a vehicle device includes: a vehicle frame; a first camera, the first camera attached to the vehicle frame and arranged to capture a first field of view, the first field of view including a portion of a cabin enclosed by the vehicle frame on at least some sides; a second camera, the second camera attached to the vehicle frame and arranged to capture a second field of view corresponding to at least a portion of the surrounding environment of the vehicle frame; and a processing system, the processing system coupled to the first camera and the second camera. The processing system may include at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations, the operations including: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user operating the vehicle; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to a second field of view of the surrounding environment of the vehicle; determining a risk map based on the second image data; determining a gaze direction of the user based on the first image data; determining a correspondence between the risk map and the gaze direction; and controlling the vehicle based on the correspondence between the risk map and the gaze direction.

[0017] In an additional aspect of the present disclosure, a vehicle device includes: a vehicle frame; a first camera, the first camera attached to the vehicle frame and arranged to capture a first field of view, the first field of view including a portion of a cabin enclosed by the vehicle frame on at least some sides; a second camera, the second camera attached to the vehicle frame and arranged to capture a second field of view corresponding to at least a portion of the surrounding environment of the vehicle frame; and a processing system, the processing system coupled to the first camera and the second camera. The processing system may include at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations, which include: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to the user's field of view; determining an area of ​​interest based on the second image data; determining a gaze direction of the user based on the first image data; and determining a concentration score based on a correspondence between the area of ​​interest and the gaze direction.

[0018] The features and technical advantages of the examples according to the present disclosure have been outlined quite extensively above so that the following specific embodiments may be better understood. Additional features and advantages will be described below. The disclosed concepts and specific examples may be easily used as a basis for modifying or designing other structures for achieving the same purpose of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. When considered in conjunction with the accompanying drawings, the characteristics of the concepts disclosed herein (both their organization and methods of operation) and the associated advantages will be better understood according to the following description. Each of the figures in the accompanying drawings is provided for the purpose of illustration and description, and not as a definition of the limitations of the claims.

[0019] Although various aspects and implementations are described in this application by way of illustration of some examples, it will be appreciated by those skilled in the art that additional implementations and use cases may be generated in many different arrangements and scenarios. The innovations described herein may be implemented across many different platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, aspects and / or uses may be generated via integrated chip implementations and other non-module component-based devices (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail / purchase devices, medical devices, devices that enable artificial intelligence (AI), etc.). Although some examples may or may not specifically point to use cases or applications, the applicability of a wide variety of the described innovations may occur.

[0020] Implementations may range from chip-level or modular components to non-modular, non-chip-level implementations, and further to aggregate, distributed, or original equipment manufacturer (OEM) devices or systems incorporating one or more aspects of the described innovations. In some practical environments, devices incorporating the various aspects and features described may also necessarily include additional components and features for implementing and practicing the various aspects claimed and described. For example, the transmission and reception of wireless signals necessarily include multiple components for analog and digital purposes (e.g., hardware components, including antennas, radio frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders / summers, etc.). The innovations described herein are intended to be practiced in a variety of devices, chip-level components, systems, distributed arrangements, end-user devices, etc., having different sizes, shapes, and configurations.

[0021] In various implementations, the techniques and apparatus may be used in wireless communication networks, such as code division multiple access (CDMA) networks, time division multiple access (TDMA) networks, frequency division multiple access (FDMA) networks, orthogonal FDMA (OFDMA) networks, single carrier FDMA (SC-FDMA) networks, LTE networks, GSM networks, fifth generation (5G) or new radio (NR) networks (sometimes referred to as “5G NR” network, system or device) and other communication networks. As described in this document, the terms “network” and “system” may be used interchangeably.

[0022] A CDMA network may implement, for example, a radio technology such as Universal Terrestrial Radio Access (UTRA), cdma2000, etc. UTRA includes Wideband CDMA (W-CDMA) and Low Chip Rate (LCR). CDMA2000 covers IS-2000, IS-95, and IS-856 standards.

[0023] For example, a TDMA network may implement a radio technology such as the Global System for Mobile Communications (GSM). The 3rd Generation Partnership Project (3GPP) defines the standard for the GSM EDGE (Enhanced Data Rates for GSM Evolution) Radio Access Network (RAN), also known as GERAN. GERAN is the radio component of GSM / EDGE together with a network connecting base stations (e.g., Ater and Abis interfaces) and base station controllers (A interfaces, etc.). The radio access network represents the component of the GSM network through which phone calls and packet data are routed from the public switched telephone network (PSTN) and the Internet to subscriber phones (also known as user terminals or user equipment (UE)) and from subscriber phones to the PSTN and the Internet. The network of a mobile phone operator may include one or more GERANs, which may be coupled with the UTRAN in the case of a UMTS / GSM network. Additionally, the operator network may also include one or more LTE networks, or one or more other networks. Various different network types may use different radio access technologies (RATs) and RANs.

[0024] An OFDMA network may implement a radio technology such as Evolved UTRA (E-UTRA), Institute of Electrical and Electronics Engineers (IEEE) 802.11, IEEE 802.16, IEEE 802.20, flash-OFDM, etc. UTRA, E-UTRA, and GSM are part of the Universal Mobile Telecommunications System (UMTS). Specifically, Long Term Evolution (LTE) is a version of UMTS that uses E-UTRA. UTRA, E-UTRA, GSM, UMTS, and LTE are described in documents provided by an organization named "3rd Generation Partnership Project" (3GPP), and cdma2000 is described in documents from an organization named "3rd Generation Partnership Project 2" (3GPP2). 5G networks include diverse deployments, diverse spectrum, and diverse services and devices that can be implemented using a unified air interface based on OFDM.

[0025] The present disclosure may describe certain aspects with reference to LTE, 4G or 5G NR technology; however, the description is not intended to be limited to a particular technology or application, and one or more aspects described with reference to one technology may be understood to be applicable to another technology. Additionally, one or more aspects of the present disclosure may relate to shared access to wireless spectrum between networks using different radio access technologies or radio air interfaces.

[0026] Devices, networks, and systems may be configured to communicate via one or more portions of the electromagnetic spectrum. The electromagnetic spectrum is typically subdivided into various categories, bands, channels, etc. based on frequency / wavelength. In 5G NR, two initial operating bands have been identified as frequency ranges designated FR1 (410MHz-7.125GHz) and FR2 (24.25GHz-52.6GHz). Frequencies between FR1 and FR2 are typically referred to as mid-band frequencies. Although a portion of FR1 is greater than 6GHz, FR1 is often (interchangeably) referred to as the "sub-6GHz" band in various documents and articles. Similar naming issues sometimes arise for FR2, which is often (interchangeably) referred to as the "millimeter wave" (mmWave) band in documents and articles, although it is different from the extremely high frequency (EHF) band (30GHz-300GHz) identified as the "mmWave" band by the International Telecommunication Union (ITU).

[0027] In view of the above aspects, unless otherwise specifically stated, it should be understood that if used in this article, the term "below 6 GHz" and the like can broadly represent frequencies that can be less than 6 GHz, can be within FR1, or can include mid-band frequencies. In addition, unless otherwise specifically stated, it should be understood that if used in this article, the term "mmWave" and the like can broadly represent frequencies that can include mid-band frequencies, can be within FR2, or can be within the EHF band.

[0028] 5G NR devices, networks, and systems can be implemented using optimized OFDM-based waveform features. These features may include scalable parameter sets and transmit time intervals (TTIs); a common flexible framework that efficiently multiplexes services and features using dynamic, low-latency time division duplex (TDD) designs or frequency division duplex (FDD) designs; and advanced wireless technologies such as massive multiple-input multiple-output (MIMO), robust mmWave transmission, advanced channel decoding, and device-centric mobility. The scalability of parameter sets in 5G NR and the scaling of subcarrier spacing can efficiently address the operation of various services across different spectrums and different deployments. For example, in various outdoor and macro coverage deployments of less than 3GHz FDD or TDD implementations, the subcarrier spacing may appear at 15kHz, such as over 1MHz, 5MHz, 10MHz, 20MHz, etc. bandwidths. For various other outdoor and small cell coverage deployments of TDD greater than 3GHz, the subcarrier spacing may appear at 30kHz over 80MHz / 100MHz bandwidths. For various other indoor broadband implementations, using TDD on the unlicensed portion of the 5GHz band, the subcarrier spacing may occur at 60kHz over a 160MHz bandwidth. Finally, for various deployments transmitting over mmWave components at 28GHz TDD, the subcarrier spacing may occur at 120kHz over a 500MHz bandwidth.

[0029] For clarity, certain aspects of the devices and techniques may be described below with reference to example 5G NR implementations or in a 5G-centric manner, and 5G terminology may be used as illustrative examples in portions of the description below; however, the description is not intended to be limited to 5G applications.

[0030] Furthermore, it should be understood that in operation, a wireless communication network adapted according to the concepts herein can operate using any combination of licensed spectrum or unlicensed spectrum depending on load and availability. Therefore, it will be apparent to one of ordinary skill in the art that the systems, devices, and methods described herein can be applied to other communication systems and applications beyond the specific examples provided.

[0031] In the following description, a large number of specific details (such as examples of specific components, circuits and processes) are set forth in order to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intermediate components or circuits. In addition, in the following description and for the purpose of explanation, specific terms are set forth in order to provide a thorough understanding of the present disclosure. However, it will be understood by those skilled in the art that these specific details may not be required to implement the teachings disclosed herein. In other examples, in order to avoid blurring the teachings of the present disclosure, known circuits and devices are shown in block diagram form.

[0032] Some portions of the following detailed description are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, procedures, logic blocks, processes, etc. are conceived as a self-consistent sequence of steps or instructions leading to a desired result. These steps are steps requiring physical manipulations of physical quantities. Typically, although not necessarily, these physical quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

[0033] In the accompanying drawings, a single box may be described as performing one or more functions. The one or more functions performed by the box may be performed in a single component or across multiple components, and / or may be performed using hardware, software, or a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, the following generally describes various exemplary components, blocks, modules, circuits, and steps in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of the present disclosure. Moreover, the example device may include components other than those shown, including known components such as processors, memories, etc.

[0034] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout this application, discussions utilizing terms such as "access," "receive," "transmit," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "set," "generate," etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission, or display devices.

[0035] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more parts that can implement at least some portions of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus may include a device or a portion of a device for performing the described operations.

[0036] As used herein, including in the claims, the term "or" used in a list of two or more items means that any one of the listed items may be employed alone, or any combination of two or more of the listed items may be employed. For example, if a composition is described as containing components A, B, or C, the composition may contain A alone; B alone; C alone; A and B combined; A and C combined; B and C combined; or A, B, and C combined.

[0037] Additionally, as used herein (including in the claims), “or” as used in a list of items beginning with “at least one of” indicates a disjunct list, so that, for example, a list of “at least one of A, B, or C” means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) or any combination of any of these.

[0038] In addition, as used herein, the term "substantially" is defined as being largely, but not necessarily completely, what is specified (and including what is specified; for example, substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by one of ordinary skill in the art. In any disclosed implementation, the term "substantially" may be replaced with "within [percentage] of" what is specified, where percentages include 0.1%, 1%, 5%, or 10%.

[0039] In addition, as used herein, unless otherwise specified, relative terms may be understood as being relative to a reference by a certain amount. For example, terms such as "higher" or "lower" or "more" or "less" may be understood as being higher, lower, more or less than a reference value by a threshold amount. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] A further understanding of the nature and advantages of the present disclosure may be achieved by reference to the following drawings. In the drawings, similar components or features may have the same reference label. If only the first reference label is used in the specification, the description applies to any of the similar components having the same first reference label, regardless of the second reference label.

[0041] Figure 1 is a perspective view of a motor vehicle having a driver monitoring system according to an embodiment of the present disclosure.

[0042] Figure 2 A block diagram of an example image processing configuration for a vehicle is shown according to one or more aspects of the present disclosure.

[0043] Figure 3 is a block diagram illustrating details of an example wireless communication system in accordance with one or more aspects.

[0044] Figure 4Ais a block diagram illustrating a system for determining operator attentiveness according to one or more aspects.

[0045] Figure 4B is a block diagram illustrating a system for determining operator concentration according to one or more aspects of the present disclosure.

[0046] Figure 5 is a flow chart illustrating an example process for performing image processing in support of vehicle monitoring operations in accordance with one or more aspects.

[0047] Fig. 6A is a block diagram illustrating a region of interest generator using an object detector according to one or more aspects.

[0048] Figure 6B is a block diagram illustrating a region of interest generator using a risk map according to one or more aspects.

[0049] Figure 7 is an illustration of a scenario having high risk areas that can be reflected in a risk map according to one or more aspects.

[0050] Figure 8 is an illustration of a vehicle display showing a vigilance alert according to one or more aspects.

[0051] The same reference numbers and designations in different drawings indicate the same elements. DETAILED DESCRIPTION

[0052] The specific embodiments described below in conjunction with the accompanying drawings are intended as descriptions of various configurations and are not intended to limit the scope of the present disclosure. Instead, the specific embodiments include specific details for providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in every case, and in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0053] The present disclosure provides systems, devices, methods, and computer-readable media that support driver monitoring and vehicle monitoring that can be used in smart vehicles and / or automated vehicles. In a particular embodiment, the vehicle can monitor the user's concentration and score the user's concentration by: determining high-risk portions of the surrounding environment, and analyzing the user's gaze direction to determine whether the user spends sufficient time monitoring high-risk portions of the surrounding environment while driving. The vehicle can warn the driver of missing high-risk areas, control the vehicle in the absence of a response from the driver, and / or report the driver's score to a remote facility.

[0054] Particular implementations of the subject matter described in this disclosure may be implemented to achieve one or more of the following potential advantages or benefits. In some aspects, this disclosure provides techniques for improving the safety of drivers operating vehicle systems and reducing vehicle collisions that result in damage to life and property.

[0055] Figure 1 1 is a perspective view of a motor vehicle with a driver monitoring system according to an embodiment of the present disclosure. The vehicle 100 may include a forward-facing camera 112 mounted inside the cabin for viewing through the windshield 102. The vehicle may also include a cabin-facing camera 114 mounted inside the cabin for viewing toward the occupants of the vehicle 100 (and in particular the driver of the vehicle 100). Although one set of mounting locations for the cameras 112 and 114 is shown for the vehicle 100, other mounting locations may also be used for the cameras 112 and 114. For example, one or more cameras may be mounted on one of the driver or passenger B-pillars 126 or one of the driver or passenger C-pillars 128, such as near the top of the pillars 126 or 128. As another example, one or more cameras may be mounted on the front of the vehicle 100, such as behind the radiator grille 130 or integral with the bumper 132. As another example, one or more cameras may be mounted as part of the driver or passenger side mirror assembly 134.

[0056] The camera 112 may be oriented such that the field of view of the camera 112 captures the scene in front of the vehicle 100 in the direction that the vehicle 100 is moving when in the driving mode or forward direction. In some embodiments, an additional camera may be located at the rear of the vehicle 100 and oriented such that the field of view of the additional camera captures the scene behind the vehicle 100 in the direction that the vehicle 100 is moving in the reverse direction. Although embodiments of the present disclosure may be described with reference to a “forward facing” camera (reference camera 112), aspects of the present disclosure may be similarly applied to a “rear facing” camera facing the reverse direction of the vehicle 100. Thus, the benefits obtained when the operator is driving the vehicle 100 in the forward direction may also be obtained when the operator is driving the vehicle 100 in the reverse direction.

[0057] Furthermore, while embodiments of the present disclosure may be described with reference to a "forward-facing" camera (reference camera 112), aspects of the present disclosure may be similarly applicable to input received from an array of cameras mounted around the vehicle 100 to provide a larger field of view, which may be as large as about 360 degrees parallel to the ground and / or as large as about 180 degrees in a vertical direction approximately perpendicular to the ground. For example, additional cameras may be mounted around the outside of the vehicle 100, such as on or integrated into a door, on or integrated into a wheel, on or integrated into a bumper, on or integrated into a bumper, on or integrated into a hood, and / or on or integrated into a roof.

[0058] The camera 114 may be oriented such that the field of view of the camera 114 captures the scene in the cabin of the vehicle and includes a user operator of the vehicle, and in particular the face of the user operator of the vehicle, with sufficient detail to discern the direction of the user operator's gaze.

[0059] One of the cameras 112 and 114 may include one, two or more image sensors, such as including a first image sensor. When there are multiple image sensors, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide-angle image sensor, and the second image sensor may be a telephoto image sensor. In another example, the first sensor is configured to obtain an image through a first lens having a first optical axis, and the second sensor is configured to obtain an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification, and the second lens may have a second magnification different from the first magnification. This configuration may occur in a camera module with a lens group, wherein multiple image sensors and associated lenses are located in offset positions within the camera module. Additional image sensors with larger, smaller, or the same field of view may be included.

[0060] Each image sensor may include components for capturing data representing a scene, such as an image sensor (including a charge coupled device (CCD), a Bayer filter sensor, an infrared (IR) detector, an ultraviolet (UV) detector, a complementary metal oxide semiconductor (CMOS) sensor), and / or a time of flight detector. The device may also include one or more components for gathering and / or focusing light into one or more image sensors (including simple lenses, compound lenses, spherical lenses, and aspherical lenses). These components may be controlled to capture a first, second, and / or more image frames. The image frames may be processed, such as by a fusion operation, to form a single output image frame, and the output image frame may be further processed according to various aspects described herein.

[0061] As used herein, an image sensor may refer to the image sensor itself and any specific other components coupled to the image sensor for generating image frames for processing by an image signal processor or other logic circuitry or for storage in a memory (whether a short-term buffer or a long-term non-volatile memory). For example, an image sensor may include other components of a camera, including a shutter, a buffer, or other readout circuitry for accessing individual pixels of the image sensor. An image sensor may also refer to an analog front end or other circuitry for converting an analog signal into a digital representation of an image frame, which is provided to a digital circuit coupled to the image sensor.

[0062] Figure 2A block diagram of an example image processing configuration for a vehicle according to one or more aspects of the present disclosure is shown. The processing system 100 may include or otherwise be coupled to an image signal processor 212 for processing image frames from one or more image sensors (such as a first image sensor 201, a second image sensor 202, and a depth sensor 240). In some implementations, the vehicle 100 also includes or is coupled to a processor (e.g., a CPU) 204 and a memory 206 storing instructions 208. The processing system 100 may also include or be coupled to a display 214 and an input / output (I / O) component 216. The I / O component 216 may be used to interact with a user, such as a touch screen interface and / or physical buttons. The I / O component 216 may also include a network interface for communicating with other devices such as other vehicles, an operator's mobile device, and / or a remote monitoring system. The network interface may include one or more of a wide area network (WAN) adapter 252, a local area network (LAN) adapter 253, and / or a personal area network (PAN) adapter 254. An example WAN adapter 252 is a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 253 is an IEEE 802.11 WiFi wireless network adapter. An example PAN adapter 254 is a Bluetooth wireless network adapter. Each of the adapters 252, 253, and / or 254 may be coupled to an antenna including multiple antennas configured for primary and diversity reception and / or configured to receive a particular frequency band. The vehicle 100 may also include or be coupled to a power source 218, such as a battery or an alternator. The vehicle 100 may also include or be coupled to a Figure 2 Additional features or components not shown in the figure. In one example, a wireless interface, which may include one or more transceivers and an associated baseband processor, may be coupled to or included in a WAN adapter 252 for wireless communication devices. In another example, an analog front end (AFE) for converting analog image frame data to digital image frame data may be coupled between image sensors 201 and 202 and image signal processor 212.

[0063] The vehicle 100 may include a sensor hub 250 for interfacing with sensors to receive data about the movement of the vehicle 100, data about the environment surrounding the vehicle 100, and / or other non-camera sensor data. An example non-camera sensor is a gyroscope, i.e., a device configured to measure rotation, orientation, and / or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, i.e., a device configured to measure acceleration, which can also be used to determine speed and distance traveled by appropriately integrating the measured acceleration, and one or more of the acceleration, speed, and / or distance may be included in the generated motion data. In further examples, the non-camera sensor may be a global positioning system (GPS) receiver, a light detection and ranging (LiDAR) system, a radio detection and ranging (RADAR) system, or other ranging system.

[0064] Image signal processor (ISP) 212 can receive image data such as for forming an image frame. In one embodiment, the local bus connection couples image signal processor 212 to the Figure 1 The first camera 203 corresponding to the camera 112 and the Figure 1 The image sensors 201 and 202 of the second camera 205 corresponding to the camera 114 are shown in FIG. 1 . In another embodiment, a wire interface can couple the image signal processor 212 to the external image sensor. In yet another embodiment, a wireless interface can couple the image signal processor 212 to the image sensors 201 and 202.

[0065] The first camera 203 may include a first image sensor 201 and a corresponding first lens 231. The second camera 205 may include a second image sensor 202 and a corresponding second lens 232. Each of the lenses 231 and 232 may be controlled by an associated auto focus (AF) algorithm 233 executed in the ISP 212, which adjusts the lenses 231 and 232 to focus on a specific focal plane at a certain scene depth from the image sensors 201 and 202. The AF algorithm 233 may be assisted by a depth sensor 240.

[0066] The first image sensor 201 and the second image sensor 202 are configured to capture one or more image frames. The lenses 231 and 232 focus light at the image sensors 201 and 202, respectively, through one or more apertures for receiving light, one or more shutters for blocking light when outside an exposure window, one or more color filter arrays (CFA) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurement results into digital information, and / or other suitable components for imaging.

[0067] In some embodiments, the image signal processor 212 may execute instructions from a memory, such as instructions 208 from the memory 206, instructions stored in a separate memory coupled to or included in the image signal processor 212, or instructions provided by the processor 204. In addition or alternatively, the image signal processor 212 may include specific hardware (such as one or more integrated circuits (ICs)) configured to perform one or more operations described in the present disclosure. For example, the image signal processor 212 may include one or more image front ends (IFEs) 235, one or more image post-processing engines (IPEs) 236, and one or more automatic exposure compensation (AEC) 234 engines. The AF 233, AEC 234, IFE 235, IPE 236 may each include dedicated circuitry embodied as software code executed by the ISP 212 and / or a combination of hardware within the ISP 212 and software code executed on the ISP.

[0068] In some implementations, the memory 206 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 208 to perform all or part of one or more operations described in the present disclosure. In some implementations, the instructions 208 include a camera application (or other suitable application) to be executed during operation of the vehicle 100 for generating images or videos. The instructions 208 may also include other applications or programs executed for the vehicle 100, such as an operating system, a mapping application, or an entertainment application. The execution of the camera application, such as by the processor 204, may enable the vehicle 100 to generate images using the image sensors 201 and 202 and the image signal processor 212. The memory 206 may also be accessed by the image signal processor 212 to store processed frames or may be accessed by the processor 204 to obtain processed frames. In some embodiments, the vehicle 100 includes a system on a chip (SoC) that combines the image signal processor 212, the processor 204, the sensor hub 250, the memory 206, and the input / output component 216 into a single package.

[0069] In some embodiments, at least one of the image signal processor 212 or the processor 204 executes instructions to perform various operations described herein, including object detection, risk map generation, driver monitoring, and driver warning operations. For example, execution of the instructions may instruct the image signal processor 212 to start or end capturing an image frame or sequence of image frames. In some embodiments, the processor 204 may include one or more general-purpose processor cores 204A capable of executing scripts or instructions of one or more software programs (such as instructions 208 stored in the memory 206). For example, the processor 204 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or videos) stored in the memory 206.

[0070] When executing the camera application, the processor 204 may be configured to instruct the image signal processor 212 to perform one or more operations with reference to the image sensor 201 or 202. For example, the camera application may receive a command to start a video preview display, and upon receiving the command, capture and process a video including a sequence of image frames from one or more image sensors 201 or 202, and display the video on an information display in the cabin of the vehicle 100.

[0071] In some embodiments, the processor 204 may include an IC or other hardware (e.g., an artificial intelligence (AI) engine 224) in addition to the ability to execute software to cause the vehicle 100 to perform multiple functions or operations, such as those described herein. In some other embodiments, the vehicle 100 does not include a processor 204, such as when all of the described functionality is configured in the image signal processor 212.

[0072] In some embodiments, the display 214 may include one or more suitable displays or screens that allow user interaction and / or present items to the user (such as previews of image frames captured by the image sensors 201 and 202). In some embodiments, the display 214 is a touch-sensitive display. The I / O component 216 may be or include any suitable mechanism, interface, or device to receive input (such as commands) from the user and provide output to the user through the display 214. For example, the I / O component 216 may include (but is not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone, a speaker, a squeezable bezel, one or more buttons (e.g., a power button), a slider, a switch, etc.

[0073] Although shown as being coupled to each other via processor 204, components such as processor 204, memory 206, image signal processor 212, display 214, and I / O components 216 may be coupled to each other in various other arrangements, such as via one or more local buses, which arrangements are not shown for simplicity. Although image signal processor 212 is illustrated as being separate from processor 204, image signal processor 212 may be a core of processor 204 that is an application processor unit (APU), included in a system on a chip (SoC), or otherwise included in processor 204. Although reference is made to vehicle 100 in the examples herein to include aspects of the present disclosure, some device components may not be present. Figure 2 100 to prevent obscuring various aspects of the present disclosure. Additionally, other components, numbers of components, or combinations of components may be included in a suitable vehicle for performing various aspects of the present disclosure. Thus, the present disclosure is not limited to a particular device or component configuration, including the vehicle 100.

[0074] The vehicle 100 can communicate as a user equipment (UE) within the wireless network 300, such as through the WAN adapter 252. Figure 3 As shown in . Figure 3 3 is a block diagram illustrating details of an example wireless communication system according to one or more aspects. Wireless network 300 may, for example, include a 5G wireless network. As will be appreciated by those skilled in the art, Figure 3 The components appearing in are likely to have related counterparts in other network arrangements, including, for example, cellular-style network arrangements and non-cellular-style network arrangements (such as device-to-device or point-to-point or ad hoc network arrangements, etc.).

[0075] Figure 3The wireless network 300 illustrated in the figure includes a base station 305 and other network entities. A base station may be a station that communicates with a UE and may also be referred to as an evolved Node B (eNB), a next generation eNB (gNB), an access point, etc. Each base station 305 may provide communication coverage for a specific geographic area. In 3GPP, the term "cell" may refer to a specific geographic coverage area of ​​a base station or a base station subsystem serving the coverage area, depending on the context in which the term is used. In a specific implementation of the wireless network 300 herein, the base station 305 may be associated with the same operator or different operators (e.g., the wireless network 300 may include multiple operator wireless networks). Additionally, in a specific implementation of the wireless network 300 herein, the base station 305 may provide wireless communication using one or more frequencies (e.g., one or more frequency bands in a licensed spectrum, an unlicensed spectrum, or a combination thereof) in the same frequency as a neighboring cell. In some examples, a separate base station 305 or UE 315 may be operated by more than one network operating entity. In other examples, each base station 305 and UE 315 may be operated by a single network operating entity.

[0076] A base station may provide communication coverage for a macro cell or a small cell (e.g., a pico cell or a femto cell), or other type of cell. A macro cell generally covers a relatively large geographic area (e.g., several kilometers in radius) and may allow unrestricted access by UEs with a service subscription with a network provider. A small cell (such as a pico cell) will generally cover a relatively small geographic area and may allow unrestricted access by UEs with a service subscription with a network provider. A small cell (such as a femto cell) will generally also cover a relatively small geographic area (e.g., a home), and in addition to unrestricted access, may also provide restricted access by UEs associated with the femto cell (e.g., UEs in a closed subscriber group (CSG), UEs of users in a home, etc.). A base station for a macro cell may be referred to as a macro base station. A base station for a small cell may be referred to as a small cell base station, a pico base station, a femto base station, or a home base station. In Figure 3 In the example shown in , base stations 305d and 305e are conventional macro base stations, while base stations 305a to 305c are macro base stations implemented using one of three-dimensional (3D) MIMO, full-dimensional (FD) MIMO, or massive MIMO. Base stations 305a to 305c use their higher-dimensional MIMO capabilities to employ 3D beamforming in both elevation and azimuth to increase coverage and capacity. Base station 305f is a small cell base station, which can be a home node or a portable access point. The base station can support one or more (e.g., two, three, four, etc.) cells.

[0077] Wireless network 300 may support synchronous or asynchronous operation. For synchronous operation, base stations may have similar frame timing, and transmissions from different base stations may be approximately aligned in time. For asynchronous operation, base stations may have different frame timing, and transmissions from different base stations may not be aligned in time. In some cases, the network may be enabled or configured to handle dynamic switching between synchronous or asynchronous operation.

[0078] UE 315 is dispersed throughout the wireless network 300, and each UE may be stationary or mobile. It should be understood that although in the standards and specifications promulgated by 3GPP, mobile devices are often referred to as UEs, such devices may be additionally or otherwise referred to by those skilled in the art as mobile stations (MS), subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals (AT), mobile terminals, wireless terminals, remote terminals, handsets, terminals, user agents, mobile clients, clients, gaming devices, augmented reality devices, vehicle components, vehicle devices or vehicle modules, or some other suitable terminology.

[0079] Some non-limiting examples of mobile devices, such as those that may include one or more specific implementations of UE 315, include mobile phones, cellular (cell) phones, smart phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, laptop computers, personal computers (PCs), notebooks, netbooks, smart books, tablet computers, personal digital assistants (PDAs), and vehicles. Although UE 315a to 315j are specifically shown as vehicles, the vehicle may employ the communication configuration described with reference to any of UE 315a to 315k.

[0080] In one aspect, the UE may be a device including a universal integrated circuit card (UICC). In another aspect, the UE may be a device that does not include a UICC. In some aspects, a UE that does not include a UICC may also be referred to as an IoE device. Figure 3 The UEs 315a to 315d of the specific implementation illustrated in FIG are examples of mobile smartphone type devices accessing the wireless network 300. A UE may also be a machine specifically configured to implement connected communications, including machine type communications (MTC), enhanced MTC (eMTC), narrowband IoT (NB-IoT), and the like. Figure 3 UEs 315e to 315k illustrated in FIG. 3 are examples of various machines configured for communication that access the wireless network 300.

[0081] A mobile device, such as UE 315, may be able to communicate with any type of base station, whether macro, pico, femto, relay, etc. Figure 3 In the figure, the communication link (represented as a lightning ball) indicates the wireless transmission between the UE and the serving base station (which is the base station designated to serve the UE on the downlink or uplink) or the expected transmission between the base stations and the backhaul transmission between the base stations. The UE can operate as a base station or other network node in some scenarios. The backhaul communication between the base stations of the wireless network 300 can be carried out using wired or wireless communication links.

[0082] In operation, at the wireless network 300, base stations 305a-305c use 3D beamforming and collaborative spatial techniques (e.g., coordinated multipoint (CoMP) or multi-connectivity) to serve UEs 315a and 315b. Macro base station 305d performs backhaul communications with base stations 305a-305c and small cells (base station 305f). Macro base station 305d also transmits multicast services that are subscribed to and received by UEs 315c and 315d. Such multicast services may include mobile TV or streaming video, or may include other services for providing community information, such as weather emergencies or alerts, such as Amber alerts or gray alerts.

[0083] The wireless network 300 of the specific implementation supports mission critical communications with ultra-reliable and redundant links for mission critical devices such as UE 315e as drones. The redundant communication links with UE 315e include links from macro base stations 305d and 305e and small cell base station 305f. Other machine type devices such as UE 315f (thermometer), UE 315g (smart meter) and UE 315h (wearable device) can communicate directly with base stations such as small cell base station 305f and macro base station 305e through the wireless network 300, or in a multi-hop configuration by communicating with another user device that relays its information to the network, such as UE 315f communicating temperature measurement information to smart meter UE 315g, which then reports it to the network through small cell base station 305f. The wireless network 300 may also provide additional network efficiency through dynamic, low-latency TDD communications or low-latency FDD communications (eg, in a vehicle-to-vehicle (V2V) mesh network between UEs 315i - 315k communicating with a macro base station 305e ).

[0084] refer to Figure 1 , Figure 2 and Figure 3 Describe and Figure 1 , Figure 2 and Figure 3 Various aspects of the vehicle system shown in FIG. 1 may be used to determine user operator concentration. For example, according to various aspects of the present disclosure, operator concentration may be determined based on input received from cameras 112 and 114. Figure 4Ais a block diagram illustrating a system for determining operator concentration according to one or more aspects of the present disclosure.

[0085] The region of interest (ROI) generator 412 may receive input from the forward-facing camera 402. The ROI generator 412 may process the image frames captured by the camera 402 to determine a region within the scene in the field of view of the camera 402 that captures a portion or all of the surrounding environment of the vehicle. The determined region may be a region of interest because these areas should receive additional attention due to the higher likelihood of vehicle collisions associated with the region. The ROI generator 412 outputs a set of regions of interest, which may be represented as a set of bounding box coordinates for the ROI. For some input scenes, there may be no regions of interest, so that the set is an empty set, which may indicate that there is no risk to the operator to assess. For example, a polygon with N points may be represented as N pairs of x-coordinates and y-coordinates corresponding to pixels within the image frames captured by the camera 402. In some embodiments, the ROI may be determined based on depth information received from a depth sensor or from another depth determination system (such as a dual image sensor system). For example, the depth information may be used to separate objects by distance within the field of view of the forward-facing camera 402.

[0086] The gaze estimator 414 can receive input from the inward-facing camera 404. The gaze estimator 414 can process the image frames captured by the camera 404 to determine the direction of the gaze of the user operator based on, for example, the position of the operator's pupils, the shape of the operator's pupils as captured in the image frames, and / or other facial features of the operator. The gaze estimator 414 outputs a gaze direction, which can be represented as a gaze vector. For example, a three-dimensional (3D) vector can be represented by three values ​​corresponding to the magnitude in each of the x-direction, y-direction, and z-direction of a reference vector. In some embodiments, the gaze estimator 414 can also output a head position corresponding to the x-position, y-position, or x-position, y-position, z-position of the operator in the cabin of the vehicle.

[0087] The gaze analyzer 422 receives the gaze vector and the ROI set, and associates the operator's gaze with a specific portion of the field of view of the camera 402 facing forward. The gaze analyzer 422 can determine whether the operator has fully observed the scene in front of or around the vehicle based on the associated gaze vector and ROI. In some embodiments, this can involve determining whether the operator has observed each ROI in the ROI and how much attention the operator has given to each ROI. For example, the gaze analyzer 422 can determine that the operator's gaze direction corresponds to the dwell time of the ROI. If the dwell time is less than a certain threshold, it can be determined that the user is not focused enough on the ROI. In some embodiments, this can involve determining whether the operator has observed a sufficient number of the highest priority ROIs. In some embodiments, this can involve using artificial intelligence to compare the operator's behavior with the ROI to determine whether the operator's behavior fully examines the scene.

[0088] The output of the gaze analyzer 422 can be used to determine information about the operator, such as whether the operator is properly focused on the high risk area in the path of the vehicle during the operation of the vehicle. In some embodiments, the gaze analyzer 422 can output a concentration score that reflects one or more ROIs in a certain percentage of ROIs observed by the operator and / or the amount of time the operator is looking at the ROI. The concentration score can indicate characteristics of the operator, such as whether the operator is sleepy or alert or whether the operator is skilled or unskilled. In some embodiments, the concentration score can be compared with the operator's baseline score to separate the operator's skills from sleepiness.

[0089] In one example operation, facial recognition can be used to determine an operator identification (ID) based on image data from the inward-facing camera 404, the operator ID can be used to look up a corresponding operator profile having an average concentration score, and the output of the gaze analyzer 422 is compared to the average concentration score. When the determined concentration score is below the average concentration score by more than a threshold amount, it can be determined that the operator is drowsy. By updating the average concentration score with the determined concentration score, the operator's profile can be updated over time to reflect the operator's changing skill level.

[0090] The warning actuator 432 can receive the output of the gaze analyzer 422 and determine the appropriate action to be performed within the vehicle. One or more criteria can be applied to the output of the gaze analyzer 422, wherein the criteria specify conditions to be met and actions to be performed based on matching these conditions. For example, the warning actuator 432 can determine that the operator is drowsy and display a warning on the dashboard of the vehicle based on the operator's concentration score being below a threshold level (or below a threshold value expected for the operator). As another example, the warning actuator 432 can control vehicle operation based on the criteria, such as applying a speed limiter to the vehicle or initiating a safety stop process when the operator is determined to be negligent.

[0091] The warning actuator 432 may receive additional information from the gaze analyzer 422, such as ROIs to which the operator does not or does not provide sufficient attention. For example, the gaze analyzer 422 may output an identifier of the ROI determined by the ROI generator 412, provide identification information such as x-coordinates, y-coordinates, etc. for the ROI, and / or provide a labeled version of the image frame captured by the forward-facing camera 402 to annotate the missing ROI in the captured scene. The warning actuator 432 may output information indicating the missing ROI to the operator. In some embodiments, the warning actuator 432 may maintain a count of the missing ROIs, and when the number of missing ROIs exceeds a threshold number within a specific time period, the operator may be instructed to take a break and / or the vehicle may be controlled to force a break by disabling vehicle propulsion.

[0092] As a supplement or alternative to the display output, the warning actuator 432 can provide feedback to the user. For example, the warning actuator 432 can communicate with a tactile device in the operator's seat to activate vibration, thereby providing tactile stimulation to the operator to make the operator more focused. In some embodiments, when the vehicle approaches a threshold distance of the ROI and the user is not looking at the ROI, the operator's seat can vibrate to warn the operator of potential danger. In some embodiments, the display in the cockpit can also display an image of the scene around the car, in which the missing ROI is highlighted in combination with tactile feedback to attract the operator's attention to a specific ROI. In some embodiments, the tactile device can be part of a mobile device carried by the operator in their pocket, where the warning actuator 432 is interfaced with the mobile device via a wireless interface such as a PAN adapter 254. In some embodiments, a head-up display (HUD) can attract the operator's attention toward the missing ROI or where the operator's attention should be focused.

[0093] The warning actuator 432 can also communicate with the user through a wireless communication system such as Figure 3For example, the warning actuator 432 can send the operator's concentration score and / or other information described with reference to FIG. 4 (including image data, ROI data, gaze vector, marked image frame, etc.) to a remote monitoring station, a user device, a nearby vehicle, and / or other UEs.

[0094] In another embodiment, the gaze analyzer 422 can be directly in the Figure 4B Operate on the risk map shown in . Figure 4B 4 is a block diagram illustrating a system for determining operator concentration according to one or more aspects of the present disclosure. A gaze analyzer 422 receives a risk map output by a risk map generator 442 and compares a gaze vector to the risk map to determine a result of a set of criteria reflected in a concentration score. Figure 4B In an embodiment within the example of , the gaze analyzer 422 can directly associate the gaze direction with the probability distribution represented by the risk map.

[0095] Figure 5 5 is a flow chart illustrating an example process 500 for performing image processing in support of vehicle monitoring operations according to one or more aspects. The operations of process 500 may be performed by a UE (such as the one described above with reference to FIG. Figure 3 4). For example, the example operations (also referred to as "blocks") of process 500 may enable a vehicle UE to support image processing and vehicle monitoring operations.

[0096] In block 502, first image data is received from a first camera, the first image data representing a scene including an action of a vehicle operator. The first camera may be arranged to capture such first image data by being attached to a vehicle frame and pointing in a direction toward a driver's seat. In some embodiments, the first camera may be configured to track the operator's face to improve the reliability of a gaze determination made based on the first image data.

[0097] In block 504, second image data is received from a second camera, the second image data representing a scene including at least a portion of the surroundings of the vehicle. The second camera may be arranged to capture such second image data by being attached to the vehicle frame and pointing in a direction away from the vehicle, which may be a direction different from the direction pointed by the first camera. The second camera may be attached to the vehicle frame, for example, by being attached to a radiator grille, to a bumper, to a door, to a side mirror, to a hood, to a roof, to a wheel, to a windshield, or to an A-pillar, B-pillar, or C-pillar.

[0098] In block 506, a set of regions of interest are determined based on the second image data; the regions of interest may reflect areas of the surrounding environment that are high risk areas for operating the vehicle. Example high risk areas are obstructed viewing areas, such as parked vehicles or blind spots, because objects may appear from behind the obstructed viewing area without warning. Operators should be aware of such high risk areas and proactively check these high risk areas.

[0099] In block 508, a gaze direction of the operator is determined based on the first image data. The first image may be analyzed to detect a face, identify an eye portion of the face, and then determine the gaze direction based on the eye portion. For example, the eye portion of the image data may be adjusted for the viewing angle from the camera so that the angle of the pupil relative to the surroundings of the vehicle may be determined.

[0100] In block 510, the gaze direction may be compared to the region of interest to determine a concentration score. In some embodiments, the concentration score may be a binary value (e.g., yes or no) indicating whether the operator has checked the region of interest for a sufficient amount of time (e.g., the total dwell time of all gazes or the number of times the gaze direction returns to the region of interest) to safely pass through the high risk area. In some embodiments, the concentration score may be a proportional value from 1 to 10 or 1 to 100 indicating a concentration rating based on the correspondence between the gaze directions and the region of interest determined over a period of time.

[0101] In certain embodiments, rules specifying criteria and consequences may be applied to the concentration score. For example, the criterion may be that the concentration score is below a threshold level and the consequence may be that an action is performed to alert the operator of their concentration score. The processing system may also take other actions in the vehicle to reduce the consequences of low concentration or to increase the operator's concentration. For example, the processing system may begin to reduce the vehicle speed, set a limit on the vehicle speed, or prohibit the propulsion of the vehicle. As another example, the processing system may flash the interior lights and / or provide tactile feedback to the operator to increase the operator's alertness.

[0102] In certain embodiments, these rules can provide an escalating response to operator inattention, such as by increasing the severity of the action if a less severe action is unsuccessful in increasing concentration. For example, an initial alert can be presented on the instrument panel, which can cause haptic feedback to be applied if unsuccessful, which can cause the brakes to be applied to stop the vehicle if unsuccessful. When the operator's gaze is detected to be toward the area of ​​interest, the escalation of the event can be terminated.

[0103] In some embodiments, the concentration score can be obtained by (such as Figure 3The scores transmitted may be used, for example, to record a history of the operator's driving behavior. The scores transmitted may be used, for example, to warn nearby pedestrians by communicating with the negligent operator's UE, to warn nearby officials by communicating with the negligent operator's police UE, and / or to alert nearby vehicles of the negligent operator so that other operators may consider the vehicle as a high-risk area of ​​interest.

[0104] Fig. 6A and Figure 6B An example technique for determining a region of interest as described in block 506 is shown in FIG. Fig. 6A is a block diagram illustrating a region of interest generator using an object detector according to one or more aspects. Fig. 6A In the ROI detector 412 of the vehicle, the object detector 602 can receive image data and / or depth data to detect objects in the surrounding environment of the vehicle. These objects can be provided to the object classifier 604, which classifies the objects into categories such as cars, people, trash cans, bicycles, animals, etc. The list of objects detected by the detector 602 and the classification of these objects determined by the classifier 604 can be provided to the risk modeler 606. The risk modeler 606 can determine whether the object or the vicinity of the object represents a risk to the vehicle based on the classification and the location of the object in the surrounding environment. Various areas of the surrounding environment can be identified as regions of interest and output from the ROI detector by the risk modeler 606. The ROI output can be a set of coordinates of a polygon (such as a rectangle) in an image frame of the specified image data, which encloses an area of ​​the image frame to indicate a high risk area. The ROI output can alternatively be a single point in the image frame.

[0105] The region of interest may also be determined using a risk map, which is a two-dimensional or three-dimensional data structure representing the surroundings of the vehicle and values ​​of perceived risk to vehicle operation at various locations in the surroundings. Figure 6B 6 is a block diagram illustrating a region of interest generator using a risk map according to one or more aspects. The machine learning model can be trained offline and deployed in a vehicle, such as within a ROI identifier 616, as a risk map generator that can generate a risk map based on one or more of image data, depth data, and vehicle state during operation of the vehicle by an operator. The trained model can be generated based on data collected from the vehicle during operation, which can include image data, depth data, and / or other sensor data.

[0106] The risk map may be used by the ROI identifier 616 to determine regions of interest from the risk map that an operator should monitor for safety reasons.

[0107] Although the depth data is described as being used for detecting Fig. 6A Object or determination Figure 6B ROI detector 412 may be an optional input to the risk map in , but other or additional data may be used instead of or in addition to image data or depth data. For example, ROI detector 412 may use input from one or more sensors of different modalities (e.g., camera, radar, or lidar). In some embodiments, ROI detector 412 may use data obtained from the cloud or other nearby vehicles.

[0108] Figure 7 An example high risk region of interest to be detected by the ROI detector 412 is shown in FIG. Figure 7 is an illustration of a scenario with high risk areas that may be reflected in a risk map according to one or more aspects. A vehicle 100 traveling on a street may have an obstructed view due to a stationary object 702 and / or a moving object 704. Therefore, each of the objects 702 and 704 may create an area of ​​interest 712 and 714, respectively, that the operator of the vehicle 100 should approach with caution. For example, the operator should visually inspect the areas in the areas of interest 712 and 714 more frequently than other areas and / or approach the street areas corresponding to the areas of interest 712 and 714 slowly due to the possibility of unseen pedestrians 722 crossing in front of the vehicle 100. If the user is not looking in the direction of the areas of interest 712 and 714, the vehicle 100 may warn the user or take action to slow down the vehicle 100 when the vehicle 100 approaches the areas of interest 712 and 714, and then allow the vehicle to return to normal speed after passing the areas of interest 712 and 714.

[0109] Figure 8 An example of such an operator warning is shown in . Figure 8 804 is an illustration of a vehicle display showing a vigilance alert according to one or more aspects. An operator of vehicle 100 may have a view of a display 802 on a dashboard of the vehicle. The dashboard or other display of vehicle 100, such as a head-up display (HUD), may display a representation of the vehicle's surroundings, such as image data from a forward-facing camera and / or a generated scene model of the surroundings. The vehicle may display an alert 804 corresponding to an area of ​​interest that has been detected but not yet fully inspected by the operator. Alert 804 may grow in urgency as vehicle 100 approaches the corresponding area of ​​interest, such as by changing color and / or flashing and / or increasing intensity. Alert 804 may be cleared when the operator sufficiently gazes in the direction of the corresponding area of ​​interest.

[0110] It should be noted that reference Figure 5 One or more blocks (or operations) described in FIG. 1 may be combined with one or more blocks (or operations) described in another figure in the reference figure. Figure 5One or more boxes (or operations) can be combined with Figure 3 As another example, Figure 5 One or more boxes associated with Figure 2 As another example, with Figure 5 The associated one or more boxes may be associated with FIG. Figure 8 Additionally or alternatively, the above reference Figures 1 to 3 One or more operations described may be performed with reference to FIG. Figure 8 Combines one or more operations described.

[0111] In one or more aspects, the technology for supporting vehicle operations may include additional aspects, such as any single aspect or any combination of aspects described below or in combination with one or more other processes or devices described elsewhere herein. In a first aspect, image processing (such as for vehicle monitoring operations) may include a device comprising one or more of the following: a vehicle frame; a first camera, the first camera is attached to the vehicle frame and arranged to capture a first field of view, the first field of view including a portion of the cabin enclosed by the vehicle frame on at least some sides; a second camera, the second camera is attached to the vehicle frame and arranged to capture a second field of view corresponding to at least a portion of the surrounding environment of the vehicle frame; and a processing system, the processing system is coupled to the first camera and the second camera. Additionally, the device may be performed or operated according to one or more aspects as described below. In some specific implementations, the device includes a wireless device, such as a UE. In some specific implementations, the device may include at least one processor, and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the device. In some other implementations, the apparatus may include a non-transitory computer-readable medium having program code recorded thereon, and the program code may be executable by a computer to cause the computer to perform the operations described herein with reference to the apparatus. In some implementations, the apparatus may include one or more components configured to perform the operations described herein. In some implementations, the method of wireless communication may include one or more operations described herein with reference to the apparatus.

[0112] In a second aspect, in combination with the first aspect, the processing system of the device is configured to perform operations, the operations comprising: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to the field of view of the user; determining an area of ​​interest based on the second image data; determining a gaze direction of the user based on the first image data; and determining a concentration score based on a correspondence between the area of ​​interest and the gaze direction.

[0113] In a third aspect, in combination with one or more of the first aspect or the second aspect, the first image data comprises a first plurality of image frames captured within a duration, and wherein the second image data comprises a second plurality of image frames captured within the duration.

[0114] In a fourth aspect, in combination with one or more of the first to third aspects, determining the concentration score includes determining a proportion of the duration in which the gaze direction of the user is associated with the region of interest.

[0115] In a fifth aspect, in combination with one or more of the first to fourth aspects, determining the region of interest comprises: determining a risk map based on the second image data, wherein the region of interest corresponds to a portion of the risk map above a threshold risk level.

[0116] In a sixth aspect, in combination with one or more of the first to fifth aspects, determining the risk map includes using a machine learning (ML) algorithm to determine risks associated with the scenario model.

[0117] In a seventh aspect, in combination with one or more of the first to sixth aspects, determining the region of interest includes: determining an object of interest in the second image data based on object detection, wherein the region of interest corresponds to the object of interest.

[0118] In an eighth aspect, in combination with one or more of the first to seventh aspects, the device is further configured to perform operations including: determining whether the concentration score satisfies a first criterion; and when the concentration score satisfies the first criterion, performing an action to warn the user.

[0119] In a ninth aspect, in combination with one or more of the first to eighth aspects, determining whether the concentration score satisfies the first criterion includes determining whether the concentration score is a predetermined threshold amount below a historical concentration score corresponding to the user.

[0120] In a tenth aspect, in combination with one or more of the first to ninth aspects, performing the action to alert the user includes displaying the second image data on a user display, wherein the region of interest is marked on the second image data.

[0121] In an eleventh aspect, in combination with one or more of the first to tenth aspects, the apparatus is further configured to perform operations including sending the concentration score via a wireless communication link.

[0122] It should be understood by those skilled in the art that any of a variety of different technologies and techniques may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0123] In this article Figures 1 to 8 The components, functional blocks and modules described include processors, electronic devices, hardware devices, electronic components, logical circuits, memories, software codes, firmware codes, etc. or any combination thereof. Software should be broadly interpreted to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, and / or functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language or other terms. In addition, the features discussed herein may be implemented via dedicated processor circuits, via executable instructions, or a combination thereof.

[0124] The technician will further understand that the various exemplary logic boxes, modules, circuits and algorithm steps described in conjunction with the disclosure of this article can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, boxes, modules, circuits and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. Those skilled in the art can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing departure from the scope of the present disclosure. The technician will also easily recognize that the order or combination of components, methods or interactions described herein are only examples, and the components, methods or interactions of various aspects of the present disclosure can be combined or performed in a manner other than the manner illustrated and described herein.

[0125] The various illustrative logics, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been generally described in terms of functionality and exemplified in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0126] The hardware and data processing devices for implementing the various exemplary logics, logic blocks, modules and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic component, a discrete hardware component or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller or state machine. In some specific implementations, the processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. In some specific implementations, specific processes and methods may be performed by circuits specific to a given function.

[0127] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware, including the structures disclosed in this specification and their structural equivalents, or any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on computer storage media for execution by data processing apparatus or for controlling the operation of data processing apparatus.

[0128] If implemented in software, the function may be stored as one or more instructions or codes on a computer-readable medium or sent through a computer-readable medium. The process of the method or algorithm disclosed herein may be implemented in a processor-executable software module that may reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, which include any media that can be implemented to transfer a computer program from one place to another. The storage medium may be any available medium that a computer can access. As an example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the required program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection may be appropriately referred to as a computer-readable medium. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as a code and instruction set, or any combination of code and instruction sets, on a machine-readable medium or computer-readable medium, which may be incorporated into a computer program product.

[0129] Various modifications to the specific implementations described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to some other specific implementations without departing from the spirit or scope of the disclosure. Therefore, the claims are not intended to be limited to the specific implementations shown herein, but are to be consistent with the broadest scope consistent with the disclosure, the principles and novel features disclosed herein.

[0130] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0131] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all illustrated operations to achieve the desired result. In addition, the accompanying drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations that are not depicted may be combined in the schematically illustrated example process. For example, one or more additional operations may be performed before, after, simultaneously or between any illustrated operations. In some environments, multitasking and parallel processing are advantageous. In addition, the separation of various system components in the specific implementation described above should not be understood as requiring such separation in all specific implementations, and it should be understood that the program components and systems described can generally be integrated together in a single software product or packaged into multiple software products. Additionally, some other specific implementations also fall within the scope of the appended claims. In some cases, the actions described in the claims can be performed in different orders and still achieve the desired result.

[0132] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Therefore, the disclosure is not intended to be limited to the examples and designs described herein, but to be consistent with the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method comprising: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to the field of view of the user; determining a set of regions of interest based on the second image data; determining a gaze direction of the user based on the first image data; as well as A concentration score is determined based on a correspondence between the set of regions of interest and the gaze directions. 2 . The method of claim 1 , wherein the first image data comprises a first plurality of image frames captured within a time duration, and wherein the second image data comprises a second plurality of image frames captured within the time duration. 3 . The method according to claim 2 , wherein determining the concentration score comprises determining a proportion of the duration in which the gaze direction of the user is associated with the set of regions of interest.

4. The method according to claim 1, wherein determining the set of regions of interest comprises: A risk map is determined based on the second image data, wherein the set of regions of interest corresponds to portions of the risk map that are above a threshold risk level. The method of claim 4 , wherein determining the risk map comprises determining the risk map using a machine learning (ML) algorithm.

6. The method according to claim 1, wherein determining the set of regions of interest comprises: An object of interest in the second image data is determined based on object detection, wherein a region of interest corresponds to the object of interest.

7. The method according to claim 1, further comprising: determining whether the concentration score satisfies a first set of criteria; as well as When the concentration score satisfies the first set of criteria, an action is performed to alert the user.

8. The method of claim 7, wherein determining whether the concentration score satisfies a first set of criteria comprises determining whether the concentration score is below a predetermined threshold amount of a historical concentration score corresponding to the user. 9 . The method of claim 7 , wherein performing the action to alert the user comprises displaying the set of regions of interest on a heads-up display (HUD).

10. The method according to claim 1, further comprising: The concentration score is transmitted via a wireless communication link.

11. An apparatus comprising: a memory storing processor readable code; as well as at least one processor coupled to the memory, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations including: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to the field of view of the user; determining a set of regions of interest based on the second image data; determining a gaze direction of the user based on the first image data; as well as A concentration score is determined based on a correspondence between the set of regions of interest and the gaze directions.

12. The apparatus of claim 11, wherein the first image data comprises a first plurality of image frames captured within a time duration, and wherein the second image data comprises a second plurality of image frames captured within the time duration. 13 . The apparatus of claim 12 , wherein determining the concentration score comprises determining a proportion of the duration in which the gaze direction of the user is associated with the set of regions of interest.

14. The apparatus according to claim 11, wherein determining the set of regions of interest comprises: A risk map is determined based on the second image data, wherein the set of regions of interest corresponds to portions of the risk map that are above a threshold risk level.

15. The apparatus of claim 14, wherein determining the risk map comprises determining the risk map using a machine learning (ML) algorithm.

16. The apparatus according to claim 11, wherein determining the set of regions of interest comprises: An object of interest in the second image data is determined based on object detection, wherein a region of interest corresponds to the object of interest.

17. The apparatus of claim 11, wherein the at least one processor is further configured to execute the processor readable code to cause the at least one processor to perform operations comprising: determining whether the concentration score satisfies a first set of criteria; and When the concentration score satisfies the first set of criteria, an action is performed to alert the user.

18. The apparatus of claim 17, wherein determining whether the concentration score satisfies a first set of criteria comprises determining whether the concentration score is below a predetermined threshold amount of a historical concentration score corresponding to the user.

19. The apparatus of claim 17, wherein performing the action to alert the user comprises displaying the set of regions of interest on a heads-up display (HUD).

20. The apparatus of claim 11, wherein the at least one processor is further configured to execute the processor readable code to cause the at least one processor to perform operations comprising: The concentration score is transmitted via a wireless communication link.

21. A vehicle device comprising: Vehicle frame; a first camera attached to the vehicle frame and arranged to capture a first field of view including a portion of the cabin enclosed on at least some sides by the vehicle frame; a second camera attached to the vehicle frame and arranged to capture a second field of view corresponding to at least a portion of a surrounding environment of the vehicle frame; as well as a processing system coupled to the first camera and the second camera, wherein the processing system comprises: a memory storing processor readable code; and at least one processor coupled to the memory, the at least one processor configured to execute the processor readable code to cause the at least one processor to perform operations including: receiving first image data from the first camera; receiving second image data from the second camera; determining a set of regions of interest based on the second image data; determining a gaze direction of a user based on the first image data; and A concentration score is determined based on a correspondence between the set of regions of interest and the gaze directions.

22. The vehicle device of claim 21, wherein: The first image data comprises a first plurality of image frames captured over a duration, The second image data comprises a second plurality of image frames captured during the duration, and Determining the concentration score includes determining a proportion of the duration in which the gaze direction of the user is associated with the set of regions of interest.

23. The vehicle apparatus of claim 21, wherein: determining the set of regions of interest comprises determining a risk map based on the second image data, the set of regions of interest corresponding to portions of the risk map above a threshold risk level; and The operations also include one or more of the following operations: determining whether the concentration score satisfies a first set of criteria; and When the concentration score satisfies the first set of criteria, an action is performed to alert the user.

24. The vehicle apparatus of claim 21, wherein the operations further comprise one or more of the following operations: determining whether the concentration score satisfies a first set of criteria; and When the concentration score satisfies the first set of criteria, an action is performed.

25. The vehicle device of claim 24, wherein the processing system further comprises a wireless communication adapter, and wherein performing the action comprises transmitting the concentration score via the wireless communication adapter.

26. A method comprising: receiving first image data from a first camera oriented in a first direction having a first field of view facing a user operating a vehicle; receiving second image data from a second camera oriented in a second direction different from the first direction, the second camera having a second field of view corresponding to a second field of view of a surrounding environment of the vehicle; determining a risk map based on the second image data; determining a gaze direction of the user based on the first image data; determining a correspondence between the risk map and the gaze direction; as well as The vehicle is controlled based on the correspondence between the risk map and the gaze direction.

27. The method of claim 26, wherein determining the correspondence between the risk map and the gaze direction comprises determining a concentration score.

28. The method of claim 26, wherein controlling the vehicle comprises generating an alarm.

29. The method of claim 26, wherein controlling the vehicle comprises displaying a region of interest on a heads-up display (HUD), the region of interest corresponding to a region of the risk map that is above a threshold level and not associated with the gaze direction.

30. The method of claim 26, wherein determining the risk map comprises: determining objects in the surroundings of the vehicle based on the second image data; determining that the object has appeared or disappeared within a threshold amount of time; as well as A value for the risk map is determined based on determining that the object has appeared or disappeared within a threshold amount of time.

Images