Eye gaze tracking calibration

Abstract

A system and method for error estimation in an eye gaze tracking system in a vehicle can include an operator monitoring system that provides measured eye gaze information corresponding to an object outside the vehicle; and an external object monitoring system that provides an error in the theoretical eye gaze information and the measured eye gaze information based on the measured eye gaze information and the theoretical eye gaze information.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to operator monitoring systems, such as motor vehicle driver monitoring systems. More particularly, the present disclosure relates to driver eye gaze determination. BACKGROUND

[0002] Eye gaze can generally refer to the direction in which a driver’s eyes are gazing at any given moment. Systems for operator eye tracking and providing operator eye gaze are known and can be used in many useful applications, including detecting driver distraction, drowsiness, situational awareness, and readiness to take vehicle control from, for example, an autonomous driving mode.

[0003] Eye tracking systems can exhibit errors in the eye gaze information they provide. One type of error can generally be categorized as a shift or bias in the indicated gaze. It is generally desirable to reduce or eliminate errors in the gaze information from eye tracking systems. SUMMARY

[0004] In one example embodiment, an apparatus for error estimation of an eye gaze tracking system in a vehicle can include an operator monitoring system that provides measured eye gaze information corresponding to objects external to the vehicle; and an external object monitoring system that provides theoretical eye gaze information and errors in the measured eye gaze information based on the measured eye gaze information and the theoretical eye gaze information.

[0005] In addition to one or more of the features described herein, the theoretical eye gaze information can be determined by the external object monitoring system based on position information corresponding to the objects external to the vehicle.

[0006] In addition to one or more of the features described herein, the measured eye gaze information can include at least one of horizontal view angle information and vertical view angle information.

[0007] In addition to one or more of the features described herein, the position information can include information of an object depth.

[0008] In addition to one or more of the features described herein, the position information can include object direction information.

[0009] In addition to one or more of the features described herein, the position information can include at least one of horizontal object angle information and vertical object angle information.

[0010] In addition to one or more of the features described herein, the external object monitoring system can include at least one forward looking camera.

[0011] In addition to one or more of the features described herein, the external object monitoring system can include at least one forward looking camera, the position information can be determined based on camera image data and relative displacement of the objects external to the vehicle.

[0012] In another example embodiment, a method for estimating an error of an eye gaze tracking system in a vehicle can include capturing a first exterior image with a forward looking camera at an earlier first time and capturing a second exterior image at a later second time. An object common to the first and second exterior images can be detected within each of the first and second exterior images whose image position has changed between the first exterior image and the second exterior image. A respective first and second direction of the object can be determined for each of the first and second exterior images. A relative displacement of the object between the first time and the second time can be determined. Based on the first direction of the object, the second direction of the object, and the relative displacement of the object, a first depth of the object at the first time and a second depth of the object at the second time can be determined. Based on a respective one of the first depth of the object and the second depth of the object and a respective one of the first direction of the object and the second direction of the object, a theoretical eye gaze of an operator of the vehicle observing the object at a selected one of the first time and the second time can be determined. A measured eye gaze of the operator of the vehicle observing the object at the selected one of the first time and the second time can be received from the eye gaze tracking system. Based on the measured eye gaze and the theoretical eye gaze, an error of the measured eye gaze of the operator can be determined.

[0013] In addition to one or more features described herein, determining the theoretical eye gaze of the operator of the vehicle observing the object at one of the first time and the second time can be based on a separation between an eye of the operator and the forward looking camera.

[0014] In addition to one or more features described herein, the object can be static, and determining the relative displacement of the object between the first time and the second time can be based on a displacement of the vehicle between the first time and the second time.

[0015] In addition to one or more features described herein, the object can be dynamic, and determining the relative displacement of the object between the first time and the second time can be based on a displacement of the vehicle between the first time and the second time and a displacement of the object between the first time and the second time.

[0016] In addition to one or more features described herein, determining the first depth of the object at the first time and the second depth of the object at the second time based on the first direction of the object, the second direction of the object, and the relative displacement of the object can include representing the relative displacement of the object as a vector and solving an injective function that includes the vector, the first direction of the object, and the second direction of the object.

[0017] In addition to one or more features described herein, the measured eye gaze of the operator can include at least one of a horizontal angle of view and a vertical angle of view.

[0018] In addition to one or more of the features described herein, the first direction of the object and the second direction of the object each can include at least one of a respective horizontal object angle and a vertical object angle.

[0019] In addition to one or more of the features described herein, the method can include recalibrating the eye gaze tracking system based on the determined error of the measured operator eye gaze.

[0020] In addition to one or more of the features described herein, determining the error can include at least one of a comparison of the measured eye gaze to a theoretical eye gaze, a statistical model, and a machine learning model.

[0021] In addition to one or more of the features described herein, recalibrating can be performed in each vehicle cycle.

[0022] In another example embodiment, an apparatus for error estimation of an eye gaze tracking system in a vehicle can include an operator monitoring system that provides measured eye gaze information corresponding to an object outside the vehicle; and an external object monitoring system that determines location information corresponding to the object, determines theoretical eye gaze information based on the determined location information, and determines an error of the measured eye gaze based on the determined theoretical eye gaze information. The determined error in the measured eye gaze can provide error estimation in the eye gaze tracking system.

[0023] In addition to one or more of the features described herein, the location information can include object depth information and object direction information.

[0024] The above features and advantages of the present disclosure, and other features and advantages, are readily apparent from the following detailed description, when taken in connection with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0025] Other features, aspects, and details are described in only an example, and with reference to the drawings, in which:

[0026] Figure 1 An example system according to the present disclosure is shown;

[0027] Figure 2 A forward-looking view from within a passenger cabin of a host vehicle according to the present disclosure is shown;

[0028] Figure 3 An example driving scenario according to the present disclosure is shown, which a forward-looking camera can capture in an external image as the host vehicle traverses a road;

[0029] Figure 4AA simplified representation of an overlay image with respect to static objects is shown in accordance with the present disclosure;

[0030] Figure 4B A top plan view representation of a driving environment is shown in accordance with the present disclosure;

[0031] Figure 5 A determination of object depth information is shown in accordance with the present disclosure;

[0032] Figure 6A A simplified representation of an image with respect to static objects is shown in accordance with the present disclosure;

[0033] Figure 6B A top plan view representation of a driving environment is shown in accordance with the present disclosure;

[0034] Figure 7A A simplified representation of an image with respect to static objects is shown in accordance with the present disclosure;

[0035] Figure 7B A side view representation of a driving environment is shown in accordance with the present disclosure; and

[0036] Figure 8 An exemplary processing flow for eye gaze error estimation is shown in accordance with the present disclosure. DETAILED DESCRIPTION

[0037] The following description is merely exemplary in nature and is not intended to limit the present disclosure, its application, or uses. Throughout the drawings, corresponding reference numerals indicate corresponding parts and features. As used herein, control module, module, control, controller, control unit, electronic control unit, processor, and like terms refer to any one or various combinations of application-specific integrated circuit (ASIC), electronic circuitry, a central processing unit (preferably a microprocessor) and associated memory and storage (read-only memory (ROM), random access memory (RAM), electrically programmable read-only memory (EPROM), hard drive, etc.) or microcontroller, combinatorial logic circuitry, input / output circuitry and devices (I / O) and appropriate signal conditioning and buffer circuitry, high-speed clock, analog-to-digital (A / D) and digital-to-analog (D / A) circuitry, and other support circuits, as well as software or firmware programs or routines executed by a processor or controllers, to provide the described functions. Control modules can include various communication interfaces, including point-to-point or discrete wires, as well as wired or wireless interfaces to networks, including wide-area and local-area networks, vehicle controller area networks, and plant and service-related networks. The functions of control modules set forth in this disclosure can be performed in a distributed control architecture among multiple networked control modules. Software, firmware, programs, instructions, routines, code, algorithms, and like terms, refer to any controller-executable set of instructions, including calibration, data structures, and lookup tables. Control modules have a set of control routines executed to provide the described functions. The routines are executed, for example, by a central processing unit, and are operable to monitor inputs from sensing devices and other networked control modules, and to execute control and diagnostic routines to control operation of actuators. Routines can be executed periodically during ongoing engine and vehicle operation. Alternatively, routines can be executed on-demand in response to occurrence of an event, software call, or request via a user interface input.

[0038] During road operation of the vehicle by the vehicle operator semi-autonomously or fully autonomously, the vehicle can be an observer in a driving scene that includes a driving environment, such as a roadway, surrounding infrastructure, objects, signs, hazards, and other road-sharing vehicles, collectively referred to herein as objects or targets. Objects can be static, such as a road sign, or dynamic, such as another vehicle crossing the roadway. The observing vehicle can be referred to herein as a host vehicle. Other vehicles sharing the roadway can be referred to herein as target vehicles. The terms driver and operator can be used interchangeably.

[0039] The host vehicle can be equipped with various sensors and communication hardware and systems. Figure 1 An exemplary host vehicle 101 is shown in FIG. 1, Figure 1An exemplary system 100 according to the present disclosure is shown. Host vehicle 101 can be a non-autonomous vehicle or an autonomous or semi-autonomous vehicle. The phrase autonomous or semi-autonomous vehicle, and any derivative terminology, broadly refers to any vehicle capable of automatically performing driving-related actions or functions without a request by a driver, and includes actions falling within the Society of Automotive Engineers (SAE) International classification system Levels 1-5. Host vehicle 101 can include a control system 102 comprising a plurality of networked electronic control units (ECUs) 117 that can be communicatively coupled via a bus structure 111 to perform control functions and information sharing, including performing control routines locally or in a distributed manner. Bus structure 111 can be part of a controller area network (CAN) or other similar network, as is well known to those of ordinary skill in the art. An exemplary ECU can include an engine control module (ECM) that performs functions related to internal combustion engine monitoring, control, and diagnostics based primarily on a plurality of inputs including CAN bus information. ECM inputs can be coupled directly to the ECM or can be provided to or determined within the ECM from various well-known sensors, computations, derivations, syntheses, other ECUs, and sensors through bus structure 111, as is well known to those of ordinary skill in the art. A battery electric vehicle (BEV) can include a propulsion system control module (PSCM) that performs functions related to the BEV powertrain, including control of wheel torque as well as charging and charge balancing of batteries within a battery pack. Those of ordinary skill in the art recognize that a plurality of other ECUs 117 can be part of a controller network on host vehicle 101 and can perform other functions related to various other vehicle systems, such as chassis, steering, braking, transmission, communications, infotainment, etc. All networked ECUs can have access to and obtain various vehicle-related information through the CAN bus, such as vehicle dynamics information such as speed, heading, steering angle, multi-axis acceleration, yaw, pitch, roll, etc. An exemplary ECU can include an external object computation module (EOCM) 113 that performs functions related to sensing the environment external to vehicle 101, and more particularly, functions related to lane, road, and object sensing. EOCM 113 receives information from various external object sensors 119 and other sources. By way of example only and without limitation, EOCM 113 can receive information from one or more radar sensors, lidar sensors, ultrasonic sensors, two-dimensional (2D) cameras, three-dimensional (3D) cameras, global positioning systems, vehicle-to-vehicle communication systems, and vehicle-to-infrastructure communication systems, as well as from on-board or off-board databases, such as maps and infrastructure databases. EOCM 113 can thus have access to location data, distance data, velocity data, and image data that can be useful in determining road and target vehicle information, such as road features and target vehicle geometry, distance, and velocity information, etc.The sensors 119 can be positioned at various perimeter points around the vehicle, including the front, rear, corners, sides, etc., as indicated by the large dots in the vehicle 101 at these locations. The positioning of the sensors 119 can be selected as appropriate to provide the desired sensor coverage for a particular application. While the sensors 119 are shown as being directly coupled to the EOCM 113, the inputs can be provided to the EOCM 113 through a bus structure 111, as is well known to those of ordinary skill in the art. Another example ECU can include a Driver Monitoring Module (DMM) 115 tasked with monitoring the driver and / or passengers within the vehicle and primarily performing functions related to sensing the environment within the vehicle 101, more particularly, functions related to driver interaction with the vehicle, driver attentiveness, occupant posture and positioning, seat belt restraint positioning, and other characteristics and functions within the vehicle. The DMM 115 can receive information from various sensors 121 and other sources. By way of example only and not by way of limitation, the DMM 115 can receive information from one or more two-dimensional (2D) cameras and / or three-dimensional (3D) cameras, including infrared (IR) and / or near-IR cameras.

[0040] The host vehicle 101 can be equipped with wireless communication capabilities, generally represented at 123, which can engage in one or more of GPS satellite 107 communication, vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication through known ad-hoc networks, vehicle-to-pedestrian (V2P) communication, vehicle-to-cloud (V2C) communication (such as through ground radio (e.g., cellular) towers 105), or vehicle-to-everything (V2X) communication. In the present disclosure, reference to V2X is understood to mean any one or more of the wireless communication capabilities that connect a vehicle to resources and systems outside the vehicle, including but not limited to V2V, V2I, V2P, V2C.

[0041] The description of the example system 100 herein is not exhaustive. Neither should the description of the various example systems be interpreted as being completely required. Thus, those of ordinary skill in the art will appreciate from the present disclosure that some, all, and additional techniques from the described example system 100 can be used in various implementations. Those of ordinary skill in the art will appreciate that the described vehicle hardware is merely illustrative of some of the more relevant hardware components for use with the apparatus and methods of the present disclosure and is not meant to be an exact or exhaustive representation of vehicle hardware for implementing the apparatus and methods of the present disclosure. Moreover, the structure or architecture of the vehicle hardware can be significantly different from that shown (e.g., individual ECUs can be integrated with one another or other devices or otherwise combined, rather than all being separate, independent components). Due to the myriad of possible arrangements, and for the sake of brevity and clarity, the description of the example system 100 is presented in connection with the description of the apparatus and methods of the present disclosure. Figure 1 The description of the example system 100 herein is not exhaustive. Neither should the description of the various example systems be interpreted as being completely required. Thus, those of ordinary skill in the art will appreciate from the present disclosure that some, all, and additional techniques from the described example system 100 can be used in various implementations. Those of ordinary skill in the art will appreciate that the described vehicle hardware is merely illustrative of some of the more relevant hardware components for use with the apparatus and methods of the present disclosure and is not meant to be an exact or exhaustive representation of vehicle hardware for implementing the apparatus and methods of the present disclosure. Moreover, the structure or architecture of the vehicle hardware can be significantly different from that shown (e.g., individual ECUs can be integrated with one another or other devices or otherwise combined, rather than all being separate, independent components). Due to the myriad of possible arrangements, and for the sake of brevity and clarity, the description of the example system 100 is presented in connection with the description of the apparatus and methods of the present disclosure. Figure 1The illustrated embodiments describe vehicle hardware, but it should be understood that the present systems and methods are not limited to such embodiments.

[0042] Figure 2 A front view is shown from within the passenger cabin 200 of the vehicle 101. The passenger cabin 200 can include one or more front-view cameras 201 (hereinafter referred to as cameras 201) configured to capture a substantially front-view driving scene through the windshield 203. The cameras 201 can be mounted on the dashboard 211, on or within the rearview mirror 213, underneath the passenger cabin roof, at the A-pillars, or any other location that provides an unobstructed view of the front-view driving scene. Alternatively, the cameras 201 can be located outside of the passenger cabin. The cameras 201 are mounted within the driver’s cabin and provide camera image data for the present systems and methods. Although the following examples describe the cameras 201 in the context of cameras that generate respective images or still frames, the cameras 201 can include any suitable camera or vision system known or used in the industry, so long as it is capable of capturing external images, representations, and / or other information about the environment outside of the vehicle. Depending on the particular application, the cameras 201 can include: still cameras, video cameras; BW and / or color cameras; analog and / or digital cameras; wide and / or narrow field-of-view (FOV) cameras; and can be part of a mono and / or stereo system, to name a few possibilities. According to non-limiting examples, the vehicle hardware includes cameras 201 that are CMOS video cameras and provide camera image data to the EOCM 113, either directly or via the bus structure 111. The cameras 201 can be wide-angle cameras (e.g., with a FOV of approximately 170° or greater), such that a full or near-full view of the relevant front driving scene can be obtained. The camera image data output by the cameras 201 can include raw video or still image data (i.e., with no or little pre-processing), or it can include pre-processed video or still image data in the case where the cameras 201 have their own image processing resources and perform pre-processing on the captured images before outputting them as camera image data.

[0043] Figure 2A passenger cabin 200 is further shown, which includes one or more rearview cameras 209 (hereinafter cameras 209) configured to capture at least an image of the driver’s eyes. The cameras 209 can be mounted on the instrument panel 211, on or within the rearview mirror 213, underneath the passenger cabin roof, the A-pillars, or any other location that provides an unobstructed view of the driver. A light source 207 can be included as part of the cameras 209 or separate from the cameras 209. The light source 207 can preferably be an IR or near-IR light source, such as the illustrated light-emitting diode (LED). According to a non-limiting example, the cameras 209 provide camera image data to the DMM 115, either directly or via the bus structure 111. The camera image data output by the cameras 209 can include raw video or still image data (i.e., with no or little pre-processing), or it can include pre-processed video or still image data in the case that the cameras 209 have their own image processing resources and perform pre-processing on the captured images before outputting them as camera image data. The DMM 115 determines the gaze direction of the driver according to known techniques. For example, one technique is known as the pupil center corneal reflection, which can use the light source 207 to illuminate the driver’s eyes and the cameras 209 to capture images of the eyes, including reflections of light from the cornea and pupil of one or both eyes. The gaze direction can be determined from the reflection geometry of the cornea and pupil, as well as additional geometric features of the reflections. The DMM 115, cameras 209, and light source 207, including known image processing algorithms (including artificial intelligence (AI)), together constitute an exemplary eye tracking system that can provide gaze information. Alternative eye tracking systems, including alternative eye tracking techniques and algorithms, can be employed within the scope of the present disclosure. The gaze information can be represented by a vector or multiple component vectors. For example, the gaze information is represented by a vector corresponding to a horizontal angle component (i.e., left / right) and a vector corresponding to a vertical angle component (i.e., up / down).

[0044] According to the present disclosure, the driver’s eye gaze can be measured and provided by an eye tracking system and compared to a theoretical eye gaze determined based on the location of external objects sensed by external sensors, such as cameras. From this comparison, a deviation of the driver’s measured eye gaze from the theoretical eye gaze can be determined and corrected as needed. As used herein, the term “compare” can refer to any evaluation of the measured eye gaze against the theoretical eye gaze, and can include, for example, simple scalar comparisons as well as statistical modeling or machine learning approaches.

[0045] Figure 3An exemplary driving scenario 300 is shown, as the host vehicle 101 traverses a roadway 315, the forward-looking camera 201 can capture the scene in an external image. The image from the camera 201 can be bounded by a top edge 303, a bottom edge 305, a left edge 307, and a right edge 309. A horizontal image centerline 311 and a vertical image centerline 313 are orthogonal image structures, and ideally correspond substantially to the horizontal and vertical directions, with proper camera 201 orientation and calibration. Similarly, the top edge 303 and the bottom edge 305 desirably correspond to the horizontal direction, while the left edge 307 and the right edge 309 desirably correspond to the vertical direction. Figure 3 An exemplary driving scenario includes a two-lane roadway 315, with adjacent lanes distinguished by lane markings 317. A shoulder 319 is illustrated as bounding the two-lane roadway 315. Figure 3 An overlay of two images separated in time (e.g., by a few seconds) is shown. Exemplary objects in the two images include a static object 321 and a dynamic object 323, which in this embodiment are a road sign and a vehicle ahead, respectively. An earlier captured image of the static object 321 at time to is designated 321A, while an earlier captured image of the dynamic object 323 at time to is designated 323A. A later captured image of the static object 321 at time ti is designated 321B, while a later captured image of the dynamic object 323 at time ti is designated 323B. The earlier captured images of the objects 321 and 323 at time to are shown in dashed lines in Figure 3 , while the later captured images of the objects 321 and 323 at time ti are shown in solid lines in Figure 3 .

[0046] Figure 4A A simplified representation of the overlaid images of Figure 3 is shown with respect to the static object 321. In this example, the static object 321 represents any static object or target that the camera 201 and EOCM 113 can capture and process at different time stamps, e.g., at an earlier time to and a later time ti that are separated by a few seconds, preferably while the host vehicle 101 traverses the roadway. The time interval can be determined by factors such as vehicle speed and relative displacement of the object between the two time stamps. The processing by the EOCM 113 can include, for example, image cropping, segmentation, object recognition, extraction and classification, and other image processing functions known to those skilled in the art. Thus, the earlier and later captured images of the static object 321 are labeled I A and I B , respectively. Also shown are the top edge 303, the bottom edge 305, the left edge 307, the right edge 309, and the vertical image centerline 313. Within the captured and overlaid images as described above, the static object I A and IB The image distances W1 and W2 are each from the earlier and later captured images. For example, image distances W1 and W2 can be scaled in pixels. Image distances W1 and W2 can be measured from the vertical scene center line 313. The vertical image center line 313 can be any anchor point or reference used to determine image distances W1 and W2, and it should be understood that other references, including the left edge 307 and the right edge 309, can be used. Similarly, the center point can be any point on the object image used to determine image distances W1 and W2, and it should be understood that other image reference points, including edges, corners, vehicle parts, such as wheels, body panels, brake lights, etc., can be used. In this disclosure, the center point is used as a reference point on the object and the object image, and can be specified by reference numerals A and B in the figures. The image plane 403 for object projection can be arbitrarily defined at a distance in front of the camera 201, such that image distances W1 and W2 can be mapped to the image plane. Figure 4B An exemplary image plane 403 is shown. Figure 4B yes Figure 3 and Figure 4A The diagram shows a top-down view of the driving environment. Image plane 403 is at an arbitrary distance D from the forward-facing camera 201, for example, 5 meters. The vertical image centerline 313 intersects image plane 403 perpendicularly. Image distances W1 and W2 can be mapped to the image plane according to a predetermined scaling factor or function to return image plane distances W1′ and W2′. Image plane distance W1′ corresponds to the projected image I. A The image plane distance W2′ corresponds to the projected image I. B From the image plane distances W1′ and W2′ and the distance D, the corresponding image plane angle α can be determined. h =tan -1 (D / W1′) and β h =tan -1 (D / W2′). It should be understood that the image plane angle shown is on the horizontal plane, and therefore is marked with the subscript h, and can be referred to as the horizontal image plane angle or the horizontal object angle. It should also be understood that the horizontal image plane angle shown is the complementary angle of the azimuth angles αh′ and βh′ corresponding to the static object 321 at the earlier and later timestamps relative to the origin O of the forward-looking camera 201 and the vertical image center line 313. Horizontal image plane angle α h and β h Or their complementary azimuth angle α h ′ and β h The orientation of static object 321 can be defined at earlier and later times t0 and t1, respectively. It should be understood that light rays... and These represent the directions of static object 321 at time t0 and time t1, respectively.

[0047] Further reference is made to Figure 5 , the relative displacement of the static object 321 can be determined. The relative displacement of the static object 321 is represented by the vector 503, which can be derived from the CAN bus vehicle position information and kinematic information from the host vehicle displacement from its first position at time t0to its second position at time t1. In the case of a static object, the vector 503 can simply be modeled as the inverse of the host vehicle 101 displacement vector. In accordance with the present disclosure, information related to the position and motion of dynamic objects can be based on external object sensors 119. In one embodiment, the EOCM 113 can receive and derive kinematic information from external object sensors 119, including one or more range / rate capable sensors, such as radar, lidar, ultrasonic, and vision sensors, which provide data directly corresponding to static and dynamic object position and its time derivatives. That is, the position, range, velocity, acceleration, and jerk of objects within the host vehicle 101 frame of reference can be obtained from such range / rate capable sensors. In addition, known range / rate sensors can also provide moving object yaw rate, also within the host vehicle 101 frame of reference. The external object sensors 119 preferably resolve the position, range, velocity, acceleration, and jerk metrics in vehicle standard longitudinal (X) and lateral (Y) components. Otherwise, such resolution can be performed in the EOCM 113. Depending on the extent of “at-sensor” signal processing, downstream sensor processing can include various filtering. In addition, in the case of external object sensors 119 that are digitally and / or topologically diverse, downstream sensor processing can include sensor fusion. Thus, it can be appreciated that moving object kinematic information can include: longitudinal position (V t P x ), velocity (V t V x ), and acceleration (V t A x ); lateral position (V t P y ), velocity (V t V y ), and acceleration (V t A y ); and yaw rate Likewise within the host vehicle 101 frame of reference, CAN bus data from the host vehicle 101 can provide host vehicle kinematic information, including: host vehicle longitudinal position (V h P x ), velocity (V h V x ), and acceleration (Vh A x ); Lateral position of the main vehicle (V) h P y ), speed (V) h V y ) and acceleration (V h A y ); and the yaw rate of the main vehicle It can also include information based on vehicle roll, pitch, and verticality.

[0048] In another embodiment, the master vehicle 101 and another vehicle on the road can be V2X capable, allowing relevant information to be transmitted from the other vehicle for the master vehicle 101 to receive using, for example, Dedicated Short Range Communication (DSRC). Thus, through V2X communication of CAN bus data with respect to the other vehicle, the other vehicle can provide its kinematic information, including V... t P x V t V x V t A x V t P y V t V y V t A y and Therefore, it should be understood that the motion information of the other vehicle provided will be within the reference frame of that other vehicle. Those skilled in the art will recognize that V2X information transmission can occur directly between vehicles or via a V2V mesh network through one or more other adjacent nodes (surrounding vehicles or infrastructure). Similarly, those skilled in the art will recognize that V2X information transmission can be performed via V2C routed communication, which may include additional cloud resources and data enhancement and processing, as well as extending the communication distance between the master vehicle 101 and other vehicles.

[0049] Therefore, the relative displacement of a dynamic object (e.g., another vehicle crossing the road) can be derived by additionally considering this vehicle kinematics from time t0 to time t1, information which can be derived from the main vehicle external object sensor 119 or provided via the V2X communication. This is for the direction of the ray representing the static object 321 at times t0 and t1. 411 and Vector within the enclosed space between 413 503, there exists a unique vector fitting solution. Therefore, the vector... 503 and timestamp direction (i.e., light direction) 411 and 413) between them exist and can be used to determine the position of the static object at the respective times t0 and ti. From the direction and relative displacement of the object, the depth dA of the static object from the origin O at time t0 and the depth dB of the static object from the origin O at time ti are thus known. The depth and direction information of the object together define the position of the object.

[0050] Figure 6A corresponding to Figure 4A but only with respect to the earlier image of the static object 321 captured at time t0. As in Figure 4B , the exemplary image plane 403 is shown in Figure 6B and is also a top-down plan view representation of the driving environment. Figure 6B includes the depth d A of the reference static object 321, the direction (i.e. the light ray 411) of the static object 321 and the corresponding horizontal image plane angle or horizontal object angle a h . Further, Figure 6B the driver’s eye 601 is shown. From the known depth d A and direction, and if needed, further utilizing the known position offset of the driver’s eye 601 from the forward looking camera 201 origin O, the theoretical horizontal viewing angle g h can be determined. It will be appreciated that the same process can be followed alternatively with the later image of the static object 321 captured at time ti.

[0051] with respect to the earlier image 321B of the static object 321 captured at time t0, Figure 7A corresponding to Figure 6A . As in Figure 6B , the exemplary image plane 403 is shown in Figure 7B ; however, Figure 7B is a side view representation of the driving environment and is a guide to the vertical analogue of the horizontal representation shown in the top-down view of Figure 6B . Figure 7B includes the depth d A of the reference static object 321, the direction (i.e. the light ray 411) of the static object 321 and the corresponding vertical image plane angle or vertical object angle a v . Further, Figure 7B the driver’s eye 601 is shown. From the known depth d A and direction, and if needed, further utilizing the known position offset of the driver’s eye 601 from the forward looking camera 201 origin O, the theoretical vertical viewing angle g v can be determined. It will be appreciated that the same process can be followed alternatively with the later image 321B of the static object 321 captured at time ti.

[0052] It should be appreciated that while a single pair of time-separated images can be sufficient to illustrate the current subject matter, multiple pairs of such time-separated images can be employed. For example, multiple pairs of images clustered around a particular pair of timestamps can be advantageously used to effectively filter noise and disturbances (e.g., due to vehicle dynamics) from the image signal information. Moreover, multiple pairs of images collected and evaluated over a wide range of gaze angles can advantageously provide information as a function of gaze angle when the operator is tracking an object.

[0053] Figure 8 An exemplary flow of eye gaze error estimation according to the present disclosure is shown. This flow is suitable for use during active periods of eye gaze determination (i.e., measured eye gaze) by the DMM 115. Thus, error correction can advantageously be done in real time, reducing the frequency of offline calibration procedures. The process 800 can be implemented primarily by the EOCM 113 by executing computer program code. However, certain steps, such as calibration requests, can require action on the part of the operator of the vehicle 101, which can be interpreted through various user interfaces, including, for example, through a dialog manager or other user interface to interface with a touchscreen display in the cab of the vehicle 101. Moreover, various computer-implemented aspects of the process 800 can be performed within one or more other ECUs, either exclusively or in a distributed manner, as previously disclosed and not necessarily limited to exclusive execution by the EOCM 113. The process 800 is shown as a flowchart having various tasks in substantially linear routines. Those skilled in the art will appreciate from the disclosure herein that the described processes can be represented in alternative ways, including, for example, state flowcharts and activity diagrams. Those skilled in the art will also appreciate that the various tasks in the process 800 flowchart can be implemented in different orders and / or concurrently, and can be merged or split apart.

[0054] The vehicle, including the DMM 115 and the EOCM 113, can be in an operational ready state when the process 800 can be initiated (801). One or more entry conditions can be evaluated (803) to determine if the process 800 is desired and capable. For example, a time or period threshold since a previous execution of the process 800 can be a required entry condition, as can a manual driver calibration request to run the process 800. The process 800 can be run multiple times in a drive cycle, or can be more limited, such as once per drive cycle. The entry conditions can be evaluated (803) periodically until satisfied, after which the process 800 can continue with reading images (805) from the forward looking camera 201. The image reading (805) can include capturing images periodically at regular or variable time stamps, or extracting still images from a continuous video feed buffer that includes time stamp information. Object detection (807) is then performed using the captured images. The image reading (805) and object detection (807) can include image cropping, segmentation, object recognition, extraction and classification, and other image processing functions. Object orientation is determined (809) for a pair of time separated images, such as horizontal and vertical image plane angles (horizontal and vertical object angles), and object depth is determined (811) for the same pair of time separated images. Vehicle motion and object motion during the time separation are used to determine relative object displacement (813). Object displacement information, object orientation information, and driver eye / camera separation information are used as needed to determine the theoretical horizontal and vertical viewing angles of the driver’s eyes to the object at a given time stamp (815). The DMM 115 horizontal and vertical viewing angle information for the object at the same time stamp is then provided and compared to the theoretical horizontal and vertical viewing angles (817), from which an estimated bias or error and a corresponding correction can be determined (819). The error determination can be made by a simple comparison between the viewing angle information provided by the DMM 115 and the theoretical viewing angle information, for example. Alternatively, a statistical model, such as a regression, can be employed using multiple incidents of the information provided by the DMM 115 and the corresponding theoretical information. The multiple incidents can correspond to time clustered time stamps or more widely distributed time stamps (e.g., related to tracking an object through a wide range of viewing angles). The latter example can be particularly useful in developing a dynamic error behavior that can vary with viewing angle, for example. Another error determination technique can include machine learning, which can be used on the vehicle, via a cloud or data center back end processing off the vehicle, or a combination thereof. The correction can then be provided to the DMM 115 for recalibration (821). The current iteration of the process 800 then ends (823).

[0055] Unless explicitly described as “direct,” when describing a relationship between first and second elements in the above disclosure, the relationship can be a direct relationship in which no other intervening elements are between the first and second elements, but can also be an indirect relationship in which one or more intervening elements (spatially or functionally) are between the first and second elements.

[0056] It should be understood that one or more steps in a method can be performed in different order (or simultaneously) and without departing from the principles of the present disclosure. Moreover, although each of the embodiments described above is described as having certain features, any one or more of those described with respect to any embodiment of the present disclosure can be implemented and / or combined in the features of any other embodiment, even if that combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with respect to the other remain within the scope of the present disclosure.

[0057] While the above disclosure has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes can be made and equivalents can be substituted for elements thereof without departing from the scope thereof. In addition, many modifications can be made to adapt a particular situation or material to the teachings of the disclosure without departing from the essential scope thereof. Therefore, it is intended that the disclosure not be limited to the particular embodiment disclosed, but that it include all embodiments falling within the scope of the disclosure.

Claims

1. An apparatus for error estimation in an eye gaze tracking system in a vehicle, comprising: An operator monitoring system providing measured eye gaze information corresponding to an object external to the vehicle; and an external object monitoring system that provides an error in theoretical eye gaze information and measured eye gaze information based on the measured eye gaze information and the theoretical eye gaze information; wherein the theoretical eye gaze information is determined by the external object monitoring system that: determines a first direction of an object at an earlier first time and a second direction of the object at a later second time; determines a relative displacement of the object between the first time and the second time; determines a first depth of the object at the first time and a second depth of the object at the second time based on the first direction of the object, the second direction of the object, and the relative displacement of the object; and determines the theoretical eye gaze information at a selected one of the first time and the second time based on a corresponding one of the first depth of the object and the second depth of the object and a corresponding one of the first direction of the object and the second direction of the object.

2. The apparatus of claim 1, wherein, the measured eye gaze information includes at least one of horizontal angle information and vertical angle information.

3. The apparatus of claim 1, wherein, the external object monitoring system includes at least one forward looking camera.

4. A method for estimating an error in an eye gaze tracking system in a vehicle, comprising: capturing a first external image with a forward looking camera at an earlier first time and a second external image at a later second time; detecting an object within each of the first and second external images that is common to the first and second external images whose image position has changed between the first external image and the second external image; determining a respective first and second direction of the object for each of the first and second external images; determining a relative displacement of the object between the first time and the second time; determining a first depth of the object at the first time and a second depth of the object at the second time based on the first direction of the object, the second direction of the object, and the relative displacement of the object; determining a theoretical eye gaze of a vehicle operator observing the object at a selected one of the first time and the second time based on a corresponding one of the first depth of the object and the second depth of the object and a corresponding one of the first direction of the object and the second direction of the object; receiving a measured eye gaze of the vehicle operator observing the object at the selected one of the first time and the second time from the eye gaze tracking system; and determining an error in the measured eye gaze of the operator based on the measured eye gaze and the theoretical eye gaze.

5. The method of claim 4, wherein, determining the first depth of the object at the first time and the second depth of the object at the second time based on the first direction of the object, the second direction of the object, and the relative displacement of the object includes representing the relative displacement of the object as a vector and solving an injective function that includes the vector, the first direction of the object, and the second direction of the object.

Images