System and method for neural network pre-training based on eye gaze direction

Abstract

Systems and methods for neural network pre-training based on eye gaze direction. A method of training a disparity estimation network. The method includes obtaining a set of eye gaze data having first images, the first images having at least one gaze direction associated with each of the first images. A gaze prediction neural network is trained based on the eye gaze data set to develop a model trained to provide gaze prediction for external images. A depth database is obtained that includes the second images having depth information associated with each of the second images. A disparity estimation neural network for object detection is trained based on the output from the gaze prediction neural network and the output from the deep database.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 419,508, filed on October 26, 2022, entitled “SYSTEM AND METHOD FOR EYE-GAZEDIRECTION BASED PRE-TRAINING OF NEURAL NETWORKS,” the entire contents of which are incorporated herein by reference. Background Art

[0003] Vehicles are commonplace in everyday life. Special purpose cameras, microcontrollers, laser technologies and sensors can be used in many different applications in vehicles. Cameras, microcontrollers and sensors can be used to enhance automated structures that provide state-of-the-art experiences and services to customers, for example in tasks such as body control, camera vision, information display, safety, autonomous control, etc. Vehicle vision systems can also be used to assist in vehicle control.

[0004] Vehicle vision systems can be used to provide information about the vehicle's surroundings to the vehicle operator. Vision systems can also be used to significantly reduce blind spot areas on the sides and rear of the vehicle. Vision systems can also be used to monitor the actions and movements of occupants, particularly vehicle operators. In particular, a driver monitoring system can include a vision system that can be used to track the position and movement of the vehicle operator's head and eyes, such as eye gaze. Eye gaze can generally refer to the direction in which the driver's eyes are looking at at any given moment. Such a system can detect operator eye gaze and can be used in many useful applications, including detecting driver distraction, drowsiness, situational awareness, and readiness to assume vehicle control from, for example, an autonomous driving mode. Summary of the invention

[0005] A method for training a disparity estimation network is disclosed herein. The method includes obtaining an eye gaze dataset having first images, the first images having at least one gaze direction associated with each of the first images. A gaze prediction neural network is trained based on the eye gaze dataset to develop a model trained to provide gaze prediction for an external image. A depth database is obtained, the depth database including second images having depth information associated with each of the second images. A disparity estimation neural network for object detection is trained based on an output from the gaze prediction neural network and an output from the depth database.

[0006] Another aspect of the present disclosure may be wherein the first images are captured by at least one optical sensor and at least one gaze direction associated with each of the first images is captured by a gaze direction system configured to determine an eye gaze direction of at least one eye.

[0007] Another aspect of the present disclosure may be where the second images are captured by at least one optical sensor and the depth information associated with each of the second images is captured by a range sensor configured to determine a distance between an object and the range sensor.

[0008] Another aspect of the disclosure may be where training a gaze prediction neural network on an eye gaze dataset includes performing dilation in at least one gaze direction.

[0009] Another aspect of the present disclosure may be wherein the dilation corresponds to a focal area of ​​the eye.

[0010] Another aspect of the disclosure may be where training the gaze prediction neural network includes associating at least one eye gaze direction with a corresponding one of the first images.

[0011] Another aspect of the disclosure may be where the output from the depth database includes converting the depth information into a normalized disparity map according to the normalized disparity labels.

[0012] Another aspect of the disclosure may be where the output from the depth database includes a second image.

[0013] Another aspect of the disclosure may be where the resolution of the normalized disparity matches a scaled version of a corresponding one of the second images.

[0014] Another aspect of the disclosure may be where training the disparity estimation neural network includes minimizing a minimum absolute deviation between a ground truth measurement and a prediction by the disparity estimation neural network.

[0015] Another aspect of the present disclosure may include performing back-propagation when training a disparity estimation neural network to minimize a minimum absolute deviation.

[0016] Another aspect of the present disclosure may be where the first image is matched with the second image.

[0017] Disclosed herein is a non-transitory computer-readable storage medium embodying programming instructions that, when executed by a processor, are operable to perform a method. The method includes obtaining an eye gaze dataset having first images, the first images having at least one gaze direction associated with each of the first images. Training a gaze prediction neural network based on the eye gaze dataset to develop a model trained to provide gaze predictions for external images. Obtaining a depth database including second images having depth information associated with each of the second images. Training a disparity estimation neural network for object detection based on output from the gaze prediction neural network and output from the depth database.

[0018] A vehicle system is disclosed herein. The system includes at least one optical sensor configured to capture an image, at least one distance sensor configured to measure the distance from at least one distance sensor, and an eye gaze measurement system configured to determine the eye position of the driver. The system also includes a controller that communicates with the at least one optical sensor, the at least one distance sensor, and the eye gaze measurement system. The controller is configured to: obtain an eye gaze dataset including a first image, the first image having at least one gaze direction associated with each of the first images; and train a gaze prediction neural network based on the eye gaze dataset to develop a model trained to provide gaze prediction for external images. The controller is also configured to: obtain a depth database including a second image, the second image having depth information associated with each of the second images; and train a disparity estimation neural network for object detection based on an output from the gaze prediction neural network and an output from the depth database.

[0019] The present invention may also include the following technical solutions:

[0020] 1. A method for training a disparity estimation network, the method comprising:

[0021] obtaining an eye gaze dataset comprising a first plurality of images, wherein at least one gaze direction is associated with each of the first plurality of images;

[0022] training a gaze prediction neural network based on an eye gaze dataset to develop a model trained to provide gaze predictions for external images;

[0023] obtaining a depth database comprising a second plurality of images having depth information associated with each of the second plurality of images; and

[0024] Based on the output from the gaze prediction neural network and the output from the depth database, a disparity estimation neural network for object detection is trained.

[0025] 2. A method according to technical solution 1, wherein the first plurality of images are captured by at least one optical sensor, and at least one gaze direction associated with each of the first plurality of images is captured by a gaze direction system, and the gaze direction system is configured to determine the eye gaze direction of at least one eye.

[0026] 3. A method according to technical solution 2, wherein the second plurality of images are captured by the at least one optical sensor, and the depth information associated with each of the second plurality of images is captured by a distance sensor, and the distance sensor is configured to determine the distance between the object and the distance sensor.

[0027] 4. A method according to technical solution 1, wherein training a gaze prediction neural network on an eye gaze dataset includes performing dilation in at least one of the gaze directions.

[0028] 5. A method according to technical solution 4, wherein the expansion corresponds to a focal area of ​​the eye.

[0029] 6. A method according to technical solution 1, wherein training the gaze prediction neural network includes associating at least one eye gaze direction with a corresponding one of the first plurality of images.

[0030] 7. The method according to technical solution 1, wherein the output from the depth database includes converting the depth information into a standardized disparity map according to the standardized disparity label.

[0031] 8. The method according to technical solution 7, wherein the output from the deep database includes the second plurality of images.

[0032] 9. The method according to technical solution 8, wherein the resolution of the normalized disparity matches a scaled version of a corresponding one of the second plurality of images.

[0033] 10. A method according to technical solution 1, wherein training the disparity estimation neural network includes minimizing the minimum absolute deviation between the ground truth measurement and the prediction by the disparity estimation neural network.

[0034] 11. The method according to technical solution 10 includes performing back propagation when training the disparity estimation neural network to minimize the minimum absolute deviation.

[0035] 12. A method according to technical solution 1, wherein the first plurality of images matches the second plurality of images.

[0036] 13. A non-transitory computer-readable storage medium embodying programming instructions, the programming instructions, when executed by a processor, being operable to perform a method comprising:

[0037] obtaining an eye gaze dataset comprising a first plurality of images, wherein at least one gaze direction is associated with each of the first plurality of images;

[0038] training a gaze prediction neural network based on an eye gaze dataset to develop a model trained to provide gaze predictions for external images;

[0039] obtaining a depth database comprising a second plurality of images having depth information associated with each of the second plurality of images; and

[0040] Based on the output from the gaze prediction neural network and the output from the depth database, a disparity estimation neural network for object detection is trained.

[0041] 14. A non-temporary computer-readable storage medium according to technical solution 13, wherein the first plurality of images are captured by at least one optical sensor, and at least one gaze direction associated with each of the first plurality of images is captured by a gaze direction system, and the gaze direction system is configured to determine the eye gaze direction of at least one eye.

[0042] 15. A non-temporary computer-readable storage medium according to technical solution 14, wherein the second plurality of images are captured by the at least one optical sensor, and the depth information associated with each of the second plurality of images is captured by a distance sensor, and the distance sensor is configured to determine the distance between the object and the distance sensor.

[0043] 16. A non-transitory computer-readable storage medium according to technical solution 13, wherein training a gaze prediction neural network on an eye gaze dataset includes performing dilation in the at least one gaze direction.

[0044] 17. A non-transitory computer-readable storage medium according to technical solution 13, wherein training a gaze prediction neural network includes associating at least one eye gaze direction with a corresponding one of the first plurality of images.

[0045] 18. The non-transitory computer-readable storage medium according to technical solution 13, wherein the output from the depth database includes converting the depth information into a standardized disparity map according to the standardized disparity label.

[0046] 19. A non-transitory computer-readable storage medium according to technical solution 18, wherein the output from the deep database includes the second plurality of images.

[0047] 20. A vehicle system, the system comprising:

[0048] at least one optical sensor configured to capture a plurality of images;

[0049] at least one distance sensor configured to measure a plurality of distances from the at least one distance sensor;

[0050] an eye gaze measurement system configured to determine the driver's eye position; and

[0051] a controller in communication with the at least one optical sensor, the at least one distance sensor, and the eye gaze measurement system, wherein the controller is configured to:

[0052] obtaining an eye gaze dataset comprising a first plurality of images, wherein at least one gaze direction is associated with each of the first plurality of images;

[0053] training a gaze prediction neural network based on an eye gaze dataset to develop a model trained to provide gaze predictions for external images;

[0054] obtaining a depth database comprising a second plurality of images having depth information associated with each of the second plurality of images; and

[0055] Based on the output from the gaze prediction neural network and the output from the depth database, a disparity estimation neural network for object detection is trained. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 An example vehicle incorporating multiple sensors is illustrated.

[0057] Figure 2 A method for developing a neural network for disparity estimation is schematically illustrated.

[0058] Figure 3A and Figure 3B An exemplary data set sample including at least one optical image and a plurality of corresponding Light Detection and Ranging (LIDAR) measurements is illustrated.

[0059] Figure 4A , Figure 4B and Figure 4C By transforming the original image ( Figure 4A ), gaze prediction( Figure 4B ), and the normalized disparity of the ground truth (GT) ( Figure 4C ) illustrating the similarity between gaze prediction and depth estimation.

[0060] Figure 5 An example eye gaze image is illustrated. DETAILED DESCRIPTION

[0061] The present disclosure allows for many different forms of embodiments. Representative examples of the present disclosure are shown in the accompanying drawings and described in detail herein as non-limiting examples of the disclosed principles. For this reason, elements and limitations described in the abstract, background technology, summary of the invention, and detailed description but not explicitly stated in the claims should not be individually or collectively incorporated into the claims by implication, inference, or otherwise.

[0062] For the purposes of this specification, unless specifically stated otherwise, the use of the singular includes the plural, and vice versa, the terms "and" and "or" shall be both conjunctions and antonyms, and the words "comprises," "comprising," "including," "having," and the like shall mean "including but not limited to." In addition, approximate words such as "approximately," "almost," "substantially," "generally," "approximately," and the like may be used herein in the sense of "at, close to, or nearly at," "within 0-5% of," or "within acceptable manufacturing tolerances," or a logical combination thereof. As used herein, a component that is "configured to" perform a specified function is capable of performing the specified function without modification, not merely having the potential to perform the specified function after further modification. In other words, the described hardware, when explicitly configured to perform a specified function, is specifically selected, created, implemented, utilized, programmed, and / or designed for the purpose of performing the specified function.

[0063] According to an exemplary embodiment, Figure 1 A vehicle 20 is shown that can be operated in an autonomous mode or an automatic mode. The vehicle 20 can be a fully autonomous vehicle or a semi-autonomous vehicle. The vehicle 20 includes a drive system 22 that controls the autonomous operation of the vehicle. The drive system 22 includes: a sensor system 24 for obtaining information about the surroundings or environment of the vehicle 20; and a controller 26 for calculating possible actions of the autonomous vehicle based on the obtained information, and for implementing one or more of the possible actions; and a human-machine interface 28 for communicating with an occupant of the vehicle (such as a driver or a passenger). The sensor system 24 may include: at least one optical sensor 30, such as at least one camera; at least one distance sensor 32, such as a depth camera (RGB-D) or a lidar; and an eye gaze monitoring system 34. In the illustrated example, the optical sensor 30 and the distance sensor 32 include at least partially overlapping fields of view so that the information captured by each sensor can be correlated.

[0064] The controller 26 may include processing circuitry that may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) executing one or more software or firmware programs and memory, combinatorial logic circuits, and / or other suitable components that provide the described functionality. The controller 26 may include a non-transitory computer readable medium storing instructions that, when processed by one or more processors of the controller 26, implement the method 100 of developing a disparity estimation neural network according to one or more embodiments detailed herein. The controller 26 may operate a program that executes artificial intelligence. However, the present disclosure is applicable to decision systems that utilize or generate vectors (such as probability or confidence vectors) in the decision process.

[0065] In the present disclosure, a method 100 of developing a neural network for object detection is described. Figure 2 ) includes a pre-training phase (box 102) and a training phase (box 104). The pre-training phase at box 102 is an at least partially unsupervised training process that uses labels automatically generated based on gaze data of real end-goal task input, as will be described below. The pre-training phase precedes a supervised training phase (such as the training phase at box 104) to provide intuition and improve neural network performance by reducing training loss without performing manual labor-intensive labeling.

[0066] In the illustrated example, the pre-training phase (block 102) includes an eye gaze database (block 106). At block 110, the eye gaze database includes external images 58 ( Figure 4A ). The external image 58 may be captured by the optical sensor 30. In one example, the external image 58 captured while the vehicle 20 is traveling is downscaled to 500×500 pixels in resolution to manage data size, and is captured by the optical sensor 30 at a rate of 20 Hz.

[0067] The eye gaze database also includes information captured by the eye gaze monitoring system 34 (block 108). The information captured by the eye gaze monitoring system 34 may include information relative to the eye gaze image 50 ( Figure 5 ) recorded gaze direction 52 of the driver of the vehicle 20. The gaze direction 52 may correspond to a single point in the eye gaze image 50, or may include a small area in the eye gaze image 50. In one example, the eye gaze monitoring system 34 includes at least one optical sensor that can determine the gaze direction of one or both eyes of the driver. If the gaze direction of each eye of the driver is being measured, this information may also be used to estimate the distance to the vehicle that the driver is looking at.

[0068] Eye gaze image 50 may also include a resolution that matches the resolution of external image 58. This allows gaze direction 52 to be mapped onto external image 58 captured by optical sensor 30. In one example, for each external image, the mapping will include corresponding x and y coordinates that associate the location of gaze direction 52 with external image 58.

[0069] The eye gaze database may also include a timestamp for each eye gaze image 50 that captured the gaze direction 52, so as to match the gaze direction 52 with the corresponding external image 58 captured by the optical sensor 30. The gaze information is dense, i.e., for each pixel in the eye gaze image 50, there is a label indicating whether the driver is looking at that point.

[0070] At block 112, the driver's gaze direction 52 is expanded to create a label 56 corresponding to an area of ​​interest, such as an area where the fovea of ​​the eye may be focused. Figure 5 ). The size and shape of the region of interest of the label 56 can be varied to be more inclusive or less inclusive. Because the label 56 is automatically generated, the pre-training at box 102 does not require human intervention other than capturing the driver's gaze direction 52 while operating the vehicle 20. This can reduce the time required to label images in the pre-training dataset used to train the neural network and allow a larger amount of data to be used for pre-training purposes.

[0071] The method 100 sends the label 56 and input 116 (such as an external image 58) to box 118 to train the gaze prediction neural network. The gaze prediction neural network matches the region of interest of the label 56 generated at box 112 to at least one corresponding external image 58 from the external image set 116.

[0072] Alternatively, each external image 58 may include more than one corresponding region of interest label 56 based on the data collection rate of the external images 58 and the gaze directions 52. For example, the external images 58 may be captured at a rate of 20 Hz, while the gaze directions 52 may be captured at a rate of 60 Hz. Thus, each external image 58 may include up to three different corresponding labels 56 as determined from the gaze directions 52. However, if the gaze directions 52 are not within the boundaries of the eye gaze images 50, it is possible that some eye gaze images 50 may include overlapping labels 56 or no region of interest at all.

[0073] In one example at box 118, a U-net may be trained using an eye gaze corpus. In one example, 200,000 batches may be digested during training. Each batch may include four pairs of external images 58 and corresponding regions of interest for 56. The training at box 118 may then output a gaze prediction network at box 120, which may be stored in a memory in the controller 26 or in another location.

[0074] The gaze prediction network from block 120 may then be used to generate a gaze prediction image 60 ( Figure 4B ), which predicts whether or not there will be a label 56 and where the label 56 will be for a given external image 58. In the illustrated example, the gaze prediction image 60 includes an area with a solid circle corresponding to the highest gaze prediction, an area with a hollow circle indicating the next lower level of gaze prediction, an empty space indicating a neutral level of gaze prediction, and an area with an X indicating the lowest level of gaze prediction. However, other scales may be used, such as color or gradients according to the present disclosure.

[0075] During the training phase (block 104), the method 100 trains a disparity estimation network (block 142) for using depth information related to the external image 58 and the gaze prediction network (block 120) generated from the pre-training phase (block 102). The gaze prediction network provides intuition when training the disparity estimation network.

[0076] The training phase at box 104 includes a depth database at box 122. The depth database includes depth information captured by the distance sensor 32 (box 124) and external images captured by the optical sensor 30 (box 126). The information in the depth database can be collected while the driver manually drives the vehicle 20.

[0077] The depth database includes depth information captured by the distance sensor 32 at box 124. In one example, the depth information is in the form of a depth map. The depth map is generated from several scans of the distance sensor 32. The depth map includes significantly less data relative to the eye gaze image 50. This is because the depth map includes information for the area used to make depth measurements.

[0078] At block 128, the depth information is converted to a normalized disparity map according to the process disclosed herein in relation to normalized disparity labels. At 130, the normalized disparity map is used as the ground truth (GT) / label during this stage for training to build a normalized disparity estimation network (block 142).

[0079] In parallel, the external image 58 is provided as input at 132 to the network feed-forward with the GT / label 130 at box 134. The external image 58 from box 126 may also be downscaled to an image resolution of 500×500 pixels to improve data management and maintain a consistent pixel size for comparison.

[0080] The training phase (box 104) aligns the external images 58 captured by the optical sensor 30 with the depth information and distances to different objects in the scene. The optical sensor 30 and the distance sensor 32 are aligned so that when the optical sensor 30 captures one of the external images, the distance sensor 32 is performing several scans around the vehicle.

[0081] By providing aligned data, the portion of the scan from the range sensor 32 in the direction the optical sensor 30 is looking, the process can provide depth information for a set of pixels in the image corresponding to the area where the depth data is collected. However, due to the limitations of the range sensor 32 scan, the data may retain sparse depth information. Figure 3A External images in 55 and Figure 3B The dataset represented in the distance scan 57 in is smaller than the eye gaze dataset. In one example, the depth dataset contains approximately 10,000 images and depth scans. 9,000 of them can be defined as a training set, and the other 1,000 can be defined as a validation set.

[0082] like Figure 4C , the normalized disparity or depth map 62 includes solid squares indicating areas closest to the distance sensor 32, hollow squares identifying additional distances further away, and empty spaces where no data was collected so that depth information is unavailable for most pixels. This is in contrast to the eye gaze image 50, which describes each pixel in the eye gaze image 50 as either including or not including eye gaze.

[0083] On the other hand, depth datasets are sparse. Most pixels do not store depth readings. One feature of the present disclosure is to overcome this fundamental mismatch. On the one hand, eye gaze information is essentially dichotomous. Each reading stores a single bit (driver is looking, not looking). On the other hand, depth information is continuous. It spans from zero to infinity.

[0084] One feature of the present disclosure is to overcome the span of depth information by making it more dichotomous, such as distinguishing between near and far, rather than zero to infinity.

[0085] At box 128, the depth data set is transformed into a change in disparity to generate a normalized disparity. In one example, the normalized disparity is proportional to the inverse of the depth, such that disparity ∝ 1 / depth. Using 1 / depth causes high depth values ​​to converge around a single value. They are located just near zero. In addition, one can define a minimum depth (denoted as minDepth or mid) to act as a depth floor. One can define a close range within which high depth values ​​are nonlinearly compressed to a region near zero. This can create a continuous version of the {near, far} dichotomy, which may be similar in nature to the {see, don't see} dichotomy. This similarity is demonstrated in 4A, 4B, and 4C.

[0086] Figure 4A , Figure 4B and Figure 4C By processing the original image 58( Figure 4A ), gaze prediction image 60( Figure 4B ), and the normalized disparity of GT 62 ( Figure 4C ) depicts the similarity between gaze prediction and depth estimation. Figure 4A , Figure 4B and Figure 4C , one can make additional definitions and present the final stage of the transform and define maxDepth or mad.

[0087] At the end of the conversion, one can adapt the dynamic range of the neurons in the output layer of the pre-trained network. One can map it to a range from minNormDisparity or mind to maxNormDisparity or mand. In one example, it is [-2.5, 2.5] and one can use the following equation.

[0088]

[0089] nd=mind+pd(mand-mind) (2)

[0090] where d, nd, and pd are the measured depth, normalized disparity, and partial disparity, respectively.

[0091] The similarity between the {near, far} dichotomy and the {see, not see} dichotomy is shown in 4A, 4B and 4C. In the original image captured by the external camera (FIG. 4), the vehicle is located in the lower right corner of the image 58. From the perspective of the vehicle, it is beneficial to identify this object.

[0092] The eye gaze prediction of the external image 58 is Figure 4BThe solid circles and X marks (respectively) represent areas that are more and less likely to be seen by the driver, while the hollow circles identify areas in the middle of the likelihood spectrum. The gaze prediction network highlights the vehicle and its surroundings in the lower right corner as areas with a high probability of being observed by the driver.

[0093] The ground truth (GT) of the depth as measured by the distance sensor 32 is Figure 4C Due to the sparsity of the depth information from the distance sensor 32, only a small number of pixels store depth values. Figure 3B In , most of the solid squares are associated with the vehicle in the lower right corner of the external image 58. One can describe Figure 4B and Figure 4C The similarity between the two graphs illustrated in .

[0094] Because the loss is sparsely estimated, only pixels with normalized disparity scores (block 136) from GT / label 130 are used for L1 score estimation (block 138) by performing an L1 loss function (least absolute deviation) or another loss function. The output of the L1 loss can be back-propagated to block 134 via 140 for further refinement to minimize the error, which is the sum of absolute differences between the ground truth and the predicted values ​​by the neural network. Once the error has been minimized to a desired level at block 138, the normalized disparity estimation network is saved at block 142.

[0095] The terms "a" and "an" do not indicate a limitation of quantity, but rather indicate the presence of at least one of the referenced items. The term "or" means "and / or" unless the context clearly indicates otherwise. References throughout the specification to "aspects" mean that a particular element (e.g., a feature, structure, step, or characteristic) described in conjunction with that aspect is included in at least one aspect described herein, and may or may not be present in other aspects. In addition, it should be understood that the described elements may be combined in a suitable manner in the various aspects.

[0096] When an element (such as a layer, film, region, or substrate) is referred to as being “on” another element, it can be directly on the other element or intervening elements may also be present. In contrast, when an element is referred to as being “directly on” another element, there are no intervening elements present.

[0097] Unless otherwise specified herein, the test standard is the most recent standard in effect as of the filing date of this application (or, if priority is claimed, the filing date of the earliest priority application in which the test standard appears).

[0098] Unless defined otherwise, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0099] Although the above disclosure has been described with reference to exemplary embodiments, it will be appreciated by those skilled in the art that various changes may be made and that equivalents may be substituted for its elements without departing from the scope thereof. In addition, many modifications may be made to adapt specific situations or materials to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is intended that the present disclosure is not limited to the specific embodiments disclosed, but will include embodiments that fall within its scope.

Claims

1. A method for training a disparity estimation network, the method comprising: obtaining an eye gaze dataset comprising a first plurality of images, wherein at least one gaze direction is associated with each of the first plurality of images; training a gaze prediction neural network based on an eye gaze dataset to develop a model trained to provide gaze predictions for external images; obtaining a depth database comprising a second plurality of images, the second plurality of images having depth information associated with each of the second plurality of images; and Based on the output from the gaze prediction neural network and the output from the depth database, a disparity estimation neural network for object detection is trained.

2. The method of claim 1 , wherein the first plurality of images are captured by at least one optical sensor and at least one gaze direction associated with each of the first plurality of images is captured by a gaze direction system configured to determine an eye gaze direction of at least one eye.

3. The method of claim 2, wherein the second plurality of images are captured by the at least one optical sensor, and the depth information associated with each of the second plurality of images is captured by a range sensor configured to determine a distance between an object and the range sensor. 4 . The method of claim 1 , wherein training a gaze prediction neural network on the eye gaze dataset comprises performing dilation in the at least one gaze direction.

5. The method of claim 4, wherein the dilation corresponds to a focal area of ​​the eye.

6. The method of claim 1, wherein training a gaze prediction neural network comprises associating at least one eye gaze direction with a corresponding one of the first plurality of images.

7. The method of claim 1, wherein the output from the depth database comprises converting the depth information into a normalized disparity map according to normalized disparity labels. The method of claim 7 , wherein the output from the deep database comprises the second plurality of images. 9 . The method of claim 8 , wherein a resolution of the normalized disparity matches a scaled version of a corresponding one of the second plurality of images.

10. The method of claim 1, wherein training the disparity estimation neural network comprises minimizing a minimum absolute deviation between a ground truth measurement and a prediction by the disparity estimation neural network.

Images