Occupancy estimation system for vehicle

Abstract

A computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform operations. Operations include generating occupancy probabilities at one or more voxels at an occupancy estimation network, predicting a gaze direction of a gaze prediction via a gaze prediction model, and identifying an object of interest based on the predicted gaze direction. The operations further include generating, via the occupancy estimation application, a gaze saliency map based on the gaze direction and the identified object of interest, and updating an occupancy probability of the one or more voxels based on the determined gaze direction and gaze saliency map.

Description

Technical Field

[0001] This disclosure generally relates to a gaze-based estimation system for vehicle occupancy estimation. Background Technology

[0002] The information provided in this section is intended to provide a general overview of the background of this disclosure. To the extent described in this section, the work of the currently named inventors, and aspects of the description that may not conform to the prior art at the time of submission, are neither explicitly nor implicitly acknowledged as prior art relative to this disclosure.

[0003] Occupation estimation is used to classify whether voxels in space are occupied in order to generate relevant images. For example, the accuracy of occupancy estimation decreases when objects are far from or relatively far from vehicles. Accuracy for distant objects is affected by low camera resolution. For example, distant objects typically project only a few pixels, resulting in a poor signal-to-noise ratio. Therefore, there is a need to improve occupancy estimation for objects at a certain distance from vehicles to improve image resolution quality. Summary of the Invention

[0004] In some aspects, a computer-implemented method causes data processing hardware to perform operations when executed by data processing hardware. The operations include generating occupancy probabilities at one or more voxels at an occupancy estimation network, tracking an occupant's gaze via a camera system, identifying objects of interest based on the tracked gaze, and determining the occupant's gaze direction based on the tracked gaze and one or more identified objects of interest. The operations also include generating a gaze saliency map based on the gaze direction and the identified objects of interest via occupancy estimation, and updating the occupancy probabilities of one or more voxels based on the determined gaze direction and the gaze saliency map.

[0005] In some examples, the operation may include training an occupancy estimation network via a gaze prediction model. In other examples, identifying objects of interest may include generating a two-dimensional Gaussian function applied to a two-dimensional image space of the gaze saliency map via occupancy estimation. The operation may also include projecting one or more voxels onto the two-dimensional image space. In some cases, updating the occupancy probability may include collecting a two-dimensional index for each projected voxel based on the one or more voxels of the projection. The operation may also include applying a threshold to the updated occupancy probability. In some configurations, the gaze may include at least one of smooth tracking, fixed gaze, and rapid saccades.

[0006] In other aspects, a computer-implemented method causes data processing hardware to perform operations when executed by data processing hardware. The operations include generating occupancy probabilities at one or more voxels at an occupancy estimation network, predicting a gaze direction via a gaze prediction model, and identifying an object of interest based on the predicted gaze direction. The operations also include generating a gaze saliency map based on the gaze direction and the identified object of interest via occupancy estimation, and updating the occupancy probabilities of one or more voxels based on the determined gaze direction and the gaze saliency map.

[0007] In some examples, the operation may include training an occupancy estimation network via a model trainer of a gaze prediction model. In some implementations, identifying objects of interest may include generating a two-dimensional Gaussian function applied to a two-dimensional image space of the gaze saliency map via occupancy estimation. The operation may also include projecting one or more voxels onto the two-dimensional image space. In some cases, updating the occupancy probability may include collecting a two-dimensional index for each projected voxel based on the one or more voxels of the projection. The operation may also include applying a threshold to the updated occupancy probability.

[0008] In a further aspect, an occupancy estimation system for a vehicle includes data processing hardware and memory hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. These operations include generating occupancy probabilities at one or more voxels at an occupancy estimation network, tracking an occupant's gaze via a camera system, identifying objects of interest based on the tracked gaze, and determining a gaze direction based on the tracked gaze and one or more identified objects of interest. The operations also include generating a gaze saliency map based on the gaze direction and the identified objects of interest via occupancy estimation, and updating the occupancy probabilities of one or more voxels based on the determined gaze direction and the occupancy map.

[0009] In some examples, the operation may include training an occupancy estimation network via a gaze prediction model. Optionally, identifying objects of interest may include generating a two-dimensional Gaussian function applied to a two-dimensional image space of the gaze saliency map via the occupancy estimation. The operation may also include projecting one or more voxels onto the two-dimensional image space. In some implementations, updating the occupancy probability may include collecting a two-dimensional index for each projected voxel based on the one or more voxels of the projection. The operation may also include applying a threshold to the updated occupancy probability. In some cases, the gaze may include at least one of smooth tracking, fixed gaze, and rapid saccade. Attached Figure Description

[0010] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0011] Figure 1 This is a schematic diagram of a vehicle occupancy estimation system based on this disclosure;

[0012] Figure 2 It is a partial interior view of a vehicle equipped with a camera system for monitoring the driver's gaze according to this disclosure;

[0013] Figure 3 This is an exemplary block diagram of an occupancy estimation system based on this disclosure;

[0014] Figure 4 This is a schematic diagram of a vehicle equipped with an occupancy estimation system according to this disclosure, the vehicle being on a road and detecting objects;

[0015] Figure 5 This is an exemplary block diagram of an occupancy estimation system according to the present disclosure, which includes a view;

[0016] Figure 6 This is another exemplary block diagram of an occupancy estimation system according to the present disclosure, which includes a gaze prediction model;

[0017] Figure 7 This is an exemplary flowchart of an occupancy estimation system based on this disclosure; and

[0018] Figure 8 This is another exemplary flowchart of a occupancy estimation system.

[0019] In all the accompanying drawings, the corresponding reference numerals indicate the corresponding parts. Detailed Implementation

[0020] The example configuration will now be described more fully with reference to the accompanying drawings. The example configuration is provided so that this disclosure will be thorough and will fully communicate the scope of this disclosure to those skilled in the art. Specific details, such as examples of specific components, devices, and methods, are set forth to provide a thorough understanding of the configuration of this disclosure. It will be apparent to those skilled in the art that the specific details are not required, that the example configuration may be implemented in many different forms, and that the specific details and exemplary configuration should not be construed as limiting the scope of this disclosure.

[0021] The terminology used herein is for the purpose of describing specific exemplary configurations only and is not intended to be limiting. As used herein, the singular articles “a,” “an,” and “the” may also be intended to include plural forms unless the context clearly indicates otherwise. The terms “comprising,” “including,” “containing,” and “having” are inclusive, thus specifying the presence of features, steps, operations, elements, and / or components, but not excluding the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein should not be construed as requiring them to be performed in the specific order discussed or shown, unless specifically identified as such. Additional or alternative steps may be employed.

[0022] When an element or layer is referred to as “on another element or layer,” “joined to,” “connected to,” “attached to,” or “linked to” another element or layer, it may be directly on, joined to, connected to, attached to, or linked to the other element or layer, or there may be intermediate elements or layers present. Conversely, when an element is referred to as “directly on another element or layer,” “directly joined to,” “directly connected to,” “directly attached to,” or “directly linked to” another element or layer, there may be no intermediate elements or layers present. Other terms used to describe relationships between elements should be interpreted in a similar manner (e.g., “between” vs. “directly between,” “adjacent” vs. “directly adjacent,” etc.). As used herein, the term “and / or” includes any and all combinations of one or more of the related listed items.

[0023] The terms first, second, third, etc., may be used herein to describe various elements, components, regions, layers, and / or parts. These elements, components, regions, layers, and / or parts should not be limited by these terms. These terms are used only to distinguish individual elements, components, regions, layers, or parts. Terms such as “first,” “second,” and other numerical terms do not imply order or sequence unless the context clearly indicates otherwise. Therefore, the first element, component, region, layer, or part discussed below may be referred to as the second element, component, region, layer, or part without departing from the teachings of the example configuration.

[0024] In this application, including the following definitions, the term "module" may be replaced by the term "circuit". The term "module" may refer to or be a part of an application-specific integrated circuit (ASIC), or include ASICs; digital, analog, or mixed-signal analog / digital discrete circuits; digital, analog, or mixed-signal analog / digital integrated circuits; combinational logic circuits; field-programmable gate arrays (FPGAs); processors (shared, dedicated, or grouped) that execute code; memory (shared, dedicated, or grouped) that stores code executed by the processor; other suitable hardware components that provide the functions described; or some or all of the above, such as in a system-on-a-chip.

[0025] The term "code" as used above can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared processor" includes a single processor that executes some or all of the code from multiple modules. The term "group processor" includes a processor, in conjunction with an additional processor, that executes some or all of the code from one or more modules. The term "shared memory" includes a single memory that stores some or all of the code from multiple modules. The term "group memory" includes memory, in conjunction with additional memory, that stores some or all of the code from one or more modules. The term "memory" can be a subset of the term "computer-readable medium." The term "computer-readable medium" does not include transient electrical and electromagnetic signals propagating through the medium and can therefore be considered tangible, non-transitory memory. Non-limiting examples of non-transitory memory include tangible computer-readable media, including non-volatile memory, magnetic memory, and optical memory.

[0026] The apparatus and methods described in this application may be implemented, in whole or in part, by one or more computer programs executed by one or more processors. The computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. The computer program may also include and / or depend on stored data.

[0027] A software application (i.e., a software resource) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0028] Non-transitory memory can be a physical device used for temporary or permanent storage of programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0029] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0030] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These different implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, at least one input device, and at least one output device, the programmable processor being dedicated or general-purpose, coupled to receive data and instructions from and send data and instructions to the storage system.

[0031] The processes and logic flows described in this specification can be executed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by manipulating input data and generating output. These processes and logic flows can also be executed by special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include or be operatively coupled to one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented or incorporated therein by dedicated logic circuitry.

[0032] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, and optional keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0033] refer to Figure 1-3Occupancy estimation system 10 is configured as part of vehicle 100 including electronic control unit (ECU) 12. ECU 12 is configured with occupancy estimation application 14, and occupancy estimation system 10 is designed to estimate objects 200 outside vehicle 100 based on the gaze 102 of occupant 104 of vehicle 100. For example, some objects 200 may be far away relative to vehicle 100, and vehicle 100's camera system 106 may otherwise capture incomplete image data 108. Incomplete image data 108 may be blurry, pixelated, or unclear. Therefore, occupancy estimation system 10 analyzes occupant 104's gaze 102 to identify objects 200, and occupancy estimation application 14 is used to more clearly identify objects 200 as objects of interest 200, as described below. Thus, occupant 104's gaze 102 can be used to increase the likelihood of a relevant object 200 (i.e., object of interest 200) located far away from vehicle 100.

[0034] Figure 2 An example of occupancy estimation system 10 capturing the gaze 102 of occupant 104 is shown. The gaze 102 is captured as image data 108 by camera system 106 and transmitted to ECU 12 for analysis via occupancy estimation application 14. Camera system 106 is configured as part of vehicle 100 such that it includes cameras 106a located within the interior compartment 110 and along the body 112 of vehicle 100. Cameras 106a can be configured as any feasible imager, including but not limited to LiDAR and infrared cameras. Interior cameras 106a are configured to monitor occupant 104 to capture the gaze 102, and exterior cameras 106a are configured to monitor the external space to locate potential objects of interest 200. Therefore, image data 108 includes the gaze 102 captured by camera system 106 and an external image 114. As part of occupancy estimation system 10, external image 114 may include the object of interest 200 and surrounding objects that would otherwise create noise. Therefore, the occupancy estimation application 14 is designed to distinguish objects of interest 200 that may be part of the external image 114 by analyzing gaze 102, as described in more detail below.

[0035] Further reference Figure 1-3The vehicle 100 may also be equipped with a navigation system 300, which is communicatively coupled to the ECU 12 to provide Global Positioning System (GPS) data 302. The ECU 12 can use the GPS data 302 to inform the vehicle 100 of its location, which can be used by the occupancy estimation application 14 when analyzing the image data 108. For example, the occupancy estimation application 14 can compare the GPS data 302 with the image data 108 received from the camera system 106 to aid in triangulation of potential objects of interest 200. The GPS data 302 may be particularly advantageous in examples where the vehicle 100 is an autonomous or semi-autonomous vehicle 100, as described below.

[0036] Now for reference Figure 2-5 Occupancy estimation application 14 is executed by data processing hardware 16 of ECU 12. ECU 12 also includes storage hardware 18 that communicates with data processing hardware 16. Memory hardware 18 stores instructions that, when executed on data processing hardware 16, cause data processing hardware 16 to perform the operations described herein. Occupancy estimation application 14 includes occupancy estimation network 20, which can communicate with eye gaze database 22 stored on memory hardware 18.

[0037] The eye gaze database 22 may be populated with image data 108 received from the camera system 106 and stored for use by the gaze estimation 24 of the occupancy estimation application 14. The eye gaze database 22 may be used to communicate with the occupancy estimation network 20 and the gaze estimation 24 to generate a gaze saliency map 26. In some cases, the eye gaze database 22 may be used for autonomous functions of the vehicle 100. For example, the vehicle 100 may be configured as an autonomous and / or semi-autonomous vehicle 100 such that when an occupant 104 is a passenger of the vehicle 100 and the occupant 104's gaze 102 is not actively captured, the vehicle 100 may utilize the eye gaze database 22 to perform the occupancy estimation application 14.

[0038] In this example, when occupant 104 operates vehicle 100, occupant 104's gaze 102 may differ from that of occupant 104. Therefore, occupancy estimation application 14 can utilize eye gaze database 22 to generate and update gaze saliency map 26 while performing occupancy estimation network 20 and gaze estimation 24, as described in more detail below. Eye gaze database 22 stores gaze patterns 102a as historical gaze data collected based on image data 108. As described herein, occupancy estimation application 14 utilizes gaze patterns 102a to estimate gaze 102 in the example of autonomous vehicle 100.

[0039] A gaze 102 may include different types of eye movements. For example, a gaze 102 includes, but is not limited to, at least one of smooth tracking, convergence and divergence, vestibular vision, fixed gaze, and rapid saccadic gaze. During smooth tracking and fixed gaze, the occupant 104 tracks the object of interest 200 and generally maintains a consistent gaze 102 in the direction of the object 200. While a constant gaze 102 can be maintained, the occupant 104 also maintains a consistent gaze pattern 102a pointing toward the road and the vehicle 100's trajectory while the gaze 102 is also monitoring the object 200. In contrast, rapid saccadic gaze corresponds to rapid eye movements of the gaze 102, allowing the occupant 104's gaze 102 to scan an external area without focusing on a specific object of interest 200. Thus, an object 200 considered important is generally associated with a gaze 102 defined as smooth tracking and / or fixed gaze because the gaze 102 involves longer and more repetitive fixation and tracking.

[0040] In some cases, a rapid scan gaze 102 may precede a fixed gaze and a smooth tracking gaze 102, such that the rapid scan gaze 102 can predict the object of interest 200 after at least one of the smooth tracking gaze 102 and the fixed gaze 102. Regarding the smooth tracking gaze 102, the smooth tracking gaze 102 indicates that one or both of the vehicle 100 and the object 200 are moving. In the case of the smooth tracking gaze 102, gaze pattern analysis 28 can triangulate the position of the object 200 based on the position of the vehicle 100, the gaze direction 30, and other relevant GPS data 302. To capture the gaze direction 30, the gaze 102 is projected into two dimensions, as described below. Therefore, gaze pattern analysis 28 can be used to determine whether the gaze 102 is pointing at the object 200 (i.e., the object of interest 200) and to identify the position of the object of interest 200 in part based on the two-dimensional gaze direction 30.

[0041] Each of the rapid saccades, steady tracking, and fixed gaze can be classified as one of the gaze patterns 102a stored in the eye gaze database 22 of the memory hardware 18. The gaze 102 can be classified with the corresponding gaze pattern 102a by time-series and / or sequence analysis performed by the gaze estimation 24 of the occupancy estimation application 14. For example, the gaze pattern analysis 28 of the gaze estimation 24 is configured to identify the gaze pattern 102a. The gaze pattern analysis 28 can include, but is not limited to, methods such as Long Short-Term Memory (LSTM) models, Markov transition fields, and classification trees.

[0042] Still referencing Figure 2-5An object of interest 200 can be identified by tracking gaze 102 and recording gaze pattern 102a in an eye gaze database 22. The eye gaze database 22 can record time-based classifications of gaze 102 to aid gaze pattern analysis 28 in identifying corresponding gaze patterns 102a. For example, time-based classification can distinguish between smooth tracking, stationary, and rapid scanning gazes 102. Gauge pattern 102a can be further analyzed by occupancy estimation application 14 by evaluating the direction and duration of gaze 102. When evaluating gaze pattern 102a, gaze estimation 24 compares gaze direction 30 with GPS data 302 received from navigation system 300, taking into account the vehicle 100's own motion, to identify tracking of stationary object 200 based on the vehicle 100's position.

[0043] Figure 4 An example of time-based gaze capture 102 is shown. For example, vehicle 100 t1 Shown in conjunction with the first gaze 102 t1 The relevant first point in time. When vehicle 100 is moving along the road, at the time when vehicle 100 t2 The second position captures the second gaze 102 t2 Finally, regarding the 100 vehicles associated with the third time point... t3 Another location captured the third gaze 102 t3 The vehicle in this example is 100. t1 -100 t3 Moving or traveling along the road, while one or more objects 200 remain stationary. A gaze at each point in time 102 t1 -102 t3 It is captured and communicated with ECU12 for potential storage in eye gaze database 22 and for use in occupancy estimation application 14. Although Figure 4 It shows vehicle 100 t1 -100 t3 This is a single example of three time points, but it is conceivable that the time points captured by the camera system 106 could be more than three and / or less than three.

[0044] Further reference Figure 2-5In some examples, vehicle 100 may be stationary, or object 200 may be directly in front of the vehicle. Therefore, gaze direction 30 can help identify gazes toward object 200. A gaze 102 (i.e., a fixed gaze) indicates the presence of object 200 of interest, which can trigger gaze pattern analysis 28. Gaze pattern analysis 28 can calculate gaze direction 30 in global coordinates to illustrate the self-motion and / or movement of object 200 relative to vehicle 100 by comparing gaze 102 with GPS data 302. Gaze pattern analysis 28 can then assign the potential importance of object 200 based on the duration of gaze pattern 102a (i.e., fixed gaze or smooth tracking), the likelihood that occupant 104 would naturally look toward object 200, and the presence of a significant gaze pattern 102a (i.e., rapid saccade) that leads to a gaze toward object 200.

[0045] For illustrative purposes, one example of the gazing behavior could include a ball rolling into the road far ahead of vehicle 100. Occupant 104 could quickly scan the ball and track (i.e., trace) its movement. Occupant 104's gaze 102 could then quickly scan an area along the road, looking for objects associated with the ball 200, and could track any object 200 associated with the ball 200. In another non-limiting example, Figure 4 An example is shown in which an occupant 104 may have a straight, stationary gaze 102 (i.e., a prolonged gaze) on an object 200 along the road. As the object 200 gets closer to the vehicle 100, the gaze pattern 102a can change from a stationary gaze 102 to a tracking gaze 102.

[0046] Continue to refer to Figure 2-5 Occupancy estimation network 20 is configured to generate a voxel grid 32, where each voxel 34, as described below, is either empty or full. For example, voxel grid 32 is binary and provides the geometry of image data 108. Voxel grid 32 is generated based on image data 108 received from camera system 106. For example, occupancy estimation network 20 receives image data 108 as input and outputs voxel grid 32. The original image data 108 transformed onto voxel grid 32 may not be aware of the occupancy of a given voxel 34.

[0047] Therefore, the occupancy estimation network 20 is configured to generate an occupancy probability 36 at one or more voxels 34. For example, the occupancy estimation network 20 can provide an occupancy probability 36 associated with the probability that each voxel 34 is occupied. The occupancy probability 36 is recorded as a score between zero (0) and one (1), where a score of 0 reflects that the voxel 34 is not occupied and a score of 1 reflects a high probability that the voxel 34 is occupied. Thus, the occupancy probability 36 represents the likelihood or probability that the object of interest 200 occupies a given voxel 34. The occupancy probability 36 can be calculated as a measure between values ​​or as a continuous value, reflected in the score associated with a given voxel 34. The occupancy probability 36 is used by the occupancy estimation application 14 in conjunction with gaze estimation 24 to track whether the object of interest 200 is identified.

[0048] The gaze direction 30 is projected and transformed into a two-dimensional (2D) image space 38. Furthermore, a 2D Gaussian function 40 is defined in the same 2D space, centered on the gaze direction, and has a standard deviation 42 stored in the memory hardware 18. Voxels 34 are data points on a three-dimensional (3D) grid (i.e., voxel grid 32). These voxels 34 are projected onto a 2D image space 38 that defines the correlation between 3D points and the 2D space. The Gaussian function 40 is used to reweight the occupancy probability 36 of each voxel 34 based on its projection into the common 2D space.

[0049] For example, in response to the identification of object of interest 200, a 2D Gaussian function 40 is created on 2D image space 38. Object of interest 200 is identified based on the gaze 102 of the tracked occupant 104 and the captured external image 110 by a camera system 106. The 2D Gaussian function 40 is defined for the corresponding pixel 44 corresponding to the projected voxel 34. For example, voxels 34 are projected onto 2D image space 38 to have corresponding pixels 44 on 2D image space 38. Occupancy estimation application 14 uses pixels 44, the 2D Gaussian function 40, gaze estimation 24, and 2D image space 38 to update the gaze saliency map 26, and thus update the occupancy probability 36. Pixels 44 can be classified by 2D indexes 46 of 2D image space 38. 2D indexes 46 are associated with each voxel 34, such that updating the occupancy probability 36 includes collecting the 2D indexes 46 of each voxel 34 based on the projected voxels 34 on 2D image space 38.

[0050] As an example and not a limitation, the occupancy estimation application 14 can use the measured gaze direction 30 and project the gaze direction 30 into the 2D image space 38. The occupancy estimation application 14 can then define a 2D Gaussian function 40 in the 2D image space 38 and present the position of the occupant 104's gaze 102 as corresponding to the position of the object 200 with a high probability. Thus, the 2D Gaussian function 40 can thus be centered on the gaze direction 30, and the occupancy estimation application 14 retrieves the relevant standard deviation 42 of the 2D Gaussian function 40.

[0051] The result is that the occupancy estimation application 14 determines a high probability that an object 200 exists within a given voxel 34. For each individual voxel 34, the occupancy estimation application 14 adjusts the occupancy probability 36 of the occupied voxel 34 based on a 2D Gaussian function 40. For example, if the first voxel 34 is projected onto the first pixel 44, then the 2D Gaussian function 40 will update the occupancy probability 36 of the first voxel 34 according to the gaze 102. The occupancy probability 34 is corrected by weighting the occupancy probability 34 according to the value of the Gaussian function 40, which defines the gaze saliency map 26, representing the probability that the gaze 102 points to a specific location. As each 3D voxel 34 is projected into a single 2D image space 38, the occupancy estimation application 14 is able to find or determine the value of the 2D Gaussian function 40 for each 3D voxel 34. After all updates are complete, a threshold 48 can be applied to the occupancy probability 36.

[0052] Now for reference Figure 5 and 6 The occupancy estimation application 14 may also include a gaze prediction model 50. The gaze prediction model 50 may be configured with a model trainer 52, configured to obtain training data 54 for training the gaze prediction model 50. The gaze prediction model 50 may be configured as a machine learning model, such that the model trainer 52 is configured to train the gaze prediction model 50 based on the training data 54. The model trainer 52 may retrieve the training data 54 from, for example, an eye gaze database 22, such that the training data 54 includes, but is not limited to, stored gaze patterns 102a of historical gazes 102. The training data 54 may also include any other type of data received during the training of the gaze prediction model 50. For example, the training data 54 may include stored image data 108. The gaze prediction model 50 is trained to predict gazes 102 based on the training data 54. Therefore, when the vehicle 100 is operating autonomously, the gaze prediction model 50 is trained to output a gaze prediction 56, which is projected to relate to the gaze 102 of the occupant 104 when the occupant 104 may be operating the vehicle 100. Thus, when the gaze prediction model 50 is executed to output the gaze prediction 56, the occupancy estimation application 14 can rely on the eye gaze database 22.

[0053] Occupation estimation application 14 may periodically enter training mode 58. During training mode 58, occupation estimation application 14 may fail to predict gaze 102 because model trainer 52 is training occupation estimation application 14 to better predict gaze 102. Since the gaze estimate 24 of occupation estimation application 14 depends on gaze 102, gaze prediction model 50 effectively determines the predicted gaze 102 based on eye gaze database 22 and image data 108 received from camera system 106. In some examples, occupation estimation application 14 may periodically execute training mode 58 to periodically update gaze saliency map 26 based on training data 54, and thus update predicted gaze 102.

[0054] Now for reference Figure 7-8 An exemplary flowchart of the occupancy estimation system 10 is shown. In a first example, the occupancy estimation system 10 tracks gaze 102 at 400 and projects gaze 102 into two dimensions at 402 to identify gaze direction 30. Then, the occupancy estimation system 10 performs gaze pattern analysis 28 using gaze direction 30 at 404. The occupancy estimation system 10 also outputs an updated occupancy probability for gaze pattern analysis 28.

[0055] After performing gaze pattern analysis 28, the occupancy estimation system 10 was at 406 ( Figure 7 The system determines whether an object of interest 200 is detected. If not, the occupancy estimation system 10 returns to tracking gaze 102. If the object of interest 200 is detected, the occupancy estimation system 10 determines the value of the 2D Gaussian function 40 at 408. The occupancy estimation system 10 also projects 3D voxels 34 onto the 2D image space 38 at 410 to obtain a 2D index 36, and samples the 2D Gaussian function 40 at the calculated 2D index 36 at 412. The occupancy estimation system 10 can then execute the 2D Gaussian function 40 at 414, which can be used to update the occupancy probability 36.

[0056] In another example, occupancy estimation system 10 executes gaze prediction model 50 at 600. Occupancy estimation system 10 can then perform gaze pattern analysis 28 at 602 and determine at 604 whether an object of interest 200 is detected. If not, occupancy estimation system 10 returns to executing gaze prediction model 50. If an object of interest 200 is detected, occupancy estimation system 10 determines the value of 2D Gaussian function 40 at 606. Occupancy estimation system 10 projects 3D voxels 34 onto 2D image space 38 at 608 to obtain 3D indices, and samples 2D Gaussian function 40 at the calculated 2D index 46 at 610. Occupancy estimation system 10 can then update occupancy probability 36 at 612 by reweighting based on the sampled value from 2D Gaussian function 40.

[0057] Refer again Figure 1-8The occupancy estimation system 10 advantageously assists in predicting and estimating the occupancy probability 36 of the object of interest 200 based on the occupant's gaze 102. Furthermore, by using a gaze prediction model 50 trained on the eye gaze database 22 and configured to analyze image data 108 received from the camera system 106, the occupancy estimation system 10 can be advantageously used in the autonomous vehicle 100. Thus, the gaze prediction model 50 can be trained to efficiently provide automatic gaze estimation for the occupancy estimation system 10, predicting the gaze 102 even in the absence of the occupant 104's gaze. The occupancy estimation system 10 ultimately provides an improved estimate of the probability of a relevant object 200 being located away from the vehicle 100.

[0058] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are also within the scope of the following claims.

[0059] The foregoing description has been provided for purposes of illustration and description. It is not intended to be exhaustive or limiting of this disclosure. Individual elements or features of a particular configuration are generally not limited to that particular configuration, but where applicable, they are interchangeable and can be used in selected configurations, even if not specifically shown or described. This can also be varied in many ways. Such variations should not be considered as departing from this disclosure, and all such modifications are intended to be included within the scope of this disclosure.

Claims

1. A computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform operations including: Generate the occupancy probability at one or more voxels at the occupancy estimation network; The system tracks the occupants' gaze via a camera system; Identify objects of interest based on the tracked gaze; A gaze salience map is generated based on the identified objects of interest through an occupancy estimation application; as well as Based on the gaze saliency map, update the occupancy probability of one or more voxels.

2. The method of claim 1, further comprising training the occupancy estimation network via a gaze prediction model.

3. The method according to claim 1, wherein, Identifying objects of interest involves generating a two-dimensional Gaussian function in the two-dimensional image space of the gaze saliency map via the occupancy estimation.

4. The method of claim 3 further includes projecting the one or more voxels onto the two-dimensional image space.

5. The method according to claim 4, wherein, Updating the occupancy probability includes collecting a two-dimensional index for each projected voxel based on one or more projected voxels.

6. The method of claim 5, further comprising applying a threshold to the updated occupancy probability.

7. The method according to claim 1, wherein, A gaze includes at least one of smooth tracking, fixed gaze, and rapid saccade.

8. The method of claim 1, further comprising determining the occupant’s gaze direction based on one or more of the tracked gaze and the identified objects of interest.

9. The method of claim 1, further comprising predicting the gaze direction of the gaze prediction via a gaze prediction model.

10. A vehicle including a controller configured to perform the method of claim 1.

Images