Augmented reality information overlay with gaze tracking

Abstract

A method for displaying information to a vehicle occupant can include generating a three-dimensional (3D) map of an environment surrounding a vehicle. The 3D map includes a plurality of 3D bounding boxes. Each of the plurality of 3D bounding boxes corresponds to one of a plurality of landmarks. The method can also include determining a pose of a vehicle-mounted camera. The method can also include determining a gaze vector of the vehicle occupant. The method can also include determining a selected landmark of the plurality of landmarks based at least in part on the plurality of 3D bounding boxes, the pose of the vehicle-mounted camera, and the gaze vector of the occupant. The method can also include displaying information about the selected landmark to the occupant using a display.

Description

Technical Field

[0001] This disclosure relates to systems and methods for displaying information to vehicle occupants. Background Technology

[0002] To provide information in vehicle applications, various display systems can be utilized. Display systems can be configured to present information such as speed, navigation instructions, system diagnostics, entertainment options, etc. In some examples, display systems are configured as touchscreens with integrated haptic feedback, allowing for intuitive user interaction. Display systems may include additional features such as adaptive brightness control to enhance visibility in various lighting conditions, voice command integration for hands-free operation, and head-up display technology. Displays can also support wireless communication protocols, allowing them to interact with mobile devices, cloud services, and other vehicle systems. Display systems can use wireless communication to retrieve information from external sources (e.g., the internet) to display to vehicle occupants. For example, a display system can be used to provide vehicle occupants with information about the external conditions of the vehicle, including, for example, weather conditions, traffic conditions, points of interest, or destination information.

[0003] While the systems and methods used to display information have achieved their intended purpose, new and improved systems and methods are still needed to provide information to vehicle occupants. Summary of the Invention

[0004] According to several aspects, a method for displaying information to vehicle occupants is provided. The method may include generating a three-dimensional (3D) map of the vehicle's surrounding environment. The 3D map includes multiple 3D bounding boxes. Each of the multiple 3D bounding boxes corresponds to one of a plurality of landmarks. The method may further include determining the pose of an onboard camera based at least in part on one or more camera images of the vehicle's surrounding environment captured using an onboard camera. The method may further include determining the gaze vector of a vehicle occupant. The method may further include determining a selected landmark among the multiple landmarks based at least in part on the multiple 3D bounding boxes, the pose of the onboard camera, and the gaze vector of the occupant. The method may further include displaying information about the selected landmark to the occupant using a display.

[0005] In another aspect of this disclosure, generating a 3D map may further include generating a 3D point cloud comprising multiple 3D points based on multiple reference images including multiple landmarks. Each of the multiple 3D points is defined by a 3D point feature vector. Generating a 3D map may further include determining each of the multiple 3D bounding boxes by clustering the 3D point cloud into multiple point cloud clusters.

[0006] In another aspect of this disclosure, clustering 3D point clouds may further include generating multiple point cloud clusters using a density-based noise applied spatial clustering (DBSCAN) algorithm. Clustering 3D point clouds may also include filtering the multiple point cloud clusters to generate multiple filtered point cloud clusters. Each of the multiple filtered point cloud clusters corresponds to one of a plurality of landmarks.

[0007] In another aspect of this disclosure, filtering multiple point cloud clusters may further include segmenting one or more of the multiple point cloud clusters to generate multiple segmented point cloud clusters. Each of the multiple segmented point cloud clusters corresponds to only one of a plurality of landmarks. The multiple filtered point cloud clusters comprise the multiple segmented point cloud clusters.

[0008] In another aspect of this disclosure, filtering multiple point cloud clusters may further include determining two or more of the multiple segmented point cloud clusters to correspond to the same landmark among multiple landmarks, based at least in part on at least one of the following: the relative size of the two or more of the multiple segmented point cloud clusters and the distance between the two or more of the multiple segmented point cloud clusters.

[0009] In another aspect of this disclosure, determining the pose of the vehicle-mounted camera may further include capturing one or more camera images using the vehicle-mounted camera. Determining the pose of the vehicle-mounted camera may further include detecting one or more detected landmarks from a plurality of landmarks in the one or more camera images. Detecting one or more detected landmarks may further include identifying a plurality of 2D points within each of the one or more detected landmarks in the one or more camera images, and determining a 2D point feature vector for each of the plurality of 2D points. Determining the pose of the vehicle-mounted camera may further include determining the pose of the vehicle-mounted camera at least in part based on matching one or more of the one or more detected landmarks with one or more of a plurality of 3D bounding boxes.

[0010] In another aspect of this disclosure, determining the pose of the vehicle-mounted camera may further include identifying multiple corresponding points between one or more of the multiple 3D points and one or more of the multiple 2D points, based at least in part on the 3D point feature vectors of each of the multiple 3D points and the 2D point feature vectors of each of the multiple 2D points. Each of the multiple corresponding points lies within one of the multiple 3D bounding boxes. Determining the pose of the vehicle-mounted camera may further include determining the pose of the vehicle-mounted camera based at least in part on the Perspective n-Point (PnP) algorithm and the Random Sample Consensus (RANSAC) algorithm used for the multiple corresponding points. The pose of the vehicle-mounted camera is constrained to have six degrees of freedom (DoF).

[0011] In another aspect of this disclosure, determining the selected landmark may further include determining the projected gaze based at least in part on the occupant's gaze vector. Determining the selected landmark may also include identifying one or more collisions between the projected gaze and one or more of a plurality of 3D bounding boxes. Determining the selected landmark may also include determining the selected landmark based at least in part on one or more collisions.

[0012] In another aspect of this disclosure, determining the projective gaze and identifying one or more collisions may further include determining the projective gaze. The projective gaze also includes a gaze cone defined by a gaze cone angle. The longitudinal axis of the gaze cone coincides with the gaze vector. Determining the projective gaze and identifying one or more collisions may further include identifying one or more collisions between the gaze cone and one or more of a plurality of 3D bounding boxes.

[0013] In another aspect of this disclosure, determining the selected landmark may further include determining the occupant's view frustum based at least in part on the occupant's gaze vector, the pose of the onboard camera, and multiple 3D bounding boxes. The view frustum includes a 2D projection of the gaze vector and 2D projections of the multiple 3D bounding boxes. Determining the selected landmark may further include determining the selected landmark based at least in part on the distance between the 2D projection of the gaze vector and the 2D projection of each of the multiple 3D bounding boxes.

[0014] According to several aspects, a system for displaying information to vehicle occupants is provided. The system may include an in-vehicle camera, an occupant monitoring system (OMS), an augmented reality (AR) display system, and a vehicle controller in electrical communication with the in-vehicle camera, OMS, and AR display system. The vehicle controller is programmed to capture one or more camera images of the vehicle's surrounding environment using the in-vehicle camera. The vehicle controller is also programmed to determine the pose of the in-vehicle camera based at least in part on the one or more camera images. The vehicle controller is further programmed to determine the gaze vectors of the vehicle occupants using the OMS. The vehicle controller is also programmed to determine a selected landmark among a plurality of landmarks based at least in part on the pose of the in-vehicle camera, the gaze vectors of the occupants, and a three-dimensional (3D) map of the vehicle's surrounding environment. The 3D map includes a plurality of 3D bounding boxes. Each of the plurality of 3D bounding boxes corresponds to one of the plurality of landmarks. The vehicle controller is also programmed to display information about the selected landmark to the occupants using the AR display system.

[0015] In another aspect of this disclosure, in order to determine the pose of the vehicle-mounted camera, the vehicle controller is also programmed to detect one or more of a plurality of landmarks in one or more camera images. To determine the pose of the vehicle-mounted camera, the vehicle controller is also programmed to determine the pose at least in part based on matching one or more of the one or more detected landmarks with one or more of a plurality of 3D bounding boxes.

[0016] In another aspect of this disclosure, to determine the pose of the vehicle-mounted camera, the vehicle controller is further programmed to identify a plurality of 2D points within each of one or more detected landmarks in one or more camera images. To determine the pose of the vehicle-mounted camera, the vehicle controller is also programmed to determine a 2D point feature vector for each of the plurality of 2D points. To determine the pose of the vehicle-mounted camera, the vehicle controller is further programmed to identify, at least in part, a plurality of corresponding points between one or more of the plurality of 3D points and one or more of the plurality of 2D points in a 3D map, based on the 3D point feature vectors of each of the plurality of 3D points and the 2D point feature vectors of each of the plurality of 2D points. Each of the plurality of corresponding points lies within one of a plurality of 3D bounding boxes. To determine the pose of the vehicle-mounted camera, the vehicle controller is also programmed to determine the pose of the vehicle-mounted camera at least in part based on the plurality of corresponding points.

[0017] In another aspect of this disclosure, to determine a selected landmark, the vehicle controller is also programmed to determine the projected gaze based at least in part on the occupant's gaze vector. To determine the selected landmark, the vehicle controller is also programmed to identify one or more collisions between the projected gaze and one or more of a plurality of 3D bounding boxes. To determine the selected landmark, the vehicle controller is also programmed to determine the selected landmark based at least in part on one or more collisions.

[0018] In another aspect of this disclosure, in order to determine the projected gaze and identify one or more collisions, the vehicle controller is also programmed to determine the projected gaze. The projected gaze also includes a gaze cone defined by a gaze cone angle. The longitudinal axis of the gaze cone coincides with the gaze vector. In order to determine the projected gaze and identify one or more collisions, the vehicle controller is also programmed to identify one or more collisions between the gaze cone and one or more of a plurality of 3D bounding boxes.

[0019] In another aspect of this disclosure, to determine a selected landmark, the vehicle controller is also programmed to determine the occupant's view frustum based at least in part on the occupant's gaze vector, the pose of the onboard camera, and a plurality of 3D bounding boxes. The view frustum includes a 2D projection of the gaze vector and 2D projections of the plurality of 3D bounding boxes. To determine the selected landmark, the vehicle controller is also programmed to determine the selected landmark based at least in part on the distance between the 2D projection of the gaze vector and the 2D projection of each of the plurality of 3D bounding boxes.

[0020] In another aspect of this disclosure, to determine a selected landmark, the vehicle controller is also programmed to determine the occupant's view frustum based at least in part on the occupant's gaze vector, the pose of the onboard camera, and a plurality of 3D bounding boxes. The view frustum includes a 2D projection of the gaze vector and 2D projections of the plurality of 3D bounding boxes. To determine the selected landmark, the vehicle controller is also programmed to determine the selected landmark based at least in part on the area of ​​the 2D projection of each of the plurality of 3D bounding boxes.

[0021] According to several aspects, a method for displaying information to vehicle occupants is provided. The method may include capturing one or more camera images using an in-vehicle camera. The method may further include detecting one or more detected landmarks from a plurality of landmarks in the one or more camera images. The method may further include determining the pose of the in-vehicle camera based at least in part on one or more camera images of the vehicle's surrounding environment captured using the in-vehicle camera and a three-dimensional (3D) map of the vehicle's surrounding environment. The method may further include determining a gaze vector of a vehicle occupant. The method may further include determining a selected landmark from a plurality of landmarks based at least in part on the 3D map, the pose of the in-vehicle camera, and the gaze vector of the occupant. The method may further include displaying information about the selected landmark to the occupant using an augmented reality (AR) display system.

[0022] In another aspect of this disclosure, determining the pose of the vehicle-mounted camera may further include identifying multiple 2D points within each of one or more detected landmarks in one or more camera images. Determining the pose of the vehicle-mounted camera may further include determining a 2D point feature vector for each of the multiple 2D points. Determining the pose of the vehicle-mounted camera may further include identifying multiple corresponding points between one or more of the multiple 3D points and one or more of the multiple 2D points in a 3D map, based at least in part on the 3D point feature vectors of each of the multiple 3D points and the 2D point feature vectors of each of the multiple 2D points. Determining the pose of the vehicle-mounted camera may further include determining the pose of the vehicle-mounted camera using a perspective n-point (PnP) algorithm and a random sample consensus (RANSAC) algorithm, based at least in part on the multiple corresponding points. The pose of the vehicle-mounted camera is defined as having six degrees of freedom (DoF).

[0023] In another aspect of this disclosure, determining a selected landmark may further include determining a projected gaze. A projected gaze includes a gaze cone defined by a gaze cone angle, wherein the vertical axis of the gaze cone coincides with the gaze vector. Determining a selected landmark may further include identifying one or more collisions between the gaze cone and one or more of a plurality of 3D bounding boxes of a 3D map. Each of the plurality of 3D bounding boxes corresponds to one of a plurality of landmarks. Determining a selected landmark may further include determining the selected landmark based at least in part on one or more collisions.

[0024] Further applicability will become apparent from the description provided herein. It should be understood that the specification and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0025] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0026] Figure 1 This is a block diagram of a system for displaying information to vehicle occupants according to an exemplary embodiment;

[0027] Figure 2 This is a flowchart of a method for displaying information to vehicle occupants according to an exemplary embodiment;

[0028] Figure 3A This is an illustration of an exemplary camera image captured by a vehicle according to an exemplary embodiment;

[0029] Figure 3B This is an illustration of an exemplary processed image according to an exemplary embodiment;

[0030] Figure 3C This illustrates an exemplary embodiment. Figure 3B A diagram illustrating the correspondence between an exemplary processed image and a 3D map;

[0031] Figure 4A It is for generating according to an exemplary embodiment Figure 3C A flowchart of a 3D mapping method;

[0032] Figure 4B This is an illustration of an exemplary 3D point cloud overlaid on an illustration of the environment, according to an exemplary embodiment;

[0033] Figure 5 This is a flowchart of a method for determining a selected landmark according to a first exemplary embodiment;

[0034] Figure 6A It is a diagram of the occupant's projected gaze according to the first exemplary embodiment;

[0035] Figure 6B This is a diagram of the occupant's projected gaze according to a second exemplary embodiment;

[0036] Figure 7 This is a flowchart of a method for determining a selected landmark according to a second exemplary embodiment; and

[0037] Figure 8 This is an illustration of an exemplary view frustum of an occupant according to an exemplary embodiment. Detailed Implementation

[0038] The following description is merely exemplary in nature and is not intended to limit this disclosure, its application, or its uses.

[0039] In various aspects of this disclosure, vehicle occupants may expect to receive information about landmarks and / or points of interest in the environment surrounding the vehicle. This disclosure provides a novel and improved system and method for displaying information to vehicle occupants with minimal occupant interaction and minimal disruption to driving tasks or passenger experience.

[0040] refer to Figure 1 A system for displaying information to vehicle occupants is shown and is generally indicated by reference numeral 10. System 10 is shown together with an exemplary vehicle 12. Although a passenger vehicle is shown, it should be understood that vehicle 12 can be any type of vehicle, including, for example, an autonomous vehicle, without departing from the scope of this disclosure. System 10 typically includes a vehicle controller 14, a plurality of vehicle sensors 16, and a display 18.

[0041] The vehicle controller 14 is used to implement a method 100 for displaying information to vehicle occupants, as described below. The vehicle controller 14 includes at least one processor 20 and a non-transitory computer-readable storage device or medium 22. The processor 20 may be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among a plurality of processors associated with the vehicle controller 14, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions.

[0042] Computer-readable storage device or medium 22 may include volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operational variables when the processor 20 is powered off. Computer-readable storage device or medium 22 may be implemented using multiple memory devices such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or other electrical, magnetic, optical, or combined memory devices capable of storing data, some of which represents executable instructions used by the vehicle controller 14 to control various systems of the vehicle 12.

[0043] The vehicle controller 14 may also include multiple controllers that are electrically communicating with each other. The vehicle controller 14 may interconnect with additional systems and / or controllers of the vehicle 12, enabling the vehicle controller 14 to access data such as the speed, acceleration, braking, and steering angle of the vehicle 12.

[0044] The vehicle controller 14 communicates electrically with a plurality of vehicle sensors 16 and a display 18. In one exemplary embodiment, electrical communication is established using, for example, a CAN network, a FLEXRAY network, a local area network (e.g., WiFi, Ethernet, etc.), a Serial Peripheral Interface (SPI) network, etc. It should be understood that various additional wired and wireless technologies and communication protocols used for communicating with the vehicle controller 14 are within the scope of this disclosure. It should also be understood that, within the scope of this disclosure, electrical communication also includes the transfer of power and / or energy between electrical devices (e.g., using wires and / or wireless power transmission technologies).

[0045] Multiple vehicle sensors 16 are used to acquire information related to the vehicle 12. In one exemplary embodiment, the multiple vehicle sensors 16 include at least an onboard camera 24 and an occupant monitoring system (OMS) 26. In another exemplary embodiment, the multiple vehicle sensors 16 also include a Global Navigation Satellite System (GNSS) 28 and / or an Inertial Measurement Unit (IMU) 30.

[0046] In another exemplary embodiment, the plurality of vehicle sensors 16 further include sensors for determining performance data about the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 further include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a brake position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission fluid temperature sensor.

[0047] In another exemplary embodiment, the plurality of vehicle sensors 16 further include sensors for determining information about the environment within the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 further include at least one of a seat occupancy sensor, a cabin air temperature sensor, a cabin motion detection sensor, a cabin camera, a cabin microphone, etc.

[0048] In another exemplary embodiment, the plurality of vehicle sensors 16 further include sensors for determining information about the environment 32 surrounding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 further include at least one of an ambient air temperature sensor, an atmospheric pressure sensor, and / or a photographic and / or video camera positioned to observe the environment 32 in front of the vehicle 12.

[0049] In another exemplary embodiment, at least one of the plurality of vehicle sensors 16 is a sensing sensor capable of sensing objects in the environment 32 surrounding the vehicle 12 and / or measuring distances in the environment 32 surrounding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 includes a stereo camera with distance measurement capability. In one example, at least one of the plurality of vehicle sensors 16 is fixed inside the vehicle 12, for example, in the roof of the vehicle 12, having a view through the windshield of the vehicle 12. In another example, at least one of the plurality of vehicle sensors 16 is fixed outside the vehicle 12, for example, on the roof of the vehicle 12, having a view of the environment 32 surrounding the vehicle 12. It should be understood that various additional types of sensing sensors, such as LiDAR sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors, are all within the scope of this disclosure. The plurality of vehicle sensors 16 are in electrical communication with the vehicle controller 14, as described above.

[0050] The vehicle-mounted camera 24 is a sensing sensor for capturing images and / or video of the environment 32 surrounding the vehicle 12. In one exemplary embodiment, the vehicle-mounted camera 24 includes a photographic and / or video camera positioned to observe the environment 32 surrounding the vehicle 12. In a non-limiting example, the vehicle-mounted camera 24 includes a camera fixed inside the vehicle 12 (e.g., in the roof of the vehicle 12) with a field of view through the windshield. In another non-limiting example, the vehicle-mounted camera 24 includes a camera fixed outside the vehicle 12 (e.g., on the roof of the vehicle 12) with a field of view of the environment 32 in front of the vehicle 12.

[0051] In another exemplary embodiment, the vehicle-mounted camera 24 is a surround-view camera system including multiple cameras (also referred to as satellite cameras) arranged to provide views of the environment 32 adjacent to all sides of the vehicle 12. In a non-limiting example, the vehicle-mounted camera 24 includes a front-view camera (e.g., mounted in the front grille of the vehicle 12), a rear-view camera (e.g., mounted on the tailgate of the vehicle 12), and two side-view cameras (e.g., mounted below each of the two side mirrors of the vehicle 12). In another non-limiting example, the vehicle-mounted camera 24 also includes an additional rear-view camera mounted near the center high-mounted brake light of the vehicle 12.

[0052] It should be understood that camera systems with additional cameras and / or additional mounting locations are within the scope of this disclosure. It should also be understood that cameras with various sensor types, including, for example, charge-coupled device (CCD) sensors, complementary metal-oxide-semiconductor (CMOS) sensors, and / or high dynamic range (HDR) sensors, are all within the scope of this disclosure. Furthermore, cameras with various lens types, including, for example, wide-angle lenses and / or narrow-angle lenses, are also within the scope of this disclosure.

[0053] Occupant Monitoring System (OMS) 26 is used to identify vehicle occupants 34 within vehicle 12. Figure 6A , 6B The direction of gaze of occupant 34. Within the scope of this disclosure. Figure 6A , 6B This includes the driver and / or passengers of vehicle 12. In one exemplary embodiment, OMS26 includes images for capturing vehicle occupants 34. Figure 6A , 6B One or more infrared (IR) cameras are located inside vehicle 12. The OMS also includes an image processor (not shown) that is in electrical communication with the IR cameras. The IR cameras capture images of occupant 34. Figure 6A , 6B High-resolution images of the faces and eyes of the occupants were obtained and the image processor analyzed the images to determine the occupants (34). Figure 6A , 6B The OMS26 uses reflected IR light from the eyes and surrounding facial features to track the orientation and position of the eyes, allowing it to calculate the gaze direction based on processed image data. In a non-limiting example, occupant 34 ( Figure 6A , 6B The gaze direction is defined by the gaze vector. The OMS26 is in electrical communication with the vehicle controller 14, as described above.

[0054] GNSS28 is used to determine the geographical location of vehicle 12. In one exemplary embodiment, GNSS28 is a Global Positioning System (GPS). In a non-limiting example, GPS includes a GPS receiver antenna (not shown) and a GPS controller (not shown) in electrical communication with the GPS receiver antenna. The GPS receiver antenna receives signals from multiple satellites, and the GPS controller calculates the geographical location of vehicle 12 based on the signals received by the GPS receiver antenna.

[0055] In one exemplary embodiment, GNSS28 also includes a map. This map includes information about infrastructure such as municipal boundaries, roads, railways, sidewalks, and buildings. Therefore, the geographic location of vehicle 12 is contextualized using the map information. In one non-limiting example, the map is retrieved from a remote source using a wireless connection. In another non-limiting example, the map is stored in GNSS28's database. It should be understood that various additional types of satellite-based radio navigation systems, such as Global Positioning System (GPS), Galileo, GLONASS, and BeiDou Navigation Satellite System (BDS), are within the scope of this disclosure. GNSS28 communicates electrically with vehicle controller 14, as described above.

[0056] The IMU 30 is used to determine the orientation, velocity, and gravity acting on the vehicle 12. In one exemplary embodiment, the IMU 30 includes several sensors, including an accelerometer, a gyroscope, and / or a magnetometer. In a non-limiting example, the IMU 30 includes a three-axis accelerometer and a three-axis gyroscope integrated into a single unit. The accelerometer measures linear acceleration along each axis, while the gyroscope measures angular velocity about each axis. The IMU 30 processes data from the sensors to calculate the vehicle 12's current orientation, velocity, heading, yaw rate (i.e., rate of change of heading), and acceleration in three-dimensional space. The IMU 30 communicates electrically with the vehicle controller 14 as described above.

[0057] Display 18 is used to display information to the occupants 34 of vehicle 12. Figure 6A , 6B ) provides information. In an exemplary embodiment, the display 18 is located in the occupant 34 ( Figure 6A , 6B A human-machine interface (HMI) that is within the field of view and capable of displaying text, graphics, and / or images. It should be understood that HMI display systems, including LCD displays, LED displays, etc., are within the scope of this disclosure. Other exemplary embodiments in which the display 18 is disposed in the rearview mirror are also within the scope of this disclosure.

[0058] In another exemplary embodiment, the display 18 includes a head-up display (HUD) configured to display text, graphics, and / or images to the occupant 34 by projecting them onto the windshield of the vehicle 12. Figure 6A , 6B Information is provided. Text, graphics, and / or images are reflected by the windshield of vehicle 12 and are visible to occupants 34. Figure 6A , 6B The display 18 is visible to the occupant 34 without requiring them to take their eyes off the road in front of the vehicle 12. In another exemplary embodiment, the display 18 includes an augmented reality (AR) display system, such as an augmented reality head-up display (AR-HUD). An AR-HUD is a type of HUD configured to display text, graphics, and / or images overlaid on the occupant 34. Figure 6A , 6B The physical objects in the environment 32 surrounding the vehicle 12 within the field of vision are used to enhance the occupants' 34 ( Figure 6A , 6B () Visual perception of the environment 32 surrounding the vehicle 12.

[0059] In one exemplary embodiment, occupant 34 ( Figure 6A , 6B The user can interact with the display 18 using a human-machine interface device (HID), such as a touchscreen, electromechanical switch, capacitive switch, knob, etc. It should be understood that the device is used to interact with the occupants 34 of vehicle 12. Figure 6A , 6B Additional systems for displaying information are also within the scope of this disclosure. The display 18 is in electrical communication with the vehicle controller 14, as described above.

[0060] refer to Figure 2 A flowchart of a method 100 for displaying information to vehicle occupants is provided. Method 100 begins at box 102 and proceeds to box 104. In box 104, a three-dimensional (3D) map 40 of the environment 32 is generated. Figure 3C In one exemplary embodiment, 3D map 40 ( Figure 3C ) includes multiple 3D bounding boxes 44 ( Figure 3C Multiple 3D points within 42 ( Figure 3C ). Multiple 3D points 42 ( Figure 3C Each of the 3D points is defined by a 3D point feature vector and its position in 3D space. Within the scope of this disclosure, a 3D point feature vector uniquely identifies a plurality of 3D points 42 ( Figure 3C A high-dimensional vector (i.e., a 128-dimensional vector) in ). In a non-restrictive example, a mathematical algorithm is used to compute the 3D point vector based on the characteristics of the subject 3D point and the surrounding 3D points.

[0061] Multiple 3D bounding boxes 44 ( Figure 3C ) defines the landmarks in three-dimensional space 46 ( Figure 3A The location of the landmark. Within the scope of this disclosure, landmarks are points of interest (POIs), such as businesses, schools, bus stops, gas stations, government buildings (e.g., police stations, fire stations, city halls), hospitals, parks, etc. Multiple 3D bounding boxes 44 ( Figure 3C Each of the ) corresponds to multiple landmarks 46 in environment 32. Figure 3A One of them. In an exemplary embodiment, 3D map 40 ( Figure 3C 3D maps are generated using an external server system (not shown) located in a centralized location (e.g., a server farm, data center, etc.) and connected to the Internet. The 3D map 40 will be discussed in more detail below. Figure 3C The generation of ). After box 104, method 100 proceeds to box 106.

[0062] In box 106, the vehicle controller 14 uses the onboard camera 24 to capture one or more camera images of the environment 32 surrounding the vehicle 12. (See reference) Figure 3A An exemplary image 50a of the environment 32 captured in box 106 is shown. See again... Figure 2 After box 106, method 100 proceeds to box 108.

[0063] Refer again Figure 3A And continue to refer to Figure 2In box 108, vehicle controller 14 detects one or more landmarks 46 in one or more images (e.g., exemplary image 50a) of the environment 32 surrounding vehicle 12 captured in box 106. Within the scope of this disclosure, the one or more landmarks 46 detected in box 108 are referred to as one or more detected landmarks 46. In one exemplary embodiment, to detect one or more detected landmarks 46, vehicle controller 14 uses a computer vision algorithm. The computer vision algorithm utilizes machine learning techniques to analyze pixel-level information of the input image to detect and classify objects or patterns of interest. In a non-limiting example, the computer vision algorithm first preprocesses the input image using techniques such as image resizing, normalization, and / or filtering to reduce noise. Subsequently, the computer vision algorithm extracts relevant features from the input image using methods such as edge detection, corner detection, and texture analysis. Then, the computer vision algorithm may utilize a machine learning model (e.g., a convolutional neural network (CNN)) to classify and label relevant objects (i.e., landmarks 46) in the input image based on learned patterns and associations.

[0064] refer to Figure 3B An exemplary processed image 50b is shown. (Reference) Figure 2 and Figure 3B In box 108, the vehicle controller 14 further identifies a plurality of two-dimensional (2D) points 54 within each of one or more detected landmarks 46. In the exemplary processed image 50b, the plurality of 2D points 54 are visualized as black dots; however, it should be understood that the plurality of 2D points 54 are arbitrary reference points selected within each of the plurality of landmarks 46. The number, density, location, distribution, etc., of the plurality of 2D points 54 can vary within the scope of this disclosure. In one exemplary embodiment, each of the plurality of 2D points 54 is defined by a 2D point feature vector and its location in 2D space. Within the scope of this disclosure, the 2D point feature vector is a high-dimensional vector (i.e., a 128-dimensional vector) that uniquely identifies one of the plurality of 2D points 54. In a non-limiting example, the 2D point vector is computed using a mathematical algorithm based on the characteristics of the subject 2D point and the surrounding 2D points. Following box 108, method 100 proceeds to box 110.

[0065] refer to Figure 3C A diagram illustrating the correspondence between an exemplary processed image 50b and a 3D map 40 is shown. (Reference) Figure 2 and Figure 3CIn box 110, the vehicle controller 14 identifies multiple corresponding points between one or more of a plurality of 3D points 42 in the 3D map 40 and one or more of a plurality of 2D points 54 in one or more images captured in box 106 (e.g., as shown in the exemplary processed image 50b). Within the scope of this disclosure, corresponding points are points indicating the same physical location within environment 32. Figure 3C In the example shown, the correspondence between corresponding points is indicated by the solid line 56. It should be understood that, although... Figure 3C The diagram shows four corresponding points, but any number of corresponding points can be identified.

[0066] In one exemplary embodiment, multiple corresponding points are identified at least in part based on the 2D point feature vectors of each of the plurality of 2D points 54 and the 3D point feature vectors of each of the plurality of 3D points 42. In a non-limiting example, the vehicle controller 14 searches the 3D map 40 to find 3D points 42 having 3D point feature vectors that substantially correspond to (i.e. match) one or more 2D point feature vectors of 2D points 54 captured in one or more images (e.g., exemplary processed image 50b) of frame 106. In one exemplary embodiment, the 3D map is searched only within the plurality of 3D bounding boxes 44 to improve the speed and accuracy of the search. Thus, each of the plurality of corresponding points is located within one of the plurality of 3D bounding boxes 44. Referring again Figure 2 After box 110, method 100 proceeds to box 112.

[0067] In block 112, the vehicle controller 14 determines the pose of the vehicle camera 24, defined as having six degrees of freedom (DoF), based at least on a plurality of corresponding points determined in block 110. Within the scope of this disclosure, the six DoFs are forward / backward (wobble), up / down (undulation), left / right (sway), yaw (rotation about the normal axis), pitch (rotation about the lateral axis), and roll (rotation about the longitudinal axis). In one exemplary embodiment, the pose of the vehicle camera 24 is determined using, for example, the perspective n-point (PnP) algorithm and / or random sample consensus (RANSAC) algorithm described in Wetzel, Johannes, “Image Based 6-DOF Camera Pose Estimation with Weighted RANSAC 3D” (Lectures on Computer Science, Vol. 8142, pp. 249–254, September 2013), the entire contents of which are incorporated herein by reference. In a non-limiting example, measurements from GNSS28 and / or IMU30 are also used to determine the six DoFs, for example, to determine sway, heave, and / or roll. Following box 112, method 100 proceeds to box 114.

[0068] In box 114, vehicle controller 14 determines occupant 34 ( Figure 6A , 6B The gaze vector of the occupant 34. In an exemplary embodiment, in order to determine the gaze vector, the vehicle controller 14 uses OMS 26 to perform gaze vector determination on the occupant 34. Figure 6A , 6B The gaze vector is measured and determined. In a non-limiting example, the gaze vector consists of a three-dimensional vector and a vector located at occupant 34 ( Figure 6A , 6B The gaze origin at the eye is defined. In an exemplary embodiment, the vehicle controller 14 uses, for example, the techniques discussed in “MPIIGaze: Real-world dataset and deep appearance-based gaze estimation” by Zhang, X. et al. (IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, No. 1, pp. 162-175, January 2019), the entire contents of which are incorporated herein by reference, to determine the gaze vector. After box 114, method 100 proceeds to box 116.

[0069] In box 116, the vehicle controller 14 is based at least in part on multiple 3D bounding boxes 44 in the 3D map 40, the pose of the onboard camera 24 within the environment 32 mapped by the 3D map, and the occupants ( Figure 6A , 6B The gaze vector 34 is used to determine the selected landmark among multiple landmarks 46. The determination of the selected landmark will be discussed in more detail below. After box 116, method 100 proceeds to box 118.

[0070] In frame 118, vehicle controller 14 uses display 18 to show information to occupant 34. Figure 6A , 6B The system displays information about the selected landmark defined in box 116. In one exemplary embodiment, this information includes the landmark's name (e.g., business name), landmark signage (e.g., business logo), landmark opening hours, services available at the landmark (e.g., services offered by the business at the landmark), information about events occurring at the landmark, historical information about the landmark, news or current events related to the landmark, etc. It should be understood that this information may include any type of information related to the landmark and may be provided in any form, including text and / or graphics. In one exemplary embodiment, the vehicle controller 14 uses the AR display system of display 18 to visually display the information. In another exemplary embodiment, the vehicle controller 14 uses text-to-speech or speech synthesis to auditorily display information to occupant 34. Figure 6A , 6B (Provide information.) After box 118, method 100 proceeds to the standby state of entering box 120.

[0071] In one exemplary embodiment, the vehicle controller 14 repeatedly exits the standby state of box 120 and restarts method 100 at box 102. In a non-limiting example, the vehicle controller 14 exits the standby state of box 120 and restarts method 100 on a timer, for example, every three hundred milliseconds.

[0072] refer to Figure 4A A flowchart of method 104a for generating a 3D map 40 in block 104 of method 100 is shown. In one exemplary embodiment, method 104a is executed by an external server system (not shown), as described above. Reference Figure 4A Continuing with reference to the preceding figures, method 104a begins at box 402. In box 402, the external server system receives multiple reference images including multiple landmarks 46 within environment 32. In one exemplary embodiment, the multiple reference images are crowdsourced from multiple vehicles, such as end-user vehicles, fleet vehicles, and / or dedicated data collection vehicles. In a non-limiting example, for instance, multiple reference images are captured from different locations within environment 32 as multiple vehicles travel through environment 32, thereby capturing multiple landmarks 46 from different angles / viewpoints. Following box 402, method 104a proceeds to box 404.

[0073] refer to Figure 4B An illustration of an exemplary 3D point cloud overlaid on an illustration of environment 32 is shown. (Reference) Figure 4A and Figure 4B In box 404, an external server system generates a 3D point cloud comprising a plurality of 3D points 42. In one exemplary embodiment, each of the plurality of 3D points 42 is defined by a 3D point vector and its position in 3D space. Within the scope of this disclosure, a 3D point feature vector is a vector (i.e., a one-dimensional matrix) that uniquely identifies one of the plurality of 3D points 42. In a non-limiting example, the 3D point vector is computed using a mathematical algorithm based on the characteristics of the subject 3D point and the surrounding 3D points. In a non-limiting example, the 3D point vector is computed by averaging information from each of a plurality of reference images. In one exemplary embodiment, the 3D point cloud is generated based on a plurality of reference images collected in box 402. In a non-limiting example, for example... The entire contents of O. et al.'s "A survey of structure from motion" (Numerical Journal, Vol. 26, pp. 305-364, May 2017) are incorporated herein by reference, which discusses the Structure of Motion (SfM) algorithm for generating 3D point clouds from multiple reference images.

[0074] After generating a 3D point cloud comprising multiple 3D points 42, each of the multiple 3D points 42 is labeled with a corresponding landmark from a plurality of landmarks 46. In one non-limiting example, each of the multiple 3D points 42 is labeled with its geographically nearest landmark, which is identified based on a map database including the coordinate locations of each of the multiple landmarks 46. In another non-limiting example, each of the multiple 3D points 42 is labeled using computer vision-based object detection on multiple reference images to identify the multiple landmarks 46. (See again...) Figure 4A After box 404, method 104a proceeds to box 406.

[0075] In box 406, multiple point cloud clusters are generated from multiple 3D points 42. In one exemplary embodiment, the density-based noise-applied spatial clustering (DBSCAN) algorithm described in, for example, Ester et al., “A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise” (Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pp. 226–231, August 1996), the entire contents of which are incorporated herein by reference, are used to generate the multiple point cloud clusters. It should be understood that alternative or additional clustering algorithms, such as Distributed DBSCAN (DDBSCAN), k-means, agglomerative clustering, mean shift, Gaussian mixture model, spectral clustering, affinity propagation, equilibrium iterative reduction and clustering using hierarchical methods (BIRCH), ordering points for identifying cluster structures (OPTICS), fuzzy c-means, etc., may be used without departing from the scope of this disclosure. After box 406, method 104a proceeds to box 408.

[0076] In box 408, multiple point cloud clusters are filtered to generate multiple filtered point cloud clusters. Each of the multiple filtered point cloud clusters corresponds to one of the multiple landmarks 46. In one exemplary embodiment, to filter the multiple point cloud clusters, irrelevant points are first removed. For example, points not associated with any of the multiple landmarks 46 are ignored.

[0077] Then, one or more of the multiple point cloud clusters are segmented to generate multiple segmented point cloud clusters, such that each of the multiple segmented point cloud clusters contains points labeled with only one of the multiple landmarks 46 (i.e., in a given segmented point cloud cluster, all points are labeled with the same landmark). In a non-limiting example, to segment multiple point cloud clusters, for each of the multiple point cloud clusters, if the point cloud cluster contains points labeled with different landmarks, the point cloud cluster is segmented into two or more segmented point cloud clusters, each of the segmented point cloud clusters having points labeled with the same landmark.

[0078] Subsequently, in a non-limiting example, two or more of the multiple segmented point cloud clusters are similarly labeled to generate multiple filtered point cloud clusters. In a non-limiting example, the multiple segmented point cloud clusters may include multiple segmented point cloud clusters corresponding to a single landmark 46 (e.g., a first segmented point cloud cluster corresponding to a sign, parking lot, entrance passage, etc., and a second segmented point cloud cluster corresponding to a building that houses the landmark itself). In an exemplary embodiment, the multiple segmented point cloud clusters are determined to correspond to the same landmark among the multiple landmarks based at least in part on at least one of the following: the relative size of two or more of the multiple segmented point cloud clusters and the distance between two or more of the multiple segmented point cloud clusters. In a non-limiting example, one or more of the multiple segmented point cloud clusters are labeled as similar to the nearest larger segmented point cloud cluster corresponding to the nearest landmark within a predetermined threshold (i.e., containing more points covering a larger area). After block 408, method 104a proceeds to block 410.

[0079] In box 410, a plurality of 3D bounding boxes 44 are determined based on a plurality of filtered point cloud clusters. In one exemplary embodiment, the boundary of each of the plurality of 3D bounding boxes 44 is determined to completely surround one of the plurality of filtered point cloud clusters. Therefore, each of the plurality of 3D bounding boxes 44 corresponds to one of a plurality of landmarks 46. The plurality of 3D bounding boxes 44 are as described above. Figure 3C The 3D map 40 is shown. In one exemplary embodiment, the completed 3D map 40 is transmitted to the vehicle 12 and stored in the medium 22 of the vehicle controller 14 for use in method 100 as described above. After block 410, method 104a ends and method 100 is performed as described above.

[0080] refer to Figure 5A flowchart of a first exemplary embodiment 116a (i.e., a method for determining selected landmarks) of block 116 of method 100 is shown. The first exemplary embodiment 116a of block 116 begins at block 502. In block 502, the vehicle controller 14 determines a projected gaze based on a gaze vector determined in block 114. Within the scope of this disclosure, a projected gaze is a projection of the gaze vector onto the coordinate system of environment 32 (i.e., the world coordinate system) and the 3D map 40 (i.e., a coordinate transformation). In one exemplary embodiment, the gaze vector is transformed based on the pose of the vehicle camera 24 determined in block 112. In a non-limiting example, the OMS 26 determines the gaze vector in the relative coordinate system of vehicle 12, and the pose of the vehicle camera 24 anchors the relative coordinate system of vehicle 12 within the absolute coordinate system of environment 32 (i.e., the world coordinate system). Therefore, a projected gaze is determined using one or more coordinate transformations and projected onto the 3D map 40.

[0081] refer to Figure 6A A diagram illustrating a first exemplary embodiment of the projected gaze of an occupant 34 in vehicle 12 is shown. Figure 6A In this context, the projected gaze is implemented as a projected gaze vector 60. The projected gaze vector 60 is similar to the gaze vectors discussed above, except that the projected gaze vector 60 is defined in the same coordinate system as the 3D map 40 (i.e., the world coordinate system).

[0082] refer to Figure 6B A diagram illustrating a second exemplary embodiment of the projected gaze of an occupant 34 in vehicle 12 is shown. Figure 6B In this implementation, besides the projection gaze vector 60, the projection gaze is also implemented as a projection gaze cone 62. The projection gaze cone 62 is defined by the gaze cone angle 64. The vertical axis of the projection gaze cone coincides with the projection gaze vector 60, as shown below. Figure 6B As shown. In one exemplary embodiment, the gaze cone angle 64 can be predetermined or adjustable, as will be discussed in more detail below. (Refer to...) Figure 5 Following box 502, the first exemplary embodiment 116a of box 116 proceeds to box 504.

[0083] In block 504, the vehicle controller 14 identifies one or more collisions between the projected gaze determined in block 502 and one or more of the plurality of 3D bounding boxes 44. In one exemplary embodiment, axis-aligned bounding box (AABB) collision detection is used to identify collisions, as is known in the field of computer graphics. In a non-limiting example, the projected gaze is further projected or simulated within the 3D map 40, and collisions with the plurality of 3D bounding boxes 44 are identified. In one exemplary embodiment, Figure 6BThe illustrated embodiment including the gaze cone 62 can be used to improve the reliability and repeatability of collision detection. The effective sensitivity of gaze collision detection can be adjusted by adjusting the gaze cone angle 64. In one exemplary embodiment, if the gaze cone 62 does not collide with any of the plurality of 3D bounding boxes 44, the gaze cone angle 64 is gradually increased until a collision with the nearest bounding box is identified. Following box 504, the first exemplary embodiment 116a of box 116 proceeds to box 506.

[0084] In block 506, vehicle controller 14 identifies a selected landmark among a plurality of landmarks 46. In one exemplary embodiment, the selected landmark is identified at least in part based on one or more collisions identified at block 504. In a non-limiting example, if the projected gaze collides with the first of the plurality of 3D bounding boxes 44, the selected landmark is determined to be the one of the plurality of landmarks 46 corresponding to the first of the plurality of 3D bounding boxes 44. If the gaze cone 62 collides with the plurality of bounding boxes, the collision closest to the center of the gaze cone 62 (i.e., the location of the projected gaze vector 60) is determined to be the selected landmark. After block 506, the first exemplary embodiment 116a of block 116 ends, and method 100 is performed as described above.

[0085] refer to Figure 7 The diagram illustrates a flowchart of a second exemplary embodiment 116b of block 116 of method 100 (i.e., a method for determining a selected landmark). The second exemplary embodiment 116b of block 116 begins at block 702. (See reference...) Figure 8 An exemplary view frustum 70 is shown. (Reference) Figure 7 and Figure 8 In box 702, the vehicle controller 14 determines the visual frustum 70 of the occupant 34 based at least in part on the gaze vector of the occupant 34, the pose of the onboard camera 24, and a 3D map 40 including multiple 3D bounding boxes 44. Within the scope of this disclosure, the visual frustum 70 is a 2D projection of the field of view of the occupant 34.

[0086] In one exemplary embodiment, as described above, the view frustum 70 is determined by first projecting gaze vectors onto a 3D map 40. Then, a perspective view of the occupant 34 within the 3D map 40 is determined based on the projection of the gaze vectors and an estimated or pre-defined field of view of the occupant 34 centered on the projection of the gaze vectors within the 3D map 40. Subsequently, the perspective view of the occupant 34 is projected onto 2D to create the view frustum 70. In a non-limiting example, the view frustum 70 includes a 2D gaze vector 72 (i.e., a 2D projection of the gaze vectors) and a plurality of 2D bounding boxes 74 (i.e., 2D projections of a plurality of 3D bounding boxes 44). In a non-limiting example, the view frustum 70 is centered on the 2D gaze vector 72. (Refer again) Figure 7Following box 702, the second exemplary embodiment 116b of box 116 proceeds to box 704.

[0087] In block 704, vehicle controller 14 determines a selected landmark among a plurality of landmarks 46. In one exemplary embodiment, vehicle controller 14 uses a view frustum 70 to identify the selected landmark. In a first exemplary embodiment, the selected landmark is determined at least in part based on the distance between the 2D gaze vector 72 and each of the plurality of 2D bounding boxes 74. In a non-limiting example, the selected landmark is determined as one of the plurality of landmarks 46 in the view frustum 70, which has a 2D bounding box 74 that is geometrically closest to the 2D gaze vector 72. In a second exemplary embodiment, the selected landmark is determined at least in part based on the area of ​​each of the plurality of 2D bounding boxes 74. In a non-limiting example, the selected landmark is determined as one of the plurality of landmarks 46 in the view frustum 70, which has a 2D bounding box 74 with the largest area. In another non-limiting example, the selected landmark is determined as one of the plurality of landmarks 46 in the view frustum 70, which has a 2D bounding box 74 with the largest area within a predetermined distance threshold of the 2D gaze vector 72. Following block 704, the second exemplary embodiment 116b of block 116 ends, and method 100 is performed as described above.

[0088] The system 10 and method 100 of this disclosure offer several advantages. By utilizing system 10 and method 100, relevant information about objects in their environment is provided to vehicle occupants based on gaze detection. Furthermore, system 10 and method 100 allow for the efficient generation of detailed 3D maps, including the locations of landmarks in the environment. By searching 3D map data based on 2D image data captured by vehicle 12, accurate and precise position and pose information about the onboard camera 24, as well as information obtained by extending vehicle 12, can be determined.

[0089] The descriptions in this disclosure are merely exemplary in nature, and variations thereof that do not depart from the spirit and scope of this disclosure are intended to fall within its scope. Such variations should not be considered as departing from the spirit and scope of this disclosure.

Claims

1. A method for displaying information to vehicle occupants, the method comprising: Generate a three-dimensional 3D map of the environment surrounding the vehicle, wherein the 3D map includes multiple 3D bounding boxes, and wherein each of the multiple 3D bounding boxes corresponds to one of multiple landmarks; The pose of the vehicle-mounted camera is determined at least in part based on one or more camera images of the vehicle's surrounding environment captured using the vehicle-mounted camera. Determine the gaze vectors of the vehicle occupants; The selected landmark among the plurality of landmarks is determined at least in part based on the plurality of 3D bounding boxes, the pose of the vehicle-mounted camera, and the gaze vector of the occupant; as well as The occupants are shown information about the selected landmark using a display.

2. The method according to claim 1, wherein, Generating the 3D map also includes: A 3D point cloud comprising multiple 3D points is generated based on multiple reference images including the plurality of landmarks, wherein each of the plurality of 3D points is defined by a 3D point feature vector; and Each of the plurality of 3D bounding boxes is determined by clustering the 3D point cloud into multiple point cloud clusters.

3. The method according to claim 2, wherein, The 3D point cloud clustering also includes: The multiple point cloud clusters were generated using the density-based noise spatial clustering DBSCAN algorithm; and The plurality of point cloud clusters are filtered to generate a plurality of filtered point cloud clusters, wherein each of the plurality of filtered point cloud clusters corresponds to one of the plurality of landmarks.

4. The method according to claim 3, wherein, Filtering the multiple point cloud clusters also includes: One or more of the plurality of point cloud clusters are segmented to generate a plurality of segmented point cloud clusters, wherein each of the plurality of segmented point cloud clusters corresponds to only one of the plurality of landmarks, and wherein the plurality of filtered point cloud clusters include the plurality of segmented point cloud clusters.

5. The method according to claim 4, wherein, Filtering the multiple point cloud clusters also includes: Two or more of the plurality of segmented point cloud clusters are determined to correspond to the same landmark among the plurality of landmarks, based at least in part on at least one of the following: the relative size of the two or more of the plurality of segmented point cloud clusters and the distance between the two or more of the plurality of segmented point cloud clusters.

6. The method according to claim 2, wherein, Determining the pose of the vehicle-mounted camera also includes: The vehicle-mounted camera is used to capture images from one or more cameras; Detecting one or more detected landmarks from among the plurality of landmarks in the one or more camera images, wherein detecting the one or more detected landmarks further includes: Identify multiple 2D points within each of the one or more detected landmarks in the one or more camera images; and Determine the 2D point feature vector of each of the plurality of 2D points; and The pose of the vehicle-mounted camera is determined at least in part based on matching one or more of the one or more detected landmarks with one or more of the plurality of 3D bounding boxes.

7. The method according to claim 6, wherein, Determining the pose of the vehicle-mounted camera also includes: At least in part, based on the 3D point feature vector of each of the plurality of 3D points and the 2D point feature vector of each of the plurality of 2D points, a plurality of corresponding points between one or more of the plurality of 3D points and one or more of the plurality of 2D points are identified, wherein each of the plurality of corresponding points is located within one of the plurality of 3D bounding boxes; and The pose of the vehicle-mounted camera is determined at least in part based on the plurality of corresponding points using the perspective n-point PnP algorithm and the random sample consensus RANSAC algorithm, wherein the pose of the vehicle-mounted camera is constrained to have six degrees of freedom (DoF).

8. The method according to claim 1, wherein, Determining the selected landmark also includes: The projected gaze is determined at least in part based on the occupant's gaze vector; Identify one or more collisions between the projected gaze and one or more of the plurality of 3D bounding boxes; and The selected landmark is determined at least in part based on the one or more collisions.

9. The method according to claim 8, wherein, Determining the projected gaze and identifying the one or more collisions further includes: Determine the projected gaze, wherein the projected gaze further includes a gaze cone defined by a gaze cone angle, and wherein the longitudinal axis of the gaze cone coincides with the gaze vector; and Identify one or more collisions between the gaze cone and one or more of the plurality of 3D bounding boxes.

10. The method according to claim 1, wherein, Determining the selected landmark also includes: The occupant's view frustum is determined at least in part based on the occupant's gaze vector, the pose of the vehicle-mounted camera, and the plurality of 3D bounding boxes, wherein the view frustum includes a 2D projection of the gaze vector and a 2D projection of the plurality of 3D bounding boxes; and The selected landmark is determined at least in part based on the distance between the 2D projection of the gaze vector and the 2D projection of each of the plurality of 3D bounding boxes.

Images