Augmented reality information overlay with eye tracking

Abstract

A method for displaying information to a vehicle occupant, which may include generating a three-dimensional (3D) map of the vehicle's surroundings. The 3D map comprises a plurality of 3D boundary boxes. Each of the plurality of 3D boundary boxes corresponds to a plurality of landmarks. The method may further include determining the position of a vehicle camera. The method may further include determining the viewing vector of the vehicle occupant. The method may further include determining a selected landmark from the plurality of landmarks, at least partially based on the plurality of 3D boundary boxes, the position of the vehicle camera, and the occupant's viewing vector. The method may further include displaying information about the selected landmark to the occupant using a display.

Description

Introduction

[0001] The present disclosure relates to systems and methods for displaying information to vehicle occupants.

[0002] Various display systems can be used to provide information in vehicle applications. Display systems can be configured to show information such as speed, navigation instructions, system diagnostics, entertainment options, and / or the like. In some examples, display systems are configured as touchscreens with integrated haptic feedback, enabling intuitive user interaction. Display systems can include additional features such as adaptive brightness controls to improve visibility under varying lighting conditions, voice command integration for hands-free operation, and head-up display technology. The display can also support wireless communication protocols, allowing it to connect with mobile devices, cloud services, and other vehicle systems.Display systems can use wireless communication to retrieve information from external sources (for example, the internet) for display to vehicle occupants. For example, display systems can be used to provide vehicle occupants with information about conditions outside the vehicle, including, for example, weather conditions, traffic conditions, points of interest or destination information, and / or the like.

[0003] While systems and procedures for displaying information achieve their intended purpose, there is a need for new and improved systems and procedures for providing information to vehicle occupants. Summary

[0004] A method for displaying information to a vehicle occupant is provided according to several aspects. The method may include generating a three-dimensional (3D) map of the vehicle's surroundings. The 3D map comprises a plurality of 3D boundary boxes. Each of these 3D boundary boxes corresponds to a plurality of reference points. The method may further include determining the position of a vehicle camera, at least partially, based on one or more camera images of the vehicle's surroundings, captured using the vehicle camera. The method may also include determining the viewing vector of the vehicle occupant.The method may further include determining a selected reference point from a multitude of reference points, at least partially based on the multitude of 3D boundary boxes, the position of the vehicle camera, and the occupant's viewing vector. The method may also include displaying information about the selected reference point to the occupant using a display.

[0005] In another aspect of the present disclosure, generating the 3D map can further include generating a 3D point cloud comprising a plurality of 3D points, based on a plurality of reference images comprising the plurality of orientation points. Each of the plurality of 3D points is defined by a 3D point feature vector. Generating the 3D map can further include determining each of the plurality of 3D boundary boxes by clustering the 3D point cloud into a plurality of point cloud clusters.

[0006] In another aspect of the present disclosure, the clustering of the 3D point cloud can further include generating the plurality of point cloud clusters using a density-based spatial clustering-of-applications-with-noise algorithm, the DBSCAN algorithm. The clustering of the 3D point cloud can further include filtering the plurality of point cloud clusters to generate a plurality of filtered point cloud clusters. Each of the plurality of filtered point cloud clusters corresponds to one of the plurality of orientation points.

[0007] In another aspect of the present disclosure, filtering the multitude of point cloud clusters can further include splitting one or more of the multitude of point cloud clusters to generate a multitude of split point cloud clusters. Each of the multitude of split point cloud clusters corresponds to only one of the multitude of reference points. The multitude of filtered point cloud clusters comprises the multitude of split point cloud clusters.

[0008] In another aspect of the present disclosure, filtering the multitude of point cloud clusters may further include determining two or more of the multitude of split point cloud clusters to correspond to an equal number of orientation points, based at least partially on at least one of: a relative size of the two or more of the multitude of split point cloud clusters and a distance between the two or more of the multitude of split point cloud clusters.

[0009] In another aspect of the present disclosure, determining the position of the vehicle camera may further comprise capturing one or more camera images using the vehicle camera. Determining the position of the vehicle camera may further comprise detecting one or more detected landmarks from the plurality of landmarks in the one or more camera images. Detecting the one or more detected landmarks may further comprise identifying a plurality of 2D points within each of the one or more detected landmarks in the one or more camera images and determining a 2D point feature vector for each of the plurality of 2D points.Determining the position of the vehicle camera may also include determining the position of the vehicle camera, at least partially, based on comparing one or more of the one or more detected reference points with one or more of the multitude of 3D boundary boxes.

[0010] In another aspect of the present disclosure, determining the position of the vehicle camera can further comprise identifying a plurality of corresponding points between one or more of the plurality of 3D points and one or more of the plurality of 2D points, at least partially based on the 3D point feature vector of each of the plurality of 3D points and the 2D point feature vector of each of the plurality of 2D points. Each of the plurality of corresponding points is located within one of the plurality of 3D bounding boxes. Determining the position of the vehicle camera can further comprise determining the position of the vehicle camera using a perspective n-point algorithm (PnP algorithm) and a random-sample consensus algorithm (RANSAC algorithm), at least partially based on the plurality of corresponding points. The position of the vehicle camera is defined with six degrees of freedom (DoF).

[0011] In another aspect of the present disclosure, determining the selected orientation point may further include determining a projected view, at least partially, based on the occupant's view vector. Determining the selected orientation point may further include identifying one or more collisions between the projected view and one or more of the plurality of 3D boundary boxes. Determining the selected orientation point may further include determining the selected orientation point, at least partially, based on the one or more collisions.

[0012] In another aspect of the present disclosure, determining the projected view and identifying one or more collisions may further include determining the projected view. The projected view further comprises a cone of view defined by a cone angle. A longitudinal axis of the cone of view coincides with the view vector. Determining the projected view and identifying one or more collisions may further include identifying one or more collisions between the cone of view and one or more of the plurality of 3D bounding boxes.

[0013] In another aspect of the present disclosure, determining the selected orientation point can further comprise determining a truncated cone of the occupant's view, at least partially, based on the occupant's gaze vector, the position of the vehicle camera, and the plurality of 3D boundary boxes. The truncated cone of view comprises a 2D projection of the gaze vector and a 2D projection of the plurality of 3D boundary boxes. Determining the selected orientation point can further comprise determining the selected orientation point, at least partially, based on a distance between the 2D projection of the gaze vector and the 2D projection of each of the plurality of 3D boundary boxes.

[0014] A system for displaying information to a vehicle occupant is provided according to several aspects. The system may include a vehicle camera, an occupant monitoring system (OMS), an augmented reality (AR) display system, and a vehicle control unit in electrical communication with the vehicle camera, the OMS, and the AR display system. The vehicle control unit is programmed to capture one or more camera images of the vehicle's surroundings using the vehicle camera. The vehicle control unit is further programmed to determine the position of a vehicle camera, at least partially, based on the one or more camera images. The vehicle control unit is also programmed to determine the occupant's gaze vector using the OMS.The vehicle control system is further programmed to determine a selected landmark from a multitude of landmarks in the environment surrounding the vehicle, at least partially based on the position of the vehicle camera, the occupant's viewing vector, and a three-dimensional (3D) map of the environment. The 3D map includes a multitude of 3D boundary boxes. Each of these 3D boundary boxes corresponds to one of the multitude of landmarks. The vehicle control system is also programmed to display information about the selected landmark to the occupant using the AR display system.

[0015] In a further aspect of the present disclosure, the vehicle control system, in order to determine the position of the vehicle camera, is further programmed to detect one or more detected reference points from the plurality of reference points in the one or more camera images. To determine the position of the vehicle camera, the vehicle control system is further programmed to determine the position of the vehicle camera, at least partially, based on comparing one or more of the one or more detected reference points with one or more of the plurality of 3D boundary boxes.

[0016] In another aspect of the present disclosure, the vehicle control system, in order to determine the position of the vehicle camera, is further programmed to identify a plurality of 2D points within each of the one or more detected reference points in the one or more camera images. To determine the position of the vehicle camera, the vehicle control system is further programmed to determine a 2D point feature vector for each of the plurality of 2D points. To determine the position of the vehicle camera, the vehicle control system is further programmed to identify a plurality of corresponding points between one or more of a plurality of 3D points of the 3D map and one or more of the plurality of 2D points, at least partially based on a 3D point feature vector for each of the plurality of 3D points and the 2D point feature vector for each of the plurality of 2D points.Each of the numerous corresponding points lies within one of the numerous 3D boundary boxes. To determine the position of the vehicle camera, the vehicle control system is further programmed to determine the position of the vehicle camera, at least partially, based on the numerous corresponding points.

[0017] In another aspect of the present disclosure, the vehicle control system, in order to determine the selected orientation point, is further programmed to determine a projected view, at least partially, based on the occupant's view vector. To determine the selected orientation point, the vehicle control system is further programmed to identify one or more collisions between the projected view and one or more of the plurality of 3D boundary boxes. To determine the selected orientation point, the vehicle control system is further programmed to determine the selected orientation point, at least partially, based on the one or more collisions.

[0018] In another aspect of the present disclosure, the vehicle control system is further programmed to determine the projected view and identify one or more collisions. The projected view further comprises a viewing cone defined by a viewing cone angle. A longitudinal axis of the viewing cone coincides with the viewing vector. To determine the projected view and identify one or more collisions, the vehicle control system is further programmed to identify one or more collisions between the viewing cone and one or more of the plurality of 3D boundary boxes.

[0019] In another aspect of the present disclosure, the vehicle control system, in order to determine the selected orientation point, is further programmed to determine a truncated view cone of the occupant, at least partially, based on the occupant's gaze vector, the position of the vehicle camera, and the plurality of 3D boundary boxes. The truncated view cone comprises a 2D projection of the gaze vector and a 2D projection of the plurality of 3D boundary boxes. To determine the selected orientation point, the vehicle control system is further programmed to determine the selected orientation point, at least partially, based on a distance between the 2D projection of the gaze vector and the 2D projection of each of the plurality of 3D boundary boxes.

[0020] In another aspect of the present disclosure, the vehicle control system, in order to determine the selected orientation point, is further programmed to determine a truncated view cone of the occupant, at least partially, based on the occupant's gaze vector, the position of the vehicle camera, and the plurality of 3D boundary boxes. The truncated view cone comprises a 2D projection of the gaze vector and a 2D projection of the plurality of 3D boundary boxes. To determine the selected orientation point, the vehicle control system is further programmed to determine the selected orientation point, at least partially, based on a portion of the 2D projection of each of the plurality of 3D boundary boxes.

[0021] A method for displaying information to a vehicle occupant is provided according to several aspects. The method may include capturing one or more camera images using a vehicle camera. The method may further include detecting one or more landmarks from a plurality of landmarks within the one or more camera images. The method may further include determining the position of a vehicle camera, at least partially, based on one or more camera images of an environment surrounding the vehicle, captured using the vehicle camera, and a three-dimensional (3D) map of the environment surrounding the vehicle. The method may further include determining the viewing vector of the vehicle occupant.The method can further include determining a selected landmark from a multitude of landmarks, at least partially based on the 3D map, the position of the vehicle camera, and the occupant's gaze vector. The method can also include displaying information about the selected landmark to the occupant using an augmented reality (AR) display system.

[0022] In another aspect of the present disclosure, determining the position of the vehicle camera may further comprise identifying a plurality of 2D points within each of the one or more detected reference points in the one or more camera images. Determining the position of the vehicle camera may further comprise determining a 2D point feature vector for each of the plurality of 2D points. Determining the position of the vehicle camera may further comprise identifying a plurality of corresponding points between one or more of the plurality of 3D points of the 3D map and one or more of the plurality of 2D points, at least partially based on a 3D point feature vector for each of the plurality of 3D points and the 2D point feature vector for each of the plurality of 2D points.Determining the position of the vehicle camera can further involve determining its position using a perspective n-point algorithm (PnP algorithm) and a random-sample consensus algorithm (RANSAC algorithm), at least partially based on the multitude of corresponding points. The position of the vehicle camera is defined with six degrees of freedom (DoF).

[0023] In another aspect of the present disclosure, determining the selected landmark may further include determining a projected view. The projected view comprises a cone of view defined by a cone angle, wherein a longitudinal axis of the cone of view coincides with the view vector. Determining the selected landmark may further include identifying one or more collisions between the cone of view and one or more of a plurality of 3D boundary boxes of the 3D map. Each of the plurality of 3D boundary boxes corresponds to one of the plurality of landmarks. Determining the selected landmark may further include determining the selected landmark at least partially based on the one or more collisions.

[0024] Further areas of application will become apparent from the description provided herein. It is understood that the description and specific examples serve only for illustrative purposes and are not intended to limit the scope of this disclosure. Brief description of the drawings

[0025] The drawings described herein serve only for illustrative purposes and are not intended to limit the scope of the present disclosure in any way. Fig. Figure 1 is a block diagram of a system for displaying information to an occupant of a vehicle according to an exemplary embodiment; Fig. Figure 2 is a flowchart of a method for displaying information to an occupant of a vehicle according to an exemplary embodiment; Fig. Figure 3A is an illustration of an exemplary camera image taken by the vehicle according to an exemplary embodiment; Fig. Figure 3B is an illustration of an exemplary processed image according to an exemplary embodiment; and Fig. 3C is a diagram that shows a match between the exemplary processed image of Fig. 3B and a 3D map illustrated, according to an exemplary embodiment; Fig. 4A is a flowchart of a procedure for generating the 3D map of Fig. 3C according to an exemplary embodiment; Fig. Figure 4B is an illustration of an exemplary 3D point cloud superimposed on an illustration of the environment, according to an exemplary embodiment; Fig. Figure 5 is a flowchart of a method for determining a selected reference point according to a first exemplary embodiment; Fig. Figure 6A is a diagram of a projected view of the occupant according to a first exemplary embodiment; Fig. Figure 6B is a diagram of a projected view of the occupant according to a second exemplary embodiment; Fig. Figure 7 is a flowchart of a method for determining a selected reference point according to a second exemplary embodiment; and Fig. Figure 8 is an illustration of an exemplary truncated cone view of the occupant according to an exemplary embodiment. Detailed description

[0026] The following description is merely exemplary and is not intended to limit the present disclosure, application or uses.

[0027] In aspects of this disclosure, vehicle occupants may wish to receive information about landmarks and / or points of interest in the environment surrounding the vehicle. This disclosure provides a new and improved system and method for displaying information to vehicle occupants with minimal occupant interaction and minimal disruption to the driving task or passenger experience.

[0028] With reference to Fig. Figure 1 illustrates a system for displaying information to a vehicle occupant and is generally indicated by reference numeral 10. System 10 is shown with an exemplary vehicle 12. Although a passenger car is illustrated, it is understood that vehicle 12 can be any type of vehicle without deviating from the scope of this disclosure, including, for example, an autonomous vehicle. System 10 generally comprises a vehicle control unit 14, a plurality of vehicle sensors 16, and a display 18.

[0029] The vehicle control unit 14 is used to implement a method 100 for displaying information to a vehicle occupant, as described below. The vehicle control unit 14 comprises at least one processor 20 and a non-volatile, computer-readable storage device or non-volatile, computer-readable storage medium 22. The processor 20 may be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the vehicle control unit 14, a microprocessor-based semiconductor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally, an instruction-executing device.

[0030] The computer-readable storage device or computer-readable storage medium 22 may, for example, include volatile and non-volatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operating variables while the processor 20 is turned off. The computer-readable storage device or computer-readable storage medium 22 may be implemented using a variety of storage devices, such as PROMs (programmable read-only memory), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which may be executable instructions used by the vehicle controller 14 to control various systems of the vehicle 12.

[0031] The vehicle control unit 14 can also comprise multiple control units that communicate electrically with each other. The vehicle control unit 14 can be connected to additional systems and / or controls of the vehicle 12, which allows the vehicle control unit 14 to access data such as speed, acceleration, braking, and steering angle of the vehicle 12.

[0032] The vehicle control unit 14 communicates electrically with the plurality of vehicle sensors 16 and the display 18. In an exemplary embodiment, the electrical communication is established, for example, using a CAN network, a FLEXRAY network, a local area network (for example, WiFi, Ethernet, and the like), a network with a serial peripheral interface (SPI), or the like. It is understood that various additional wired and wireless techniques and communication protocols for communicating with the vehicle control unit 14 are within the scope of this disclosure. It is further understood that, within the scope of this disclosure, the electrical communication also includes the supply of power and / or energy transfer between electrical devices (for example, using conductive wires and / or wireless power transfer techniques).

[0033] The plurality of vehicle sensors 16 is used to acquire information relevant to the vehicle 12. In one exemplary embodiment, the plurality of vehicle sensors 16 comprises at least one vehicle camera 24 and an occupant monitoring system (OMS) 26. In another exemplary embodiment, the plurality of vehicle sensors 16 further comprises a global navigation satellite system (GNSS) 28 and / or an inertial measurement unit (IMU) 30.

[0034] In another exemplary embodiment, the plurality of vehicle sensors 16 further comprises sensors for determining performance data about the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 further comprises at least one of an engine speed sensor, an engine torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a brake position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor.

[0035] In another exemplary embodiment, the plurality of vehicle sensors 16 further comprises sensors for determining information about an environment inside the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 further comprises at least one of a seat occupancy sensor, a cabin air temperature sensor, a cabin motion detection sensor, a cabin camera, a cabin microphone and / or the like.

[0036] In another exemplary embodiment, the plurality of vehicle sensors 16 further comprises sensors for determining information about an environment 32 surrounding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 further comprises at least one ambient air temperature sensor, an air pressure sensor, and / or a photo and / or video camera positioned to view the environment 32 in front of the vehicle 12.

[0037] In another exemplary embodiment, at least one of the plurality of vehicle sensors 16 is a perception sensor capable of perceiving objects and / or measuring distances in the environment 32 surrounding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 comprises a stereoscopic camera with distance-measuring capabilities. In one example, at least one of the plurality of vehicle sensors 16 is mounted inside the vehicle 12, for example, in a headliner of the vehicle 12, providing a view through a windshield of the vehicle 12. In another example, at least one of the plurality of vehicle sensors 16 is mounted outside the vehicle 12, for example, on a roof of the vehicle 12, providing a view of the environment 32 surrounding the vehicle 12.It is understood that various additional types of perception sensors, such as LiDAR sensors, ultrasonic distance sensors, radar sensors, and / or time-of-flight sensors, are within the scope of this disclosure. The multitude of vehicle sensors 16 are in electrical communication with the vehicle control unit 14, as discussed above.

[0038] The vehicle camera 24 is a perception sensor used to capture images and / or videos of the environment 32 surrounding the vehicle 12. In one exemplary embodiment, the vehicle camera 24 comprises a photo and / or video camera positioned to view the environment 32 surrounding the vehicle 12. In a non-limiting example, the vehicle camera 24 comprises a camera mounted inside the vehicle 12, for example, in the headliner of the vehicle 12, providing a view through the windshield. In another non-limiting example, the vehicle camera 24 comprises a camera mounted outside the vehicle 12, for example, on the roof of the vehicle 12, providing a view of the environment 32 in front of the vehicle 12.

[0039] In another exemplary embodiment, the vehicle camera 24 is a surround-view camera system comprising a plurality of cameras (also known as satellite cameras) arranged to provide a view of the surroundings 32 adjacent to all sides of the vehicle 12. In a non-limiting example, the vehicle camera 24 comprises a forward-facing camera (mounted, for example, in a front grille of the vehicle 12), a rear-facing camera (mounted, for example, on a tailgate of the vehicle 12), and two side-facing cameras (mounted, for example, below each of two side mirrors of the vehicle 12). In another non-limiting example, the vehicle camera 24 further comprises an additional reversing camera mounted near a centrally mounted high-mounted brake light of the vehicle 12.

[0040] It is understood that camera systems comprising additional cameras and / or additional mounting points are within the scope of this disclosure. It is further understood that cameras with different sensor types, including, for example, CCD sensors (CCD = charge-coupled device), CMOS sensors (CMOS = complementary metal oxide semiconductor), and / or HDR sensors (HDR = high dynamic range), are within the scope of this disclosure. Furthermore, cameras with different lens types, including, for example, wide-angle lenses and / or narrow-angle lenses, are also within the scope of this disclosure.

[0041] The occupant monitoring system (OMS) 26 is used to determine the direction of view of the vehicle occupant 34 ( Fig. 6A, Fig. 6B) within the vehicle 12 to determine. To the extent of the present disclosure, the occupant comprises 34 ( Fig. 6A, Fig. 6B) a driver and / or a passenger of the vehicle 12. In an exemplary embodiment, the OMS 26 comprises one or more infrared cameras (IR cameras) positioned inside the vehicle 12 to capture images of the vehicle occupant 34 ( Fig. 6A, Fig. 6B). The OMS further includes an image processor (not shown) in electrical communication with the IR cameras. The IR cameras capture high-resolution images of the face and eyes of occupant 34 ( Fig. 6A, Fig. 6B) and the image processor analyzes the images to determine the occupant's gaze direction 34 ( Fig. 6A, Fig. 6B). The OMS 26 uses reflected IR light from the eyes and surrounding facial features to track the orientation and position of the eyes, enabling the OMS 26 to calculate the gaze direction based on the processed image data. In a non-restrictive example, the gaze direction of occupant 34 ( Fig. 6A, Fig. 6B) defined by a viewing vector. The OMS 26 is in electrical communication with the vehicle control unit 14, as discussed above.

[0042] The GNSS 28 is used to determine the geographic location of the vehicle 12. In an exemplary embodiment, the GNSS 28 is a global positioning system (GPS). In a non-limiting example, the GPS comprises a GPS receiver antenna (not shown) and a GPS controller (not shown) in electrical communication with the GPS receiver antenna. The GPS receiver antenna receives signals from a variety of satellites, and the GPS controller calculates the geographic location of the vehicle 12 based on the signals received by the GPS receiver antenna.

[0043] In one exemplary embodiment, the GNSS 28 additionally includes a map. The map contains information about infrastructure such as municipal boundaries, roads, railways, sidewalks, buildings, and the like. Therefore, the geographic location of the vehicle 12 is contextualized using the map information. In a non-limiting example, the map is retrieved from a remote source using a wireless connection. In another non-limiting example, the map is stored in a database of the GNSS 28. It is understood that various additional types of satellite-based radio navigation systems, such as the Global Positioning System (GPS), Galileo, GLONASS, and the BeiDou Navigation Satellite System (BDS), are within the scope of this disclosure. The GNSS 28 is in electrical communication with the vehicle control unit 14, as discussed above.

[0044] The IMU 30 is used to determine the orientation, velocity, and gravitational forces acting on the vehicle 12. In one exemplary embodiment, the IMU 30 comprises multiple sensors, including accelerometers, gyroscopes, and / or magnetometers. In a non-limiting example, the IMU 30 comprises tri-axis accelerometers and tri-axis gyroscopes integrated into a single unit. The accelerometers measure the linear acceleration along each axis, while the gyroscopes measure the angular velocity about each axis. The IMU 30 processes data from the sensors to calculate the current orientation, velocity, direction, yaw rate (that is, rate of change of direction), and acceleration of the vehicle 12 in three-dimensional space. The IMU 30 communicates electrically with the vehicle controller 14, as discussed above.

[0045] Display 18 is used to show occupant 34 ( Fig. 6A, Fig. 6B) of the vehicle 12 to provide information. In an exemplary embodiment, the display 18 is a human-machine interface (HMI) located within the view of the occupant 34 ( Fig. 6A, Fig. 6B) and can display text, graphics and / or images. It is understood that HMI display systems, including LCD displays, LED displays and the like, are within the scope of this disclosure. Further exemplary embodiments in which the display 18 is arranged in a rearview mirror are also within the scope of this disclosure.

[0046] In another exemplary embodiment, the display 18 comprises a head-up display (HUD) configured to show the occupant 34 ( Fig. 6A, Fig. 6B) To provide information by projecting text, graphics and / or images onto the windscreen of vehicle 12. The text, graphics and / or images are reflected from the windscreen of vehicle 12 and are visible to the occupant 34 ( Fig. 6A, Fig. 6B) visible without looking away from a roadway in front of the vehicle 12. In another exemplary embodiment, the display 18 comprises an augmented reality (AR) display system, such as an augmented reality head-up display (AR-HUD). The AR-HUD is a type of HUD configured to enhance the view of the environment 32 around the vehicle 12 for the occupant 34 ( Fig. 6A, Fig. 6B) to improve by applying text, graphics and / or images to physical objects in the environment 32 around the vehicle 12 within a field of vision of the occupant 34 ( Fig. 6A, Fig. 6B) are superimposed.

[0047] In an exemplary embodiment, the occupant 34 ( Fig. 6A, Fig. 6B) interacting with the display 18 using a human interface device (HID), which includes, for example, a touchscreen, an electromechanical switch, a capacitive switch, a rotary knob, and the like. It is understood that additional systems for displaying information to the occupant 34 ( Fig. 6A, Fig. 6B) of the vehicle 12 also fall within the scope of this disclosure. The display 18 is in electrical communication with the vehicle control unit 14, as discussed above.

[0048] With reference to Fig. Figure 2 shows a flowchart of procedure 100 for displaying information to a vehicle occupant. Procedure 100 begins at block 102 and proceeds to block 104. At block 104, a three-dimensional (3D) map 40 ( Fig. 3C) of the environment 32. In an exemplary embodiment, the 3D map 40 comprises ( Fig. 3C) a multitude of 3D points 42 ( Fig. 3C) within a multitude of 3D bounding boxes 44 ( Fig. 3C). Each of the multitude of 3D points 42 ( Fig. 3C) is defined by a 3D point feature vector and a location in 3D space. Within the scope of this disclosure, a 3D point feature vector is a high-dimensional vector (that is, a 128-dimensional vector) that represents one of the plurality of 3D points 42 ( Fig. 3C) uniquely identified. In a non-restrictive example, 3D point vectors are calculated using a mathematical algorithm based on properties of the 3D point in question and surrounding 3D points.

[0049] The multitude of 3D boundary boxes 44 ( Fig. 3C) define locations of landmarks 46 ( Fig. 3A) in three-dimensional space. For the purposes of this disclosure, a landmark is a point of interest (POI), such as a business, a school, a bus stop, a gas station, a government building (for example, a police station, a fire station, a town hall), a hospital, a park, and / or the like. Each of the multitude of 3D boundary boxes 44 ( Fig. 3C) corresponds to one of a multitude of landmarks 46 ( Fig. 3A) in the environment 32. In an exemplary embodiment, the 3D map 40 ( Fig. 3C) is generated using an external server system (not shown) located at a centralized site (for example, a server farm, data center, or the like) and connected to the internet. The generation of the 3D map 40 ( Fig. 3C) is discussed in more detail below. After Block 104, Procedure 100 proceeds to Block 106.

[0050] In block 106, the vehicle control unit 14 uses the vehicle camera 24 to capture one or more camera images of the environment 32 surrounding the vehicle 12. With reference to Fig. Image 3A is an example image 50a of the area 32, taken at Block 106. With further reference to Fig. 2. Procedure 100 proceeds from block 106 to block 108.

[0051] With renewed reference to Fig. 3A and with continued reference to Fig. 2. The vehicle control unit 14 detects one or more landmarks 46 at block 108 in the one or more images of the environment 32 surrounding the vehicle 12, which were captured at block 106 (for example, the exemplary image 50a). For the purposes of this disclosure, the one or more landmarks 46 detected at block 108 are referred to as one or more detected landmarks 46. In an exemplary embodiment, the vehicle control unit 14 uses a computer vision algorithm to detect the one or more detected landmarks 46. The computer vision algorithm uses machine learning techniques to analyze pixel-level information from an input image in order to detect and classify objects or patterns of interest.In a non-restrictive example, the computer vision algorithm begins by preprocessing the input image using techniques such as resizing, normalization, and / or filtering to reduce noise. The algorithm then extracts relevant features from the input image using methods such as edge detection, corner detection, texture analysis, and / or similar techniques. Finally, the algorithm can employ a machine learning model, such as a convolutional neural network (CNN), to classify and label relevant objects (i.e., the landmarks) of the input image based on learned patterns and associations.

[0052] With reference to Fig. Image 3B is an example of a processed image, as shown in image 50b. With reference to Fig. 2 and Fig. 3B further identifies the vehicle control unit 14 at block 108 a plurality of two-dimensional (2D) points 54 within each of the one or more detected reference points 46. In the exemplary processed image 50b, the plurality of 2D points 54 is visualized as black dots; however, it is understood that the plurality of 2D points 54 are arbitrary reference points selected within each of the plurality of reference points 46. The quantity, density, position, distribution, and / or the like of the plurality of 2D points 54 may vary within the scope of this disclosure. In an exemplary embodiment, each of the plurality of 2D points 54 is defined by a 2D point feature vector and a location in 2D space.Within the scope of this disclosure, a 2D point feature vector is a high-dimensional vector (that is, a 128-dimensional vector) that uniquely identifies one of the plurality of 2D points 54. In a non-restrictive example, 2D point vectors are computed using a mathematical algorithm based on properties of the 2D point in question and surrounding 2D points. After Block 108, the procedure proceeds to Block 110.

[0053] With reference to Fig. Figure 3C shows a diagram illustrating the correspondence between the exemplary processed image 50b and the 3D map 40. With reference to Fig. 2 and Fig. 3C identifies the vehicle control unit 14 at block 110 as a plurality of corresponding points between one or more of the plurality of 3D points 42 in the 3D map 40 and one or more of the plurality of 2D points 54 in the one or more images taken at block 106 (for example, as illustrated in the exemplary processed image 50b). For the purposes of this disclosure, corresponding points are points that indicate the same physical location in the environment 32. In the Fig. In the example shown in 3C, the correspondence between the corresponding points is illustrated by the solid lines 56. It is understood that, although there are four corresponding points in Fig. As illustrated in 3C, any number of corresponding points can be identified.

[0054] In an exemplary embodiment, the plurality of corresponding points is identified at least partially based on the 2D point feature vector of each of the plurality of 2D points 54 and the 3D point feature vector of each of the plurality of 3D points 42. In a non-restrictive example, the vehicle controller 14 searches the 3D map 40 to find 3D points 42 with 3D point feature vectors that substantially correspond to (i.e., match) one or more of the 2D point feature vectors of the 2D points 54 in the one or more images acquired at block 106 (for example, the exemplary processed image 50b). In an exemplary embodiment, the 3D map is searched only within the plurality of 3D bounding boxes 44 to increase the speed and accuracy of the search. Therefore, each of the multitude of corresponding points is located within one of the multitude of 3D boundary boxes 44.With renewed reference to . Fig. 2. Procedure 100 proceeds from block 110 to block 112.

[0055] At block 112, the vehicle control unit 14 determines a position of the vehicle camera 24, which is defined with six degrees of freedom (DoF), at least based on the plurality of corresponding points determined at block 110. Within the scope of this disclosure, the six DoF are forward / backward (pumping), up / down (lifting), left / right (swaying), yaw (rotation about the normal axis), pitch (rotation about the transverse axis), and roll (rotation about the longitudinal axis). In an exemplary embodiment, the position of the vehicle camera 24 is determined using a perspective n-point algorithm (PnP algorithm) and / or a random-sample consensus algorithm (RANSAC algorithm), as described, for example, in "Image Based 6-DOF Camera Pose Estimation with Weighted RANSAC 3D" by Johannes Wetzel. (Lecture Notes in Computer Science, Vol. 8142, pp. 249-254, Sep. 2013) described, the entire content of which is hereby incorporated by reference.In a non-restrictive example, measurements from the GNSS 28 and / or the IMU 30 are also used to determine the six DoFs, for example, to determine pumping, lifting, and / or swaying. After Block 112, Procedure 100 transitions to Block 114.

[0056] In block 114, the vehicle control unit 14 determines the occupant's (34) gaze vector. Fig. 6A, Fig. 6B). In an exemplary embodiment, the vehicle control unit 14 uses the OMS 26 to determine the gaze vector in order to take measurements of the occupant 34 ( Fig. 6A, Fig. 6B), and determines the gaze vector. In a non-restrictive example, the gaze vector is given by a three-dimensional vector and a gaze origin point located at the eyes of occupant 34 ( Fig. 6A, Fig. 6B). In an exemplary embodiment, the vehicle control unit 14 determines the gaze vector using techniques discussed, for example, in “MPIIGaze: Real-world dataset and deep appearance-based gaze estimation” by Zhang, X., et al. (IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, No. 1, pp. 162–175, Jan. 2019), the entire content of which is hereby incorporated by reference. After Block 114, the method 100 transitions to Block 116.

[0057] In block 116, the vehicle control unit 14 determines a selected orientation point from the multitude of orientation points 46 at least partially based on the multitude of 3D boundary boxes 44 in the 3D map 40, the position of the vehicle camera 24 within the environment 32, which is mapped by the 3D map 40, and the viewing vector of the occupant 34 ( Fig. 6A, Fig. 6B). The determination of the selected landmark is discussed in more detail below. After Block 116, Procedure 100 continues to Block 118.

[0058] At block 118, the vehicle control unit 14 uses the display 18 to provide information about the selected landmark determined at block 116 for the occupant 34 ( Fig. 6A, Fig. 6B) to display. In one exemplary embodiment, the information includes a name of the landmark (for example, a company name), a logo of the landmark (for example, a company logo), opening hours of the landmark, services available at the landmark (for example, services offered by a business at the landmark), information about events occurring at the landmark, historical information about the landmark, news or current events relating to the landmark, and / or the like. It is understood that the information may include any type of information relating to the landmark and that the information may be provided in any form, including text and / or graphics.In one exemplary embodiment, the vehicle control unit 14 uses the AR display system of the display 18 to visually display the information. In another exemplary embodiment, the vehicle control unit 14 uses text-to-speech or speech synthesis to provide the information audibly to the occupant 34. Fig. 6A, Fig. 6B). After block 118, procedure 100 transitions to enter a standby state at block 120.

[0059] In one exemplary embodiment, the vehicle control unit 14 repeatedly leaves the standby state 120 and restarts the procedure 100 at block 102. In a non-restrictive example, the vehicle control unit 14 leaves the standby state 120 and restarts the procedure 100 on a timer, for example, every three hundred milliseconds.

[0060] With reference to Fig. Figure 4A shows a flowchart of a method 104a for generating the 3D map 40 at block 104 of method 100. In an exemplary embodiment, the method 104a is performed by the external server system (not shown), as discussed above. With reference to Fig. 4A and with continued reference to the preceding figures, method 104a begins at block 402. At block 402, the external server system receives a plurality of reference images comprising the plurality of landmarks 46 in the environment 32. In one exemplary embodiment, the plurality of reference images is crowdsourced from multiple vehicles, such as end-user vehicles, fleet vehicles, and / or dedicated data collection vehicles. In a non-limiting example, the plurality of reference images is acquired from varying positions in the environment 32, for example, while the multiple vehicles drive through the environment 32, thereby acquiring the plurality of landmarks 46 from varying angles / perspectives. After block 402, method 104a proceeds to block 404.

[0061] With reference to Fig. Figure 4B shows an illustration of an exemplary 3D point cloud superimposed on an illustration of the environment shown in Figure 32. With reference to Fig. 4A and Fig. 4B The external server system at block 404 generates a 3D point cloud comprising the plurality of 3D points 42. In an exemplary embodiment, each of the plurality of 3D points 42 is defined by a 3D point vector and a location in 3D space. Within the scope of this disclosure, a 3D point feature vector is a vector (that is, a one-dimensional matrix) that uniquely identifies one of the plurality of 3D points 42. In a non-limiting example, 3D point vectors are computed using a mathematical algorithm based on properties of the 3D point in question and surrounding 3D points. In another non-limiting example, the 3D point vectors are computed by averaging information from each of the plurality of reference images. In an exemplary embodiment, the 3D point cloud is generated based on the plurality of reference images collected at block 402.In a non-restrictive example, a StructurefromMotion (SfM) algorithm is used to generate the 3D point cloud based on the multitude of reference images, as discussed, for example, in “A survey of structure from motion” by Özyeşil, O., et al. (Acta Numerica, Vol. 26, pp. 305-364, May 2017), the entire content of which is hereby incorporated by reference.

[0062] After generating the 3D point cloud containing the multitude of 3D points 42, each of the multitude of 3D points 42 is labeled with a corresponding landmark from the multitude of landmarks 46. In one non-restrictive example, each of the multitude of 3D points 42 is labeled with a geographically nearest landmark, as identified based on a map database containing the coordinate locations of each of the multitude of landmarks 46. In another non-restrictive example, each of the multitude of 3D points 42 is labeled using computer vision-based object detection on the multitude of reference images to identify the multitude of landmarks 46. With further reference to Fig. 4A, procedure 104a passes from block 404 to block 406.

[0063] Block 406 generates a multitude of point cloud clusters from the multitude of 3D points 42. In an exemplary embodiment, the multitude of point cloud clusters is generated using a density-based spatial clustering-of-applications-with-noise algorithm (DBSCAN algorithm), as described, for example, in "A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise" by Ester et al. (Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, Pgs. 226-231, Aug. 1996), the entire content of which is hereby incorporated by reference.It is understood that alternative or additional clustering algorithms, such as distributed DBSCAN (DDBSCAN), k-means, agglomerative clustering, mean shift, Gaussian mixture models, spectral clustering, affinity propagation, balanced iterative reduction, and clustering using hierarchies (BIRCH), ordering points to identify the clustering structure (OPTICS), fuzzy c-means, and / or the like, may be used without deviating from the scope of this disclosure. After Block 406, Method 104a transitions to Block 408.

[0064] In block 408, the multitude of point cloud clusters is filtered to generate a multitude of filtered point cloud clusters. Each of the multitude of filtered point cloud clusters corresponds to one of the multitude of orientation points 46. In an exemplary embodiment, extraneous points are first removed to filter the multitude of point cloud clusters. For example, points that are not assigned to any of the multitude of orientation points 46 are disregarded.

[0065] Then, one or more of the plurality of point cloud clusters are split to generate a plurality of split point cloud clusters such that each of the plurality of split point cloud clusters contains points labeled with only one of the plurality of orientation points 46 (that is, in a given split point cloud cluster, all points are labeled with the same orientation point). In a non-restrictive example, to split the plurality of point cloud clusters, for each of the plurality of point cloud clusters, if the point cloud cluster contains points labeled with different orientation points, the point cloud cluster is split into two or more split point cloud clusters, each containing points labeled with the same orientation point.

[0066] Subsequently, in a non-restrictive example, two or more of the multitude of split point cloud clusters are labeled identically to generate the multitude of filtered point cloud clusters. In a non-restrictive example, the multitude of split point cloud clusters can include multiple split point cloud clusters corresponding to a single landmark 46 (for example, a first split point cloud cluster corresponding to a sign, a parking lot, an entrance, and / or the like, and a second split point cloud cluster corresponding to a building that houses the landmark itself).In an exemplary embodiment, the multiple split point cloud clusters are determined to correspond to one of the plurality of reference points, based at least partially on at least one of: a relative size of the two or more of the plurality of split point cloud clusters and a distance between the two or more of the plurality of split point cloud clusters. In a non-restrictive example, the one or more of the plurality of split point cloud clusters are identified as corresponding to a nearest larger split point cloud cluster (that is, containing more points and covering a larger area) that corresponds to a nearest reference point within a predetermined threshold distance. Following Block 408, Method 104a transitions to Block 410.

[0067] In block 410, the plurality of 3D boundary boxes 44 is determined based on the plurality of filtered point cloud clusters. In an exemplary embodiment, the boundaries of each of the plurality of 3D boundary boxes 44 are determined to completely enclose one of the plurality of filtered point cloud clusters. Therefore, each of the plurality of 3D boundary boxes 44 corresponds to one of the plurality of orientation points 46. The plurality of 3D boundary boxes 44 is in the Fig. Figure 3C illustrates the 3D map 40 as discussed above. In an exemplary embodiment, the completed 3D map 40 is transferred to the vehicle 12 and stored in the medium 22 of the vehicle control unit 14 for use in the method 100, as discussed above. According to Block 410, the method 104a is completed and the method 100 continues as discussed above.

[0068] With reference to Fig. Figure 5 shows a flowchart of a first exemplary embodiment 116a of block 116 of method 100 (that is, a method for determining a selected orientation point). The first exemplary embodiment 116a of block 116 begins at block 502. At block 502, the vehicle control unit 14 determines a projected view based on the view vector determined at block 114. Within the scope of this disclosure, the projected view is a projection (that is, a coordinate transformation) of the view vector into a coordinate system of the environment 32 (that is, a world coordinate system) and the 3D map 40. In one exemplary embodiment, the view vector is transformed based on the position of the vehicle camera 24 determined at block 112.In a non-restrictive example, the OMS 26 determines the viewing vector in a relative coordinate system of the vehicle 12, and the position of the vehicle camera 24 anchors the relative coordinate system of the vehicle 12 within an absolute coordinate system of the environment 32 (that is, a world coordinate system). Therefore, the projected view is determined using one or more coordinate transformations and projected onto the 3D map 40.

[0069] With reference to Fig. Figure 6A shows a diagram of a first exemplary embodiment of the projected view with the occupant 34 in the vehicle 12. Fig. 6A, the projected view is realized as a projected view vector 60. The projected view vector 60 is analogous to the view vector discussed above, except that the projected view vector 60 is defined in the same coordinate system as the 3D map 40 (i.e., a world coordinate system).

[0070] With reference to Fig. Figure 6B shows a diagram of a second exemplary embodiment of the projected view with the occupant 34 in the vehicle 12. Fig. 6B, the projected gaze is realized as a projected cone of vision 62 in addition to the projected gaze vector 60. The projected cone of vision 62 is defined by a cone angle 64. A longitudinal axis of the projected cone of vision coincides with the projected gaze vector 60, as shown in Fig. Figure 6B shows that in an exemplary embodiment, the viewing cone angle 64 can be predetermined or adjustable, as discussed in more detail below. With further reference to Fig. 5 the first exemplary embodiment 116a goes from block 116 to block 502 to block 504.

[0071] At block 504, the vehicle controller 14 identifies one or more collisions between the projected view determined at block 502 and one or more of the plurality of 3D bounding boxes 44. In an exemplary embodiment, axis-aligned bounding box (AABB) collision detection is used to identify collisions, as is known in the field of computer graphics. In a non-restrictive example, the projected view is further projected or simulated within the 3D map 40, and collisions with the plurality of 3D bounding boxes 44 are identified. In an exemplary embodiment, the Fig. The embodiment shown in Figure 6B, which includes the viewing cone 62, can be used to increase the reliability and repeatability of the collision detection. By adjusting the viewing cone angle 64, the effective sensitivity of the viewing collision detection can be tuned. In an exemplary embodiment, if the viewing cone 62 does not collide with one of the plurality of 3D boundary boxes 44, the viewing cone angle 64 is incrementally increased until a collision with the nearest boundary box is identified. Following Block 504, the first exemplary embodiment 116a transitions from Block 116 to Block 506.

[0072] At block 506, the vehicle control 14 identifies the selected orientation point from the plurality of orientation points 46. In an exemplary embodiment, the selected orientation point is identified at least partially based on the one or more collisions identified at block 504. In a non-restrictive example, if the projected view collides with a first of the plurality of 3D boundary boxes 44, the selected orientation point is determined to be the one of the plurality of orientation points 46 that corresponds to the first of the plurality of 3D boundary boxes 44. If the view cone 62 collides with multiple boundary boxes, the collision closest to the center of the view cone 62 (i.e., a location of the projected view vector 60) is determined to be the selected orientation point.Following Block 506, the first exemplary embodiment 116a of Block 116 is completed and the method 100 proceeds as discussed above.

[0073] With reference to Fig. Figure 7 shows a flowchart of a second exemplary embodiment 116b of block 116 of method 100 (that is, a method for determining a selected reference point). The second exemplary embodiment 116b of block 116 begins at block 702. With reference to Fig. Figure 8 shows an exemplary view of a truncated cone 70. With reference to Fig. 7 and Fig. 8 The vehicle control unit 14 at block 702 determines a truncated cone of view 70 of the occupant 34 at least partially based on the occupant 34's gaze vector, the position of the vehicle camera 24, and the 3D map 40, which comprises the plurality of 3D boundary boxes 44. Within the scope of this disclosure, the truncated cone of view 70 is a 2D projection of the occupant 34's field of vision.

[0074] In an exemplary embodiment, the truncated view cone 70 is determined by first projecting the view vector onto the 3D map 40, as discussed above. The perspective view of the occupant 34 within the 3D map 40 is then determined based on the projection of the view vector and an estimated or preset field of view of the occupant 34 centered on the projection of the view vector within the 3D map 40. Subsequently, the perspective view of the occupant 34 is projected onto 2D to generate the truncated view cone 70. In a non-restrictive example, the truncated view cone 70 comprises a 2D view vector 72 (that is, a 2D projection of the view vector) and a plurality of 2D bounding boxes 74 (that is, a 2D projection of the plurality of 3D bounding boxes 44). In a non-restrictive example, the truncated view cone 70 is centered on the 2D view vector 72. Referring again to Fig.7 the second exemplary embodiment 116b goes from block 116 to block 702 to block 704.

[0075] In block 704, the vehicle control unit 14 determines the selected orientation point from the plurality of orientation points 46. In an exemplary embodiment, the vehicle control unit 14 identifies the selected orientation point using the truncated cone of view 70. In a first exemplary embodiment, the selected orientation point is determined at least partially based on a distance between the 2D viewing vector 72 and each of the plurality of 2D boundary boxes 74. In a non-restrictive example, it is determined that the selected orientation point is one of the plurality of orientation points 46 in the truncated cone of view 70 that has a 2D boundary box 74 that is geometrically closest to the 2D viewing vector 72. In a second exemplary embodiment, the selected orientation point is determined at least partially based on an area of ​​each of the plurality of 2D boundary boxes 74.In one non-restrictive example, it is determined that the selected orientation point is one of the plurality of orientation points 46 in the truncated cone of view 70, which has a 2D bounding box 74 with a largest area. In another non-restrictive example, it is determined that the selected orientation point is one of the plurality of orientation points 46 in the truncated cone of view 70, which has a 2D bounding box 74 with a largest area within a predetermined distance threshold of the 2D viewing vector 72. According to Block 704, the second exemplary embodiment 116b of Block 116 is completed, and Method 100 proceeds as discussed above.

[0076] System 10 and method 100 of the present disclosure offer several advantages. By using System 10 and method 100, vehicle occupants are provided with relevant information about objects in their environment based on gaze detection. Furthermore, System 10 and method 100 enable the efficient generation of detailed 3D maps that include the locations of landmarks in the environment. By searching the 3D map data based on 2D image data acquired by the vehicle 12, accurate and precise location and position information can be determined via the vehicle camera 24 and by extending the vehicle 12.

[0077] The description of the present revelation is merely exemplary, and variations that do not deviate from the core of the present revelation are to be considered within the scope of the present revelation. Such variations should not be regarded as a deviation from the spirit and scope of the present revelation. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Image Based 6-DOF Camera Pose Estimation with Weighted RANSAC 3D.” by Wetzel, Johannes. (Lecture Notes in Computer Science, Vol. 8142, pp. 249-254, Sep. 2013

[0055] Zhang, X., et al. (IEEE Transactions on Pattern Analysis and Machine Intelligence, Bd. 41, Nr. 1, S. 162-175, Jan. 2019

[0056] A survey of structure from motion“ von Özyeşil, O., et al. (Acta Numerica, Bd. 26, S. 305-364, Mai 2017

[0061] A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise“ von Ester et al. (Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, Pgs. 226-231, Aug. 1996

[0063]

Claims

[1] Method for displaying information to an occupant of a vehicle, the method comprising: Generating a three-dimensional, 3D, map of an environment surrounding the vehicle, wherein the 3D map comprises a plurality of 3D boundary boxes and wherein each of the plurality of 3D boundary boxes corresponds to one of a plurality of landmarks; Determining the position of a vehicle camera, at least partially, based on one or more camera images of an environment surrounding the vehicle, captured using the vehicle camera; Determining the gaze vector of the vehicle's occupant; Determining a selected reference point from the multitude of reference points, at least partially based on the multitude of 3D boundary boxes, the position of the vehicle camera, and the occupant's viewing vector; and Displaying information about the selected landmark for the occupant using a display. [2] Method according to claim 1, wherein the generation of the 3D map further comprises: Generating a 3D point cloud comprising a multitude of 3D points, based on a multitude of reference images comprising the multitude of orientation points, where each of the multitude of 3D points is defined by a 3D point feature vector; and Determine each of the multitude of 3D boundary boxes by clustering the 3D point cloud into a multitude of point cloud clusters. [3] Method according to claim 2, wherein the clustering of the 3D point cloud further comprises: Generating the multitude of point cloud clusters using a density-based spatial clustering-of-applications-with-noise algorithm, DBSCAN algorithm; and Filtering the multitude of point cloud clusters to generate a multitude of filtered point cloud clusters, where each of the multitude of filtered point cloud clusters corresponds to one of the multitude of orientation points. [4] Method according to claim 3, wherein filtering the plurality of point cloud clusters further comprises: Splitting one or more of the plurality of point cloud clusters to generate a plurality of split point cloud clusters, wherein each of the plurality of split point cloud clusters corresponds to only one of the plurality of orientation points, and wherein the plurality of filtered point cloud clusters comprises the plurality of split point cloud clusters. [5] Method according to claim 4, wherein filtering the plurality of point cloud clusters further comprises: Determining two or more of the plurality of split point cloud clusters to correspond to an equal of the plurality of reference points, based at least partially on at least one of: a relative size of the two or more of the plurality of split point cloud clusters and a distance between the two or more of the plurality of split point cloud clusters. [6] The method of claim 2, wherein determining the position of the vehicle camera further comprises: Capturing one or more camera images using the vehicle camera; Detecting one or more detected landmarks from the multitude of landmarks in the one or more camera images, wherein the detection of the one or more detected landmarks further comprises: Identifying a multitude of 2D points within each of the one or more detected landmarks in the one or more camera images; and Determining a 2D point feature vector for each of the plurality of 2D points; and Determining the position of the vehicle camera, at least partially, based on comparing one or more of the detected reference points with one or more of the multitude of 3D boundary boxes. [7] The method of claim 6, wherein determining the position of the vehicle camera further comprises: Identifying a plurality of corresponding points between one or more of the plurality of 3D points and one or more of the plurality of 2D points, at least partially based on the 3D point feature vector of each of the plurality of 3D points and the 2D point feature vector of each of the plurality of 2D points, wherein each of the plurality of corresponding points is located within one of the plurality of 3D bounding boxes; and Determining the position of the vehicle camera using a perspective n-point algorithm, PnP algorithm, and a random-sample consensus algorithm, RANSAC algorithm, at least partially based on the multitude of corresponding points, wherein the position of the vehicle camera is defined with six degrees of freedom, DoF. [8] Method according to claim 1, wherein determining the selected reference point further comprises: Determining a projected gaze at least partially based on the gaze vector of the occupant; Identifying one or more collisions between the projected view and one or more of the multitude of 3D bounding boxes; and Determining the selected reference point, at least partially, based on one or more collisions. [9] Method according to claim 8, wherein determining the projected view and identifying the one or more collisions further comprises: Determining the projected gaze, wherein the projected gaze further comprises a cone of vision defined by a cone of vision angle, and wherein a longitudinal axis of the cone of vision coincides with the gaze vector; and Identifying one or more collisions between the cone of vision and one or more of the multitude of 3D boundary boxes. [10] Method according to claim 1, wherein determining the selected orientation point further comprises: Determining a truncated view cone of the occupant at least partially based on the occupant's gaze vector, the position of the vehicle camera, and the plurality of 3D boundary boxes, wherein the truncated view cone comprises a 2D projection of the gaze vector and a 2D projection of the plurality of 3D boundary boxes; and determining the selected orientation point at least partially based on a distance between the 2D projection of the gaze vector and the 2D projection of each of the plurality of 3D boundary boxes.

Images