Providing information about point of interest to vehicle occupant

Abstract

A system for providing information to a vehicle occupant includes a global navigation satellite system (GNSS), a vehicle communication system, a display system configured to provide information to the vehicle occupant, and a vehicle controller in electrical communication with the GNSS, the vehicle communication system, and the display system. The vehicle controller is programmed to determine a position of the vehicle using the GNSS. The vehicle controller is also programmed to transmit the location of the vehicle to the remote server system using the vehicle communication system. The vehicle controller is also programmed to display graphics using the display system. The graphics are received from a remote server system. The vehicle controller is also programmed to transmit the engagement indicator of the graph to the remote server system using the vehicle communication system.

Description

Technical Field

[0001] The present disclosure relates to an augmented reality heads-up display for displaying information to vehicle occupants. Background Art

[0002] Augmented reality (AR) involves augmenting the real world with virtual elements that are displayed in three-dimensional space and allow real-time interaction with the user. A head-up display (HUD) displays information such as vehicle speed and navigation instructions directly on the vehicle's windshield in the occupant's forward field of view. The head-up display thus provides information to the occupant without the occupant having to take their eyes off the road. One possible implementation of augmented reality is an augmented reality head-up display (AR-HUD) for vehicles. By superimposing images on the windshield, the AR-HUD enhances the occupant's view of the environment outside the vehicle and strengthens environmental awareness. However, current AR-HUDs may not be able to provide occupants with additional information about points of interest (POIs) in the vehicle's surroundings.

[0003] Therefore, while current augmented reality heads-up displays achieve their intended purpose, there is a need in the art for an improved method of providing information to vehicle occupants. Summary of the invention

[0004] According to several aspects, a system for providing information to a vehicle occupant is provided. The system may include a global navigation satellite system (GNSS), a vehicle communication system, a display system configured to provide information to the vehicle occupant, and a vehicle controller in electrical communication with the GNSS, the vehicle communication system, and the display system. The vehicle controller is programmed to determine the location of the vehicle using the GNSS. The vehicle controller is also programmed to send the location of the vehicle to a remote server system using the vehicle communication system. The vehicle controller is also programmed to display a graphic using the display system. The graphic is received from the remote server system. The vehicle controller is also programmed to send an engagement indicator of the graphic to the remote server system using the vehicle communication system.

[0005] In another aspect of the present disclosure, the engagement metric includes at least an impression metric. To send the engagement metric, the vehicle controller is further programmed to determine the impression metric. The impression metric is a binary true or false value. To send the engagement metric, the vehicle controller is further programmed to send the impression metric to a remote server system using the vehicle communication system.

[0006] In another aspect of the present disclosure, the system also includes an occupant position tracking device. In order to determine the impression indicator, the vehicle controller is also programmed to determine the occupant's gaze direction using the occupant position tracking device. The gaze direction includes one of the following: a graphic gaze direction and a non-graphic gaze direction. In order to determine the impression indicator, the vehicle controller is also programmed to determine the occupant's gaze time in response to determining that the gaze direction is a graphic gaze direction. In order to determine the impression indicator, the vehicle controller is also programmed to compare the occupant's gaze time with a predetermined gaze time threshold. In order to determine the impression indicator, the vehicle controller is also programmed to determine that the impression indicator is true in response to determining that the occupant's gaze time is greater than or equal to a predetermined gaze time threshold. In order to determine the impression indicator, the vehicle controller is also programmed to determine that the impression indicator is false in response to determining that the occupant's gaze time is less than a predetermined gaze time threshold.

[0007] In another aspect of the present disclosure, a remote server system includes a server communication system and a server database in electrical communication with a server controller. The server database includes a plurality of database records. Each of the plurality of database records includes a graphic display location, a point of interest (POI) location, and a POI graphic. The server controller is programmed to receive the location of the vehicle using the server communication system. The server controller is also programmed to select a graphic based at least in part on the location of the vehicle. The server controller is also programmed to send the graphic to the vehicle communication system using the server communication system. The server controller is also programmed to receive an impression indicator using the server communication system and store the impression indicator in the server database.

[0008] In another aspect of the present disclosure, to select the graphic, the server controller is further programmed to select a first database record in the plurality of database records. The location of the vehicle is within a graphic display location of the first database record in the plurality of database records. To select the graphic, the server controller is further programmed to select the graphic. The graphic is a POI graphic from the first database record in the plurality of database records.

[0009] In another aspect of the present disclosure, to receive the impression metric, the server controller is further programmed to receive the impression metric using the server communication system and store the impression metric in the server database. To receive the impression metric, the server controller is further programmed to determine a closure metric in response to determining that the impression metric is true. The closure metric is a binary true or false value. To receive the impression metric, the server controller is further programmed to store the closure metric in the server database.

[0010] In another aspect of the present disclosure, to determine the end indicator, the server controller is further programmed to compare the vehicle's location to a POI location of a first database record in the plurality of database records. To determine the end indicator, the server controller is further programmed to determine that the end indicator is true in response to determining that the vehicle's location is within the POI location of the first database record in the plurality of database records.

[0011] In another aspect of the present disclosure, the display system includes an augmented reality head-up display (AR-HUD) system in electrical communication with a vehicle controller. The AR-HUD system includes an occupant position tracking device and an AR-HUD projector. To display the graphics, the vehicle controller is also programmed to use the occupant position tracking device to determine the position of the occupant. To display the graphics, the vehicle controller is also programmed to calculate the size, shape, and position of the graphics based on the position of the occupant. To display the graphics, the vehicle controller is also programmed to display the graphics on the windshield of the vehicle using the augmented reality head-up display system based on the size, shape, and position.

[0012] In another aspect of the present disclosure, the display system also includes a transparent windshield display (TWD) system in electrical communication with the vehicle controller. The transparent windshield display system includes a transparent phosphor embedded in the windshield of the vehicle and a TWD projector. To display graphics, the vehicle controller is also programmed to display graphics on the windshield of the vehicle using the TWD system based on size, shape, and location.

[0013] In another aspect of the present disclosure, the display system further includes a human machine interface (HMI) in electrical communication with the vehicle controller. To display the graphics, the vehicle controller is further programmed to display the graphics using the HMI.

[0014] According to several aspects, a method for providing information to a vehicle occupant is provided. The method may include determining a location of the vehicle using a global navigation satellite system (GNSS). The method may also include determining a graphic based at least in part on the location of the vehicle. The method may also include displaying the graphic using a display system. The method may also include determining an engagement indicator of the occupant with the graphic.

[0015] In another aspect of the present disclosure, determining a graphic may also include selecting a first database record in a plurality of database records. Each of the plurality of database records includes a graphic display location, a point of interest (POI) location, and a POI graphic. The location of the vehicle is within the graphic display location of a first database record in the plurality of database records. Determining a graphic may also include determining a graphic, wherein the graphic is a POI graphic from the first database record in the plurality of database records.

[0016] In another aspect of the present disclosure, the engagement indicator includes at least an impression indicator. The impression indicator is a binary true or false value. Determining the engagement indicator may also include determining the gaze direction of the occupant using an occupant position tracking device. The gaze direction includes one of the following: a graphic gaze direction and a non-graphic gaze direction. Determining the engagement indicator may also include determining the gaze time of the occupant in response to determining that the gaze direction is a graphic gaze direction. Determining the engagement indicator may also include comparing the gaze time of the occupant with a predetermined gaze time threshold. Determining the engagement indicator may also include determining that the impression indicator is true in response to determining that the gaze time of the occupant is greater than or equal to the predetermined gaze time threshold. Determining the engagement indicator may also include determining that the impression indicator is false in response to determining that the gaze time of the occupant is less than the predetermined gaze time threshold.

[0017] In another aspect of the present disclosure, determining the engagement indicator may further include determining an end indicator in response to determining that the impression indicator is true. The end indicator is a binary true or false value.

[0018] In another aspect of the present disclosure, determining the end indicator may also include comparing the vehicle's position to a POI position of a first database record in a plurality of database records. Determining the end indicator may also include determining that the end indicator is true in response to determining that the vehicle's position is within the POI position of the first database record in the plurality of database records.

[0019] In another aspect of the present disclosure, a display system includes an augmented reality head-up display (AR-HUD) system. The AR-HUD system includes an occupant position tracking device and an AR-HUD projector. Displaying graphics may also include determining the position of the occupant using the occupant position tracking device. Displaying graphics may also include calculating the size, shape, and position of the graphics based on the position of the occupant. Displaying graphics may also include displaying the graphics on the windshield of the vehicle using the augmented reality head-up display system based on the size, shape, and position.

[0020] In another aspect of the present disclosure, the display system also includes a transparent windshield display (TWD) system. The transparent windshield display system includes a transparent phosphor embedded in the windshield of the vehicle and a TWD projector. Displaying graphics can also include displaying graphics on the windshield of the vehicle using the TWD system based on size, shape, and position.

[0021] According to several aspects, a system for providing information to vehicle occupants is provided. The system may include a global navigation satellite system (GNSS), a vehicle communication system, and a display system configured to provide information to vehicle occupants. The display system includes an augmented reality head-up display (AR-HUD) system. The AR-HUD system includes an occupant position tracking device and an AR-HUD projector. The system may also include a vehicle controller that communicates electrically with the GNSS, the vehicle communication system, and the display system. The vehicle controller is programmed to determine the position of the vehicle using the GNSS. The vehicle controller is also programmed to send the position of the vehicle to a remote server system using the vehicle communication system. The vehicle controller is also programmed to receive graphics from the remote server system. The vehicle controller is also programmed to determine the position of the occupant using the occupant position tracking device. The vehicle controller is also programmed to calculate the size, shape, and position of the graphics based on the position of the occupant. The vehicle controller is also programmed to display graphics on the windshield of the vehicle using the augmented reality head-up display system based on the size, shape, and position. The vehicle controller is also programmed to send the impression indicator of the graphics to the remote server system using the vehicle communication system.

[0022] In another aspect of the present disclosure, in order to send the impression indicator, the vehicle controller is also programmed to determine the gaze direction of the occupant using the occupant position tracking device. The gaze direction includes one of the following: a graphic gaze direction and a non-graphic gaze direction. In order to send the impression indicator, the vehicle controller is also programmed to determine the gaze time of the occupant in response to determining that the gaze direction is a graphic gaze direction. In order to send the impression indicator, the vehicle controller is also programmed to compare the gaze time of the occupant with a predetermined gaze time threshold. In order to send the impression indicator, the vehicle controller is also programmed to determine that the impression indicator is true in response to determining that the gaze time of the occupant is greater than or equal to the predetermined gaze time threshold. In order to send the impression indicator, the vehicle controller is also programmed to determine that the impression indicator is false in response to determining that the gaze time of the occupant is less than the predetermined gaze time threshold. In order to send the impression indicator, the vehicle controller is also programmed to send the impression indicator to a remote server system using the vehicle communication system.

[0023] In another aspect of the present disclosure, a remote server system includes a server communication system and a server database in electrical communication with a server controller. The server database includes a plurality of database records. Each of the plurality of database records includes a graphical display location, a point of interest (POI) location, and a POI graphic. The server controller is programmed to receive the location of the vehicle using the server communication system. The server controller is also programmed to select a first database record in the plurality of database records. The location of the vehicle is within the graphical display location of the first database record in the plurality of database records. The server controller is also programmed to select a graphic. The graphic is a POI graphic from the first database record in the plurality of database records. The server controller is also programmed to send the graphic to the vehicle communication system using the server communication system. The server controller is also programmed to receive an impression indicator using the server communication system and store the impression indicator in the server database. The server controller is also programmed to compare the location of the vehicle with the POI location of the first database record in the plurality of database records in response to determining that the impression indicator is true. The server controller is also programmed to determine that the end indicator is true in response to determining that the location of the vehicle is within the POI location of the first database record in the plurality of database records.

[0024] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0026] Figure 1 is a schematic diagram of a system for providing information to a vehicle occupant according to an exemplary embodiment;

[0027] Figure 2 is a schematic diagram of an AR-HUD system for use by an exemplary occupant according to an exemplary embodiment;

[0028] Figure 3 is a schematic front view of a dual focal plane augmented reality display according to an exemplary embodiment, highlighting a second image plane of the dual focal plane augmented reality display;

[0029] Figure 4 is a schematic diagram of a second image plane of a dual-focal-plane augmented reality display according to an exemplary embodiment;

[0030] Figure 5 is a flow chart of a method for providing information to a vehicle occupant according to an exemplary embodiment;

[0031] Figure 6 is a first exemplary view through a windshield of a vehicle according to an exemplary embodiment; and

[0032] Figure 7 is a second exemplary view through a windshield of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0034] While driving, a vehicle occupant may wish to receive information about points of interest (POIs) in the vehicle's surroundings. However, taking one's eyes off the road to receive the information may reduce the occupant's awareness. Accordingly, the present disclosure provides a new and improved system and method for providing information about POIs to a vehicle occupant.

[0035] refer to Figure 1 , a system for providing information to vehicle occupants is shown, generally indicated by reference numeral 10. The system 10 is shown with an exemplary vehicle 12. Although a passenger vehicle is shown, it should be understood that the vehicle 12 may be any type of vehicle without departing from the scope of the present disclosure. The system 10 generally includes a vehicle controller 14, a plurality of vehicle sensors 16, an augmented reality head-up display (AR-HUD) system 18, a transparent windshield display (TWD) system 20, and a human-machine interface (HMI) 22.

[0036] The vehicle controller 14 is used to implement the method 100 for providing information to vehicle occupants, as will be described below. The vehicle controller 14 includes at least one processor 30 and a non-transitory computer-readable storage device or medium 32. The processor 30 can be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor in a plurality of processors associated with the vehicle controller 14, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions. The computer-readable storage device or medium 32 can include volatile and non-volatile storage in, for example, read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the processor 30 is powered off. The computer readable storage device or medium 32 may be implemented using a plurality of storage devices, such as a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), an EEPROM (Electrically Erasable PROM), flash memory, or other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which represent executable instructions, used by the vehicle controller 14 to control various systems of the vehicle 12. The vehicle controller 14 may also be comprised of multiple controllers that are in electrical communication with each other. The vehicle controller 14 may be interconnected with additional systems and / or controllers of the vehicle 12, thereby allowing the vehicle controller 14 to access data, such as the speed, acceleration, braking, and steering angle of the vehicle 12.

[0037] The vehicle controller 14 is in electrical communication with a plurality of vehicle sensors 16, an augmented reality head-up display (AR-HUD) system 18, a transparent windshield display (TWD) system 20, and a human-machine interface (HMI) 22. In an exemplary embodiment, the electrical communication is established using, for example, a CAN network, a FLEXRAY network, a local area network (e.g., WiFi, Ethernet, etc.), a serial peripheral interface (SPI) network, etc. It should be understood that various additional wired and wireless technologies and communication protocols for communicating with the vehicle controller 14 are within the scope of the present disclosure.

[0038] The plurality of vehicle sensors 16 are used to obtain information related to the vehicle 12. In an exemplary embodiment, the plurality of vehicle sensors 16 include at least a camera system 34, a vehicle communication system 36, and a global navigation satellite system (GNSS) 38.

[0039] In another exemplary embodiment, the plurality of vehicle sensors 16 further include sensors for determining performance data regarding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 further include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a brake position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor.

[0040] In another exemplary embodiment, the plurality of vehicle sensors 16 also include sensors for determining information about the environment within the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 also include at least one of a seat occupancy sensor, a cabin air temperature sensor, a cabin motion detection sensor, a cabin camera, a cabin microphone, etc.

[0041] In another exemplary embodiment, the plurality of vehicle sensors 16 also include sensors for determining information about the environment 40 surrounding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 also include at least one of an ambient air temperature sensor, an atmospheric pressure sensor, and / or a camera and / or video camera positioned to view the environment 40 in front of the vehicle 12.

[0042] In another exemplary embodiment, at least one of the plurality of vehicle sensors 16 is a perception sensor capable of perceiving objects in the environment 40 surrounding the vehicle 12 and / or measuring distances in the environment 40 surrounding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 include a stereo camera with distance measurement capabilities. In one example, at least one of the plurality of vehicle sensors 16 is fixed to the interior of the vehicle 12, for example, fixed to the roof lining of the vehicle 12, with a view through the windshield 42 of the vehicle 12. In another example, at least one of the plurality of vehicle sensors 16 is fixed to the exterior of the vehicle 12, for example, fixed to the roof of the vehicle 12, with a view of the environment 40 surrounding the vehicle 12. It should be understood that various additional types of perception sensors, such as LiDAR sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors are within the scope of the present disclosure. The plurality of vehicle sensors 16 are in electrical communication with the vehicle controller 14, as described above.

[0043] The camera system 34 is a perception sensor for capturing images and / or video of the environment 40 surrounding the vehicle 12. In an exemplary embodiment, the camera system 34 includes a camera and / or video camera positioned to view the environment 40 surrounding the vehicle 12. In a non-limiting example, the camera system 34 includes a camera fixed to the interior of the vehicle 12 (e.g., in the headliner of the vehicle 12) that has a field of view through the windshield 42. In another non-limiting example, the camera system 34 includes a camera fixed to the exterior of the vehicle 12 (e.g., on the roof of the vehicle 12) that has a field of view of the environment 40 in front of the vehicle 12.

[0044] In another exemplary embodiment, the camera system 34 is a surround view camera system that includes multiple cameras (also referred to as satellite cameras) that are arranged to provide views of the environment 40 adjacent to all sides of the vehicle 12. In a non-limiting example, the camera system 34 includes a front camera (e.g., mounted in the front grille of the vehicle 12), a rear camera (e.g., mounted on the tailgate of the vehicle 12), and two side cameras (e.g., mounted below each of the two side mirrors of the vehicle 12). In another non-limiting example, the camera system 34 also includes an additional rear view camera mounted near the center high mounted brake light of the vehicle 12.

[0045] It should be understood that camera systems with additional cameras and / or additional mounting locations are within the scope of the present disclosure. It should also be understood that cameras with various sensor types, including, for example, charge coupled device (CCD) sensors, complementary metal oxide semiconductor (CMOS) sensors, and / or high dynamic range (HDR) sensors, are within the scope of the present disclosure. In addition, cameras with various lens types, including, for example, wide angle lenses and / or narrow angle lenses, are also within the scope of the present disclosure.

[0046] GNSS 38 is used to determine the geographic location of vehicle 12. In an exemplary embodiment, GNSS 38 is a global positioning system (GPS). In a non-limiting example, GPS includes a GPS receiver antenna (not shown) and a GPS controller (not shown) in electrical communication with the GPS receiver antenna. The GPS receiver antenna receives signals from a plurality of satellites, and the GPS controller calculates the geographic location of vehicle 12 based on the signals received by the GPS receiver antenna. In an exemplary embodiment, GNSS 38 also includes a map. The map includes information about infrastructure, such as municipal boundaries, roads, railways, sidewalks, buildings, etc. Therefore, the map information is used to put the geographic location of vehicle 12 into context. In a non-limiting example, the map is retrieved from a remote source using a wireless connection. In another non-limiting example, the map is stored in a database of GNSS 38. It should be understood that various additional types of satellite-based radio navigation systems, such as global positioning system (GPS), Galileo, GLONASS, and BeiDou navigation satellite system (BDS) are within the scope of the present invention. It should be appreciated that the GNSS 38 may be integrated with the vehicle controller 14 (eg, integrated on the same circuit board as the vehicle controller 14 or otherwise be a part of the vehicle controller 14 ) without departing from the scope of the present disclosure.

[0047] refer to Figure 2 , a system diagram of an AR-HUD system 18 for use with an exemplary occupant 44 is shown. Within the scope of the present disclosure, in a non-limiting example, the occupant 44 includes a driver, a passenger, and / or any other person in the vehicle 12. The AR-HUD system 18 is used to display AR-HUD graphics 46 (i.e., notification symbols that provide visual information to the occupant 44) on the windshield 42 of the vehicle 12. The AR-HUD system 18 includes an AR-HUD projector 48 and an occupant position tracking device 50. The AR-HUD system 18 is in electrical communication with the vehicle controller 14, as described above.

[0048] The AR-HUD projector 48 is used to project the AR-HUD graphics 46 onto the windshield 42 of the vehicle 12. It should be understood that various devices designed for projecting images, including, for example, optical collimators, laser projectors, digital light projectors (DLP), etc., are within the scope of the present disclosure.

[0049] The occupant position tracking device 50 is used to determine the position of the occupant 44 in the vehicle 12. For example, the occupant position tracking device 50 can track the position of the head 52a or the eyes 52b of the occupant 44. The position of the occupant 44 in the vehicle 12 obtained from the occupant position tracking device 50 is used to position the AR-HUD graphics 46 on the windshield 42 of the vehicle 12. In an exemplary embodiment, the occupant position tracking device 50 is one or more cameras disposed in the vehicle 12.

[0050] To operate the AR-HUD system 18, the vehicle controller 14 includes a plurality of software modules, including a system manager 54. During operation of the AR-HUD system 18, the system manager 54 receives at least a first input 56, a second input 58, and a third input 60. The first input 56 indicates the position of the vehicle 12 in space (i.e., the geographic location of the vehicle 12), the second input 58 indicates the position of the occupant 44 in the vehicle 12 (e.g., the position of the head 52a and / or eyes 52b of the occupant 44 in the vehicle 12), and the third input 60 is data related to a point of interest (POI), which will be discussed in more detail below. The first input 56 may include data such as GNSS data (e.g., GPS data), vehicle speed, road curvature, and vehicle steering, and the data is collected from a plurality of vehicle sensors 16. The second input 58 is received from the occupant position tracking device 50. The third input 60 is data related to the POI. The system manager 54 is configured to determine (e.g., calculate) the type, size, shape, and color of the AR-HUD graphics 46 displayed using the AR-HUD projector 48 based on the first input 56 (i.e., the position of the vehicle in the environment 40), the second input 58 (e.g., the position of the eyes 52b and / or the head 52a of the occupant 44 in the vehicle 12), and the third input 60 (i.e., data related to the POI). The system manager 54 instructs the image engine 62 to display the AR-HUD graphics 46 using the AR-HUD projector 48, wherein the image engine 62 is a software module or integrated circuit of the AR-HUD projector 48 or the vehicle controller 14. The image engine 62 displays the AR-HUD graphics 46 on the windshield 42 of the vehicle 12 using the AR-HUD projector 48 based on the type, size, shape, and color of the AR-HUD graphics 46 determined by the system manager 54. The AR-HUD graphics 46 are projected onto the windshield 42 by the AR-HUD projector 48 to display the AR-HUD graphics 46 along the road surface 64 .

[0051] In an exemplary embodiment of the present disclosure, the AR-HUD system 18 is a dual focal plane AR-HUD system. Figure 3 and Figure 4 , and continue to refer to Figure 2 , the AR-HUD system 18 has a first image plane 66 and a second image plane 68. The first image plane 66 shows a view of the outside world, and the second image plane 68 is reserved for displaying the AR-HUD graphics 46. The second image plane 68 spans multiple lanes, and the AR-HUD graphics 46 appear at a farther position on the road surface 64 relative to the first image plane 66. For example, Figure 3 and Figure 4As shown, the second image plane 68 covers the left lane 70, the center lane 72, and the right lane 74. As a non-limiting example, in the center lane 72, the second image plane 68 starts at a first predetermined distance D1 (e.g., twenty-five meters) from the vehicle 12 and ends at a second predetermined distance D2 (e.g., ninety meters) from the vehicle 12. Regardless of the specific distance, the second predetermined distance D2 is greater than the first predetermined distance D1 to help the occupant 44 see the AR-HUD graphics 46 displayed using the AR-HUD projector 48. In the left lane 70 and the right lane 74, the second image plane 68 is bounded by a sloped boundary that starts at the first predetermined distance D1 from the vehicle 12 and ends at a third predetermined distance D3 (e.g., fifty meters) from the vehicle 12. The third predetermined distance D3 is greater than the first predetermined distance D1 and less than the second predetermined distance D2 to help the occupant 44 see the AR-HUD graphics 46 displayed using the AR-HUD projector 48. As used herein, the term "dual focal plane AR-HUD" refers to an AR-HUD system that presents images in a first image plane and a second image plane, wherein the first image plane is located at a different position than the second image plane. It is desirable to configure the AR-HUD system 18 as a dual focal plane AR-HUD to facilitate manipulation of the AR-HUD graphics 46 over the view of the outside world. For example, by using a dual focal plane AR-HUD, the size, position, and characteristics of the AR-HUD graphics 46 may be changed based on, for example, the position of the eye 52b of the occupant 44.

[0052] The TWD system 20 is used to display images on the windshield 42 of the vehicle 12. In an exemplary embodiment, the AR-HUD system 18 can display AR-HUD graphics 46 in predefined areas of the windshield 42 (e.g., in the first image plane 66 and the second image plane 68). The TWD system 20 can display TWD graphics (not shown) in any area of ​​the windshield 42. Therefore, by operating the AR-HUD system 18 and the TWD system 20 in combination, the vehicle controller 14 can display graphics in any area of ​​the windshield 42. In an exemplary embodiment, the TWD system 20 includes a transparent phosphor (not shown) embedded in the windshield 42 and a TWD projector 76 ( Figure 1 ). The TWD system 20 is in electrical communication with the vehicle controller 14, as described above.

[0053] The transparent phosphor is a luminescent particle that fluoresces in response to being excited by the TWD projector 76. In an exemplary embodiment, the transparent phosphor is a red, green and blue (RGB) phosphor, allowing full color operation of the TWD system 20. The use of monochromatic and / or bichromatic phosphors is also within the scope of the present disclosure. When the excitation light is absorbed by the transparent phosphor, the transparent phosphor emits visible light. The excitation light can be, for example, violet light (ranging from about 380 nanometers to 450 nanometers) and / or ultraviolet light in the visible spectrum.

[0054] The TWD projector 76 is used to excite the transparent phosphors in a predetermined pattern to produce a TWD pattern on the windshield 42. In an exemplary embodiment, the TWD projector 76 is a violet / ultraviolet laser projector disposed near the ceiling of the vehicle 12. The TWD projector 76 includes three lasers, each of which is configured to excite one of the red, green, or blue transparent phosphors.

[0055] In an exemplary embodiment, in addition to the AR-HUD system 18 and the TWD system 20, the HMI 22 is used to display information to the occupants 44 of the vehicle 12. In another exemplary embodiment, the HMI 22 is used instead of the AR-HUD system 18 and / or the TWD system 20 to display information to the occupants 44 of the vehicle 12. In the aforementioned exemplary embodiments, the HMI 22 is a display that is located in the field of view of the occupants 44 and is capable of displaying text, graphics and / or images. It should be understood that HMI displays including LCD displays, LED displays, etc. are within the scope of the present disclosure. Other exemplary embodiments in which the HMI 22 is set in the rearview mirror are also within the scope of the present disclosure. As described above, the HMI 22 is in electrical communication with the vehicle controller 14.

[0056] Reference again Figure 1 , a remote server system is shown and generally indicated by reference numeral 80. The remote server system 80 includes a server controller 82 in electrical communication with a server database 84 and a server communication system 86. In a non-limiting example, the remote server system 80 is located in a server farm, data center, etc., and is connected to the Internet.

[0057] The server controller 82 includes at least one server processor 88a and a server non-transitory computer readable storage device or server medium 88b. The description of types and configurations given above for the vehicle controller 14 also applies to the server controller 82. In some examples, the server controller 82 differs from the vehicle controller 14 in that the server controller 82 can have a higher processing speed, include more memory, include more input / output, etc. In a non-limiting example, the server processor 88a and server medium 88b of the server controller 82 are similar in structure and / or function to the processor 30 and medium 32 of the vehicle controller 14, as described above.

[0058] The server database 84 is used to store information about points of interest (POIs). In an exemplary embodiment, the server database 84 includes one or more mass storage devices, such as hard disk drives, tape drives, magneto-optical drives, optical disks, solid-state drives, and / or other devices operable to store data in a persistent and machine-readable manner. In some examples, one or more mass storage devices may be configured to provide redundancy in the event of hardware failure and / or data corruption using, for example, a redundant array of independent disks (RAID). In a non-limiting example, the server controller 82 may execute software such as a database management system (DBMS) to allow organization and access to data stored on one or more mass storage devices.

[0059] In an exemplary embodiment, the server database 84 is populated with a plurality of database records. Each of the plurality of database records includes a graphical display location, a point of interest (POI) location, and a POI graphic. The graphical display location is a location in the environment 40. In a non-limiting example, the graphical display location is a point location. In another non-limiting example, the graphical display location is a geo-fenced two-dimensional area (e.g., a portion of a road, a parking lot, etc.). In another non-limiting example, the graphical display location is a geo-fenced three-dimensional area. The POI location is the location of a point of interest in the environment 40. Within the scope of the present disclosure, a point of interest (POI) is a specific point location in the environment 40 that the occupant 44 may find useful or interesting, including, for example, businesses, restaurants, vehicle service centers, hospitals, police stations, gas stations, vehicle charging stations, recreational areas, nature reserves, museums, theaters, etc. The POI graphic is a graphic to be displayed when the vehicle 12 is located within the graphical display location. In a non-limiting example, the graphic provides information related to the POI location, such as business hours, product / service availability, advertising promotions, etc. In one non-limiting example, the POI graphics are predetermined and loaded into the server database 84. In another non-limiting example, the server controller 82 uses the server communication system 86 (e.g., by connecting to the Internet) to periodically retrieve current information about the POIs (e.g., updated opening hours) and save the current information in a POI graphics field of each of the plurality of database records.

[0060] The server communication system 86 is used to communicate with external systems, such as the vehicle controller 14, through the vehicle communication system 36. In a non-limiting example, the server communication system 86 is similar in structure and / or function to the vehicle communication system 36, as described above. In some examples, the server communication system 86 differs from the vehicle communication system 36 in that the server communication system 86 is capable of higher power signal transmission, more sensitive signal reception, higher bandwidth transmission, additional transmission / reception protocols, etc.

[0061] refer to Figure 5 , a flow chart of a method 100 for providing information to a vehicle occupant is provided. The method 100 begins at box 102 and proceeds to box 104. At box 104, the vehicle controller 14 determines the location of the vehicle 12 using the GNSS 38. The vehicle controller 14 then sends the location of the vehicle 12 to the remote server system 80 using the vehicle communication system 36. After box 104, the method 100 proceeds to box 106. At box 106, the remote server system 80 receives the location of the vehicle 12 using the server communication system 86, and selects a graphic for display by the vehicle 12 based at least in part on the location of the vehicle 12.

[0062] In an exemplary embodiment, to select a graphic, the server controller 82 searches a plurality of database records stored in the server database 84 based on the location of the vehicle 12. The server controller 82 identifies a first database record in the plurality of database records, wherein the first database record in the plurality of database records has a graphic display location that includes the location of the vehicle 12 (i.e., the location of the vehicle 12 is within the graphic display location of the first database record in the plurality of database records). The graphic is selected as the POI graphic for the first database record in the plurality of database records. The server controller 82 then sends the POI graphic for the first database record in the plurality of database records to the vehicle controller 14 using the server communication system 86. After frame 106, the method 100 proceeds to frame 108.

[0063] At box 108, the vehicle controller 14 receives the graphic using the vehicle communication system 36 and displays the graphic using the display system. In an exemplary embodiment, the display system includes at least one of the following: the AR-HUD system 18, the TWD system 20, and the HMI 22. To display the graphic, the vehicle controller 14 first determines the position of the occupant 44 using the occupant position tracking device 50, as described above. After determining the position of the occupant 44, the vehicle controller 14 calculates the size, shape, and position of the graphic, as described above.

[0064] After calculating the size, shape, and position of the graphic, the vehicle controller 14 displays the graphic using at least one of the AR-HUD system 18 and the TWD system 20, as described above. In an exemplary embodiment, the vehicle controller 14 additionally uses the HMI 22 to provide an augmented view of the environment 40 surrounding the vehicle 12, including the graphic.

[0065] In an exemplary embodiment, graphics may be selectively displayed based on one or more vehicle states or vehicle conditions. Within the scope of the present disclosure, one or more vehicle states or vehicle conditions include, for example, the location of the vehicle 12, the environmental conditions surrounding the vehicle 12, the status of the occupants in the vehicle 12, etc. In a non-limiting example, graphics are selectively displayed based on the road the vehicle 12 is traveling on (e.g., graphics are only displayed on highways, major road categories, and / or straight sections of road). In another non-limiting example, graphics are selectively displayed based on the attention level of the occupant 44 (e.g., the graphics are only displayed when the occupant 44's gaze profile data indicates that the occupant 44 is not busy looking at vehicle traffic, is impaired, distracted, or fatigued).

[0066] In another non-limiting example, the graphics are selectively displayed based on traffic conditions around the vehicle 12 (e.g., the graphics are only displayed in free-flowing traffic conditions and not displayed when the vehicle 12 is in a construction zone). In another non-limiting example, the graphics are selectively displayed based on weather conditions around the vehicle 12 (e.g., the graphics are only displayed in dry ambient weather conditions). In another non-limiting example, the graphics are selectively displayed based on the autonomous driving state of the vehicle 12 (e.g., the graphics are only displayed when the vehicle 12 is operating in a Level 2 or higher autonomous driving state).

[0067] In another non-limiting example, the graphics are selectively displayed based on the vehicle maneuver state (e.g., graphics are not displayed during active vehicle maneuvers, when a turn signal is active, when a turn is in progress, when a lane change is in progress, etc.). In another non-limiting example, the graphics are selectively displayed based on the traffic control state of the vehicle 12 (e.g., graphics are only displayed when the vehicle 12 is in an active traffic control situation, such as when the vehicle 12 is stopped at a red light with a predetermined minimum remaining time in the traffic phase).

[0068] In another non-limiting example, the graphic is selectively displayed based on a diagnostic status of the vehicle 12 (e.g., not displaying the graphic if a vehicle malfunction has occurred, such as if low tire pressure is detected, a malfunction indicator light is activated, etc.). In another non-limiting example, the graphic is selectively displayed based on occupant type (e.g., not displaying the graphic to occupants under a certain age, not displaying the graphic to commercial / fleet occupants, etc.).

[0069] In a non-limiting example, one or more vehicle states or vehicle conditions are determined by the vehicle controller 14 using the plurality of vehicle sensors 16. It should be appreciated that the foregoing one or more vehicle states or vehicle conditions are merely exemplary in nature and that the graphic may be selectively displayed at block 108 based on additional factors without departing from the scope of the present disclosure. After block 108, the method 100 proceeds to block 110.

[0070] refer to Figure 6 , a first exemplary view 90a is shown through the windshield 42 of the vehicle 12. The first exemplary view 90a includes a first exemplary graphic 92. In a non-limiting example, the first exemplary graphic 92 includes text and / or an image that provides information about the location of the POI.

[0071] refer to Figure 7, a second exemplary view 90b through the windshield 42 of the vehicle 12 is shown. The second exemplary view 90b includes a second exemplary graphic 94. In a non-limiting example, the second exemplary graphic 94 is overlaid on an existing roadside sign 96. For example, the second exemplary graphic 94 can provide information about POI locations customized according to the preferences of the occupant 44, such as a display that reorders the names of businesses that appear on the roadside sign 96 based on the preferences of the occupant 44. In an exemplary embodiment, the plurality of vehicle sensors 16 are used to determine factors such as gender, age, etc., in order to infer the preferences of the occupant 44. In another exemplary embodiment, the plurality of vehicle sensors 16 are used to determine the purpose of the vehicle trip (e.g., leisure, business, shopping, etc.) based on the occupants of the vehicle.

[0072] It should be appreciated that the system 10 and method 100 are applicable to displaying additional information, such as the opening hours of the POI, special events at the POI, information about yard sales, etc. It should also be appreciated that the plurality of vehicle sensors 16 may be used to sense information about the environment 40 and provide additional context. For example, the plurality of vehicle sensors 16 may be used to identify objects, such as vehicles, in the environment 40, and the system 10 and method 100 may be used to provide information about the identified objects to the occupants 44. Furthermore, it should be appreciated that the system 10 and method 100 are also applicable to displaying graphics on other windows of the vehicle 12, such as side windows, rear windows, rear windshield, etc.

[0073] Reference again Figure 5 At box 110, the vehicle controller 14 determines the gaze direction of the occupant 44. The gaze direction includes one of the following: a graphic gaze direction and a non-graphic gaze direction. Within the scope of the present disclosure, the graphic gaze direction means that the occupant 44 is determined to be looking at the graphics displayed at box 108. The non-graphic gaze direction means that the occupant 44 is determined to be not looking at the graphics displayed at box 108 (i.e., the occupant 44 is determined to be looking at anything other than the graphics displayed at box 108). In an exemplary embodiment, the gaze direction is determined using the occupant position tracking device 50. If the gaze direction is determined to be a non-graphic gaze direction, the method 100 continues to enter the standby state at box 112. If the gaze direction is determined to be a graphic gaze direction, the method 100 proceeds to box 114.

[0074] At block 114, the vehicle controller 14 determines the gaze time of the occupant 44 in response to determining that the gaze direction determined at block 110 is the graphic gaze direction. Within the scope of the present disclosure, the gaze time is the length of time that the occupant 44 has gazed at the graphic displayed at block 108. If the gaze time is less than a predetermined gaze time threshold (e.g., two seconds), the method 100 returns to block 110. If the gaze time is greater than or equal to the predetermined gaze time threshold, the method 100 proceeds to block 116.

[0075] At box 116, the vehicle controller 14 determines an impression indicator. Within the scope of the present disclosure, the impression indicator is a binary true or false value that quantifies whether the graphics displayed at box 108 left an impression on the occupant 44 (i.e., whether the occupant 44 saw, read, and understood the graphics). The impression indicator is an engagement indicator. Within the scope of the present disclosure, the engagement indicator quantifies the engagement of the occupant 44 with the graphics displayed at box 108. The engagement indicator may include other components in addition to the impression indicator, as will be discussed in more detail below. In an exemplary embodiment, at box 116, in response to determining that the gaze time is greater than or equal to a predetermined gaze time threshold, the impression indicator is determined to be true. In addition, the impression indicator is sent to the remote server system 80 using the vehicle communication system 36. After box 116, the method 100 proceeds to box 118.

[0076] At box 118, the vehicle controller 14 determines whether a predetermined time (e.g., two days) has passed since the graphic was displayed at box 108. If the predetermined time has passed since the graphic was displayed, the method 100 continues to the standby state at box 112. If the predetermined time has not passed since the graphic was displayed, the method 100 proceeds to box 120.

[0077] At box 120 , the location of the vehicle 12 is determined using the GNSS 38 and sent to the remote server system 80 using the vehicle communication system 36 . After box 120 , the method 100 proceeds to box 122 .

[0078] At box 122, the server communication system 86 receives the location of the vehicle 12 sent at box 120 and compares the location of the vehicle 12 to the POI location of the first of the plurality of database records in the server database 84 identified at box 106. If the location of the vehicle 12 is not within the POI location of the first of the plurality of database records, the method 100 returns to box 118. If the location of the vehicle 12 is within the POI location of the first of the plurality of database records, the method 100 proceeds to box 124.

[0079] At box 124, the server controller 82 determines an end indicator. Within the scope of the present disclosure, the end indicator is a binary true or false value that quantifies whether the graphic displayed at box 108 affects the behavior of the occupant 44 (i.e., whether the occupant 44 changes their behavior based on the graphic). The end indicator is a type of engagement indicator. Therefore, within the scope of the present disclosure, the engagement indicator may include one or both of the impression indicator and / or the end indicator. In an exemplary embodiment, at box 124, in response to determining that the vehicle 12 visited the POI location within a predetermined time, the end indicator is determined to be true. After box 124, the method 100 continues to enter the standby state at box 112.

[0080] In an exemplary embodiment, the engagement metrics (i.e., impression metrics and end metrics) are stored in the server database 84 and linked to the first of the plurality of database records. In a non-limiting example, the engagement metrics are later retrieved for use in analyzing the effectiveness of the graphic and / or for transmission to interested parties (e.g., operators of the POI).

[0081] In an exemplary embodiment, the vehicle controller 14 repeatedly exits the standby state 112 and restarts the method 100 at any stage of the method 100, including, for example, at blocks 102, 110, or 118. In a non-limiting example, the vehicle controller 14 exits the standby state 112 and restarts the method 100 on a timer, such as once every three hundred milliseconds.

[0082] The system 10 and method 100 of the present disclosure provide several advantages. For example, the system 10 and method 100 allow for increased awareness of POIs in the environment 40 by the occupants, thereby increasing the convenience of the occupants. By providing current information about the POI, the system 10 and method 100 inform the occupants of relevant information. By tracking engagement metrics (i.e., impression metrics and end metrics), interested parties (e.g., operators of POIs) can quantify and evaluate the effectiveness of graphical notifications to occupants. The remote server system 80 can send engagement metrics to interested parties. In addition, engagement metrics can be improved by customizing graphics according to the preferences of the occupants. Using a display system including an AR-HUD 18 and a TWD system 20 allows occupants to receive information from the system 10 and method 100 while maintaining focus on the road ahead.

[0083] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A system for providing information to a vehicle occupant, the system comprising: Global Navigation Satellite System (GNSS); Vehicle communication systems; a display system configured to provide information to an occupant of the vehicle; and a vehicle controller in electrical communication with the GNSS, the vehicle communication system, and the display system, wherein the vehicle controller is programmed to: determining a position of the vehicle using the GNSS; transmitting the position of the vehicle to a remote server system using the vehicle communication system; displaying a graphic using the display system, wherein the graphic is received from the remote server system; and An engagement indicator for the graphic is transmitted to the remote server system using the vehicle communication system.

2. The system according to claim 1, wherein: The engagement metric includes at least an impression metric, and wherein to transmit the engagement metric, the vehicle controller is further programmed to: determining the impression indicator, wherein the impression indicator is a binary true or false value; and The impression metrics are sent to the remote server system using the vehicle communication system.

3. The system according to claim 2, wherein: The system further includes an occupant position tracking device, and wherein to determine the impression indicator, the vehicle controller is further programmed to: determining a gaze direction of the occupant using the occupant position tracking device, wherein the gaze direction comprises one of: a graphic gaze direction and a non-graphic gaze direction; In response to determining that the gaze direction is the graphic gaze direction, determining a gaze time of the occupant; comparing the gaze time of the occupant to a predetermined gaze time threshold; determining that the impression indicator is true in response to determining that the gaze time of the occupant is greater than or equal to the predetermined gaze time threshold; as well as In response to determining that the gaze time of the occupant is less than the predetermined gaze time threshold, the impression indicator is determined to be false.

4. The system according to claim 3, wherein: The remote server system includes a server communication system and a server database in electronic communication with a server controller, wherein the server database includes a plurality of database records, wherein each of the plurality of database records includes a graphical display location, a point of interest (POI) location, and a POI graphic, and wherein the server controller is programmed to: receiving, using the server communication system, a location of the vehicle; selecting the graphic based at least in part on a location of the vehicle; transmitting the graphic to the vehicle communication system using the server communication system; as well as The impression metrics are received using the server communication system and stored in the server database.

5. The system according to claim 4, wherein: To select the graphic, the server controller is also programmed to: selecting a first one of the plurality of database records, wherein the position of the vehicle is within the graphically displayed position of the first one of the plurality of database records; as well as The graphic is selected, wherein the graphic is the POI graphic from the first database record of the plurality of database records.

6. The system according to claim 5, wherein: In order to receive the impression metrics, the server controller is further programmed to: receiving the impression metrics using the server communication system and storing the impression metrics in the server database; In response to determining that the impression indicator is true, determining an ending indicator, wherein the ending indicator is a binary true or false value; and The end indicator is stored in the server database.

7. The system according to claim 6, wherein: To determine the end indicator, the server controller is further programmed to: comparing the location of the vehicle to the POI location of the first database record of the plurality of database records; as well as The end indicator is determined to be true in response to determining that the location of the vehicle is within the POI location of the first database record of the plurality of database records.

8. The system according to claim 1, wherein: The display system includes an augmented reality head-up display (AR-HUD) system in electrical communication with the vehicle controller, wherein the AR-HUD system includes an occupant position tracking device and an AR-HUD projector, and wherein to display the graphics, the vehicle controller is further programmed to: determining the position of the occupant using the occupant position tracking device; calculating a size, shape, and position of the graphic based on the position of the occupant; as well as Based on the size, shape, and position, the graphic is displayed on a windshield of the vehicle using the augmented reality heads-up display system.

9. The system according to claim 8, wherein: The display system further includes a transparent windshield display (TWD) system in electrical communication with the vehicle controller, wherein the transparent windshield display system includes a transparent phosphor embedded in the windshield of the vehicle and a TWD projector, and wherein to display the graphics, the vehicle controller is further programmed to: Based on the size, shape, and location, the graphic is displayed on the windshield of the vehicle using the TWD system.

10. The system according to claim 9, wherein: The display system further includes a human machine interface (HMI) in electrical communication with the vehicle controller, and wherein to display the graphic, the vehicle controller is further programmed to: The graphic is displayed using the HMI.

Images