Hybrid Augmented Reality Head-Up Display for Creating Edge-to-Edge Augmented Reality Views
By using luminescent particles and multiple projection devices working together within a transparent windshield, the limited field of view of AR-HUD is solved, achieving augmented reality display with a wide field of view and small package size, thus enhancing the driver's environmental awareness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GM GLOBAL TECHNOLOGY OPERATIONS LLC
- Filing Date
- 2022-10-24
- Publication Date
- 2026-08-04
AI Technical Summary
Existing augmented reality head-up displays (AR-HUDs) have limited field of view, making it difficult to effectively highlight features in the driver's peripheral vision. Furthermore, increasing the field of view is challenging, and the size of optical components is limited by eye distance, resulting in large package sizes that are difficult to accommodate.
The system employs a hybrid augmented reality head-up display system, utilizing luminescent particles within a transparent windshield and two graphic projection devices. The first device generates an image based on visible light, while the second device generates a fluorescent image based on excitation light, collaboratively displaying an edge-to-edge augmented reality view of the vehicle's surroundings.
It achieves a wide field of view display of the vehicle's surrounding environment, eliminates the limitations of the peephole, enhances the driver's environmental awareness, and has a small system package size, making it suitable for use as a primary instrument.
Smart Images
Figure CN117124850B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a hybrid augmented reality head-up display including a first graphics projection device and a second graphics projection device working together to create an edge-to-edge augmented reality view of the environment surrounding the vehicle. Background Technology
[0002] Augmented reality (AR) involves using virtual elements displayed in 3D space and allowing real-time interaction with the user to enhance the real world. A head-up display (HUD) displays information such as vehicle speed and navigation instructions directly on the windshield within the driver's field of vision. Therefore, the HUD provides information to the driver without requiring them to take their eyes off the road. One possible implementation of augmented reality is an augmented reality head-up display (AR-HUD) for vehicles. By overlaying images onto the windshield, the AR-HUD enhances the driver's view of the external environment, thus creating a stronger sense of environmental awareness.
[0003] One challenge AR-HUDs may face is the limited field of view, sometimes referred to as the keyhole problem. Specifically, an AR-HUD may only be able to highlight a relatively small area of the vehicle's windshield, known as the keyhole. Generally, the horizontal field of view of an AR-HUD is approximately 10 to 15 degrees. Therefore, an AR-HUD may not be able to highlight features within the driver's peripheral field of vision. For example, if an animal is running in front of the vehicle, the AR-HUD may not be able to highlight the animal and draw the driver's attention until the animal is directly visible in front of the vehicle. Furthermore, an AR-HUD cannot highlight objects outside the keyhole within the driver's peripheral field of vision. Additionally, because HUD peepholes tend to be relatively small, they may not be used as the primary instrument. A peephole is a volume inside the vehicle where the driver's eyes receive an acceptable view relative to a standard set of images.
[0004] Because a relatively large eye distance is typically required in many vehicles, simply increasing the field of view of an AR-HUD can be challenging. Eye distance refers to the distance measured between the driver's eyes and the windshield. Furthermore, the size of the various optical components that make up an AR-HUD projector depends on this eye distance; reducing the eye distance results in larger optical components. Therefore, reducing this eye distance could lead to a very large package size for the AR-HUD, which, even if accommodated, could be extremely difficult. Finally, there is a growing need for a wider field of view across the entire windshield.
[0005] Therefore, while current AR-HUDs have achieved their intended purpose, there is still a need in the field for an improved AR-HUD with a larger field of view. Summary of the Invention
[0006] According to several aspects, a hybrid augmented reality head-up display system for displaying graphics on a vehicle's windshield is disclosed, wherein the windshield includes a transparent substrate comprising luminescent particles dispersed therein. The hybrid augmented reality head-up display system includes a first graphics projection device for generating a first set of images on the vehicle's windshield based on visible light, wherein the first set of images is displayed on a first area of the windshield. The hybrid augmented reality head-up display system also includes a second graphics projection device for generating a second set of images on a second area of the vehicle's windshield based on excitation light, wherein, in response to absorbing the excitation light, the luminescent particles in the windshield emit visible light; and wherein the first set of images displayed on the first area of the windshield and the second set of images displayed on the second area of the windshield cooperate to create an edge-to-edge augmented reality view of the vehicle's surroundings.
[0007] On one hand, the mixed augmented reality head-up display system also includes a first graphics processing unit that communicates electronically with a first graphics projection device and a second graphics processing unit that communicates electronically with a second graphics projection device.
[0008] On the other hand, the mixed augmented reality head-up display system also includes one or more controllers that communicate electronically with the first graphics processing unit and the second graphics processing unit.
[0009] In another aspect, one or more controllers execute instructions to determine that the object of interest is located within a second area of the windshield, and in response to determining that the object of interest is located within the second area of the windshield, instruct a second graphics processing unit to determine a graphic representation of a second set of images generated by a second graphics projection device.
[0010] On one hand, the second set of images includes one or more original graphic objects.
[0011] On the other hand, one or more original graphic objects include one or more of the following: points and lines.
[0012] In another aspect, one or more controllers execute instructions to instruct a second graphics processing unit to follow the object of interest with a graphical representation of a second set of images as the object of interest moves from a second region of the windshield to a first region of the windshield.
[0013] On one hand, one or more controllers execute instructions to determine that an object of interest has entered a first area of the windshield, and in response to determining that an object of interest has entered a first area of the windshield, instruct a first graphics processing unit to determine a graphic representation of a first set of images generated by a first graphics projection device.
[0014] On the other hand, the first set of images are augmented reality graphics that overlap and align with objects in the vehicle's surrounding environment.
[0015] In another aspect, the first graphics projection device is a single-image planar augmented reality head-up display that includes a digital light projector (DLP) optical system.
[0016] On one hand, the first graphic projection device is a fixed dual-image planar holographic augmented reality head-up display.
[0017] On the other hand, the first graphics projection device generates a dual image plane including a near-field image plane and a far-field image plane.
[0018] On the other hand, the first set of images includes cluster content information projected onto the near-field image plane and augmented reality graphics projected onto the far-field image plane.
[0019] On one hand, the excitation light is one or more of the following: violet light in the visible spectrum or ultraviolet light that induces fluorescence in luminescent particles.
[0020] In one aspect, the hybrid augmented reality head-up display system for a vehicle includes a windshield comprising a transparent substrate in which luminescent particles are dispersed. The hybrid augmented reality head-up display system also includes a first graphics projection device for generating a first set of images on the windshield of the vehicle based on visible light, wherein the first set of images is displayed on a first area of the windshield. The hybrid augmented reality head-up display system also includes a second graphics projection device for generating a second set of images on a second area of the windshield of the vehicle based on excitation light, wherein luminescent particles in the windshield emit visible light in response to absorption of the excitation light; and wherein the first set of images displayed on the first area of the windshield and the second set of images displayed on the second area of the windshield cooperate to create an edge-to-edge augmented reality view of the vehicle's surroundings. The hybrid augmented reality head-up display system includes a first graphics processing unit electronically communicating with the first graphics projection device, a second graphics processing unit electronically communicating with the second graphics projection device, and one or more controllers electronically communicating with both the first and second graphics projection units.
[0021] On one hand, the luminescent particles are red, green, and blue (RGB) phosphors.
[0022] On the other hand, one or more controllers execute instructions to determine that the object of interest is located within a second area of the windshield, and in response to determining that the object of interest is located within the second area of the windshield, instruct a second graphics processing unit to determine a graphic representation of a second set of images generated by a second graphics projection device.
[0023] In another aspect, one or more controllers execute instructions to instruct a second graphics processing unit to follow the object of interest with a graphical representation of a second set of images as the object of interest moves from a second region of the windshield to a first region of the windshield.
[0024] On one hand, one or more controllers execute instructions to determine that an object of interest has entered a first area of the windshield, and in response to determining that an object of interest has entered a first area of the windshield, instruct a first graphics processing unit to determine a graphic representation of a first set of images generated by a first graphics projection device.
[0025] On the other hand, the first set of images are augmented reality graphics that overlap and align with objects in the environment surrounding the vehicle.
[0026] Further applicability will become apparent from the description provided herein. It should be understood that the specification and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description
[0027] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.
[0028] Figure 1 This is a schematic diagram of a hybrid augmented reality head-up display system disclosed according to an exemplary embodiment for displaying graphics on the windshield of a vehicle;
[0029] Figure 2A This is an embodiment of the windshield as viewed from the perspective of a vehicle passenger, according to an exemplary embodiment;
[0030] Figure 2B This is another embodiment of a windshield as viewed from an occupant's perspective, according to an exemplary embodiment;
[0031] Figure 2C This is an enlarged view of a portion of a windshield comprising a plurality of light-emitting particles dispersed within a substantially transparent substrate, according to an exemplary embodiment.
[0032] Figure 3 This is an illustration of a windshield according to an exemplary embodiment, wherein a first set of images is displayed on a first area of the windshield, and a second set of images is displayed on a second area of the windshield; and
[0033] Figure 4 This is a process flow diagram illustrating a method for highlighting an object of interest located in the environment surrounding a vehicle using a hybrid augmented reality head-up display system, according to an exemplary embodiment. Detailed Implementation
[0034] The following description is merely exemplary in nature and is not intended to limit this disclosure, its application, or its uses.
[0035] refer to Figure 1 An exemplary hybrid augmented reality head-up display system 10 for displaying graphics on the windshield 12 of a vehicle 14 is shown. As described below, the hybrid augmented reality head-up display system 10 includes a first graphics projection device 26 and a second graphics projection device 28 working together to provide an augmented reality experience for passengers of the vehicle 14. It should be understood that the vehicle 14 can be any type of vehicle, such as, but not limited to, a sedan, truck, SUV, van, or motorhome. The hybrid augmented reality head-up display system 10 includes one or more controllers 20 in electronic communication with one or more image capture devices 22, an eye positioning system 24, a first graphics processing unit 30 corresponding to the first graphics projection device 26, and a second graphics processing unit 32 corresponding to the second graphics projection device 28. The image capture device 22 may be a camera that acquires periodic or continuous images representing a view of the surrounding environment 34 of the vehicle 14. The eye positioning system 24 includes one or more sensors for determining the position of the driver's head and the direction or gaze position of the driver's eyes.
[0036] Figure 2A This is an embodiment of a front view of the windshield 12 as seen from the perspective of a passenger in vehicle 14. Figure 2B yes Figure 2A Alternative embodiment of the center windshield 12. Figure 2C This is an enlarged view of a portion of the windshield 12. (See image below.) Figure 2C As shown, the windshield 12 includes a plurality of light-emitting particles 36 dispersed within a substantially transparent substrate 38. When excitation light is absorbed by the light-emitting particles 36, the light-emitting particles 36 generate visible light. In one embodiment, the light-emitting particles are red, green, and blue (RGB) phosphors for full-color operation; however, it should be understood that monochromatic or dual-color phosphors may also be used.
[0037] refer to Figure 1 and Figure 2A The windshield 12 includes a first region 40 and a second region 42, wherein a first graphic projection device 26 generates a first set of images 44 on the first region 40 of the windshield 12 based on visible light, and a second graphic projection device generates a second set of images 46 on the second region 42 of the vehicle windshield 12 based on excitation light. Specifically, in response to absorbing the excitation light emitted by the second graphic projection device 28, luminescent particles 36 dispersed within the windshield 12 ( Figure 2CIt emits visible light. As explained below, a first set of images 44 displayed on a first area 40 of the windshield 12 and a second set of images 46 displayed on a second area 42 of the windshield 12 cooperate to create an edge-to-edge augmented reality view of the vehicle 14's surrounding environment 34. Figure 2A As shown, the first region 40 of the windshield 12 includes only the portion of the windshield 12 with a limited field of view, while the second region 42 of the windshield 12 includes the remaining portion of the windshield 12 excluding the portion that is part of the first region 40. Combining the first region 40 and the second region 42 yields an augmented reality view of the surrounding environment 34 of the vehicle 14, which spans the opposite side edges 50 of the windshield 12.
[0038] Still referencing Figures 1 to 2A The first graphics processing unit 30 communicates electronically with the first graphics projection device 26, wherein the first graphics processing unit 30 translates image-based instructions from one or more controllers 20 into a graphical representation of a first set of images 44 generated by the first graphics projection device 26. The first set of images 44 are augmented reality graphics 48 that overlap and align with one or more objects of interest located in the surrounding environment 34 of the vehicle 14, thereby providing passengers with a virtual reality experience. Figure 2A In the example shown, augmented reality graphics 48 include target pins and text displayed as "0.1 mi", which overlap and align with the object of interest, namely building 52. It should be understood that, as Figure 2A The target pins and text shown are merely exemplary and other types of augmented reality graphics can also be used. Some examples of augmented reality graphics 48 include, for example, circles or boxes around an object of interest, arrows, warning symbols, text, numbers, colors, lines, indicators, and logos.
[0039] The first graphic projection device 26 includes a visible light source configured to generate a first set of images 44 on the windshield 12. The visible light source may be, for example, a laser or a light-emitting diode (LED). Figure 2A In the illustrated embodiment, the first region 40 of the windshield 12 includes a single image plane 60, and the first graphics projection device 26 is a single image plane augmented reality head-up display including a digital light projector (DLP) optical system. In one embodiment, the first region 40 of the windshield 12 includes a horizontal field of view of 10° and a vertical field of view of up to 5°.
[0040] In such Figure 2B In the alternative embodiment shown, the first region 40 of the windshield 12 includes a dual image plane 62, and a first graphic projection device 26 ( Figure 1The device is a fixed dual-image-plane holographic augmented reality head-up display. In one embodiment, the dual image planes 62 may include a first near-field image plane 62A and a second far-field image plane 62B. In this example, the first set of images 44 includes cluster content information 54 projected onto the near-field image plane 62A and augmented reality graphics 48 projected onto the far-field image plane 62B. The cluster content information 54 informs the driver of vehicle 14 of driving conditions, such as, but not limited to, vehicle speed, speed limit, gear, fuel level, current location, and navigation instructions. Figure 2B In the example shown, cluster content information 54 includes vehicle speed.
[0041] refer to Figure 1 and Figure 2A The second graphics processing unit 32 communicates electronically with the second graphics projection device 28. The second graphics processing unit 32 translates image-based instructions from the controller 20 into a graphical representation of a second set of images 46 generated by the second graphics projection device 28. The second set of images 46 includes one or more original graphic objects 64. (The last sentence appears to be incomplete and possibly refers to a different context.) Figure 2A In the example shown, the original graphic object 64 is a point or pixel. Other examples of the original graphic object 64 include a straight line. As explained below, an object in the surrounding environment 34 first appears in the driver's peripheral field of vision, located in the second area 42 of the windshield 12, as the original graphic object 64. By highlighting the object with the second set of images 46 (i.e., the original graphic object 64), the object of interest located in the second area 42 of the windshield 12 first attracts the driver's attention. Once the object of interest moves from the second area 42 of the windshield 12 to the first area 40 of the windshield 12, the object of interest is instead highlighted by the first set of images 44 (i.e., the augmented reality graphics 48).
[0042] The second graphic projection device 28 includes an excitation light source configured to generate a second set of images on the windshield 12. Specifically, in response to absorbing the excitation light emitted by the second graphic projection device 28, luminescent particles 36 dispersed within the windshield 12 ( Figure 2CThe excitation light is visible light. In an embodiment, the excitation light is violet light in the visible spectrum (ranging from approximately 380 nm to 450 nm) or ultraviolet light that induces fluorescence in the luminescent particles 36. It should be understood that since the luminescent particles 36 are dispersed throughout the windshield 12, the fluorescence irradiated by the luminescent particles 36 is non-directional. Therefore, the fluorescence is always visible regardless of the passenger's location within the vehicle 14. In other words, there is no peephole, and the disclosed hybrid augmented reality head-up display system 10 can be used as the primary instrument. The excitation light source can be, for example, a laser or an LED. In an embodiment, the second graphics projection device 28 is a microprojector with a relatively small package size and weight. For example, in one embodiment, the microprojector weighs approximately 120 grams.
[0043] refer to Figure 1 , Figure 2A and Figure 2B In one embodiment, the second graphic projection device 28 may be positioned along the roof 70 of the vehicle 14. The projection distance D from the windshield 12 to the projection lens 58 of the second graphic projection device 28 was measured. The projection distance D was designed such that a second region 42 of the windshield 12 spans across opposite side edges 50 of the windshield 12 and between the top edge 66 and the bottom edge 68 of the windshield 12.
[0044] Figure 3 The illustration highlights the windshield 12, which is a deer, within the surrounding environment 34 of the vehicle 14, highlighting the object of interest 80. Although a deer is depicted, it should be understood that... Figure 3 Essentially just an example, the object of interest 80 can be any object in the surrounding environment 34 of the vehicle 14, and it is also an object that the disclosed hybrid augmented reality head-up display system 10 may want to draw the driver's attention to, such as a child riding a bicycle, another vehicle, weather conditions, road conditions, obstacles, pedestrians, emergency vehicles, points of interest, road signs, billboards, and destinations.
[0045] In such Figure 3 In the example shown, the hybrid augmented reality heads-up display system 10 is tracking a deer that flies out from the right peripheral view 82 of the surrounding environment 34 and walks directly in front of the vehicle 14. Specifically, refer to Figure 1 and 3One or more controllers 20 first detect an object of interest 80 within a second region 42 of the windshield 12. In response to detecting the object of interest 80, one or more controllers 20 draw the driver's attention to the object of interest 80 by highlighting it with a second set of images 46. The object of interest 80 is highlighted by a primitive graphic object 64 (i.e., a dot). As the object of interest 80 moves toward the first region 40, the primitive graphic object 64 follows or tracks the object of interest 80. Once the object of interest 80 enters the first region 40 of the windshield 12, one or more controllers 20 seamlessly switch from highlighting the object of interest 80 with the primitive graphic object 64 to highlighting the object of interest 80 with augmented reality graphics 48. Figure 3 In the example shown, augmented reality graphics 48 include a circle surrounding object of interest 80.
[0046] Figure 4 This is a process flow diagram illustrating a method 200 for highlighting an object of interest 80 located within the surrounding environment 34 of a vehicle 14 (see [link]). Figure 3 ). refer to Figure 1 , Figure 3 and Figure 4 The method 200 begins at block 202. In block 202, one or more controllers 20 monitor image data representing the surrounding environment 34 of the vehicle 14 from the image capture device 22. The one or more controllers 20 continue monitoring the image data until it is determined, based on the image data, that the object of interest 80 is located within a second region 42 of the windshield 12. The method 200 then proceeds to block 204.
[0047] In frame 204, in response to determining that the object of interest 80 is located within the second region 42 of the windshield 12, one or more controllers 20 instruct the second graphics processing unit 32 to determine a graphical representation of the second set of images 46 generated by the second graphics projection device 28. Figure 3 As shown, the second set of images 46 includes one or more original graphic objects 64. Specifically, in Figure 3 In the example shown, the original graphic object 64 is a point. Method 200 then proceeds to box 206.
[0048] In frame 206, as the object of interest 80 moves from the second region 42 of the windshield 12 to the first region 40 of the windshield 12, one or more controllers 20 instruct the second graphics processing unit 32 to follow the object of interest 80 using a graphical representation of a second set of images 46. Figure 3 In the example shown, the deer flies from the right peripheral view 82 of the surrounding environment 34 into the first region 40 of the windshield 12. The method 200 then proceeds to frame 208.
[0049] In block 208, one or more controllers 20 determine that the object of interest 80 has entered the first region 40 of the windshield 12. The method 200 then proceeds to block 210.
[0050] In frame 210, in response to determining that the object of interest 80 has entered the first region 40 of the windshield 12, one or more controllers 20 instruct the first graphics processing unit 30 to determine a graphical representation of the first set of images 44 generated by the first graphics projection device 26. Figure 3 In the example shown, the first set of images 44 includes a circle surrounding the deer. The method 200 can then terminate.
[0051] Referring to the accompanying drawings, the disclosed hybrid augmented reality head-up display system offers various technical effects and benefits. Specifically, a first set of images displayed by a first graphic projection device collaborates with a second set of images displayed by a second graphic projection device to create an edge-to-edge augmented reality view of the vehicle's surroundings. It should be understood that since the fluorescence emitted by the luminescent particles in the windshield is non-directional, a peephole is not required. Therefore, unlike conventional augmented reality systems currently available, the disclosed hybrid augmented reality head-up display can be used as the primary instrument.
[0052] A controller can refer to electronic circuitry, combinational logic circuitry, a field-programmable gate array (FPGA), a processor (shared, dedicated, or grouped) that executes code, or a combination of some or all of the above, such as in a system-on-a-chip. Alternatively, the controller can be microprocessor-based, such as a computer having at least one processor, memory (RAM and / or ROM), and associated input and output buses. The processor can run under the control of an operating system residing in memory. The operating system can manage computer resources such that computer program code, embodied as one or more computer software applications, such as applications residing in memory, can have instructions executed by the processor. In alternative embodiments, the processor can directly run the application, in which case the operating system can be omitted.
[0053] The description in this disclosure is merely exemplary in nature, and any changes that do not depart from the spirit and scope of this disclosure are intended to fall within its scope. Such changes should not be considered as departing from the spirit and scope of this disclosure.
Claims
1. A hybrid augmented reality head-up display system for displaying graphics on the windshield of a vehicle, wherein, The windshield includes a transparent substrate, the transparent substrate including light-emitting particles dispersed therein, and the hybrid augmented reality head-up display system includes: A first graphic projection device is used to generate a first set of images on the windshield of a vehicle based on visible light, wherein the first set of images is displayed on a first area of the windshield; and A second graphics projection device is used to generate a second set of images on a second area of the windshield of a vehicle based on excitation light, wherein, in response to absorbing the excitation light, light-emitting particles in the windshield emit visible light, and wherein, a first set of images displayed on a first area of the windshield and a second set of images displayed on a second area of the windshield cooperate to create an edge-to-edge augmented reality view of the vehicle's surrounding environment. The first graphics processing unit communicates electronically with the first graphics projection device. The second graphics processing unit communicates electronically with the second graphics projection device; and The controller communicates electronically with the first graphics processing unit and the second graphics processing unit. The controller executes instructions as follows: The object of interest is located within a second area of the windshield; and In response to determining that the object of interest is located within a second area of the windshield, the second graphics processing unit is instructed to determine a graphic representation of the second set of images generated by the second graphics projection device; When the object of interest moves from the second region of the windshield to the first region of the windshield, the second graphics processing unit is instructed to follow the object of interest through the graphic representation of the second set of images; It has been determined that the object of interest has entered the first area of the windshield; and In response to determining that the object of interest has entered a first area of the windshield, the first graphics processing unit is instructed to determine a graphic representation of the first set of images generated by the first graphics projection device.
2. The hybrid augmented reality head-up display system according to claim 1, wherein, The second set of images includes one or more original graphic objects.
3. The hybrid augmented reality head-up display system according to claim 2, wherein, The one or more original graphic objects include one or more of the following: points and lines.
4. The hybrid augmented reality head-up display system according to claim 1, wherein, The first set of images is augmented reality graphics that overlap and align with objects in the environment surrounding the vehicle.
5. The hybrid augmented reality head-up display system according to claim 1, wherein, The first graphics projection device is a single-image planar augmented reality head-up display that includes a digital light projector (DLP) optical system.