Hybrid head-up display calibration

Abstract

A method of calibrating a head-up display system includes training a neural network to predict an occupant perspective distortion map of a single graphic image projected onto a windshield of a vehicle, projecting the graphic image onto the windshield with a digital light projector, capturing the graphic image projected onto the windshield with a camera proximate to the digital light projector, and calibrating the head-up display system. A passenger monitoring system is used to position a passenger eye ellipse position, a neural network is used to predict a passenger visual angle distortion graph based on the position of a digital light projector, the passenger eye ellipse position and the shape of a windshield, and a controller is used to apply the distortion graph to a graphic image projected by the digital light projector. And projecting, using the digital light projector, the corrected graphic image based on the predicted warp map.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for displaying graphics in a vehicle, and more particularly to a graphics projector for use in a vehicle. Background Art

[0002] To increase occupant awareness and convenience, a vehicle may be equipped with a display system that is configured to provide information to the occupant about the vehicle's surroundings. The display system may provide information to the occupant using a human-machine interface device (e.g., a touch screen). The display system may also provide information to the occupant using a head-up display (HUD). To control and / or illuminate the HUD, a graphic projector may be used to project graphics onto a surface of the vehicle (e.g., a windshield). Full windshield displays embedded with transparent phosphors require calibration to correct for errors due to keystone distortion, windshield shape variations, and installation errors.

[0003] Therefore, while current systems achieve their intended purpose, a need exists for a new and improved system and method for correcting displayed images to compensate for image distortion by computing a warp map and then encoding the distortion information into a two-dimensional image projected onto the windshield. Summary of the invention

[0004] According to several aspects of the present disclosure, a method for calibrating a head-up display system in a vehicle includes: training a neural network in communication with a controller to predict an occupant perspective distortion map of a single graphic image projected onto an inner surface of a vehicle windshield; projecting the graphic image onto the inner surface of the vehicle windshield using a digital light projector located in the vehicle; capturing the graphic image projected onto the inner surface of the windshield using a camera close to the digital light projector; locating an occupant eye ellipse position using an occupant monitoring system in the vehicle; predicting an occupant perspective distortion map of the projected graphic image based on a position of the digital image light projector, an occupant eye ellipse position, and a shape of the windshield in the vehicle using the neural network; applying the distortion map to the graphic image projected by the digital light projector using the controller; and projecting a corrected graphic image based on the predicted distortion map using the digital light projector.

[0005] According to another aspect, training a neural network within a controller to predict an occupant perspective distortion map for a single graphical image projected onto an interior surface of a vehicle windshield further comprises: projecting a source image onto an interior surface of a reference windshield using a reference digital light projector of a reference system; capturing the source image from a perspective of the reference occupant eyellipse location using a first reference camera located at a reference occupant eyellipse location; capturing the source image from a perspective of the reference digital light projector using a second reference camera proximate to the reference digital light projector; calculating, using a reference controller, a transformation matrix between the source image captured from a perspective of the reference occupant eyellipse location and an image captured from a perspective of the reference digital light projector; and creating a reference distortion map for the captured source image.

[0006] According to another aspect, training a neural network within a controller to predict an occupant perspective distortion map for a single graphical image projected onto an interior surface of a vehicle windshield further includes creating a plurality of reference distortion maps for a plurality of different source images, projecting each of the plurality of different source images from a plurality of different reference digital light projector locations within a reference system to a plurality of different locations on the interior surface of a reference windshield, and storing the plurality of reference distortion maps within the neural network.

[0007] According to another aspect, training a neural network within a controller to predict an occupant perspective distortion map for a single graphical image projected onto an interior surface of a vehicle windshield further includes: creating a plurality of reference distortion maps for a plurality of different reference windshields; and storing the plurality of reference distortion maps within the neural network.

[0008] According to another aspect, using a neural network to predict an occupant perspective distortion map of a projected graphic image based on an occupant eye ellipse position also includes: using a neural network to apply an interpolation algorithm to predict the distortion map of the projected graphic image based on a position of a digital light projector, an occupant eye ellipse position, and a shape of a windshield shape within a vehicle.

[0009] According to another aspect, applying the distortion map to the graphic image projected by the digital light projector using the controller further includes encoding distortion information of the distortion map to the graphic image projected by the digital light projector using the controller.

[0010] According to another aspect, the method further includes initiating calibration of a heads-up display in the vehicle upon selective actuation by an occupant of the vehicle.

[0011] According to another aspect, the method further includes automatically initiating calibration of a heads-up display in the vehicle.

[0012] According to another aspect, projecting a graphic image onto the inner surface of a vehicle windshield using a digital light projector also includes: projecting excitation light using a light source; focusing the excitation light from the light source using a condenser lens; separating the focused excitation light into red, green, and blue light using a color filter; focusing the excitation light passing through the color filter using a shaping lens; redirecting the excitation light using a digital micromirror device (DMD); and receiving the excitation light from the digital micromirror device using a projection lens, and projecting the excitation light onto the inner surface of the windshield using the projection lens.

[0013] According to another aspect, projecting a graphic image onto an inner surface of a vehicle windshield using a digital light projector within the vehicle further comprises projecting excitation light using the digital light projector, wherein luminescent particles within the windshield emit visible light in response to absorbing the excitation light.

[0014] According to several aspects of the present disclosure, a head-up display system in a vehicle includes: a controller suitable for calibrating the head-up display; a neural network in communication with the controller, which is suitable for predicting a distortion map of a single graphic image of an occupant's perspective projected onto an inner surface of a vehicle windshield; a digital light projector located in the vehicle, which is suitable for projecting the graphic image onto the inner surface of the vehicle windshield; a camera close to the digital light projector, which is suitable for capturing the graphic image projected onto the inner surface of the windshield; and an occupant monitoring system in the vehicle, which is suitable for locating an occupant's eye ellipse position; the neural network is further suitable for predicting a projected occupant's perspective distortion map based on a graphic image of a position of the digital light projector, an occupant's eye ellipse position, and a shape of the windshield in the vehicle, the controller is further suitable for applying the distortion map to the graphic image projected by the digital light projector, and the digital light projector is further suitable for projecting a corrected graphic image based on the predicted distortion map.

[0015] According to another aspect, the system further comprises a reference system comprising: a reference digital light projector adapted to project a source image onto an inner surface of a reference windshield; a first reference camera located at a reference occupant eye ellipse position adapted to capture the source image from the perspective of the reference occupant eye ellipse position; a second reference camera proximate the reference digital light projector adapted to capture the source image from the perspective of the reference digital light projector; and a reference controller adapted to calculate a transformation matrix between the source image captured from the perspective of the reference occupant eye ellipse position and the source image captured from the perspective of the reference digital light projector, and to create a reference distortion map for the captured source image.

[0016] According to another aspect, the reference controller is further adapted to create a plurality of reference distortion maps for a plurality of different source images, project each of the plurality of different source images from a plurality of different reference digital light projector locations within the reference system to a plurality of different locations on the interior surface of the reference windshield, and store the plurality of reference distortion maps within the neural network.

[0017] According to another aspect, the reference controller is further adapted to create a plurality of reference distortion maps for a plurality of different reference windshields and store the plurality of reference distortion maps within the neural network.

[0018] According to another aspect, when predicting an occupant perspective distortion map of a projected graphic image based on the occupant eyellipse location, the neural network is further adapted to apply an interpolation algorithm to predict the distortion map of the projected graphic image based on the location of the digital light projector, the occupant eyellipse location, and the shape of the windshield shape within the vehicle.

[0019] According to another aspect, when the distortion map is applied to a graphic image projected by the digital light projector, the controller is further adapted to encode distortion information of the distortion map into the graphic image projected by the digital light projector.

[0020] According to another aspect, the controller is further adapted to initiate calibration of a head-up display in the vehicle upon selective actuation by an occupant of the vehicle, and to automatically initiate calibration.

[0021] According to another aspect, a digital light projector includes: a light source suitable for projecting excitation light; a condensing lens suitable for focusing the excitation light from the light source; a color filter suitable for separating the focused excitation light into red, green and blue light; a shaping lens suitable for focusing the excitation light passing through the color filter; a digital micromirror device (DMD) suitable for redirecting the excitation light; and a projection lens suitable for receiving the excitation light from the digital micromirror device and projecting the excitation light onto the inner surface of the windshield.

[0022] According to another aspect, the digital light projector is adapted to project an excitation light, wherein the light emitting particles within the windshield emit visible light in response to absorbing the excitation light.

[0023] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0025] Figure 1 is a schematic diagram of a vehicle having a system according to an exemplary embodiment of the present disclosure;

[0026] Figure 2 is a schematic diagram of a system according to an exemplary embodiment of the present disclosure;

[0027] Figure 3 is a schematic diagram of a substrate for a vehicle windshield according to an exemplary embodiment;

[0028] Figure 4 yes Figure 3 The enlarged part, such as Figure 3 The middle mark is " Figure 4 " is shown in the circle;

[0029] Figure 5 is a schematic diagram of a reference system according to an exemplary embodiment of the present disclosure;

[0030] Figure 6 is a schematic diagram showing a source image projected from the perspective of a second reference camera, a source image projected from the perspective of a first reference camera, a distortion map, and a corrected graphic image;

[0031] Figure 7 is a flowchart illustrating a method according to an exemplary embodiment of the present disclosure;

[0032] Figure 8 It is shown Figure 7 A flowchart of the method steps of block 202; and

[0033] Fig. 9 It is shown Figure 7 Flow chart of the method steps of blocks 204 and 214.

[0034] The drawings are not necessarily drawn to scale, and some features may be exaggerated or minimized, for example, to show details of a particular component. In some cases, well-known components, systems, materials, or methods are not described in detail to avoid obscuring the present disclosure. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to adopt the present disclosure in different ways. DETAILED DESCRIPTION

[0035] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application or use. In addition, it is not intended to be constrained by any express or implied theory presented in the aforementioned technical field, background technology, brief overview or the following detailed description. It should be understood that throughout the drawings, the corresponding reference numerals represent similar or corresponding components and features. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device, individually or in any combination, including but not limited to: application specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or groups) and memories that execute one or more software or firmware programs, combinational logic circuits and / or other suitable components that provide the described functions. Although the drawings shown herein depict examples with certain element arrangements, additional intermediate elements, devices, features or components may be present in actual embodiments. It should also be understood that the drawings are merely illustrative and may not be drawn to scale.

[0036] As used herein, the term "vehicle" is not limited to automobiles. Although this article mainly describes the present technology in conjunction with automobiles, the present technology is not limited to automobiles. These concepts can be used in a variety of applications, such as aircraft, ships, other vehicles, and consumer electronic components.

[0037] According to an exemplary embodiment, Figure 1 A vehicle 10 and an associated hybrid head-up display system 11 within the vehicle 10 are shown according to various embodiments. Generally speaking, the hybrid head-up display system 11 works in conjunction with other systems within the vehicle 10 to display various information and infotainment content to passengers. The vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and substantially surrounds the components of the vehicle 10. The body 14 and the chassis 12 may together form a vehicle frame. The front wheels 16 and the rear wheels 18 are each rotatably connected to the chassis 12 near respective corners of the body 14.

[0038] In various embodiments, the vehicle 10 is an autonomous vehicle and the system 11 is incorporated into the autonomous vehicle 10. The autonomous vehicle 10 is, for example, a vehicle 10 that is automatically controlled to transport passengers from one location to another. The vehicle 10 is depicted as a passenger car in the illustrated embodiment, but it should be understood that any other vehicle may also be used, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), etc. In an exemplary embodiment, the vehicle 10 is equipped with a so-called level 4 or level 5 automation system. The level 4 system represents "high automation" and refers to the driving mode-specific performance of the autonomous driving system in all aspects of dynamic driving tasks, even if the human driver does not respond appropriately to the intervention request. The level 5 system represents "full automation" and refers to the full-time performance of the autonomous driving system in all aspects of dynamic driving tasks under all road and environmental conditions that the human driver can manage.

[0039] As shown, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, a vehicle controller 34, and a communication system 36. In an embodiment where the vehicle 10 is an electric vehicle, there may be no transmission system 22. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transmit power from the propulsion system 20 to the front wheels 16 and rear wheels 18 of the vehicle according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a stepped automatic transmission, a continuously variable transmission, or other suitable transmissions. The braking system 26 is configured to provide braking torque to the front wheels 16 and rear wheels 18 of the vehicle. In various embodiments, the braking system 26 may include friction brakes, wire control brakes, regenerative braking systems (e.g., motors), and / or other suitable braking systems. The steering system 24 affects the position of the front wheels 16 and rear wheels 18. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of the present disclosure, the steering system 24 may not include a steering wheel.

[0040] The sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external environment and / or the internal environment of the autonomous vehicle 10. The sensing devices 40a-40n may include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal camera, ultrasonic sensor, and / or other sensors. The camera may include two or more digital cameras spaced at a selected distance from each other, wherein the two or more digital cameras are used to obtain stereo images of the surrounding environment so as to obtain a three-dimensional image. The plurality of sensing devices 40a-40n are used to determine information about the environment around the vehicle 10. In an exemplary embodiment, the plurality of sensing devices 40a-40n include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor. In another exemplary embodiment, the plurality of sensing devices 40a-40n also include sensors for determining information about the environment around the vehicle 10, such as an ambient air temperature sensor, an atmospheric pressure sensor, and / or a camera and / or video camera positioned to observe the environment in front of the vehicle 10. In another exemplary embodiment, at least one of the plurality of sensing devices 40a-40n is capable of measuring distances in the environment surrounding the vehicle 10. In a non-limiting example where the plurality of sensing devices 40a-40n include cameras, the plurality of sensing devices 40a-40n measure distances using an image processing algorithm configured to process images from the cameras and determine distances between objects. In another non-limiting example, the plurality of vehicle sensors 40a-40n include stereo cameras with distance measurement capabilities. In one example, at least one of the plurality of sensing devices 40a-40n is fixed to the interior of the vehicle 10, for example, fixed in the roof of the vehicle 10, with a view through the windshield 56 of the vehicle 10. In another example, at least one of the plurality of sensing devices 40a-40n is fixed to the exterior of the vehicle 10, for example, fixed on the roof of the vehicle 10, with a view of the environment surrounding the vehicle 10. It should be understood that various additional types of sensing devices, such as LiDAR sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors are within the scope of the present disclosure. The actuator system 30 includes one or more actuator devices 42 a - 42 n that control one or more features of the vehicle 10 , such as, but not limited to, the propulsion system 20 , the transmission system 22 , the steering system 24 , and the braking system 26 .

[0041] The vehicle controller 34 includes at least one processor 44 and a computer readable storage device or medium 46. The at least one data processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor in a plurality of processors associated with the vehicle controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or any device generally used to execute instructions. The computer readable storage device or medium 46 can include volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the at least one data processor 44 is powered off. The computer readable storage device or medium 46 can be implemented using any of a variety of known storage devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combination storage device capable of storing data used by the controller 34 in controlling the vehicle 10, some of which represents executable instructions.

[0042] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by at least one processor 44, the instructions receive and process signals from the sensor system 28, perform logic, calculations, methods and / or algorithms for automatically controlling components of the vehicle 10, and generate control signals to the actuator system 30 to automatically control components of the vehicle 10 based on the logic, calculations, methods and / or algorithms. Although Figure 1 Only one controller 34 is shown, but embodiments of the vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication mediums and collaborate to process sensor signals, execute logic, calculations, methods and / or algorithms, and generate control signals that automatically control features of the autonomous vehicle 10.

[0043] In various embodiments, one or more instructions of the vehicle controller 34 are embodied in a trajectory planning system and, when executed by at least one data processor 44, generate trajectory outputs that address kinematic and dynamic constraints of the environment. For example, the instructions receive process sensor and map data as input. The instructions execute a graph-based approach with a customized cost function to handle different road scenarios of cities and highways.

[0044] The communication system 36 is configured to wirelessly communicate information with other remote entities 48, such as, but not limited to, other vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems, remote servers, cloud computers, and / or personal devices. In an exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate over a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communications. However, additional or alternative communication methods such as dedicated short-range communication (DSRC) channels are also considered to be within the scope of the present disclosure. A DSRC channel refers to a one-way or two-way short-range to medium-range wireless communication channel designed specifically for automotive use and a set of corresponding protocols and standards.

[0045] The vehicle controller 34 is a non-universal electronic control device having a pre-programmed digital computer or processor, a memory or non-transitory computer-readable medium for storing data such as control logic, software applications, instructions, computer code, data, look-up tables, etc., and a transceiver [or input / output port]. Computer-readable media include any type of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard drive, compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer-readable media excludes wired, wireless, optical or other communication links that transmit temporary electrical or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store and subsequently rewrite data, such as rewritable optical disks or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code.

[0046] refer to Figure 2 The head-up display system 11 according to an exemplary embodiment communicates with a vehicle controller 34, and the vehicle controller 34 is adapted to calibrate the head-up display system 11. A digital light projector (DLP) 50 is positioned within the vehicle 10 and is adapted to project a graphic image 52 onto an interior surface 54 of a windshield 56 of the vehicle 10.

[0047] The DLP 50 projects a graphic image 52 that includes various information / infotainment applications. The DLP 50 is capable of projecting two-dimensional and three-dimensional images depicting objects such as pedestrians, buildings, and other vehicles in the environment surrounding the vehicle 10. In an exemplary embodiment, the DLP 50 includes a light source 62 adapted to project an excitation light 64, a focusing lens 66 adapted to focus the excitation light 64 from the light source 62, a color filter 68 (color wheel) adapted to separate the excitation light 64 into red, green, and blue light, a shaping lens 70 adapted to focus the excitation light 64 transmitted through the color filter 68, a digital micromirror device (DMD) 72 adapted to redirect the excitation light 64, and a projection lens 74 adapted to receive the excitation light 64 from the DMD 72 and project the excitation light 64 onto the inner surface 54 of the windshield 56.

[0048] The vehicle controller 34 includes a DLP engine 76. The DLP engine 76 includes a programmable processor that includes programming for monitoring various inputs and determining which information is suitable for display. The DLP engine 76 can communicate directly with various systems and components, or the DLP engine 76 can communicate via a LAN / CAN system instead or in addition. The DLP engine 76 utilizes information about the operating environment of the vehicle 10 derived from multiple inputs. Various sensing devices 40a-40n collect and provide information. The sensing devices 40a-40n include cameras or image capture devices that capture periodic or sequential images representing the view of the vehicle 10. The windshield 56 has the feature of being able to display images projected thereon while remaining transparent or substantially transparent, so that the occupants of the vehicle 10 can clearly observe the exterior of the vehicle 10 through the windshield 56. It should be understood that, as shown in the figure, the windshield 56 is located at the front of the vehicle 10, and other surfaces in the vehicle 10 including the side windows and the rear window can also be used for projection. Additionally, the view on the front windshield 56 may continue as a continuous image onto the front vehicle "A-pillars" and onto the side windows.

[0049] The DLP engine 76 includes display software or programming to translate requests to display information from the DLP engine 76 in a graphical representation describing the information. The DLP engine 76 includes programming to compensate for the curves and slopes of the windshield 56 and any other surface onto which the first graphic image 52 is to be projected. The DLP engine 76 controls the light source 62, which includes a laser or projector device that produces the excitation light 64 to project the graphic image 52.

[0050] The occupant monitoring system 78 includes sensors known in the art to simulate the position of the occupant's head and further simulate the orientation or gaze position of the occupant's eyes. Based on the output of the occupant monitoring system 78 and the input data of the tracked position information about the environment surrounding the vehicle 10, the DLP engine 76 can accurately position the graphic image 52 so that the occupant 80 sees the graphic image 52 overlaid with the visual image through the windshield 56. The eye tracking system in the vehicle is suitable for locating the occupant eye ellipsoid position.

[0051] The system 11 described above includes an eye sensing device and a head sensing device that can estimate eye position and can register an image on the windshield 56 so that the image corresponds to the field of view of the operator 80. In addition, the occupant monitoring system 78 allows the system 11 to locate the position of the occupant's eyellipse 58. The eyellipse 58 is a three-dimensional graphical depiction of a multivariate normal distribution that is used to simulate the distribution of 80 eye positions of an occupant in the vehicle 10. The eyellipse 58 is represented by two three-dimensional ellipses, one for the right eye and one for the left eye.

[0052] It should be understood that the estimation of head and eye positions can be achieved by a variety of methods. For example, in a process similar to adjusting a rearview mirror, the operator can use a calibration routine upon entering the vehicle to align the graphic with the detected object. In another embodiment, the longitudinal seat position in the vehicle 10 can be used to estimate the position of the driver's head. In another embodiment, manual adjustment of one or more rearview mirrors can be used to estimate the position of the operator's eyes. It should be understood that a combination of methods such as seat position and mirror adjustment viewing angles can be utilized to estimate the operator's head position with improved accuracy. Many methods are contemplated for achieving accurate registration of the graphic on the windshield 56 with the occupant eye ellipse 58 position, and the present disclosure is not intended to be limited to the specific embodiments described herein.

[0053] The windshield 56 serves as a medium through which the relevant features can be observed, and also serves as a display device on which the first graphic image 52 can be displayed. The windshield 56 is transparent and is capable of displaying the image projected by the excitation light 64. Figure 3 and Figure 4 , the occupant 80 can see any object (e.g., cube 82) through the substrate 84 located on the windshield 56. The substrate 84 can be transparent or substantially transparent. When the occupant 80 sees the arbitrary object 82 through the substrate 84, the occupant 80 can also see the image (e.g., circle 86 and triangle 88) created at the substrate 84. The substrate 84 can be a portion of the windshield 56, a glass substrate, a plastic substrate, a polymer substrate, or other transparent (or substantially transparent) medium understood by those of ordinary skill in the art. Other substrates can supplement the substrate 84 to provide coloring, substrate protection, light filtering (e.g., filtering external ultraviolet light), and other functions.

[0054] Figure 4 The illumination of the transparent display irradiated with excitation light 64 (e.g., ultraviolet light) from the light source 62 is depicted. The substrate 84 receives the excitation light 64 from the light source 62. The received excitation light 64 can be absorbed by the luminescent material 90 at the substrate 84. When the luminescent material receives the excitation light 64, the luminescent material 90 emits visible light. Therefore, the substrate 84 can be selectively illuminated with the excitation light 64 to produce an image (e.g., a circle 86 and a triangle 88) at the substrate 84.

[0055] In an exemplary embodiment, the luminescent material 90 includes a transparent phosphor embedded in the substrate 84. The transparent phosphor is a luminescent particle that emits fluorescence in response to excitation by the excitation light 64. In an exemplary embodiment, the transparent phosphor is a red, green, and blue (RGB) phosphor that allows full color. The use of monochromatic and / or dual-color phosphors is also within the scope of the present disclosure. When the excitation light 64 is absorbed by the transparent phosphor, the transparent phosphor emits visible light. The excitation light 64 is provided by the light source 62. The use of the substrate 84 and the luminescent material 90 to display graphics is discussed in more detail in U.S. Patent No. 17 / 749,464, entitled "Mixed Augmented Reality Head-Up Display for Creating Edge-to-Edge Enhanced Display Field of View", filed on May 20, 2022, the entire contents of which are incorporated herein by reference.

[0056] According to an embodiment of the present disclosure, the excitation light 64 may be ultraviolet light. If the excitation light 64 is ultraviolet light, when the luminescent material 90 emits visible light in response to the ultraviolet light, a down-conversion physical phenomenon occurs. Specifically, ultraviolet light has a shorter wavelength and higher energy than visible light. Therefore, when the luminescent material 90 absorbs ultraviolet light and emits lower energy visible light, the ultraviolet light is down-converted into visible light because its energy level decreases when the ultraviolet light is converted into visible light.

[0057] It should be understood that the excitation wavelengths described herein are specific for a given phosphor sheet.Other technologies may also be used, such as holographic grating (Bragg grating) based technologies using visible light projectors, where RGB LEDs are used to make vector graphics projectors and RGB-DLP projectors.

[0058] like Figure 2As shown, the excitation light 64 is output by the light source 62 of the DLP 50. In an exemplary embodiment, the light source 62 is a micromirror array (MMA) projector (e.g., a digital light processing (DLP) projector 50). In the DLP 50, the first graphic image 52 is created by microscopic mirrors arranged in a matrix on a semiconductor chip in the DMD 72. The MMA projector that outputs ultraviolet light can be similar to the MMA projector that outputs visible light, except that the color filter 68 has a filter customized for the ultraviolet spectrum. In other embodiments, the DLP 50 is a liquid crystal display (LCD) projector. In an embodiment, the DLP 50 can be a liquid crystal on silicon (LCOS) projector. In an embodiment, the DLP 50 can be an analog projector (e.g., a slide projector or a movie film projector). Those of ordinary skill in the art will understand other types of projectors that can be used to project ultraviolet light onto the substrate 84.

[0059] Reference again Figure 2 , the system 11 also includes a neural network 92 in communication with the controller 34. The neural network 92 is adapted to predict a vehicle occupant perspective distortion map based on a single graphical image captured by a camera 94 located near (embedded within) the digital light projector 50, as will be discussed in detail below.

[0060] The camera 94 is located within the vehicle 10 proximate to the digital light projector 50 and is adapted to capture the graphic image 52 projected onto the interior surface 54 of the windshield 56 from the perspective of the digital light projector 50. The camera 94 may be integrated within the digital light projector 50. The neural network 92 is further adapted to predict a distortion map of the projected graphic image 52 from the perspective of the driver / occupant based on the position of the digital light projector 50, the position of the occupant eye ellipses 58, and the shape of the windshield 56 within the vehicle 10. The controller 34 is further adapted to apply the predicted distortion map to the graphic image 52 projected by the digital light projector 50, and the digital light projector 50 is further adapted to project a corrected graphic image 98 based on the predicted distortion map.

[0061] refer to Figure 2 and Figure 5In an exemplary embodiment, the system 11 includes a reference system 100. The reference system 100 includes a reference digital light projector 102 adapted to project a source image 104 onto an inner surface 106 of a reference windshield 108. A first reference camera 110 is located at a reference occupant eyellipse position 112 and adapted to capture the source image 104 from a perspective of the reference occupant eyellipse position 112. A second reference camera 114 is located within the vehicle 10 proximate to the reference digital light projector 102 and adapted to capture the source image 104 from a perspective of the reference digital light projector 102. A reference controller 116 is adapted to calculate a transformation matrix between the source image 104 captured from a perspective of the reference occupant eyellipse 58 position and the source image 104 captured from a perspective of the reference digital light projector 102, and to create a reference distortion map 118 for the captured source image 104.

[0062] refer to Figure 6 , the reference controller 116 projects the source image 104 onto the reference windshield 108 using the reference DLP 102. As shown at 120, the reference controller captures the source image 104 from the perspective of the reference digital light projector 102 using the second reference camera 114. Within the scope of the present disclosure, the source image 104 includes a plurality of projection points 122 arranged in a uniform two-dimensional array or grid (i.e., the distance between each point and the surrounding points is consistent). Each of the plurality of projection points 122 has an expected x-coordinate and an expected y-coordinate defined relative to the projection field 124. The source image 104 is represented as an expected source image matrix. Within the scope of the present disclosure, the expected source image matrix is ​​a two-dimensional matrix having a plurality of elements. Each of the plurality of elements of the expected source image matrix defines the position of one of the plurality of projection points 122 by storing the expected x-coordinate and the expected y-coordinate of one of the plurality of projection points 122. For Figure 6 For the exemplary source image 104 shown, the expected source image matrix is:

[0063]

[0064] Where I is the expected source image matrix, x n is the expected x coordinate of one of the plurality of projection points 122 located in the n columns of the expected source image matrix, y m is the expected y coordinate of one of the plurality of projected points 122 located in the m rows in the columns of the expected source image matrix. It should be understood that the source image 104 may have any number of points and the expected source image matrix may have any number of rows and / or columns without departing from the scope of the present disclosure.

[0065] The reference controller 116 captures the source image 104 from the perspective of the reference occupant eyellipse location 112 using the first reference camera 110. Due to manufacturing variations of the reference windshield 108 (e.g., curvature variations, glass thickness uniformity variations, glass surface quality variations, glass coating uniformity, etc.), and due to the position of the reference occupant eyellipse location 112 relative to the reference windshield 108, as shown at 126, the source image 104 appears distorted. In other words, as shown at 130, the position of each of the plurality of projection points 122 of the source image 104 from the perspective of the second reference camera 114 (as shown at 120) does not correspond to the position of each of the plurality of projection points 126 of the source image 104 from the perspective of the first reference camera 110 (as shown at 128).

[0066] In an exemplary embodiment, as shown at 128, the reference controller 116 uses a computer vision algorithm to identify a plurality of projection points 126 of the source image 104 from the perspective of the first reference camera 110. Within the scope of the present disclosure, identifying the plurality of projection points 126 means that the reference controller 116 isolates the plurality of projection points 126 from other elements (e.g., background objects, dust particles, camera artifacts, etc.) in the captured image 128, so that the position of each of the plurality of projection points 126 can be determined, as discussed below. In an exemplary embodiment, the computer vision algorithm is trained by providing it with a plurality of sub-images including points at different positions and perspectives. After sufficient training of the computer vision algorithm, the reference controller 116 can identify the plurality of projection points 126 with high accuracy and precision. After identifying the plurality of projection points 126 of the source image 104 from the perspective of the first reference camera 110, the reference controller 116 determines the actual x-coordinate and the actual y-coordinate of each of the plurality of projection points 126 of the source image 104 from the perspective of the first reference camera 110. In an exemplary embodiment, actual coordinates of each of the plurality of projection points 126 of the source image 104 from the perspective of the first reference camera 110 are identified by performing pixel measurements on each of the plurality of projection points 126 of the source image 104 from the perspective of the first reference camera 110 relative to the projection field 124.

[0067] The reference controller 116 generates an actual source image matrix. Within the scope of the present disclosure, the actual source image matrix contains the actual x-coordinates and the actual y-coordinates of each of the plurality of projection points 126 of the source image 104 from the perspective of the first reference camera 110. The reference controller 116 then calculates a transformation matrix. Within the scope of the present disclosure, the transformation matrix encodes the difference between the position of each of the plurality of projection points 122 of the source image 104 from the perspective of the second reference camera 114, as shown at 120, and the position of each of the plurality of projection points 126 of the image 104 from the perspective of the first reference camera 110, as shown at 128, and defines a reference warp map 118. In an exemplary embodiment, to calculate the transformation matrix, the actual source image matrix is ​​subtracted from the expected source image matrix.

[0068] When applied to the source image 104 projected by the reference DLP 102, the reference distortion map 118 adjusts the projected source image 104 to compensate for distortion due to glass manufacturing variations of the reference windshield 108 and due to the position of the reference occupant eye ellipse position 112 relative to the reference windshield 108, as shown at 132. The reference system 100 is calibrated using the reference distortion map so that the position of each of the plurality of projected points 126 of the source image 104 from the perspective of the first reference camera 110 corresponds to the position of each of the plurality of projected points 122 of the original source image 104.

[0069] The distortion map provides a calibration tool for a specific position (position / viewing angle) of the first reference camera, the second reference camera, and the reference DLP 102, as well as features of a reference windshield manufactured and installed within the reference system. The reference controller 116 is further adapted to create a plurality of reference distortion maps 118a-118n for a plurality of different source sub-images 104a-104n, each of which is projected onto a plurality of different positions on the interior surface 106 of the reference windshield 108 on the image and from a plurality of different reference digital light projector 102 positions within the reference system 100. Furthermore, the reference controller 116 is adapted to create a plurality of reference distortion maps 118a-118n for a plurality of different reference windshields 108a-108n. The plurality of reference distortion maps 118a-118n are stored within the neural network 92.

[0070] In an exemplary embodiment, when predicting the distortion map of the projected graphic image 52 based on the position of the occupant eyellipse 58, the neural network 92 is further adapted to apply an interpolation algorithm to predict the distortion map of the projected graphic image 52 based on the position of the digital light projector 50, the position of the occupant eyellipse 58, and the shape of the windshield 56 in the vehicle 10. The neural network 92 receives information related to the position of the digital light projector 50 based on feedback from the camera 94 and based on information about the shape of the windshield 56 of the graphic image 52 captured from the perspective of the camera 94. In the vehicle 10, no camera is placed at the position of the occupant eyellipse 58, so the image 52 from the perspective of the camera 94 cannot be compared with the image 52 from the perspective of the position of the occupant eyellipse 58. Therefore, the neural network 92 looks at the plurality of reference distortion maps 118a-118n stored therein and uses machine learning to predict the distortion map of the current environment.

[0071] The set of reference distortion maps 118a-118n within the neural network 92 includes collected object space images and corrected object space images for various first reference camera positions (position and angle of the first reference camera 110 near the reference DLP 102, and the position and angle of the reference DLP 102), source image 104 sizes, and windshield areas. The neural network 92 predicts distortion maps to correct the image from the perspective of the occupant eye ellipse 58 position using a single image captured by the camera 94 near the DLP 50. The neural network 92 is trained to predict distortion maps for intermediate camera 94 positions (position and perspective of the camera 94 near the DLP 50, and the position and angle of the DLP 50) given the images captured by the camera 94 in real time. The neural network 92 uses this information and the interpolation algorithm stored therein to predict the distortion map.

[0072] When applying the distortion map to the graphic image 52 projected by the digital light projector 50, the controller 34 is further adapted to encode distortion information from the distortion map into the graphic image 52 projected by the digital light projector 50. Thus, the digital light projector 50 is able to project a corrected graphic image 52 that will compensate for the distortion of the graphic image 52 from the perspective of the occupant eyellipse 58 position and calibrate the system 11 to properly display the graphic image 52 for the occupant 80.

[0073] In an exemplary embodiment, the controller 34 is further adapted to initiate calibration of the head-up display system 11 in the vehicle 10 when selectively driven by an occupant 80 in the vehicle 10. Alternatively, the controller 34 is adapted to automatically initiate calibration of the head-up display system 11, such as each time the vehicle 10 is used, or periodically, such as once a minute or once an hour.

[0074] refer to Figure 7 , Figure 8 and Fig. 9 The method 200 for calibrating the head-up display system 11 in the vehicle 10 includes, beginning at box 202 , training the neural network 92 in communication with the controller 34 to predict a distortion map based on a single graphic image 52 projected onto the interior surface 54 of the windshield 56 of the vehicle 10 . Moving to box 204, the method includes projecting a graphic image 52 onto an inner surface 54 of a windshield 56 of the vehicle 10 using a digital light projector 50 located within the vehicle 10; moving to box 206, capturing the graphic image 52 projected onto the inner surface 54 of the windshield 56 using a camera 94 proximate to the digital light projector 50; moving to box 208, locating a position of an occupant eyellipse 58 using an occupant monitoring system 78 within the vehicle 10; moving to box 210, predicting a distortion map of the projected graphic image 52 based on the position of the digital light projector 50, the position of the occupant eyellipse 58, and the shape of the windshield 56 within the vehicle 10 using the neural network 92; moving to box 212, applying the distortion map to the graphic image 52 projected by the digital light projector 50 using the controller 34; and moving to box 214, projecting a corrected graphic image 98 based on the predicted distortion map using the digital light projector 50.

[0075] In an exemplary embodiment, training the neural network 92 within the controller 34 at block 202 to predict the distortion map from the single graphical image 52 projected onto the interior surface 54 of the windshield 56 of the vehicle 10 further includes: moving to block 216, projecting the source image 104 onto the interior surface 106 of the reference windshield 108 using the reference digital light projector 102 of the reference system 100; moving to block 218, projecting the source image 104 onto the interior surface 106 of the reference windshield 108 using the first reference camera 110 located at the reference occupant eye ellipse position 112; and moving to block 219, capturing the source image 104 from the reference occupant eye ellipse using the first reference camera 110 located at the reference occupant eye ellipse position 112. ; moving to block 220, capturing the source image 104 from the perspective of the reference digital light projector 102 using a second reference camera 114 proximate to the reference digital light projector 102; moving to box 222, calculating a transformation matrix between the source image 128 captured from the perspective of the reference occupant eye ellipse position 112 and the image 120 captured from the perspective of the reference digital light projector 102 using the reference controller 116; and moving to box 224, creating a reference distortion map 118 for the captured source image 104.

[0076] In an exemplary embodiment, training the neural network 92 within the controller 34 at box 202 to predict a distortion map from a single graphical image 52 projected onto the interior surface 54 of the windshield 56 of the vehicle 10 also includes: following the steps set forth in boxes 216-224, creating a plurality of reference distortion maps 118a-118n for a plurality of different source images 104, wherein, for a plurality of different reference windshields 108a-108n, each of the plurality of different source images is projected to a plurality of different locations on the interior surface 106 of the reference windshield 108 from a plurality of different reference digital light projector 102 locations within the reference system 100, and moving to box 226, storing the plurality of reference distortion maps 118a-118n within the neural network 92.

[0077] In an exemplary embodiment, predicting the distortion map of the projected graphic image 52 based on the occupant eyellipse 58 position using the neural network 92 at block 210 further includes: predicting the distortion map of the graphic image 52 based on the position of the digital light projector 50, the occupant eyellipse 58 position, and the shape of the windshield 56 within the vehicle 10 using the neural network 92. In addition, applying the distortion map to the graphic image 52 projected by the digital light projector 50 using the controller 34 at block 212 further includes: encoding distortion information from the distortion map into the graphic image 52 projected by the digital light projector 50 using the controller 34.

[0078] In another exemplary embodiment, projecting the graphic image 50 onto the inner surface 54 of the windshield 56 of the vehicle 10 using the digital light projector 50 at frame 204 and projecting the corrected graphic image 50 onto the inner surface 54 of the windshield 56 of the vehicle 10 using the digital light projector 50 at frame 214 further includes: moving to frame 228, projecting the excitation light 64 using the light source 62; moving to frame 230, focusing the excitation light 64 from the light source 62 using the condenser lens 66; moving to frame 232, focusing the excitation light 64 using the color filter 68; The focused excitation light 64 is divided into red, green and blue light; moves to frame 234, uses a shaping lens 70 to focus the excitation light 64 passing through the color filter 68; moves to frame 236, uses a digital micromirror device (DMD) 72 to redirect the excitation light 64; and moves to frame 238, uses a projection lens 74 to receive the excitation light 64 from the digital micromirror device 72, and uses the projection lens 74 to project the excitation light 64 onto the inner surface 54 of the windshield 56, wherein the luminescent particles 90 in the windshield 56 emit visible light 64 in response to the absorption excitation.

[0079] Typically, calibration of the head-up display is performed relative to a camera located at the location of the occupant's eyellipse 58, such as described above with respect to the reference system 100, where multiple reference distortion maps 118a-118n are generated for various sub-images and projector positions. The head-up display system 11 of the present disclosure allows calibration using a single image captured by the camera 94 located at the digital light projector 50, without the need to use a camera located at the location of the occupant's eyellipse 58, by using machine learning and interpolation algorithms to predict intermediate distortion maps based on the stored reference distortion maps 118a-118n.

[0080] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A method for calibrating a head-up display system in a vehicle, comprising: training a neural network in communication with the controller to predict an occupant perspective distortion map of a single graphical image projected onto an interior surface of a windshield of the vehicle; projecting a graphic image onto an inner surface of a windshield of the vehicle using a digital light projector located within the vehicle; capturing a graphic image projected onto the inner surface of the windshield using a camera proximate to the digital light projector; locating the eye ellipses of the occupants using an occupant monitoring system in the vehicle; using the neural network to predict an occupant perspective distortion map of a projected graphic image based on a position of the digital light projector, a position of the occupant eye ellipses, and a shape of a windshield within the vehicle; applying, with the controller, the distortion map to a graphic image projected by the digital light projector; as well as The digital light projector is used to project a corrected graphic image based on the predicted distortion map.

2. The method according to claim 1, wherein: The training of a neural network within a controller to predict an occupant perspective distortion map of a single graphical image projected onto an interior surface of a vehicle windshield further comprises: projecting a source image onto an interior surface of a reference windshield using a reference digital light projector of the reference system; capturing the source image from a perspective of the reference occupant eyellipse position using a first reference camera located at the reference occupant eyellipse position; capturing the source image from the perspective of the reference digital light projector using a second reference camera proximate to the reference digital light projector; calculating, with a reference controller, a transformation matrix between a source image captured from the perspective of the reference occupant eyellipse position and an image captured from the perspective of the reference digital light projector; and Create a reference warp map for the captured source image.

3. The method according to claim 2, wherein: The training of a neural network within a controller to predict an occupant perspective distortion map of a single graphical image projected onto an interior surface of a vehicle windshield further comprises: creating a plurality of reference distortion maps for a plurality of different source images, each of the plurality of different source images being projected from a plurality of different reference digital light projector locations within the reference system to a plurality of different locations on the reference windshield interior surface; and The plurality of reference warp maps are stored within the neural network.

4. The method according to claim 3, wherein: The training of a neural network within a controller to predict an occupant perspective distortion map of a single graphical image projected onto an interior surface of a vehicle windshield further comprises: creating multiple reference distortion maps for multiple different reference windshields; and The plurality of reference warp maps are stored within the neural network.

5. The method according to claim 4, wherein: The method of using the neural network to predict the occupant's perspective distortion map of the projected graphic image based on the occupant's eye ellipse position also includes: An interpolation algorithm is applied using the neural network to predict a distortion map of a projected graphic image based on the position of the digital light projector, the occupant eyellipse position, and the shape of the windshield within the vehicle.

6. The method according to claim 5, wherein: The step of applying the distortion map to the graphic image projected by the digital light projector using a controller further comprises: Distortion information of the warp map is encoded into a graphic image projected by the digital light projector using the controller. 7 . The method of claim 1 , further comprising initiating calibration of the heads-up display in the vehicle upon selective actuation by an occupant of the vehicle. 8 . The method of claim 1 , further comprising automatically initiating calibration of the heads-up display in the vehicle.

9. The method according to claim 1, wherein: The projecting of the graphic image onto the inner surface of the vehicle windshield by a digital light projector further comprises: Projecting excitation light using a light source; Focusing the excitation light from the light source using a condenser lens; Use color filters to separate the focused excitation light into red, green, and blue light; Focusing the excitation light passing through the color filter by using a shaping lens; Redirecting the excitation light using a digital micromirror device (DMD); and The projection lens is used to receive the excitation light from the digital micromirror device, and the projection lens is used to project the excitation light onto the inner surface of the windshield.

10. The method according to claim 1, wherein: The step of projecting a graphic image onto the inner surface of the vehicle windshield using a digital projector in the vehicle further comprises: The digital light projector is used to project excitation light, wherein the light-emitting particles in the windshield emit visible light in response to absorbing the excitation light.

Images